free_sized() and free_aligned_sized() forward straight to sdallocx(), which
expects a non-NULL pointer and asserts on it in debug builds. C23 says both
should accept NULL and do nothing, like free(NULL) does, so a NULL argument
either trips that assert or feeds NULL into the dealloc path in release builds.
It is not hard to hit. glibc 2.41 ships free_sized()/free_aligned_sized(), and
a C++ sized delete of a null pointer compiles down to a free_sized() call. Once
jemalloc is preloaded its versions take over, and that NULL call takes down the
process. I ran into it with GTK4/GLib apps under LD_PRELOAD.
Check for NULL first, the way free() already does, and add an integration test
covering the NULL case for both functions.
While here, give free_aligned_sized() its own core.free_aligned_sized.entry
and .exit logging and call je_sdallocx_impl() directly rather than the
je_sdallocx() wrapper, so it mirrors free_sized() and no longer logs under
sdallocx. The C++ sized-delete paths (sizedDeleteImpl, alignedSizedDeleteImpl)
get the same treatment: log entry/exit unconditionally and guard the call with
likely(ptr != nullptr).
It is unused and no longer needed: hpa_dirty_mult, hpa_purge_threshold,
hpa_min_purge_interval_ms, and hpa_min_purge_delay_ms now provide finer
control over the purge rate.
Add a small extent cache in front of the PAC ecaches. Allocs and dallocs
that fit are served from per-shard SEC bins without taking the ecache
mutex; overflow falls through to the backing ecaches, including
ecache_pinned for pinned extents.
The feature is gated behind experimental_pac_sec_nshards (default 0,
disabled). To support independent HPA and PAC SEC instances,
sec_alloc/sec_dalloc/sec_fill take an explicit shard argument, with HPA
and PAC using separate TSD shard slots.
Per-bin SEC stats (bytes_cur, nmisses, nhits, ndalloc_flush,
ndalloc_noflush, noverfills) become atomic_zu_t fields directly inside
sec_bin_t. Writers continue to hold bin->mtx; sec_stats_merge now reads
them lock-free with ATOMIC_RELAXED instead of acquiring bin->mtx.
This removes a lock-rank reversal that would otherwise occur whenever
stats aggregation runs while holding arena->stats.mtx.
base_block_alloc() grows new base blocks along the page size-class series to
reduce the number of disjoint VM ranges. This works well when new base blocks
are rare.
Under high thread churn, many threads can miss the base free pool in parallel
while allocating metadata. base_extent_alloc() drops base->mtx after mapping a
new block, but before splitting and inserting the rest into the reuse pool.
Therefore, each parallel miss can map its own block and each completed
allocation then advances base->pind_last. The result is that small metadata
requests can drive the growth heuristic to increasingly large mmap() sizes, far
beyond the actual allocation demand.
Cap the heuristic growth size at 128 MiB. This preserves the usual amortization
benefit while bounding the rare pathological case where parallel misses rapidly
advance the growth series. Large individual requests are still honored because
min_block_size continues to override the cap.
The READ/WRITE/READONLY/WRITEONLY/VERIFY_READ/ASSURED_WRITE/MIB_UNSIGNED/
NEITHER_READ_NOR_WRITE/READ_XOR_WRITE macros hid control flow (goto
label_return) inside the mallctl handlers. Convert them to memcpy-based
helper functions.
Add error-path tests for the converted handlers and direct unit tests for
the helpers
The runtime option aborted on every OOM, breaking new(std::nothrow)
semantics. Replace with configure-time --enable-cxx-infallible-new
(default off): when on, throwing new aborts (size logged) and
nothrow returns null; when off, standard new_handler + bad_alloc /
null behavior is preserved. Under LTO the on-path lets the compiler
prove operator new is no-throw.
The split managed one ordering constraint: arena_choose() had to be
defined before arena_choose_maybe_huge() but after the tsd/tcache
inlines it depends on. After the malloc_dispatch refactor moved the
heaviest tcache-pulling inlines out of arena_inlines_b.h, the
remaining arena-side inlines all belong together. The merged
arena_inlines.h explicitly includes jemalloc_internal_inlines_a.h
and tcache.h (previously transitively pulled).
Convert the production source files in src/ (69 .c/.cpp) and
test/jemalloc_test.h.in to list the headers they actually use, then
delete the umbrella. Three consolidated headers (peak_event.h,
prof_sys.h, sz.h) also gain explicit transitive includes.
Every translation unit now declares what it uses. A missing include
now fails at the failing file rather than silently working because
something upstream pulled in the world.
Both components had a four-way split (_types, _structs, _externs,
_inlines) that predates explicit per-file includes. With the edata
<-> prof_types coupling broken in the prior commit, merging _types +
_structs + _externs no longer risks an include cycle.
- prof.h replaces prof_types.h + prof_structs.h + prof_externs.h.
- tcache.h replaces tcache_types.h + tcache_structs.h + tcache_externs.h.
prof_inlines.h and tcache_inlines.h stay separate: prof_inlines.h
sits at the bottom of the dependency layering, and tcache_inlines.h's
include of arena_externs.h is the asymmetric cycle-breaker that keeps
the arena <-> tcache symbol cycle from becoming an include cycle.
Fold historical *_types/_structs/_externs/_inlines splits where the
layering is no longer load-bearing.
- large_externs.h -> large.h: rename; it was a single-purpose
function-prototype file.
- background_thread_structs.h + background_thread_externs.h ->
background_thread.h: merge. background_thread_inlines.h stays
separate because it depends on arena_inlines_a.h.
- bin_types.h -> bin.h: BIN_SHARDS_MAX / N_BIN_SHARDS_DEFAULT join the
bin_t struct and the bin_dalloc_locked_info_t type. bin_inlines.h
stays separate so TUs that only need the bin_t type don't pull in
the tcache-flush inline closure. The two bin_stats_* helpers move
from bin.h into bin_inlines.h so all bin inlines live together.
- tsd_binshards.h (new): houses tsd_binshards_t and its zero
initializer. Lets tsd_internals.h pull it in for X-macro expansion
without dragging mutex.h -- mutex.h depends on TSD machinery, which
would form a cycle through bin.h.
jemalloc_internal_includes.h drops the references to the deleted/
merged headers.
Pull the tcache-aware routing helpers out of arena into a layer that
sits directly below the public malloc interface:
arena_malloc -> malloc_dispatch_malloc
arena_palloc -> malloc_dispatch_palloc
arena_ralloc -> malloc_dispatch_ralloc
arena_dalloc* -> malloc_dispatch_dalloc*
arena_sdalloc* -> malloc_dispatch_sdalloc*
arena_dalloc_promoted -> malloc_dispatch_dalloc_promoted
The new module (malloc_dispatch.h, malloc_dispatch_inlines.h,
src/malloc_dispatch.c) owns the tcache-vs-fall-through decision; the
only consumer is jemalloc_internal_inlines_c.h. arena keeps a narrower
arena_prof_demote() for the sampled-allocation demotion path.
Fix FreeBSD postfork child handler never being called: FreeBSD's libthr
calls _malloc_postfork in both parent and child (see freebsd-src
lib/libthr/thread/thr_fork.c), but jemalloc mapped it to the parent
handler only. Detect the child via getpid() and route to
jemalloc_postfork_child, which resets nthreads and rebuilds the
descriptor queue.
Remove the child_survivor_bytes vs pre_survivor_bytes comparison: on
macOS where jemalloc registers as the default zone, internal allocations
during the postfork handler (pthread_mutex_init) can inflate the
surviving thread's tcache.
Add double-fork test to verify prefork pid is refreshed correctly when a
child process forks again.
This change includes the following improvements:
- Remove the hpa_sec_batch_fill_extra parameter.
- Refactor the hpa_alloc() code and helper functions to be able to
allocate more than one extent out of a single pageslab. This way
we can amortize the per-pageslab costs (active bitmap iteration,
pageslab metadata updates) across multiple extents.
- Decide on a min and max number of extents that will be allocated
in hpa_alloc(). The code will try to allocate at least the min
and allocate up to the max as long as we can allocate additional
ones from the pageslab we already have, as additional allocations
are relatively cheap.
- Add extent allocation distribution stats.
- Amend hpa_sec_integration.c unit test.
The orchestrator looks up the surviving descriptor via
tcache_postfork_arena_descriptor and threads it into
arena_postfork_child, eliminating arena's call into tcache. Also reset
cache_bin_array_descriptor_ql_mtx right before the queue rebuild it
protects.
tcache.c was reaching into arena->cache_bin_array_descriptor_ql{,_mtx}
directly to register / unregister / postfork-relink its descriptor.
That queue and mutex are owned by arena, so the locking and ql_*
operations belong in arena.c.
After tcache_init runs, tcache_slow->tcache == tcache always holds, so
the tcache_t parameter to the three association helpers is derivable
from tcache_slow.
Drop the duplicate arena->tcache_ql; stats merging walks the
cache_bin_array_descriptor_ql directly. Rename the protecting mutex
from tcache_ql_mtx to cache_bin_array_descriptor_ql_mtx to match. Add
an assertion in test_thread_migrate_arena that the dissociate-time
flush zeros cache_bin->tstats.nrequests.
