Page id enablement relies on prctl in its implementation. When prctl is
not available, current codes still shows page id enabled, which is
not true and misleading. This commit fixes the misleading definition.
Migrates the mutex backend selection out of mutex.h/mutex.c, and the
background thread's condition-variable and signal-masking code out of
background_thread.c, onto the OS layer dispatcher pattern.
Introduce the `os.h` dispatcher pattern under
`include/jemalloc/internal/os/`, with no modules wired in yet.
- os.h - documents the os/<module>.h dispatcher / os/posix/
default / os/<os>/ override pattern that every
facility will follow.
- os/detect.h - JEMALLOC_OS_POSIX platform detection, shared by every
module dispatcher.
The extents and hpa-nonfull "ind" columns are emitter_type_unsigned but were
set via col_ind.size_val -- correct only by little-endian union overlap.
Output is byte-identical. Pre-existing (predates the refactor).
mutex_stats_read_global and mutex_stats_read_arena had byte-identical
bodies; merge them into a single mutex_stats_read used by both the global
and per-arena mutex tables. Pure cleanup; no functional or output change.
Table-output change: label size classes above 8*PAGE as a range
"prev_size+1..size" (e.g. "32769..40960") in the extents / lextents /
hpa-nonfull tables, instead of a single size. Text only; JSON is unchanged
(the ordered size-class metadata already implies the range).
Table-output change: fold the HPA "In full slabs:" / "In empty slabs:" prose
blocks into the per-size-class table as two symbolic rows (size "full"/"empty")
and retitle it "pageslabs:". JSON is byte-identical; only the human-readable
table changes.
Table-output change: emit the HPA shard counters as structured key:value rows
keyed by the JSON field names, instead of the previous free-form prose. JSON
output is byte-identical; only the human-readable table changes.
src/stats.c fused mallctl reads with dual-format (text/JSON) emission inside a
few very large, bottom-up functions, behind dense macros, with the public entry
point at the very bottom. Rework it to be output-preserving but much easier to
read and extend:
- Separate gathering from emission where it is worth it: most sections become a
small gather (stats_gather_*, the mallctl reads into a typed struct) plus an
emit that renders it, with the plumbing macros (CTL_*, COL*) and gather
structs moved to internal header (stats_internal.h). This is a balance,
not a rule: the per-size-class tables stay O(1) streaming (per-row gather+emit,
no large stack buffers, avoiding heap buffering), the mutex rows keep reading
straight into emitter columns, and the already-readable config/opt probes are
left as they are.
- Read like the output: open with a table-of-contents comment and the public
stats_print(), then the section functions top-down, keeping the
interval/boot/fork plumbing at the bottom. The ToC names the function that
prints each section, and its JSON sub-path.
- Keep one emission engine: the emitter remains the single dual-format renderer,
so each section has one rendering path rather than parallel renderers (the two
formats are not field-identical). Move some table-only concerns into it --
gap-collapsing ("---") and a dual columnar row op (emitter_row) -- so section
code carries less table-format detail.
Output-preserving: text is byte-identical and JSON keys/values are unchanged.
The frameptr-based unwinder was an interim solution while issues with
libunwind were resolved. It's better if we remove it,
so we don't have to support it moving forward.
New tests cover:
- background thread states after arena_reset
- background thread stats
- toggle background thread on and off during parallel stress
allocs/dallocs
- background thread fork behavior
Add background-thread fork unit test
test_fork_background_thread (in fork.c): with the background thread enabled,
fork and assert the child comes up with it disabled yet usable (re-enable +
alloc round-trips), while the parent keeps its threads. Placed in fork.c to
reuse its wait_for_child_exit helper and fork/WIN32 handling instead of
duplicating them in a separate file.
Add unit tests to cover public interface of pac decay & deferral
relevant functions. To align with the decopuling between arena and pac,
rename arena_decay to pac_decay here.
Currently there are deferral-relevant constants in arena.c and
background_thread.c. This commit moves them out into corresponding PAC
and HPA headers. Those commonly needed by both are put into a new header
deferral.h where the contract of deferral are clearly stated.
The LG_VADDR detection block accumulated duplicated logic and the rdtscp
feature probe was interleaved with virtual-address-bit detection.
Consolidate the detection:
- Wrap the whole case in a single "if LG_VADDR = detect" guard instead
of repeating it in every "case" section.
- Define LG_VADDR exactly once, after the case.
- Check the cache variable directly for the x86_64 error path.
- Move the rdtscp probe into its own x86_64 case so it stays independent
of LG_VADDR detection (it must run even when --with-lg-vaddr is given).
There is no functional change. Verified on x86_64 that LG_VADDR and
JEMALLOC_HAVE_RDTSCP are identical to before for auto-detection, and that
--with-lg-vaddr still probes rdtscp.
* Allow resuming per-CPU arena selection via thread.arena
With percpu_arena enabled, thread.arena control is one-directional. A
thread can be bound to a manually created arena (an index at or above the
per-CPU auto range) to route a bounded region of work to a dedicated,
long-lived arena, but there is no way back: thread_arena_ctl returns EPERM
for any index within the auto range, and arena_choose_impl only re-selects
a per-CPU arena for threads whose current arena is already in that range.
So once a thread is bound to a manual arena it stays pinned there forever,
and its later allocations land there instead of following the CPU.
Treat setting thread.arena to an index within the per-CPU range as a
request to resume automatic per-CPU selection: hand the thread back to
percpu management (rebinding it to the current CPU's arena via
percpu_arena_update) and return 0 instead of EPERM. Binding to a manual
arena is unchanged. The requested index is advisory; under percpu the
thread is governed by its current CPU, so it resumes on the current CPU's
arena regardless of the value passed.
Add test/unit/percpu_arena_resume covering the manual-arena to resume
round trip with allocs & dallocs happening meanwhile, and update
test_thread_arena, which asserted the old EPERM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
background_thread_sleep() computed the pthread_cond_timedwait deadline
from gettimeofday() (CLOCK_REALTIME). If the wall clock jumps backward
(NTP correction, VM snapshot, leap second), the deadline ends up far in
the future and the background thread stops purging dirty pages -
potentially for hours or days. This was observed in production where
NTP corrections caused unbounded RSS growth due to the background
thread stalling.
Switch deadline computation to nstime_init_update() (which already
handles the monotonic vs realtime fallback), and set CLOCK_MONOTONIC
on the condvar via pthread_condattr_setclock at both init sites
(background_thread_boot1 and background_thread_postfork_child), with
fallback to CLOCK_REALTIME if condattr init/setclock fails.
This only changes behavior for the tsd_generic path, where
pthread_getspecific() can report no TSD after the pthread key destructor
has finished. Other TSD backends keep tsd_teardown_done() as a constant
false, so the added checks compile out there.
Avoid recreating TSD for late deallocations. Preserve existing
reincarnation behavior for late allocations and nonzero reallocations.
Add Linux CI coverage for force_tls=0, with and without --enable-debug,
to exercise the Android-equivalent generic TSD path.
Reformat the malloc_getcpu Apple comment to the block style and note that the
tpidr_el0/sidt scheme requires macOS 12+: macOS 11 and earlier kept the CPU
number in tpidrro_el0's low 3 bits (now retired). Cite xnu's
__TPIDR_CPU_NUM_MASK / MACHDEP_TPIDR_CPUNUM_MASK, kept in sync with
_os_cpu_number, as the source of truth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Apple branch read the CPU number from the low 3 bits of `tpidrro_el0`.
That layout no longer holds on Apple Silicon: those bits read back as 0, so
`malloc_getcpu` returned 0 for every thread. `percpu_arena` then funneled all
allocations into arena 0, or was disabled outright at init.
Read the CPU number the same way as libplatform's `_os_cpu_number`: the low 12
bits of `tpidr_el0` on arm64, or of the IDT base (via `sidt`) on x86. Also move
the Apple branch ahead of the `rdtscp` one so Apple x86 uses `sidt` rather than
`rdtscp`, whose `ecx` is not the CPU id under XNU.
https://github.com/apple-oss-distributions/xnu/blob/main/libsyscall/os/tsd.h
Verified on a 12-CPU M2 Pro: with `percpu_arena:percpu`, `opt.percpu_arena` now
stays `percpu` and `thread.arena` spreads across all 12 arenas {0..11}; the old
code collapsed every thread onto arena 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
free_sized() and free_aligned_sized() forward straight to sdallocx(), which
expects a non-NULL pointer and asserts on it in debug builds. C23 says both
should accept NULL and do nothing, like free(NULL) does, so a NULL argument
either trips that assert or feeds NULL into the dealloc path in release builds.
It is not hard to hit. glibc 2.41 ships free_sized()/free_aligned_sized(), and
a C++ sized delete of a null pointer compiles down to a free_sized() call. Once
jemalloc is preloaded its versions take over, and that NULL call takes down the
process. I ran into it with GTK4/GLib apps under LD_PRELOAD.
Check for NULL first, the way free() already does, and add an integration test
covering the NULL case for both functions.
While here, give free_aligned_sized() its own core.free_aligned_sized.entry
and .exit logging and call je_sdallocx_impl() directly rather than the
je_sdallocx() wrapper, so it mirrors free_sized() and no longer logs under
sdallocx. The C++ sized-delete paths (sizedDeleteImpl, alignedSizedDeleteImpl)
get the same treatment: log entry/exit unconditionally and guard the call with
likely(ptr != nullptr).
