romenskiy2012/jemalloc
1
0
Fork 0
mirror of https://github.com/jemalloc/jemalloc.git synced 2026-08-24 16:13:38 +03:00
Code Issues Projects Releases Packages Wiki Activity

Guard TSD thread-locals against LTO hoisting (JEMALLOC_TLS_ADDR)

Browse source 
`tsd_get`/`tsd_set` took the tsd address as a plain `&tsd_tls` -- `thread_pointer
+ const_offset`, which the compiler treats as loop-invariant.  Under
whole-program LTO it is hoisted out of the inlined malloc/free and kept in a
callee-saved register across a user-space context switch (swapcontext,
boost::context fibers); resumed on a different OS thread, the cached tsd/tcache
still belongs to the previous thread and the two race -- heap corruption that
reproduces only under LTO and is invisible to sanitizers.  The write side is
worse: tsd_set (from tsd_fetch_slow, on a thread's first allocation after
migration) would copy the new thread's tsd over the previous thread's live one
and register the wrong cleanup key.

Route tsd_get/tsd_set in every GNU backend through JEMALLOC_TLS_ADDR(tsd_tls): a
per-variable accessor that takes the address inside a noinline `memory`-barrier
function, opaque to the optimizer.  Correct for every TLS model, one call per
access; MSVC keeps the plain address.

The accessor is declared in every TU (JEMALLOC_TLS_ADDR_DECLARE) but defined
once in src/tsd.c (JEMALLOC_TLS_ADDR_DEFINE, under JEMALLOC_TSD_C_): a per-TU
`static` body is emitted at -O0 even when unused, pulling an undefined reference
to the internal thread-local into the integration-test util objects, which link
only the public shared library.  DECLARE/DEFINE are invoked directly, never
forwarded, so `##tlsvar` pastes the literal name rather than the expanded
`je_tsd_tls`.
This commit is contained in:
Azat Khuzhin 2026-06-22 12:47:46 +02:00
parent 1903f63aaa
commit 7b9853c4a1
5 changed files with 72 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

34
include/jemalloc/internal/tsd_internals.h
View file

@ -25,6 +25,40 @@
typedef struct arena_s arena_t;
typedef struct prof_tdata_s prof_tdata_t;
/*
* JEMALLOC_TLS_ADDR(tlsvar): take a thread-local's address so the compiler
* cannot cache it across a user-space context switch. A raw `&tlsvar` is
* loop-invariant; inlined into malloc/free under LTO it can be hoisted across a
* swapcontext and reused on the OS thread a fiber migrated to -- a stale
* tsd/tcache. Route all tsd access through this macro; never take `&tsd_tls`
* raw. See https://github.com/jemalloc/jemalloc/issues/2890
*
* The accessor takes the address behind a noinline `memory` barrier, opaque to
* the optimizer. MSVC has no inline asm, so it keeps the plain address; an MSVC
* build whose fibers run under /GL must instead compile with /GT (fiber-safe
* TLS). DECLARE is emitted in every TU;
* DEFINE (the out-of-line body) only under JEMALLOC_TSD_C_, i.e. once in
* src/tsd.c. Both must be invoked directly, not forwarded through another
* macro, or `##tlsvar` pastes the macro-expanded `je_tsd_tls` instead of the
* literal name.
*/
#if defined(__GNUC__)
# define JEMALLOC_TLS_ADDR_DECLARE(tlsvar) \
__typeof__(&(tlsvar)) jemalloc_tls_addr_##tlsvar(void);
# define JEMALLOC_TLS_ADDR_DEFINE(tlsvar) \
JEMALLOC_NOINLINE __typeof__(&(tlsvar)) \
jemalloc_tls_addr_##tlsvar(void) { \
__typeof__(&(tlsvar)) tls_addr = &(tlsvar); \
__asm__ __volatile__("" : "+r"(tls_addr) : : "memory"); \
return tls_addr; \
}
# define JEMALLOC_TLS_ADDR(tlsvar) (jemalloc_tls_addr_##tlsvar())
#else
# define JEMALLOC_TLS_ADDR_DECLARE(tlsvar)
# define JEMALLOC_TLS_ADDR_DEFINE(tlsvar)
# define JEMALLOC_TLS_ADDR(tlsvar) (&(tlsvar))
#endif
/*
* Thread-Specific-Data layout
*