Commit graph

1715 commits

Author SHA1 Message Date
Slobodan Predolac
176ea0a801 Remove experimental.thread.activity_callback 2026-04-01 16:23:41 -07:00
Slobodan Predolac
19bbefe136 Remove dead code: extent_commit_wrapper, large_salloc, tcache_gc_dalloc event waits
These functions had zero callers anywhere in the codebase:
- extent_commit_wrapper: wrapper never called, _impl used directly
- large_salloc: trivial wrapper never called
- tcache_gc_dalloc_new_event_wait: no header declaration, no callers
- tcache_gc_dalloc_postponed_event_wait: no header declaration, no callers
2026-04-01 17:48:19 -04:00
Tony Printezis
1d018d8fda improve hpdata_assert_consistent()
A few ways this consistency check can be improved:
* Print which conditions fail and associated values.
* Accumulate the result so that we can print all conditions that fail.
* Turn hpdata_assert_consistent() into a macro so, when it fails,
  we can get line number where it's called from.
2026-03-26 10:39:23 -07:00
Carl Shapiro
ad726adf75 Separate out the configuration code from initialization 2026-03-10 18:14:33 -07:00
Carl Shapiro
a056c20d67 Handle tcache init failures gracefully
tsd_tcache_data_init() returns true on failure but its callers ignore
this return value, leaving the per-thread tcache in an uninitialized
state after a failure.

This change disables the tcache on an initialization failure and logs
an error message.  If opt_abort is true, it will also abort.

New unit tests have been added to test tcache initialization failures.
2026-03-10 18:14:33 -07:00
Carl Shapiro
0ac9380cf1 Move bin inline functions from arena_inlines_b.h to bin_inlines.h
This is a continuation of my previous clean-up change, now focusing on
the inline functions defined in header files.
2026-03-10 18:14:33 -07:00
Carl Shapiro
1cc563f531 Move bin functions from arena.c to bin.c
This is a clean-up change that gives the bin functions implemented in
the area code a prefix of bin_ and moves them into the bin code.

To further decouple the bin code from the arena code, bin functions
that had taken an arena_t to check arena_is_auto now take an is_auto
parameter instead.
2026-03-10 18:14:33 -07:00
Slobodan Predolac
34ace9169b Remove prof_threshold built-in event. It is trivial to implement it as user event if needed 2026-03-10 18:14:33 -07:00
Slobodan Predolac
d4908fe44a Revert "Experimental configuration option for fast path prefetch from cache_bin"
This reverts commit f9fae9f1f8.
2026-03-10 18:14:33 -07:00
Slobodan Predolac
6016d86c18 [SEC] Make SEC owned by hpa_shard, simplify the code, add stats, lock per bin 2026-03-10 18:14:33 -07:00
Slobodan Predolac
8a06b086f3 [EASY] Extract hpa_central component from hpa source file 2026-03-10 18:14:33 -07:00
Slobodan Predolac
355774270d [EASY] Encapsulate better, do not pass hpa_shard when hooks are enough, move shard independent actions to hpa_utils 2026-03-10 18:14:33 -07:00
Slobodan Predolac
47aeff1d08 Add experimental_enforce_hugify 2026-03-10 18:14:33 -07:00
Shirui Cheng
6d4611197e move fill/flush pointer array out of tcache.c 2026-03-10 18:14:33 -07:00
Slobodan Predolac
3678a57c10 When extracting from central, hugify_eager is different than start_as_huge 2026-03-10 18:14:33 -07:00
guangli-dai
2cfa41913e Refactor init_system_thp_mode and print it in malloc stats. 2026-03-10 18:14:33 -07:00
Slobodan Predolac
5e49c28ef0 [EASY] Spelling in the comments 2026-03-10 18:14:33 -07:00
Slobodan Predolac
a199278f37 [HPA] Add ability to start page as huge and more flexibility for purging 2026-03-10 18:14:33 -07:00
Slobodan Predolac
2688047b56 Revert "Do not dehugify when purging"
This reverts commit 16c5abd1cd.
2026-03-10 18:14:33 -07:00
Slobodan Predolac
5d5f76ee01 Remove pidfd_open call handling and rely on PIDFD_SELF 2026-03-10 18:14:33 -07:00
Slobodan Predolac
2a66c0be5a [EASY][BUGFIX] Spelling and format 2026-03-10 18:14:33 -07:00
lexprfuncall
38b12427b7 Define malloc_{write,read}_fd as non-inline global functions
The static inline definition made more sense when these functions just
dispatched to a syscall wrapper.  Since they acquired a retry loop, a
non-inline definition makes more sense.
2026-03-10 18:14:33 -07:00
lexprfuncall
9fdc1160c5 Handle interruptions and retries of read(2) and write(2) 2026-03-10 18:14:33 -07:00
Shirui Cheng
2114349a4e Revert PR #2608: Manually revert commits 70c94d..f9c0b5
Closes: #2707
2026-03-10 18:14:33 -07:00
Slobodan Predolac
d73de95f72 Experimental configuration option for fast path prefetch from cache_bin 2026-03-10 18:14:33 -07:00
lexprfuncall
a156e997d7 Do not dehugify when purging
Giving the advice MADV_DONTNEED to a range of virtual memory backed by
a transparent huge page already causes that range of virtual memory to
become backed by regular pages.
2026-03-10 18:14:33 -07:00
Slobodan Predolac
4246475b44 [process_madvise] Make init lazy so that python tests pass. Reset the pidfd on fork 2026-03-10 18:14:33 -07:00
Slobodan Predolac
711fff750c Add experimental support for usdt systemtap probes 2026-03-10 18:14:33 -07:00
guangli-dai
6200e8987f Reformat the codebase with the clang-format 18. 2026-03-10 18:14:33 -07:00
Shirui Cheng
a952a3b8b0 Update the default value for opt_experimental_tcache_gc and opt_calloc_madvise_threshold 2026-03-10 18:14:33 -07:00
Slobodan Predolac
015b017973 [thread_event] Add support for user events in thread events when stats are enabled 2026-03-10 18:14:33 -07:00
Slobodan Predolac
e6864c6075 [thread_event] Remove macros from thread_event and replace with dynamic event objects 2026-03-10 18:14:33 -07:00
Jason Evans
27d7960cf9 Revert "Extend purging algorithm with peak demand tracking"
This reverts commit ad108d50f1.
2025-06-02 10:44:37 -07:00
guangli-dai
edaab8b3ad Turn clang-format off for codes with multi-line commands in macros 2025-05-28 19:22:21 -07:00
guangli-dai
fd60645260 Add one more check to double free validation. 2025-05-28 19:21:49 -07:00
Xin Yang
5e460bfea2 Refactor: use the cache_bin_sz_t typedef instead of direct uint16_t
any future changes to the underlying data type for bin sizes
(such as upgrading from `uint16_t` to `uint32_t`) can be achieved
by modifying only the `cache_bin_sz_t` definition.

Signed-off-by: Xin Yang <yangxin.dev@bytedance.com>
2025-05-22 10:43:33 -07:00
Xin Yang
9169e9272a Fix: Adjust CACHE_BIN_NFLUSH_BATCH_MAX size to prevent assert failures
The maximum allowed value for `nflush_batch` is
`CACHE_BIN_NFLUSH_BATCH_MAX`. However, `tcache_bin_flush_impl_small`
could potentially declare an array of `emap_batch_lookup_result_t`
of size `CACHE_BIN_NFLUSH_BATCH_MAX + 1`. leads to a `VARIABLE_ARRAY`
assertion failure, observed when `tcache_nslots_small_max` is
configured to 2048. This patch ensures the array size does not exceed
the allowed maximum.

Signed-off-by: Xin Yang <yangxin.dev@bytedance.com>
2025-05-22 10:27:09 -07:00
guangli-dai
37bf846cc3 Fixes to prevent static analysis warnings. 2025-05-06 14:47:35 -07:00
guangli-dai
8347f1045a Renaming limit_usize_gap to disable_large_size_classes 2025-05-06 14:47:35 -07:00
Guangli Dai
01e9ecbeb2 Remove build-time configuration 'config_limit_usize_gap' 2025-05-06 14:47:35 -07:00
Slobodan Predolac
852da1be15 Add experimental option force using SYS_process_madvise 2025-04-28 18:45:30 -07:00
Slobodan Predolac
1956a54a43 [process_madvise] Use process_madvise across multiple huge_pages 2025-04-25 19:19:03 -07:00
Slobodan Predolac
0dfb4a5a1a Add output argument to hpa_purge_begin to count dirty ranges 2025-04-25 19:19:03 -07:00
Slobodan Predolac
cfa90dfd80 Refactor hpa purging to prepare for vectorized call across multiple pages 2025-04-25 19:19:03 -07:00
guangli-dai
c20a63a765 Silence the uninitialized warning from clang. 2025-04-16 10:38:10 -07:00
Slobodan Predolac
f19f49ef3e if process_madvise is supported, call it when purging hpa 2025-04-04 13:57:42 -07:00
Kaspar M. Rohrer
80e9001af3 Move `extern "C" specifications for C++ to where they are needed
This should fix errors when compiling C++ code with modules enabled on clang.
2025-03-31 10:41:51 -07:00
Shirui Cheng
3688dfb5c3 fix assertion error in huge_arena_auto_thp_switch() when b0 is deleted in unit test 2025-03-20 12:45:23 -07:00
Shirui Cheng
e1a77ec558 Support THP with Huge Arena in PAC 2025-03-17 16:06:43 -07:00
Guangli Dai
773b5809f9 Fix frame pointer based unwinder to handle changing stack range 2025-03-13 17:15:42 -07:00