summaryrefslogtreecommitdiffstats
path: root/src/extent.c
Commit message (Collapse)AuthorAgeFilesLines
* Add the missing unlock in the error path of extent_register.Qi Wang2019-03-291-0/+1
|
* Eagerly purge oversized merged extents.Qi Wang2019-03-151-0/+7
| | | | This change improves memory usage slightly, at virtually no CPU cost.
* Avoid potential issues on extent zero-out.Qi Wang2019-01-121-4/+21
| | | | | | When custom extent_hooks or transparent huge pages are in use, the purging semantics may change, which means we may not get zeroed pages on repopulating. Fixing the issue by manually memset for such cases.
* Avoid touching all pages in extent_recycle for debug build.Qi Wang2018-11-131-2/+3
| | | | | We may have a large number of pages with *zero set (since they are populated on demand). Only check the first page to avoid paging in all of them.
* Optimize large deallocation.Qi Wang2018-11-081-17/+41
| | | | | | | | | | | We eagerly coalesce large buffers when deallocating, however the previous logic around this introduced extra lock overhead -- when coalescing we always lock the neighbors even if they are active, while for active extents nothing can be done. This commit checks if the neighbor extents are potentially active before locking, and avoids locking if possible. This speeds up large_dalloc by ~20%. It also fixes some undesired behavior: we could stop coalescing because a small buffer was merged, while a large neighbor was ignored on the other side.
* Bypass extent_dalloc when retain is enabled.Qi Wang2018-11-081-8/+18
| | | | | | | When retain is enabled, the default dalloc hook does nothing (since we avoid munmap). But the overhead preparing the call is high, specifically the extent de-register and re-register involve locking and extent / rtree modifications. Bypass the call with retain in this diff.
* Add stats for the size of extent_avail heapTyler Etzel2018-08-021-0/+2
|
* Add extents information to mallocstats outputTyler Etzel2018-08-021-0/+36
| | | | - Show number/bytes of extents of each size that are dirty, muzzy, retained.
* SC: Remove global data.David Goldblatt2018-07-231-11/+9
| | | | | | | The global data is mostly only used at initialization, or for easy access to values we could compute statically. Instead of consuming that space (and risking TLB misses), we can just pass around a pointer to stack data during bootstrapping.
* SC: Make some key size classes static.David Goldblatt2018-07-131-3/+3
| | | | | | The largest small class, smallest large class, and largest large class may all be needed down fast paths; to avoid the risk of touching another cache line, we can make them available as constants.
* Hide size class computation behind a layer of indirection.David Goldblatt2018-07-131-23/+28
| | | | | | | | | This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
* Clean compilation -Wextragnzlbg2018-07-101-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
* Avoid a resource leak down extent split failure paths.David Goldblatt2018-04-181-10/+8
| | | | | | Previously, we would leak the extent and memory associated with a salvageable portion of an extent that we were trying to split in three, in the case where the first split attempt succeeded and the second failed.
* Fix arguments passed to extent_init.Qi Wang2018-04-091-1/+1
|
* extents: Remove preserve_lru feature.Dave Watson2018-04-021-46/+18
| | | | | | | | preserve_lru feature adds lots of complication, for little value. Removing it means merged extents are re-added to the lru list, and may take longer to madvise away than they otherwise would. Canaries after removal seem flat for several services (no change).
* Add opt.thp which allows explicit hugepage usage.Qi Wang2018-03-081-6/+6
| | | | | | | | "always" marks all user mappings as MADV_HUGEPAGE; while "never" marks all mappings as MADV_NOHUGEPAGE. The default setting "default" does not change any settings. Note that all the madvise calls are part of the default extent hooks by design, so that customized extent hooks have complete control over the mappings including hugepage settings.
* Improve the fit for aligned allocation.Qi Wang2018-01-051-10/+61
| | | | | | | We compute the max size required to satisfy an alignment. However this can be quite pessimistic, especially with frequent reuse (and combined with state-based fragmentation). This commit adds one more fit step specific to aligned allocations, searching in all potential fit size classes.
* Over purge by 1 extent always.Qi Wang2017-12-181-4/+2
| | | | | | | | | When purging, large allocations are usually the ones that cross the npages_limit threshold, simply because they are "large". This means we often leave the large extent around for a while, which has the downsides of: 1) high RSS and 2) more chance of them getting fragmented. Given that they are not likely to be reused very soon (LRU), let's over purge by 1 extent (which is often large and not reused frequently).
* Fix extent deregister on the leak path.Qi Wang2017-12-091-4/+14
| | | | On leak path we should not adjust gdump when deregister.
* Add more tests for extent hooks failure paths.Qi Wang2017-11-291-0/+3
|
* Add missing deregister before extents_leak.Qi Wang2017-11-201-0/+1
| | | | This fixes an regression introduced by 211b1f3 (refactor extent split).
* Avoid setting zero and commit if split fails in extent_recycle.Qi Wang2017-11-201-14/+10
|
* Eagerly coalesce large extents.Qi Wang2017-11-161-1/+15
| | | | | | Coalescing is a small price to pay for large allocations since they happen less frequently. This reduces fragmentation while also potentially improving locality.
* Fix an extent coalesce bug.Qi Wang2017-11-161-7/+13
| | | | | When coalescing, we should take both extents off the LRU list; otherwise decay can grab the existing outer extent through extents_evict.
* Add opt.lg_extent_max_active_fitQi Wang2017-11-161-0/+9
| | | | | | | | | | When allocating from dirty extents (which we always prefer if available), large active extents can get split even if the new allocation is much smaller, in which case the introduced fragmentation causes high long term damage. This new option controls the threshold to reuse and split an existing active extent. We avoid using a large extent for much smaller sizes, in order to reduce fragmentation. In some workload, adding the threshold improves virtual memory usage by >10x.
* Use extent_heap_first for best fit.Qi Wang2017-11-161-1/+1
| | | | | extent_heap_any makes the layout less predictable and as a result incurs more fragmentation.
* Fix unbounded increase in stash_decayed.Qi Wang2017-11-091-2/+3
| | | | | | Added an upper bound on how many pages we can decay during the current run. Without this, decay could have unbounded increase in stashed, since other threads could add new pages into the extents.
* Add arena.i.retain_grow_limitQi Wang2017-11-031-3/+4
| | | | | | | This option controls the max size when grow_retained. This is useful when we have customized extent hooks reserving physical memory (e.g. 1G huge pages). Without this feature, the default increasing sequence could result in fragmented and wasted physical memory.
* Add a "dumpable" bit to the extent state.David Goldblatt2017-10-161-6/+13
| | | | | Currently, this is unused (i.e. all extents are always marked dumpable). In the future, we'll begin using this functionality.
* Factor out extent-splitting core from extent lifetime management.David Goldblatt2017-10-161-81/+149
| | | | | | | Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
* Document some of the internal extent functions.David Goldblatt2017-10-161-0/+35
|
* Use ph instead of rb tree for extents_avail_Dave Watson2017-10-041-1/+1
| | | | | | | | | | There does not seem to be any overlap between usage of extent_avail and extent_heap, so we can use the same hook. The only remaining usage of rb trees is in the profiling code, which has some 'interesting' iteration constraints. Fixes #888
* Relax constraints on reentrancy for extent hooks.Qi Wang2017-08-311-1/+12
| | | | | | If we guarantee no malloc activity in extent hooks, it's possible to make customized hooks working on arena 0. Remove the non-a0 assertion to enable such use cases.
* Bypass extent_alloc_wrapper_hard for no_move_expand.Qi Wang2017-07-311-0/+9
| | | | | | When retain is enabled, we should not attempt mmap for in-place expansion (large_ralloc_no_move), because it's virtually impossible to succeed, and causes unnecessary syscalls (which can cause lock contention under load).
* Check arena in current context in pre_reentrancy.Qi Wang2017-06-231-34/+34
|
* Set reentrancy when invoking customized extent hooks.Qi Wang2017-06-231-8/+76
| | | | | Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
* Fix extent_hooks in extent_grow_retained().Qi Wang2017-06-141-3/+12
| | | | | | | This issue caused the default extent alloc function to be incorrectly used even when arena.<i>.extent_hooks is set. This bug was introduced by 411697adcda2fd75e135cdcdafb95f2bd295dc7f (Use exponential series to size extents.), which was first released in 5.0.0.
* Remove assertions on extent_hooks being default.Qi Wang2017-06-051-16/+0
| | | | | It's possible to customize the extent_hooks while still using part of the default implementation.
* Take background thread lock when setting extent hooks.Qi Wang2017-06-051-2/+12
|
* Header refactoring: Pull size helpers out of jemalloc module.David Goldblatt2017-05-311-11/+12
|
* Header refactoring: unify and de-catchall mutex_pool.David Goldblatt2017-05-311-0/+1
|
* Header refactoring: unify and de-catchall extent_mmap module.David Goldblatt2017-05-311-0/+1
|
* Header refactoring: unify and de-catchall extent_dss.David Goldblatt2017-05-311-0/+1
|
* Header refactoring: unify and de-catchall rtree module.David Goldblatt2017-05-311-0/+1
|
* Fix extent_grow_next management.Jason Evans2017-05-301-146/+206
| | | | | | | | | | | | | Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
* Fix OOM paths in extent_grow_retained().Jason Evans2017-05-301-2/+8
|
* Header refactoring: unify and de-catchall mutex moduleDavid Goldblatt2017-05-241-0/+1
|
* Header refactoring: unify and de-catchall witness code.David Goldblatt2017-05-241-18/+36
|
* Protect the rtree/extent interactions with a mutex pool.David Goldblatt2017-05-191-99/+145
| | | | | | | | | | | | | | | | | | Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
* Allow mutexes to take a lock ordering enum at construction.David Goldblatt2017-05-191-1/+2
| | | | | | | This lets us specify whether and how mutexes of the same rank are allowed to be acquired. Currently, we only allow two polices (only a single mutex at a given rank at a time, and mutexes acquired in ascending order), but we can plausibly allow more (e.g. the "release uncontended mutexes before blocking").
generated by cgit v0.12 at 2026-01-07 17:08:30 (GMT)