summaryrefslogtreecommitdiffstats
path: root/src/extent.c
Commit message (Collapse)AuthorAgeFilesLines
* Improve the fit for aligned allocation.Qi Wang2018-01-051-10/+61
| | | | | | | We compute the max size required to satisfy an alignment. However this can be quite pessimistic, especially with frequent reuse (and combined with state-based fragmentation). This commit adds one more fit step specific to aligned allocations, searching in all potential fit size classes.
* Over purge by 1 extent always.Qi Wang2017-12-181-4/+2
| | | | | | | | | When purging, large allocations are usually the ones that cross the npages_limit threshold, simply because they are "large". This means we often leave the large extent around for a while, which has the downsides of: 1) high RSS and 2) more chance of them getting fragmented. Given that they are not likely to be reused very soon (LRU), let's over purge by 1 extent (which is often large and not reused frequently).
* Fix extent deregister on the leak path.Qi Wang2017-12-091-4/+14
| | | | On leak path we should not adjust gdump when deregister.
* Add more tests for extent hooks failure paths.Qi Wang2017-11-291-0/+3
|
* Add missing deregister before extents_leak.Qi Wang2017-11-201-0/+1
| | | | This fixes an regression introduced by 211b1f3 (refactor extent split).
* Avoid setting zero and commit if split fails in extent_recycle.Qi Wang2017-11-201-14/+10
|
* Eagerly coalesce large extents.Qi Wang2017-11-161-1/+15
| | | | | | Coalescing is a small price to pay for large allocations since they happen less frequently. This reduces fragmentation while also potentially improving locality.
* Fix an extent coalesce bug.Qi Wang2017-11-161-7/+13
| | | | | When coalescing, we should take both extents off the LRU list; otherwise decay can grab the existing outer extent through extents_evict.
* Add opt.lg_extent_max_active_fitQi Wang2017-11-161-0/+9
| | | | | | | | | | When allocating from dirty extents (which we always prefer if available), large active extents can get split even if the new allocation is much smaller, in which case the introduced fragmentation causes high long term damage. This new option controls the threshold to reuse and split an existing active extent. We avoid using a large extent for much smaller sizes, in order to reduce fragmentation. In some workload, adding the threshold improves virtual memory usage by >10x.
* Use extent_heap_first for best fit.Qi Wang2017-11-161-1/+1
| | | | | extent_heap_any makes the layout less predictable and as a result incurs more fragmentation.
* Fix unbounded increase in stash_decayed.Qi Wang2017-11-091-2/+3
| | | | | | Added an upper bound on how many pages we can decay during the current run. Without this, decay could have unbounded increase in stashed, since other threads could add new pages into the extents.
* Add arena.i.retain_grow_limitQi Wang2017-11-031-3/+4
| | | | | | | This option controls the max size when grow_retained. This is useful when we have customized extent hooks reserving physical memory (e.g. 1G huge pages). Without this feature, the default increasing sequence could result in fragmented and wasted physical memory.
* Add a "dumpable" bit to the extent state.David Goldblatt2017-10-161-6/+13
| | | | | Currently, this is unused (i.e. all extents are always marked dumpable). In the future, we'll begin using this functionality.
* Factor out extent-splitting core from extent lifetime management.David Goldblatt2017-10-161-81/+149
| | | | | | | Before this commit, extent_recycle_split intermingles the splitting of an extent and the return of parts of that extent to a given extents_t. After it, that logic is separated. This will enable splitting extents that don't live in any extents_t (as the grow retained region soon will).
* Document some of the internal extent functions.David Goldblatt2017-10-161-0/+35
|
* Use ph instead of rb tree for extents_avail_Dave Watson2017-10-041-1/+1
| | | | | | | | | | There does not seem to be any overlap between usage of extent_avail and extent_heap, so we can use the same hook. The only remaining usage of rb trees is in the profiling code, which has some 'interesting' iteration constraints. Fixes #888
* Relax constraints on reentrancy for extent hooks.Qi Wang2017-08-311-1/+12
| | | | | | If we guarantee no malloc activity in extent hooks, it's possible to make customized hooks working on arena 0. Remove the non-a0 assertion to enable such use cases.
* Bypass extent_alloc_wrapper_hard for no_move_expand.Qi Wang2017-07-311-0/+9
| | | | | | When retain is enabled, we should not attempt mmap for in-place expansion (large_ralloc_no_move), because it's virtually impossible to succeed, and causes unnecessary syscalls (which can cause lock contention under load).
* Check arena in current context in pre_reentrancy.Qi Wang2017-06-231-34/+34
|
* Set reentrancy when invoking customized extent hooks.Qi Wang2017-06-231-8/+76
| | | | | Customized extent hooks may malloc / free thus trigger reentry. Support this behavior by adding reentrancy on hook functions.
* Fix extent_hooks in extent_grow_retained().Qi Wang2017-06-141-3/+12
| | | | | | | This issue caused the default extent alloc function to be incorrectly used even when arena.<i>.extent_hooks is set. This bug was introduced by 411697adcda2fd75e135cdcdafb95f2bd295dc7f (Use exponential series to size extents.), which was first released in 5.0.0.
* Remove assertions on extent_hooks being default.Qi Wang2017-06-051-16/+0
| | | | | It's possible to customize the extent_hooks while still using part of the default implementation.
* Take background thread lock when setting extent hooks.Qi Wang2017-06-051-2/+12
|
* Header refactoring: Pull size helpers out of jemalloc module.David Goldblatt2017-05-311-11/+12
|
* Header refactoring: unify and de-catchall mutex_pool.David Goldblatt2017-05-311-0/+1
|
* Header refactoring: unify and de-catchall extent_mmap module.David Goldblatt2017-05-311-0/+1
|
* Header refactoring: unify and de-catchall extent_dss.David Goldblatt2017-05-311-0/+1
|
* Header refactoring: unify and de-catchall rtree module.David Goldblatt2017-05-311-0/+1
|
* Fix extent_grow_next management.Jason Evans2017-05-301-146/+206
| | | | | | | | | | | | | Fix management of extent_grow_next to serialize operations that may grow retained memory. This assures that the sizes of the newly allocated extents correspond to the size classes in the intended growth sequence. Fix management of extent_grow_next to skip size classes if a request is too large to be satisfied by the next size in the growth sequence. This avoids the potential for an arbitrary number of requests to bypass triggering extent_grow_next increases. This resolves #858.
* Fix OOM paths in extent_grow_retained().Jason Evans2017-05-301-2/+8
|
* Header refactoring: unify and de-catchall mutex moduleDavid Goldblatt2017-05-241-0/+1
|
* Header refactoring: unify and de-catchall witness code.David Goldblatt2017-05-241-18/+36
|
* Protect the rtree/extent interactions with a mutex pool.David Goldblatt2017-05-191-99/+145
| | | | | | | | | | | | | | | | | | Instead of embedding a lock bit in rtree leaf elements, we associate extents with a small set of mutexes. This gets us two things: - We can use the system mutexes. This (hypothetically) protects us from priority inversion, and lets us stop doing a backoff/sleep loop, instead opting for precise wakeups from the mutex. - Cuts down on the number of mutex acquisitions we have to do (from 4 in the worst case to two). We end up simplifying most of the rtree code (which no longer has to deal with locking or concurrency at all), at the cost of additional complexity in the extent code: since the mutex protecting the rtree leaf elements is determined by reading the extent out of those elements, the initial read is racy, so that we may acquire an out of date mutex. We re-check the extent in the leaf after acquiring the mutex to protect us from this race.
* Allow mutexes to take a lock ordering enum at construction.David Goldblatt2017-05-191-1/+2
| | | | | | | This lets us specify whether and how mutexes of the same rank are allowed to be acquired. Currently, we only allow two polices (only a single mutex at a given rank at a time, and mutexes acquired in ascending order), but we can plausibly allow more (e.g. the "release uncontended mutexes before blocking").
* Add extent_destroy_t and use it during arena destruction.Jason Evans2017-04-291-1/+46
| | | | | | | | | | Add the extent_destroy_t extent destruction hook to extent_hooks_t, and use it during arena destruction. This hook explicitly communicates to the callee that the extent must be destroyed or tracked for later reuse, lest it be permanently leaked. Prior to this change, retained extents could unintentionally be leaked if extent retention was enabled. This resolves #560.
* Refactor !opt.munmap to opt.retain.Jason Evans2017-04-291-1/+1
|
* Replace --disable-munmap with opt.munmap.Jason Evans2017-04-251-1/+1
| | | | | | | | | Control use of munmap(2) via a run-time option rather than a compile-time option (with the same per platform default). The old behavior of --disable-munmap can be achieved with --with-malloc-conf=munmap:false. This partially resolves #580.
* Trim before commit in extent_recycle().Jason Evans2017-04-201-2/+6
| | | | | | | | | | This avoids creating clean committed pages as a side effect of aligned allocation. For configurations that decommit memory, purged pages are decommitted, and decommitted extents cannot be coalesced with committed extents. Unless the clean committed pages happen to be selected during allocation, they cause unnecessary permanent extent fragmentation. This resolves #766.
* Header refactoring: move assert.h out of the catch-allDavid Goldblatt2017-04-191-1/+2
|
* Prefer old/low extent_t structures during reuse.Jason Evans2017-04-171-10/+11
| | | | | | Rather than using a LIFO queue to track available extent_t structures, use a red-black tree, and always choose the oldest/lowest available during reuse.
* Track extent structure serial number (esn) in extent_t.Jason Evans2017-04-171-2/+1
| | | | This enables stable sorting of extent_t structures.
* Header refactoring: Split up jemalloc_internal.hDavid Goldblatt2017-04-111-1/+3
| | | | | | | | | | | | | | This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
* Header refactoring: break out ph.h dependenciesDavid Goldblatt2017-04-111-0/+2
|
* Convert extent module to use C11-style atomcisDavid Goldblatt2017-04-051-8/+10
|
* Transition arena struct fields to C11 atomicsDavid Goldblatt2017-04-051-6/+9
|
* Move arena_slab_data_t's nfree into extent_t's e_bits.Jason Evans2017-03-281-1/+2
| | | | | | | | | | | | | | Compact extent_t to 128 bytes on 64-bit systems by moving arena_slab_data_t's nfree into extent_t's e_bits. Cacheline-align extent_t structures so that they always cross the minimum number of cacheline boundaries. Re-order extent_t fields such that all fields except the slab bitmap (and overlaid heap profiling context pointer) are in the first cacheline. This resolves #461.
* Use a bitmap in extents_t to speed up search.Jason Evans2017-03-251-11/+30
| | | | | Rather than iteratively checking all sufficiently large heaps during search, maintain and use a bitmap in order to skip empty heaps.
* Use first fit layout policy instead of best fit.Jason Evans2017-03-251-12/+42
| | | | | | | | | For extents which do not delay coalescing, use first fit layout policy rather than first-best fit layout policy. This packs extents toward older virtual memory mappings, but at the cost of higher search overhead in the common case. This resolves #711.
* Embed root node into rtree_t.Jason Evans2017-03-231-1/+1
| | | | This avoids one atomic operation per tree access.
* Incorporate szind/slab into rtree leaves.Jason Evans2017-03-231-48/+79
| | | | | | Expand and restructure the rtree API such that all common operations can be achieved with minimal work, regardless of whether the rtree leaf fields are independent versus packed into a single atomic pointer.
generated by cgit v0.12 at 2025-04-08 01:00:10 (GMT)