summaryrefslogtreecommitdiffstats
path: root/src/chunk.c
Commit message (Collapse)AuthorAgeFilesLines
* Add/use chunk_merge_wrapper().Jason Evans2016-06-031-32/+47
|
* Add/use chunk_commit_wrapper().Jason Evans2016-06-031-0/+9
|
* Add/use chunk_decommit_wrapper().Jason Evans2016-06-031-0/+9
|
* Merge chunk_alloc_base() into its only caller.Jason Evans2016-06-031-20/+0
|
* Replace extent_tree_szad_* with extent_heap_*.Jason Evans2016-06-031-32/+53
|
* Use rtree rather than [sz]ad trees for chunk split/coalesce operations.Jason Evans2016-06-031-152/+221
|
* Remove redundant chunk argument from chunk_{,de,re}register().Jason Evans2016-06-031-10/+12
|
* Replace extent_achunk_[gs]et() with extent_slab_[gs]et().Jason Evans2016-06-031-5/+5
|
* Add extent_active_[gs]et().Jason Evans2016-06-031-5/+6
| | | | Always initialize extents' runs_dirty and chunks_cache linkage.
* Set/unset rtree node for last chunk of extents.Jason Evans2016-06-031-4/+41
| | | | | Set/unset rtree node for last chunk of extents, so that the rtree can be used for chunk coalescing.
* Refactor rtree to always use base_alloc() for node allocation.Jason Evans2016-06-031-12/+5
|
* Use rtree-based chunk lookups rather than pointer bit twiddling.Jason Evans2016-06-031-0/+9
| | | | | | | | | | | Look up chunk metadata via the radix tree, rather than using CHUNK_ADDR2BASE(). Propagate pointer's containing extent. Minimize extent lookups by doing a single lookup (e.g. in free()) and propagating the pointer's extent into nearly all the functions that may need it.
* Add element acquire/release capabilities to rtree.Jason Evans2016-06-031-7/+5
| | | | | | | | This makes it possible to acquire short-term "ownership" of rtree elements so that it is possible to read an extent pointer *and* read the extent's contents with a guarantee that the element will not be modified until the ownership is released. This is intended as a mechanism for resolving rtree read/write races rather than as a way to lock extents.
* Rename extent_node_t to extent_t.Jason Evans2016-05-161-95/+93
|
* Remove Valgrind support.Jason Evans2016-05-131-10/+0
|
* Resolve bootstrapping issues when embedded in FreeBSD libc.Jason Evans2016-05-111-73/+73
| | | | | | | | | | | | | b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
* Add the stats.retained and stats.arenas.<i>.retained statistics.Jason Evans2016-05-041-2/+11
| | | | This resolves #367.
* Add witness, a simple online locking validator.Jason Evans2016-04-141-89/+97
| | | | This resolves #358.
* Fix potential chunk leaks.Jason Evans2016-03-311-35/+16
| | | | | | | | | | | | Move chunk_dalloc_arena()'s implementation into chunk_dalloc_wrapper(), so that if the dalloc hook fails, proper decommit/purge/retain cascading occurs. This fixes three potential chunk leaks on OOM paths, one during dss-based chunk allocation, one during chunk header commit (currently relevant only on Windows), and one during rtree write (e.g. if rtree node allocation fails). Merge chunk_purge_arena() into chunk_purge_default() (refactor, no change to functionality).
* Move retaining out of default chunk hooksbuchgr2016-02-261-11/+25
| | | | | | | This fixes chunk allocation to reuse retained memory even if an application-provided chunk allocation function is in use. This resolves #307.
* Refactor arenas array (fixes deadlock).Jason Evans2016-02-251-3/+1
| | | | | | | | | | | | Refactor the arenas array, which contains pointers to all extant arenas, such that it starts out as a sparse array of maximum size, and use double-checked atomics-based reads as the basis for fast and simple arena_get(). Additionally, reduce arenas_lock's role such that it only protects against arena initalization races. These changes remove the possibility for arena lookups to trigger locking, which resolves at least one known (fork-related) deadlock. This resolves #315.
* Attempt mmap-based in-place huge reallocation.Jason Evans2016-02-251-7/+4
| | | | | | | | Attempt mmap-based in-place huge reallocation by plumbing new_addr into chunk_alloc_mmap(). This can dramatically speed up incremental huge reallocation. This resolves #335.
* Silence miscellaneous 64-to-32-bit data loss warnings.Jason Evans2016-02-241-2/+2
|
* Refactor jemalloc_ffs*() into ffs_*().Jason Evans2016-02-241-1/+1
| | | | Use appropriate versions to resolve 64-to-32-bit data loss warnings.
* Fix a strict aliasing violation.Jason Evans2015-08-121-1/+6
|
* Fix chunk_dalloc_arena() re: zeroing due to purge.Jason Evans2015-08-121-1/+1
|
* Arena chunk decommit cleanups and fixes.Jason Evans2015-08-111-2/+2
| | | | | Decommit arena chunk header during chunk deallocation if the rest of the chunk is decommitted.
* Implement chunk hook support for page run commit/decommit.Jason Evans2015-08-071-46/+80
| | | | | | | | | Cascade from decommit to purge when purging unused dirty pages, so that it is possible to decommit cleaned memory rather than just purging. For non-Windows debug builds, decommit runs rather than purging them, since this causes access of deallocated runs to segfault. This resolves #251.
* Generalize chunk management hooks.Jason Evans2015-08-041-100/+246
| | | | | | | | | | | | | | | | | | | | Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks allow control over chunk allocation/deallocation, decommit/commit, purging, and splitting/merging, such that the application can rely on jemalloc's internal chunk caching and retaining functionality, yet implement a variety of chunk management mechanisms and policies. Merge the chunks_[sz]ad_{mmap,dss} red-black trees into chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries to honor the dss precedence setting; prior to this change the precedence setting was also consulted when recycling chunks. Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead deallocate them in arena_unstash_purged(), so that the dirty memory linkage remains valid until after the last time it is used. This resolves #176 and #201.
* Implement support for non-coalescing maps on MinGW.Jason Evans2015-07-251-0/+6
| | | | | | | | - Do not reallocate huge objects in place if the number of backing chunks would change. - Do not cache multi-chunk mappings. This resolves #213.
* Revert to first-best-fit run/chunk allocation.Jason Evans2015-07-161-35/+9
| | | | | | | | | This effectively reverts 97c04a93838c4001688fe31bf018972b4696efe2 (Use first-fit rather than first-best-fit run/chunk allocation.). In some pathological cases, first-fit search dominates allocation time, and it also tends not to converge as readily on a steady state of memory layout, since precise allocation order has a bigger effect than for first-best-fit.
* Use jemalloc_ffs() rather than ffs().Jason Evans2015-07-081-4/+12
|
* Optimizations for WindowsMatthijs2015-06-251-1/+16
| | | | | | | - Set opt_lg_chunk based on run-time OS setting - Verify LG_PAGE is compatible with run-time OS setting - When targeting Windows Vista or newer, use SRWLOCK instead of CRITICAL_SECTION - When targeting Windows Vista or newer, statically initialize init_lock
* Fix two valgrind integration regressions.Jason Evans2015-06-221-2/+8
| | | | The regressions were never merged into the master branch.
* Implement dynamic per arena control over dirty page purging.Jason Evans2015-03-191-2/+35
| | | | | | | | | | | | | | Add mallctls: - arenas.lg_dirty_mult is initialized via opt.lg_dirty_mult, and can be modified to change the initial lg_dirty_mult setting for newly created arenas. - arena.<i>.lg_dirty_mult controls an individual arena's dirty page purging threshold, and synchronously triggers any purging that may be necessary to maintain the constraint. - arena.<i>.chunk.purge allows the per arena dirty page purging function to be replaced. This resolves #93.
* Fix a chunk_recycle() regression.Jason Evans2015-03-071-4/+15
| | | | | | This regression was introduced by 97c04a93838c4001688fe31bf018972b4696efe2 (Use first-fit rather than first-best-fit run/chunk allocation.).
* Use first-fit rather than first-best-fit run/chunk allocation.Jason Evans2015-03-071-4/+39
| | | | | | | This tends to more effectively pack active memory toward low addresses. However, additional tree searches are required in many cases, so whether this change stands the test of time will depend on real-world benchmarks.
* Quantize szad trees by size class.Jason Evans2015-03-071-1/+1
| | | | | | | Treat sizes that round down to the same size class as size-equivalent in trees that are used to search for first best fit, so that there are only as many "firsts" as there are size classes. This comes closer to the ideal of first fit.
* Fix a compilation error and an incorrect assertion.Jason Evans2015-02-191-2/+2
|
* Fix chunk cache races.Jason Evans2015-02-191-35/+79
| | | | | | These regressions were introduced by ee41ad409a43d12900a5a3108f6c14f84e4eb0eb (Integrate whole chunks into unused dirty page purging machinery.).
* Rename "dirty chunks" to "cached chunks".Jason Evans2015-02-181-22/+23
| | | | | | | | | | Rename "dirty chunks" to "cached chunks", in order to avoid overloading the term "dirty". Fix the regression caused by 339c2b23b2d61993ac768afcc72af135662c6771 (Fix chunk_unmap() to propagate dirty state.), and actually address what that change attempted, which is to only purge chunks once, and propagate whether zeroed pages resulted into chunk_record().
* Fix chunk_unmap() to propagate dirty state.Jason Evans2015-02-181-3/+3
| | | | | | | | | | | | Fix chunk_unmap() to propagate whether a chunk is dirty, and modify dirty chunk purging to record this information so it can be passed to chunk_unmap(). Since the broken version of chunk_unmap() claimed that all chunks were clean, this resulted in potential memory corruption for purging implementations that do not zero (e.g. MADV_FREE). This regression was introduced by ee41ad409a43d12900a5a3108f6c14f84e4eb0eb (Integrate whole chunks into unused dirty page purging machinery.).
* Simplify extent_node_t and add extent_node_init().Jason Evans2015-02-171-14/+11
|
* Integrate whole chunks into unused dirty page purging machinery.Jason Evans2015-02-171-53/+91
| | | | | | | | | | | | Extend per arena unused dirty page purging to manage unused dirty chunks in aaddtion to unused dirty runs. Rather than immediately unmapping deallocated chunks (or purging them in the --disable-munmap case), store them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially allocate dirty chunks. When excessive unused dirty pages accumulate, purge runs and chunks in ingegrated LRU order (and unmap chunks in the --enable-munmap case). Refactor extent_node_t to provide accessor functions.
* add missing check for new_addr chunk sizeDaniel Micay2015-02-121-1/+1
| | | | | | 8ddc93293cd8370870f221225ef1e013fbff6d65 switched this to over using the address tree in order to avoid false negatives, so it now needs to check that the size of the free extent is large enough to satisfy the request.
* Move centralized chunk management into arenas.Jason Evans2015-02-121-172/+103
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Migrate all centralized data structures related to huge allocations and recyclable chunks into arena_t, so that each arena can manage huge allocations and recyclable virtual memory completely independently of other arenas. Add chunk node caching to arenas, in order to avoid contention on the base allocator. Use chunks_rtree to look up huge allocations rather than a red-black tree. Maintain a per arena unsorted list of huge allocations (which will be needed to enumerate huge allocations during arena reset). Remove the --enable-ivsalloc option, make ivsalloc() always available, and use it for size queries if --enable-debug is enabled. The only practical implications to this removal are that 1) ivsalloc() is now always available during live debugging (and the underlying radix tree is available during core-based debugging), and 2) size query validation can no longer be enabled independent of --enable-debug. Remove the stats.chunks.{current,total,high} mallctls, and replace their underlying statistics with simpler atomically updated counters used exclusively for gdump triggering. These statistics are no longer very useful because each arena manages chunks independently, and per arena statistics provide similar information. Simplify chunk synchronization code, now that base chunk allocation cannot cause recursive lock acquisition.
* Refactor rtree to be lock-free.Jason Evans2015-02-051-12/+13
| | | | | | | | | | | | | | | | | | Recent huge allocation refactoring associates huge allocations with arenas, but it remains necessary to quickly look up huge allocation metadata during reallocation/deallocation. A global radix tree remains a good solution to this problem, but locking would have become the primary bottleneck after (upcoming) migration of chunk management from global to per arena data structures. This lock-free implementation uses double-checked reads to traverse the tree, so that in the steady state, each read or write requires only a single atomic operation. This implementation also assures that no more than two tree levels actually exist, through a combination of careful virtual memory allocation which makes large sparse nodes cheap, and skipping the root node on x64 (possible because the top 16 bits are all 0 in practice).
* Refactor base_alloc() to guarantee demand-zeroed memory.Jason Evans2015-02-051-7/+10
| | | | | | | | | | | | Refactor base_alloc() to guarantee that allocations are carved from demand-zeroed virtual memory. This supports sparse data structures such as multi-page radix tree nodes. Enhance base_alloc() to keep track of fragments which were too small to support previous allocation requests, and try to consume them during subsequent requests. This becomes important when request sizes commonly approach or exceed the chunk size (as could radix tree node allocations).
* Fix chunk_recycle()'s new_addr functionality.Jason Evans2015-02-051-2/+6
| | | | | | | | Fix chunk_recycle()'s new_addr functionality to search by address rather than just size if new_addr is specified. The functionality added by a95018ee819abf897562d9d1f3bc31d4dd725a8d (Attempt to expand huge allocations in-place.) only worked if the two search orders happened to return the same results (e.g. in simple test cases).
* Implement the prof.gdump mallctl.Jason Evans2015-01-261-1/+2
| | | | | | | | This feature makes it possible to toggle the gdump feature on/off during program execution, whereas the the opt.prof_dump mallctl value can only be set during program startup. This resolves #72.
generated by cgit v0.12 at 2025-03-30 15:29:57 (GMT)