summaryrefslogtreecommitdiffstats
path: root/include/jemalloc/internal/tcache.h
Commit message (Collapse)AuthorAgeFilesLines
* Break up headers into constituent partsDavid Goldblatt2017-01-121-459/+0
| | | | | | | | | | This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
* Do not (recursively) allocate within tsd_fetch().Jason Evans2016-10-211-1/+1
| | | | | | | Refactor tsd so that tsdn_fetch() does not trigger allocation, since allocation could cause infinite recursion. This resolves #458.
* Make tsd cleanup functions optional, remove noop cleanup functions.Jason Evans2016-06-061-1/+0
|
* Rename huge to large.Jason Evans2016-06-061-12/+12
|
* Move slabs out of chunks.Jason Evans2016-06-061-1/+1
|
* Use huge size class infrastructure for large size classes.Jason Evans2016-06-061-21/+12
|
* Refactor rtree to always use base_alloc() for node allocation.Jason Evans2016-06-031-2/+2
|
* Use rtree-based chunk lookups rather than pointer bit twiddling.Jason Evans2016-06-031-1/+1
| | | | | | | | | | | Look up chunk metadata via the radix tree, rather than using CHUNK_ADDR2BASE(). Propagate pointer's containing extent. Minimize extent lookups by doing a single lookup (e.g. in free()) and propagating the pointer's extent into nearly all the functions that may need it.
* Resolve bootstrapping issues when embedded in FreeBSD libc.Jason Evans2016-05-111-17/+16
| | | | | | | | | | | | | b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
* Fix huge_palloc() regression.Jason Evans2016-05-041-4/+3
| | | | | | | | | | Split arena_choose() into arena_[i]choose() and use arena_ichoose() for arena lookup during internal allocation. This fixes huge_palloc() so that it always succeeds during extent node allocation. This regression was introduced by 66cd953514a18477eb49732e40d5c2ab5f1b12c5 (Do not allocate metadata via non-auto arenas, nor tcaches.).
* Do not allocate metadata via non-auto arenas, nor tcaches.Jason Evans2016-04-221-4/+6
| | | | | This assures that all internally allocated metadata come from the first opt_narenas arenas, i.e. the automatically multiplexed arenas.
* Add witness, a simple online locking validator.Jason Evans2016-04-141-11/+11
| | | | This resolves #358.
* Add JEMALLOC_ALLOC_JUNK and JEMALLOC_FREE_JUNK macrosChris Peterson2016-03-311-3/+4
| | | | | Replace hardcoded 0xa5 and 0x5a junk values with JEMALLOC_ALLOC_JUNK and JEMALLOC_FREE_JUNK macros, respectively.
* Make *allocx() size class overflow behavior defined.Jason Evans2016-02-251-5/+5
| | | | | | | Limit supported size and alignment to HUGE_MAXCLASS, which in turn is now limited to be less than PTRDIFF_MAX. This resolves #278 and #295.
* Make nhbins unsigned rather than size_t.Jason Evans2016-02-241-1/+1
|
* Implement decay-based unused dirty page purging.Jason Evans2016-02-201-1/+1
| | | | | | | | | | | | | | | | This is an alternative to the existing ratio-based unused dirty page purging, and is intended to eventually become the sole purging mechanism. Add mallctls: - opt.purge - opt.decay_time - arena.<i>.decay - arena.<i>.decay_time - arenas.decay_time - stats.arenas.<i>.decay_time This resolves #325.
* Use ticker for incremental tcache GC.Jason Evans2016-02-201-4/+2
|
* Fix a comment.Jason Evans2015-11-121-1/+1
|
* Fast-path improvement: reduce # of branches and unnecessary operations.Qi Wang2015-11-101-36/+80
| | | | | | - Combine multiple runtime branches into a single malloc_slow check. - Avoid calling arena_choose / size2index / index2size on fast path. - A few micro optimizations.
* Rename index_t to szind_t to avoid an existing type on Solaris.Jason Evans2015-08-191-9/+9
| | | | This resolves #256.
* Impose a minimum tcache count for small size classes.Jason Evans2015-05-201-0/+5
| | | | | | Now that small allocation runs have fewer regions due to run metadata residing in chunk headers, an explicit minimum tcache count is needed to make sure that tcache adequately amortizes synchronization overhead.
* If MALLOCX_ARENA(a) is specified, use it during tcache fill.Jason Evans2015-02-131-13/+15
|
* Implement explicit tcache support.Jason Evans2015-02-101-33/+69
| | | | | | | | | Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be used in conjunction with the *allocx() API. Add the tcache.create, tcache.flush, and tcache.destroy mallctls. This resolves #145.
* Introduce two new modes of junk filling: "alloc" and "free".Guilherme Goncalves2014-12-151-5/+5
| | | | | | | | In addition to true/false, opt.junk can now be either "alloc" or "free", giving applications the possibility of junking memory only on allocation or deallocation. This resolves #172.
* Style and spelling fixes.Jason Evans2014-12-091-1/+1
|
* Add configure options.Jason Evans2014-10-101-2/+2
| | | | | | | | | | | | Add: --with-lg-page --with-lg-page-sizes --with-lg-size-class-group --with-lg-quantum Get rid of STATIC_PAGE_SHIFT, in favor of directly setting LG_PAGE. Fix various edge conditions exposed by the configure options.
* Refactor/fix arenas manipulation.Jason Evans2014-10-081-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Abstract arenas access to use arena_get() (or a0get() where appropriate) rather than directly reading e.g. arenas[ind]. Prior to the addition of the arenas.extend mallctl, the worst possible outcome of directly accessing arenas was a stale read, but arenas.extend may allocate and assign a new array to arenas. Add a tsd-based arenas_cache, which amortizes arenas reads. This introduces some subtle bootstrapping issues, with tsd_boot() now being split into tsd_boot[01]() to support tsd wrapper allocation bootstrapping, as well as an arenas_cache_bypass tsd variable which dynamically terminates allocation of arenas_cache itself. Promote a0malloc(), a0calloc(), and a0free() to be generally useful for internal allocation, and use them in several places (more may be appropriate). Abstract arena->nthreads management and fix a missing decrement during thread destruction (recent tsd refactoring left arenas_cleanup() unused). Change arena_choose() to propagate OOM, and handle OOM in all callers. This is important for providing consistent allocation behavior when the MALLOCX_ARENA() flag is being used. Prior to this fix, it was possible for an OOM to result in allocation silently allocating from a different arena than the one specified.
* Normalize size classes.Jason Evans2014-10-061-25/+27
| | | | | | | | | | Normalize size classes to use the same number of size classes per size doubling (currently hard coded to 4), across the intire range of size classes. Small size classes already used this spacing, but in order to support this change, additional small size classes now fill [4 KiB .. 16 KiB). Large size classes range from [16 KiB .. 4 MiB). Huge size classes now support non-multiples of the chunk size in order to fill (4 MiB .. 16 MiB).
* Don't disable tcache for lazy-lock.Jason Evans2014-10-041-2/+0
| | | | | | Don't disable tcache when lazy-lock is configured. There already exists a mechanism to disable tcache, but doing so automatically due to lazy-lock causes surprising performance behavior.
* Fix tsd cleanup regressions.Jason Evans2014-10-041-16/+5
| | | | | | | | | | | | | | | | Fix tsd cleanup regressions that were introduced in 5460aa6f6676c7f253bfcb75c028dfd38cae8aaf (Convert all tsd variables to reside in a single tsd structure.). These regressions were twofold: 1) tsd_tryget() should never (and need never) return NULL. Rename it to tsd_fetch() and simplify all callers. 2) tsd_*_set() must only be called when tsd is in the nominal state, because cleanup happens during the nominal-->purgatory transition, and re-initialization must not happen while in the purgatory state. Add tsd_nominal() and use it as needed. Note that tsd_*{p,}_get() can still be used as long as no re-initialization that would require cleanup occurs. This means that e.g. the thread_allocated counter can be updated unconditionally.
* Convert to uniform style: cond == false --> !condJason Evans2014-10-031-4/+4
|
* Convert all tsd variables to reside in a single tsd structure.Jason Evans2014-09-231-51/+37
|
* Apply likely()/unlikely() to allocation/deallocation fast paths.Jason Evans2014-09-121-16/+16
|
* mark some conditions as unlikelyDaniel Micay2014-09-111-1/+1
| | | | | | | | | | | | * assertion failure * malloc_init failure * malloc not already initialized (in malloc_init) * running in valgrind * thread cache disabled at runtime Clang and GCC already consider a comparison with NULL or -1 to be cold, so many branches (out-of-memory) are already correctly considered as cold and marking them is not important.
* Refactor small_size2bin and small_bin2size.Jason Evans2014-04-171-2/+2
| | | | | Refactor small_size2bin and small_bin2size to be inline functions rather than directly accessed arrays.
* Merge pull request #73 from bmaurer/smallmallocJason Evans2014-04-161-36/+5
|\ | | | | Smaller malloc hot path
| * Create a const array with only a small bin to size mapBen Maurer2014-04-161-3/+3
| |
| * outline rare tcache_get codepathsBen Maurer2014-04-161-33/+2
| |
* | Optimize Valgrind integration.Jason Evans2014-04-151-6/+1
|/ | | | | | | | | | | Forcefully disable tcache if running inside Valgrind, and remove Valgrind calls in tcache-specific code. Restructure Valgrind-related code to move most Valgrind calls out of the fast path functions. Take advantage of static knowledge to elide some branches in JEMALLOC_VALGRIND_REALLOC().
* Remove support for non-prof-promote heap profiling metadata.Jason Evans2014-04-111-1/+1
| | | | | | | | | | | | | | | Make promotion of sampled small objects to large objects mandatory, so that profiling metadata can always be stored in the chunk map, rather than requiring one pointer per small region in each small-region page run. In practice the non-prof-promote code was only useful when using jemalloc to track all objects and report them as leaks at program exit. However, Valgrind is at least as good a tool for this particular use case. Furthermore, the non-prof-promote code is getting in the way of some optimizations that will make heap profiling much cheaper for the predominant use case (sampling a small representative proportion of all allocations).
* Don't junk-fill reallocations unless usize changes.Jason Evans2013-12-161-0/+1
| | | | | | | | | | | Don't junk fill reallocations for which the request size is less than the current usable size, but not enough smaller to cause a size class change. Unlike malloc()/calloc()/realloc(), *allocx() contractually treats the full usize as the allocation, so a caller can ask for zeroed memory via mallocx() and a series of rallocx() calls that all specify MALLOCX_ZERO, and be assured that all newly allocated bytes will be zeroed and made available to the application without danger of allocator mutation until the size class decreases enough to cause usize reduction.
* Fix a Valgrind integration flaw.Jason Evans2013-10-201-2/+2
| | | | | | Fix a Valgrind integration flaw that caused Valgrind warnings about reads of uninitialized memory in internal zero-initialized data structures (relevant to tcache and prof code).
* Fix Valgrind integration.Jason Evans2013-02-011-2/+2
| | | | | Fix Valgrind integration to annotate all internally allocated memory in a way that keeps Valgrind happy about internal data structure access.
* Add and use JEMALLOC_ALWAYS_INLINE.Jason Evans2013-01-221-9/+9
| | | | | Add JEMALLOC_ALWAYS_INLINE and use it to guarantee that the entire fast paths of the primary allocation/deallocation functions are inlined.
* Tighten valgrind integration.Jason Evans2013-01-221-0/+2
| | | | | | | Tighten valgrind integration such that immediately after memory is validated or zeroed, valgrind is told to forget the memory's 'defined' state. The only place newly allocated memory should be left marked as 'defined' is in the public functions (e.g. calloc() and realloc()).
* Optimize malloc() and free() fast paths.Jason Evans2012-05-021-63/+8
| | | | | | | | | | Embed the bin index for small page runs into the chunk page map, in order to omit [...] in the following dependent load sequence: ptr-->mapelm-->[run-->bin-->]bin_info Move various non-critcal code out of the inlined function chain into helper functions (tcache_event_hard(), arena_dalloc_small(), and locking).
* Make arena_salloc() an inline function.Jason Evans2012-04-201-6/+7
|
* Implement Valgrind support, redzones, and quarantine.Jason Evans2012-04-111-10/+19
| | | | | | | | | | | | | Implement Valgrind support, as well as the redzone and quarantine features, which help Valgrind detect memory errors. Redzones are only implemented for small objects because the changes necessary to support redzones around large and huge objects are complicated by in-place reallocation, to the point that it isn't clear that the maintenance burden is worth the incremental improvement to Valgrind support. Merge arena_salloc() and arena_salloc_demote(). Refactor i[v]salloc() to expose the 'demote' option.
* Add a0malloc(), a0calloc(), and a0free().Jason Evans2012-04-041-1/+1
| | | | | Add a0malloc(), a0calloc(), and a0free(), which are used by FreeBSD's libc to allocate/deallocate TLS in static binaries.
* Clean up *PAGE* macros.Jason Evans2012-04-021-5/+5
| | | | | | | | | | | s/PAGE_SHIFT/LG_PAGE/g and s/PAGE_SIZE/PAGE/g. Remove remnants of the dynamic-page-shift code. Rename the "arenas.pagesize" mallctl to "arenas.page". Remove the "arenas.chunksize" mallctl, which is redundant with "opt.lg_chunk".
generated by cgit v0.12 at 2025-03-31 08:40:23 (GMT)