summaryrefslogtreecommitdiffstats
path: root/include
Commit message (Collapse)AuthorAgeFilesLines
* Simplify run quantization.Jason Evans2016-05-162-17/+1
|
* Refactor runs_avail.Jason Evans2016-05-164-11/+43
| | | | | | | | Use pszind_t size classes rather than szind_t size classes, and always reserve space for NPSIZES elements. This removes unused heaps that are not multiples of the page size, and adds (currently) unused heaps for all huge size classes, with the immediate benefit that the size of arena_t allocations is constant (no longer dependent on chunk size).
* Implement pz2ind(), pind2sz(), and psz2u().Jason Evans2016-05-133-11/+115
| | | | | | | These compute size classes and indices similarly to size2index(), index2size() and s2u(), respectively, but using the subset of size classes that are multiples of the page size. Note that pszind_t and szind_t are not interchangeable.
* Initialize arena_bin_info at compile time rather than at boot time.Jason Evans2016-05-132-14/+64
| | | | This resolves #370.
* Implement BITMAP_INFO_INITIALIZER(nbits).Jason Evans2016-05-131-19/+67
| | | | This allows static initialization of bitmap_info_t structures.
* Remove redzone support.Jason Evans2016-05-134-80/+16
| | | | This resolves #369.
* Remove quarantine support.Jason Evans2016-05-137-113/+13
|
* Remove Valgrind support.Jason Evans2016-05-135-138/+0
|
* Use TSDN_NULL rather than NULL as appropriate.Jason Evans2016-05-131-2/+2
|
* Optimize witness fast path.Jason Evans2016-05-113-14/+153
| | | | | | | | | | | Short-circuit commonly called witness functions so that they only execute in debug builds, and remove equivalent guards from mutex functions. This avoids pointless code execution in witness_assert_lockless(), which is typically called twice per allocation/deallocation function invocation. Inline commonly called witness functions so that optimized builds can completely remove calls as dead code.
* Resolve bootstrapping issues when embedded in FreeBSD libc.Jason Evans2016-05-1115-270/+362
| | | | | | | | | | | | | b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
* Add LG_QUANTUM definition for the RISC-V architecture.Jason Evans2016-05-071-0/+3
|
* Update private_symbols.txt.Jason Evans2016-05-061-2/+11
|
* Optimize the fast paths of calloc() and [m,d,sd]allocx().Jason Evans2016-05-063-55/+21
| | | | | | | | This is a broader application of optimizations to malloc() and free() in f4a0f32d340985de477bbe329ecdaecd69ed1055 (Fast-path improvement: reduce # of branches and unnecessary operations.). This resolves #321.
* Modify pages_map() to support mapping uncommitted virtual memory.Jason Evans2016-05-063-2/+13
| | | | | | | | | | | If the OS overcommits: - Commit all mappings in pages_map() regardless of whether the caller requested committed memory. - Linux-specific: Specify MAP_NORESERVE to avoid unfortunate interactions with heuristic overcommit mode during fork(2). This resolves #193.
* Add the stats.retained and stats.arenas.<i>.retained statistics.Jason Evans2016-05-042-0/+9
| | | | This resolves #367.
* Fix huge_palloc() regression.Jason Evans2016-05-043-6/+27
| | | | | | | | | | Split arena_choose() into arena_[i]choose() and use arena_ichoose() for arena lookup during internal allocation. This fixes huge_palloc() so that it always succeeds during extent node allocation. This regression was introduced by 66cd953514a18477eb49732e40d5c2ab5f1b12c5 (Do not allocate metadata via non-auto arenas, nor tcaches.).
* Fix witness/fork() interactions.Jason Evans2016-04-262-2/+4
| | | | | | Fix witness to clear its list of owned mutexes in the child if platform-specific malloc_mutex code re-initializes mutexes rather than unlocking them.
* Fix fork()-related lock rank ordering reversals.Jason Evans2016-04-265-5/+21
|
* Fix degenerate mb_write() compilation error.Jason Evans2016-04-231-3/+3
| | | | This resolves #375.
* Implement the arena.<i>.reset mallctl.Jason Evans2016-04-223-1/+7
| | | | | | | This makes it possible to discard all of an arena's allocations in a single operation. This resolves #146.
* Do not allocate metadata via non-auto arenas, nor tcaches.Jason Evans2016-04-226-23/+47
| | | | | This assures that all internally allocated metadata come from the first opt_narenas arenas, i.e. the automatically multiplexed arenas.
* Fix malloc_mutex_assert_[not_]owner() for --enable-lazy-lock case.Jason Evans2016-04-181-2/+2
|
* Update private_symbols.txt.Jason Evans2016-04-181-8/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | Change test-related mangling to simplify symbol filtering. The following commands can be used to detect missing/obsolete symbol mangling, with the caveat that the full set of symbols is based on the union of symbols generated by all configurations, some of which are platform-specific: ./autogen.sh --enable-debug --enable-prof --enable-lazy-lock make all tests nm -a lib/libjemalloc.a src/*.jet.o \ |grep " [TDBCR] " \ |awk '{print $3}' \ |sed -e 's/^\(je_\|jet_\(n_\)\?\)\([a-zA-Z0-9_]*\)/\3/g' \ |LC_COLLATE=C sort -u \ |grep -v \ -e '^\(malloc\|calloc\|posix_memalign\|aligned_alloc\|realloc\|free\)$' \ -e '^\(m\|r\|x\|s\|d\|sd\|n\)allocx$' \ -e '^mallctl\(\|nametomib\|bymib\)$' \ -e '^malloc_\(stats_print\|usable_size\|message\)$' \ -e '^\(memalign\|valloc\)$' \ -e '^__\(malloc\|memalign\|realloc\|free\)_hook$' \ -e '^pthread_create$' \ > /tmp/private_symbols.txt
* Update private_symbols.txtRajat Goel2016-04-181-0/+4
| | | Add 4 missing symbols
* Fix style nits.Jason Evans2016-04-171-1/+1
|
* Fix malloc_mutex_[un]lock() to conditionally check witness.Jason Evans2016-04-171-2/+2
| | | | Also remove tautological cassert(config_debug) calls.
* s/MALLOC_MUTEX_RANK_OMIT/WITNESS_RANK_OMIT/Jason Evans2016-04-141-1/+1
| | | | | | | | This fixes a compilation error caused by b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.). This resolves #375.
* Fix a compilation error.Jason Evans2016-04-141-2/+2
| | | | | | Fix a compilation error that occurs if Valgrind is not enabled. This regression was caused by b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.).
* Add witness, a simple online locking validator.Jason Evans2016-04-1415-187/+353
| | | | This resolves #358.
* Fix a style nit.Jason Evans2016-04-131-1/+2
|
* Simplify RTREE_HEIGHT_MAX definition.Jason Evans2016-04-111-29/+4
| | | | | Use 1U rather than ZU(1) in macro definitions, so that the preprocessor can evaluate the resulting expressions.
* Support --with-lg-page values larger than actual page size.Jason Evans2016-04-112-4/+4
| | | | | | | | | | | | | | During over-allocation in preparation for creating aligned mappings, allocate one more page than necessary if PAGE is the actual page size, so that trimming still succeeds even if the system returns a mapping that has less than PAGE alignment. This allows compiling with e.g. 64 KiB "pages" on systems that actually use 4 KiB pages. Note that for e.g. --with-lg-page=21, it is also necessary to increase the chunk size (e.g. --with-malloc-conf=lg_chunk:22) so that there are at least two "pages" per chunk. In practice this isn't a particularly compelling configuration because so much (unusable) virtual memory is dedicated to chunk headers.
* Clean up char vs. uint8_t in junk filling code.Jason Evans2016-04-111-2/+2
| | | | Consistently use uint8_t rather than char for junk filling code.
* Refactor/fix ph.Jason Evans2016-04-114-266/+331
| | | | | | | | | | | | | | | | | | | | | Refactor ph to support configurable comparison functions. Use a cpp macro code generation form equivalent to the rb macros so that pairing heaps can be used for both run heaps and chunk heaps. Remove per node parent pointers, and instead use leftmost siblings' prev pointers to track parents. Fix multi-pass sibling merging to iterate over intermediate results using a FIFO, rather than a LIFO. Use this fixed sibling merging implementation for both merge phases of the auxiliary twopass algorithm (first merging the aux list, then replacing the root with its merged children). This fixes both degenerate merge behavior and the potential for deep recursion. This regression was introduced by 6bafa6678fc36483e638f1c3a0a9bf79fb89bfc9 (Pairing heap). This resolves #371.
* Reduce differences between alternative bitmap implementations.Jason Evans2016-04-061-1/+1
|
* Fix bitmap_sfu() regression.Jason Evans2016-04-061-1/+1
| | | | | | | | | | | | Fix bitmap_sfu() to shift by LG_BITMAP_GROUP_NBITS rather than hard-coded 6 when using linear (non-USE_TREE) bitmap search. In practice this affects only 64-bit systems for which sizeof(long) is not 8 (i.e. Windows), since USE_TREE is defined for 32-bit systems. This regression was caused by b8823ab02607d6f03febd32ac504bb6188c54047 (Use linear scan for small bitmaps). This resolves #368.
* Add JEMALLOC_ALLOC_JUNK and JEMALLOC_FREE_JUNK macrosChris Peterson2016-03-312-3/+8
| | | | | Replace hardcoded 0xa5 and 0x5a junk values with JEMALLOC_ALLOC_JUNK and JEMALLOC_FREE_JUNK macros, respectively.
* Fix potential chunk leaks.Jason Evans2016-03-312-7/+1
| | | | | | | | | | | | Move chunk_dalloc_arena()'s implementation into chunk_dalloc_wrapper(), so that if the dalloc hook fails, proper decommit/purge/retain cascading occurs. This fixes three potential chunk leaks on OOM paths, one during dss-based chunk allocation, one during chunk header commit (currently relevant only on Windows), and one during rtree write (e.g. if rtree node allocation fails). Merge chunk_purge_arena() into chunk_purge_default() (refactor, no change to functionality).
* Remove unused arenas_extend() function declaration.Chris Peterson2016-03-261-1/+0
| | | | | | The arenas_extend() function was renamed to arenas_init() in commit 8bb3198f72fc7587dc93527f9f19fb5be52fa553, but its function declaration was not removed from jemalloc_internal.h.in.
* Use abort() for fallback implementations of unreachable().Jason Evans2016-03-241-2/+2
|
* Constify various internal arena APIs.Jason Evans2016-03-232-49/+82
|
* Always inline performance-critical rtree operations.Jason Evans2016-03-231-9/+10
|
* Optimize rtree_get().Jason Evans2016-03-231-35/+131
| | | | | | | Specialize fast path to avoid code that cannot execute for dependent loads. Manually unroll.
* Refactor out signed/unsigned comparisons.Jason Evans2016-03-151-2/+2
|
* Add (size_t) casts to MALLOCX_ALIGN().Jason Evans2016-03-111-4/+4
| | | | | | | | Add (size_t) casts to MALLOCX_ALIGN() macros so that passing the integer constant 0x80000000 does not cause a compiler warning about invalid shift amount. This resolves #354.
* typecast address to pointer to byte to avoid unaligned memory access errorRajeev Misra2016-03-111-2/+2
|
* Convert arena_bin_t's runs from a tree to a heap.Jason Evans2016-03-081-14/+8
|
* Use pairing heap for arena->runs_availDave Watson2016-03-082-5/+25
| | | | | | | | | | | | | | Use pairing heap instead of red black tree in arena runs_avail. The extra links are unioned with the bitmap_t, so this change doesn't use any extra memory. Canaries show this change to be a 1% cpu win, and 2% latency win. In particular, large free()s, and small bin frees are now O(1) (barring coalescing). I also tested changing bin->runs to be a pairing heap, but saw a much smaller win, and it would mean increasing the size of arena_run_s by two pointers, so I left that as an rb-tree for now.
* Refactor ph_merge_ordered() out of ph_merge().Jason Evans2016-03-082-17/+23
|
generated by cgit v0.12 at 2025-09-03 03:43:26 (GMT)