summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* Remove --enable-ivsalloc.Jason Evans2017-04-215-42/+23
| | | | | | | | | | | | Continue to use ivsalloc() when --enable-debug is specified (and add assertions to guard against 0 size), but stop providing a documented explicit semantics-changing band-aid to dodge undefined behavior in sallocx() and malloc_usable_size(). ivsalloc() remains compiled in, unlike when #211 restored --enable-ivsalloc, and if JEMALLOC_FORCE_IVSALLOC is defined during compilation, sallocx() and malloc_usable_size() will still use ivsalloc(). This partially resolves #580.
* Remove --disable-tls.Jason Evans2017-04-216-43/+4
| | | | | | | This option is no longer useful, because TLS is correctly configured automatically on all supported platforms. This partially resolves #580.
* Use openat syscall if availableJim Chen2017-04-212-0/+8
| | | | | | | | | | Some architectures like AArch64 may not have the open syscall because it was superseded by the openat syscall, so check and use SYS_openat if SYS_open is not available. Additionally, Android headers for AArch64 define SYS_open to __NR_open, even though __NR_open is undefined. Undefine SYS_open in that case so SYS_openat is used.
* Remove --disable-tcache.Jason Evans2017-04-2125-416/+192
| | | | | | | | | | | Simplify configuration by removing the --disable-tcache option, but replace the testing for that configuration with --with-malloc-conf=tcache:false. Fix the thread.arena and thread.tcache.flush mallctls to work correctly if tcache is disabled. This partially resolves #580.
* Bypass extent tracking for auto arenas.Qi Wang2017-04-216-32/+49
| | | | | | | | Tracking extents is required by arena_reset. To support this, the extent linkage was used for tracking 1) large allocations, and 2) full slabs. However modifying the extent linkage could be an expensive operation as it likely incurs cache misses. Since we forbid arena_reset on auto arenas, let's bypass the linkage operations for auto arenas.
* Trim before commit in extent_recycle().Jason Evans2017-04-202-3/+9
| | | | | | | | | | This avoids creating clean committed pages as a side effect of aligned allocation. For configurations that decommit memory, purged pages are decommitted, and decommitted extents cannot be coalesced with committed extents. Unless the clean committed pages happen to be selected during allocation, they cause unnecessary permanent extent fragmentation. This resolves #766.
* Output 4 counters for bin mutexes instead of just 2.Qi Wang2017-04-191-8/+24
|
* Support --with-lg-page values larger than system page size.Jason Evans2017-04-198-111/+155
| | | | | | | | | All mappings continue to be PAGE-aligned, even if the system page size is smaller. This change is primarily intended to provide a mechanism for supporting multiple page sizes with the same binary; smaller page sizes work better in conjunction with jemalloc's design. This resolves #467.
* Revert "Remove BITMAP_USE_TREE."Jason Evans2017-04-195-0/+307
| | | | | | | | | Some systems use a native 64 KiB page size, which means that the bitmap for the smallest size class can be 8192 bits, not just 512 bits as when the page size is 4 KiB. Linear search in bitmap_{sfu,ffu}() is unacceptably slow for such large bitmaps. This reverts commit 7c00f04ff40a34627e31488d02ff1081c749c7ba.
* Header refactoring: unify spin.h and move it out of the catch-all.David Goldblatt2017-04-199-49/+42
|
* Header refactoring: unify nstime.h and move it out of the catch-allDavid Goldblatt2017-04-1910-29/+21
|
* Header refactoring: move jemalloc_internal_types.h out of the catch-allDavid Goldblatt2017-04-197-3/+10
|
* Header refactoring: move assert.h out of the catch-allDavid Goldblatt2017-04-1922-7/+34
|
* Header refactoring: move util.h out of the catchallDavid Goldblatt2017-04-1911-1/+18
|
* Header refactoring: move malloc_io.h out of the catchallDavid Goldblatt2017-04-199-1/+14
|
* Header refactoring: move bit_util.h out of the catchallDavid Goldblatt2017-04-195-1/+6
|
* Move CPP_PROLOGUE and CPP_EPILOGUE to the .cppDavid Goldblatt2017-04-195-21/+8
| | | | This lets us avoid having to specify them in every C file.
* Only disable munmap(2) by default on 64-bit Linux.Jason Evans2017-04-172-5/+11
| | | | | | | This reduces the likelihood of address space exhaustion on 32-bit systems. This resolves #350.
* Fix LD_PRELOAD_VAR configuration logic for 64-bit AIX.Jason Evans2017-04-171-1/+1
|
* Remove the function alignment of prof_backtrace.Qi Wang2017-04-171-1/+0
| | | | | This was an attempt to avoid triggering slow path in libunwind, however turns out to be ineffective.
* Prefer old/low extent_t structures during reuse.Jason Evans2017-04-1710-37/+80
| | | | | | Rather than using a LIFO queue to track available extent_t structures, use a red-black tree, and always choose the oldest/lowest available during reuse.
* Track extent structure serial number (esn) in extent_t.Jason Evans2017-04-177-48/+121
| | | | This enables stable sorting of extent_t structures.
* Allocate increasingly large base blocks.Jason Evans2017-04-172-27/+44
| | | | | Limit the total number of base block by leveraging the exponential size class sequence, similarly to extent_grow_retained().
* Update base_unmap() to match extent_dalloc_wrapper().Jason Evans2017-04-171-10/+10
| | | | | | | Reverse the order of forced versus lazy purging attempts in base_unmap(), in order to match the order in extent_dalloc_wrapper(), which was reversed by 64e458f5cdd64f9b67cb495f177ef96bf3ce4e0e (Implement two-phase decay-based purging.).
* Improve rtree cache with a two-level cache design.Qi Wang2017-04-175-35/+97
| | | | | | | | Two levels of rcache is implemented: a direct mapped cache as L1, combined with a LRU cache as L2. The L1 cache offers low cost on cache hit, but could suffer collision under circumstances. This is complemented by the L2 LRU cache, which is slower on cache access (overhead from linear search + reordering), but solves collison of L1 rather well.
* Skip percpu arena when choosing iarena.Qi Wang2017-04-171-1/+1
|
* Switch to fine-grained reentrancy support.Qi Wang2017-04-158-83/+87
| | | | | | | Previously we had a general detection and support of reentrancy, at the cost of having branches and inc / dec operations on fast paths. To avoid taxing fast paths, we move the reentrancy operations onto tsd slow state, and only modify reentrancy level around external calls (that might trigger reentrancy).
* Bundle 3 branches on fast path into tsd_state.Qi Wang2017-04-1410-78/+170
| | | | | | Added tsd_state_nominal_slow, which on fast path malloc() incorporates tcache_enabled check, and on fast path free() bundles both malloc_slow and tcache_enabled branches.
* Pass alloc_ctx down profiling path.Qi Wang2017-04-129-82/+133
| | | | | | With this change, when profiling is enabled, we avoid doing redundant rtree lookups. Also changed dalloc_atx_t to alloc_atx_t, as it's now used on allocation path as well (to speed up profiling).
* Pass dalloc_ctx down the sdalloc path.Qi Wang2017-04-125-24/+41
| | | | This avoids redundant rtree lookups.
* Header refactoring: move atomic.h out of the catch-allDavid Goldblatt2017-04-1111-1/+20
|
* Header refactoring: Split up jemalloc_internal.hDavid Goldblatt2017-04-1138-1337/+1396
| | | | | | | | | | | | | | This is a biggy. jemalloc_internal.h has been doing multiple jobs for a while now: - The source of system-wide definitions. - The catch-all include file. - The module header file for jemalloc.c This commit splits up this functionality. The system-wide definitions responsibility has moved to jemalloc_preamble.h. The catch-all include file is now jemalloc_internal_includes.h. The module headers for jemalloc.c are now in jemalloc_internal_[externs|inlines|types].h, just as they are for the other modules.
* Header refactoring: break out ql.h dependenciesDavid Goldblatt2017-04-1111-2/+20
|
* Header refactoring: break out qr.h dependenciesDavid Goldblatt2017-04-113-1/+4
|
* Header refactoring: break out rb.h dependenciesDavid Goldblatt2017-04-114-4/+8
|
* Header refactoring: break out ph.h dependenciesDavid Goldblatt2017-04-115-1/+8
|
* Header refactoring: Add CPP_PROLOGUE and CPP_EPILOGUE macrosDavid Goldblatt2017-04-111-4/+8
|
* Pass dealloc_ctx down free() fast path.Qi Wang2017-04-118-34/+62
| | | | This gets rid of the redundent rtree lookup down fast path.
* Turn on -Werror for travis CI buildsDavid Goldblatt2017-04-112-29/+31
|
* Port CPU_SPINWAIT to __powerpc64__Rafael Folco2017-04-101-1/+2
| | | | | | | | | Hyper-threaded CPUs may need a special instruction inside spin loops in order to yield to another virtual CPU. The 'pause' instruction that is available for x86 is not supported on Power. Apparently the extended mnemonics like yield, mdoio, and mdoom are not actually implemented on POWER8, although mentioned in the ISA 2.07 document. The recommended magic bits are an 'or 31,31,31'.
* Move reentrancy_level to the beginning of TSD.Qi Wang2017-04-073-9/+9
|
* Add basic reentrancy-checking support, and allow arena_new to reenter.David Goldblatt2017-04-0710-45/+168
| | | | | | | | | This checks whether or not we're reentrant using thread-local data, and, if we are, moves certain internal allocations to use arena 0 (which should be properly initialized after bootstrapping). The immediate thing this allows is spinning up threads in arena_new, which will enable spinning up background threads there.
* Add hooking functionalityDavid Goldblatt2017-04-0719-11/+183
| | | | | This allows us to hook chosen functions and do interesting things there (in particular: reentrancy checking).
* Optimizing TSD and thread cache layout.Qi Wang2017-04-0710-99/+185
| | | | | | | | | | 1) Re-organize TSD so that frequently accessed fields are closer to the beginning and more compact. Assuming 64-bit, the first 2.5 cachelines now contains everything needed on tcache fast path, expect the tcache struct itself. 2) Re-organize tcache and tbins. Take lg_fill_div out of tbin, and reduce tbin to 24 bytes (down from 32). Split tbins into tbins_small and tbins_large, and place tbins_small close to the beginning.
* Bypass witness_fork in TSD when !config_debug.Qi Wang2017-04-071-0/+9
| | | | | With the tcache change, we plan to leave some blank space when !config_debug (unused tbins, witnesses) at the end of the tsd. Let's not touch the memory.
* Get rid of tcache_enabled_t as we have runtime init support.Qi Wang2017-04-076-24/+14
|
* Integrate auto tcache into TSD.Qi Wang2017-04-0716-172/+294
| | | | | | | | | The embedded tcache is initialized upon tsd initialization. The avail arrays for the tbins will be allocated / deallocated accordingly during init / cleanup. With this change, the pointer to the auto tcache will always be available, as long as we have access to the TSD. tcache_available() (called in tcache_get()) is provided to check if we should use tcache.
* Remove the pre-C11-atomics API, which is now unusedDavid Goldblatt2017-04-052-90/+0
|
* Make prof's cum_gctx a C11-style atomicDavid Goldblatt2017-04-051-2/+2
|
* Make the mutex n_waiting_thds field a C11-style atomicDavid Goldblatt2017-04-054-7/+19
|
generated by cgit v0.12 at 2025-03-21 00:00:29 (GMT)