summaryrefslogtreecommitdiffstats
path: root/src/jemalloc.c
Commit message (Collapse)AuthorAgeFilesLines
* Avoid check_min for opt_lg_extent_max_active_fit.Qi Wang2019-03-291-1/+1
| | | | This fixes a compiler warning.
* Allow low values of oversize_threshold to disable the feature.Qi Wang2019-03-291-2/+2
| | | | | We should allow a way to easily disable the feature (e.g. not reserving the arena id at all).
* Rename huge_threshold to oversize_threshold.Qi Wang2019-01-251-3/+3
| | | | | The keyword huge tend to remind people of huge pages which is not relevent to the feature.
* Set huge_threshold to 8M by default.Qi Wang2019-01-241-1/+8
| | | | | | This feature uses an dedicated arena to handle huge requests, which significantly improves VM fragmentation. In production workload we tested it often reduces VM size by >30%.
* Un-experimental the huge_threshold feature.Qi Wang2019-01-161-2/+1
|
* Avoid creating bg thds for huge arena lone.Qi Wang2019-01-161-11/+7
| | | | | | For low arena count settings, the huge threshold feature may trigger an unwanted bg thd creation. Given that the huge arena does eager purging by default, bypass bg thd creation when initializing the huge arena.
* Store the bin shard selection in TSD.Qi Wang2018-12-041-3/+8
| | | | | This avoids having to choose bin shard on the fly, also will allow flexible bin binding for each thread.
* Add opt.bin_shards to specify number of bin shards.Qi Wang2018-12-041-7/+32
| | | | | The option uses the same format as "slab_sizes" to specify number of shards for each bin size.
* Add support for sharded bins within an arena.Qi Wang2018-12-041-0/+3
| | | | | | | | | This makes it possible to have multiple set of bins in an arena, which improves arena scalability because the bins (especially the small ones) are always the limiting factor in production workload. A bin shard is picked on allocation; each extent tracks the bin shard id for deallocation. The shard size will be determined using runtime options.
* Add a free() and sdallocx(where flags=0) fastpathDave Watson2018-11-121-11/+86
| | | | | | | | | | | | | Add unsized and sized deallocation fastpaths. Similar to the malloc() fastpath, this removes all frame manipulation for the majority of free() calls. The performance advantages here are less than that of the malloc() fastpath, but from prod tests seems to still be half a percent or so of improvement. Stats and sampling a both supported (sdallocx needs a sampling check, for rtree lookups slab will only be set for unsampled objects). We don't support flush, any flush requests go to the slowpath.
* malloc: Add a fastpathDave Watson2018-10-181-8/+89
| | | | | | | | | | | | | | | | | This diff adds a fastpath that assumes size <= SC_LOOKUP_MAXCLASS, and that we hit tcache. If either of these is false, we fall back to the previous codepath (renamed 'malloc_default'). Crucially, we only tail call malloc_default, and with the same kind and number of arguments, so that both clang and gcc tail-calling will kick in - therefore malloc() gets treated as a leaf function, and there are *no* caller-saved registers. Previously malloc() contained 5 caller saved registers on x64, resulting in at least 10 extra memory-movement instructions. In microbenchmarks this results in up to ~10% improvement in malloc() fastpath. In real programs, this is a ~1% CPU and latency improvement overall.
* drop bump_empty_alloc option. Size class lookup support used instead.Dave Watson2018-10-171-16/+1
|
* Make `smallocx` symbol name depend on the `JEMALLOC_VERSION_GID`gnzlbg2018-10-171-5/+10
| | | | | | This comments concatenates the `JEMALLOC_VERSION_GID` to the `smallocx` symbol name, such that the symbol ends up exported as `smallocx_{git_hash}`.
* Hide smallocx even when enabled from the library APIgnzlbg2018-10-171-0/+5
| | | | | | | | | The experimental `smallocx` API is not exposed via header files, requiring the users to peek at `jemalloc`'s source code to manually add the external declarations to their own programs. This should reinforce that `smallocx` is experimental, and that `jemalloc` does not offer any kind of backwards compatiblity or ABI gurantees for it.
* Add experimental API: smallocx_return_t smallocx(size, flags)gnzlbg2018-10-171-1/+76
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | --- Motivation: This new experimental memory-allocaction API returns a pointer to the allocation as well as the usable size of the allocated memory region. The `s` in `smallocx` stands for `sized`-`mallocx`, attempting to convey that this API returns the size of the allocated memory region. It should allow C++ P0901r0 [0] and Rust Alloc::alloc_excess to make use of it. The main purpose of these APIs is to improve telemetry. It is more accurate to register `smallocx(size, flags)` than `smallocx(nallocx(size), flags)`, for example. The latter will always line up perfectly with the existing size classes, causing a loss of telemetry information about the internal fragmentation induced by potentially poor size-classes choices. Instrumenting `nallocx` does not help much since user code can cache its result and use it repeatedly. --- Implementation: The implementation adds a new `usize` option to `static_opts_s` and an `usize` variable to `dynamic_opts_s`. These are then used to cache the result of `sz_index2size` and similar functions in the code paths in which they are unconditionally invoked. In the code-paths in which these functions are not unconditionally invoked, `smallocx` calls, as opposed to `mallocx`, these functions explicitly. --- [0]: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0901r0.html
* remove malloc_init() off the fastpathDave Watson2018-10-151-3/+16
|
* Bootstrapping: don't overwrite opt_prof_prefix.David Goldblatt2018-09-131-3/+8
|
* Allow the use of readlinkat over readlink.David Goldblatt2018-08-031-0/+5
| | | | This can be useful in situations where readlink is disallowed.
* Add logging for sampled allocationsTyler Etzel2018-08-011-0/+1
| | | | | - prof_opt_log flag starts logging automatically at runtime - prof_log_{start,stop} mallctl for manual control
* TSD: Add fork support to tsd_nominal_tsds.David Goldblatt2018-07-271-0/+5
| | | | | In case of multithreaded fork, we want to leave the child in a reasonable state, in which tsd_nominal_tsds is either empty or contains only the forking thread.
* SC: Remove global data.David Goldblatt2018-07-231-7/+12
| | | | | | | The global data is mostly only used at initialization, or for easy access to values we could compute statically. Instead of consuming that space (and risking TLB misses), we can just pass around a pointer to stack data during bootstrapping.
* Tolerate experimental features for abort_conf.Qi Wang2018-07-181-0/+5
| | | | | Not aborting with unrecognized experimental options. This helps us testing experimental features with abort_conf enabled.
* SC: Make some key size classes static.David Goldblatt2018-07-131-25/+25
| | | | | | The largest small class, smallest large class, and largest large class may all be needed down fast paths; to avoid the risk of touching another cache line, we can make them available as constants.
* Add MALLOC_CONF parsing for dynamic slab sizes.David T. Goldblatt2018-07-131-0/+68
| | | | This actually enables us to change the values.
* Bootstrapping: Parse MALLOC_CONF before using slab sizes.David T. Goldblatt2018-07-131-1/+8
| | | | | | | | | | I.e., parse before booting the bin module or sz module. This lets us tweak size class settings before committing to them by letting them leak into other modules. This commit does not actually do any tweaking of the size classes; it *just* chanchanges bootstrapping order; this may help bisecting any bootstrapping failures on poorly-tested architectures.
* Hide size class computation behind a layer of indirection.David Goldblatt2018-07-131-32/+46
| | | | | | | | | This class removes almost all the dependencies on size_classes.h, accessing the data there only via the new module sc.h, which does not depend on any configuration options. In a subsequent commit, we'll remove the configure-time size class computations, doing them at boot time, instead.
* Clean compilation -Wextragnzlbg2018-07-101-1/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before this commit jemalloc produced many warnings when compiled with -Wextra with both Clang and GCC. This commit fixes the issues raised by these warnings or suppresses them if they were spurious at least for the Clang and GCC versions covered by CI. This commit: * adds `JEMALLOC_DIAGNOSTIC` macros: `JEMALLOC_DIAGNOSTIC_{PUSH,POP}` are used to modify the stack of enabled diagnostics. The `JEMALLOC_DIAGNOSTIC_IGNORE_...` macros are used to ignore a concrete diagnostic. * adds `JEMALLOC_FALLTHROUGH` macro to explicitly state that falling through `case` labels in a `switch` statement is intended * Removes all UNUSED annotations on function parameters. The warning -Wunused-parameter is now disabled globally in `jemalloc_internal_macros.h` for all translation units that include that header. It is never re-enabled since that header cannot be included by users. * locally suppresses some -Wextra diagnostics: * `-Wmissing-field-initializer` is buggy in older Clang and GCC versions, where it does not understanding that, in C, `= {0}` is a common C idiom to initialize a struct to zero * `-Wtype-bounds` is suppressed in a particular situation where a generic macro, used in multiple different places, compares an unsigned integer for smaller than zero, which is always true. * `-Walloc-larger-than-size=` diagnostics warn when an allocation function is called with a size that is too large (out-of-range). These are suppressed in the parts of the tests where `jemalloc` explicitly does this to test that the allocation functions fail properly. * adds a new CI build bot that runs the log unit test on CI. Closes #1196 .
* Rename huge_threshold to experimental, and tweak documentation.Qi Wang2018-06-291-1/+3
|
* Refactor arena_is_auto.Qi Wang2018-06-291-2/+6
|
* Implement huge arena: opt.huge_threshold.Qi Wang2018-06-291-3/+10
| | | | | | | | | The feature allows using a dedicated arena for huge allocations. We want the addtional arena to separate huge allocation because: 1) mixing small extents with huge ones causes fragmentation over the long run (this feature reduces VM size significantly); 2) with many arenas, huge extents rarely get reused across threads; and 3) huge allocations happen way less frequently, therefore no concerns for lock contention.
* Optimize ixalloc by avoiding a size lookup.Qi Wang2018-06-061-4/+4
|
* Hooks: hook the realloc pathways that move/expand.David Goldblatt2018-05-181-16/+28
|
* Hooks: hook the realloc paths that act as pure malloc/free.David Goldblatt2018-05-181-1/+11
|
* Hooks: hook the pure-expand function.David Goldblatt2018-05-181-0/+6
|
* Hooks: hook the pure-deallocation functions.David Goldblatt2018-05-181-0/+6
|
* Hooks: hook the pure-allocation functions.David Goldblatt2018-05-181-5/+61
|
* Add "hook" module.David Goldblatt2018-05-181-0/+2
| | | | | | | | The hook module allows a low-reader-overhead way of finding hooks to invoke and calling them. For now, none of the allocation pathways are tied into the hooks; this will come later.
* Fix abort_conf processing.Qi Wang2018-04-181-6/+4
| | | | | When abort_conf is set, make sure we always error out at the end of the options processing loop.
* Add UNUSED to avoid compiler warnings.Qi Wang2018-04-161-1/+1
|
* background_thread: add max thread count configDave Watson2018-04-101-0/+4
| | | | | Looking at the thread counts in our services, jemalloc's background thread is useful, but mostly idle. Add a config option to tune down the number of threads.
* Add opt.thp which allows explicit hugepage usage.Qi Wang2018-03-081-2/+22
| | | | | | | | "always" marks all user mappings as MADV_HUGEPAGE; while "never" marks all mappings as MADV_NOHUGEPAGE. The default setting "default" does not change any settings. Note that all the madvise calls are part of the default extent hooks by design, so that customized extent hooks have complete control over the mappings including hugepage settings.
* Add opt.lg_extent_max_active_fitQi Wang2017-11-161-0/+3
| | | | | | | | | | When allocating from dirty extents (which we always prefer if available), large active extents can get split even if the new allocation is much smaller, in which case the introduced fragmentation causes high long term damage. This new option controls the threshold to reuse and split an existing active extent. We avoid using a large extent for much smaller sizes, in order to reduce fragmentation. In some workload, adding the threshold improves virtual memory usage by >10x.
* Delay background_thread_ctl_init to right before thread creation.Qi Wang2017-10-061-1/+6
| | | | | ctl_init sets isthreaded, which means it should be done without holding any locks.
* Logging: capitalize log macro.David Goldblatt2017-10-031-48/+48
| | | | Dodge a name-conflict with the math.h logarithm function. D'oh.
* Put static keyword first.Qi Wang2017-09-211-1/+1
| | | | Fix a warning by -Wold-style-declaration.
* Change opt.metadata_thp to [disabled,auto,always].Qi Wang2017-08-301-1/+17
| | | | | | | | To avoid the high RSS caused by THP + low usage arena (i.e. THP becomes a significant percentage), added a new "auto" option which will only start using THP after a base allocator used up the first THP region. Starting from the second hugepage (in a single arena), "auto" behaves the same as "always", i.e. madvise hugepage right away.
* Implement opt.metadata_thpQi Wang2017-08-111-0/+1
| | | | | This option enables transparent huge page for base allocators (require MADV_HUGEPAGE support).
* Only read szind if ptr is not paged aligned in sdallocx.Qi Wang2017-07-311-2/+22
| | | | | | If ptr is not page aligned, we know the allocation was not sampled. In this case use the size passed into sdallocx directly w/o accessing rtree. This improve sdallocx efficiency in the common case (not sampled && small allocation).
* Logging: log using the log var names directly.David Goldblatt2017-07-241-151/+47
| | | | | | | | | | | Currently we have to log by writing something like: static log_var_t log_a_b_c = LOG_VAR_INIT("a.b.c"); log (log_a_b_c, "msg"); This is sort of annoying. Let's just write: log("a.b.c", "msg");
* Logging: allow logging with empty varargs.David Goldblatt2017-07-221-7/+7
| | | | | | Currently, the log macro requires at least one argument after the format string, because of the way the preprocessor handles varargs macros. We can hide some of that irritation by pushing the extra arguments into a varargs function.
generated by cgit v0.12 at 2026-01-11 14:13:50 (GMT)