summaryrefslogtreecommitdiffstats
path: root/include/jemalloc/internal/prof.h
Commit message (Collapse)AuthorAgeFilesLines
* Break up headers into constituent partsDavid Goldblatt2017-01-121-568/+0
| | | | | | | | | | This is part of a broader change to make header files better represent the dependencies between one another (see https://github.com/jemalloc/jemalloc/issues/533). It breaks up component headers into smaller parts that can be made to have a simpler dependency graph. For the autogenerated headers (smoothstep.h and size_classes.h), no splitting was necessary, so I didn't add support to emit multiple headers.
* Do not (recursively) allocate within tsd_fetch().Jason Evans2016-10-211-4/+4
| | | | | | | Refactor tsd so that tsdn_fetch() does not trigger allocation, since allocation could cause infinite recursion. This resolves #458.
* Add a missing prof_alloc_rollback() call.Jason Evans2016-06-061-0/+1
| | | | | | In the case where prof_alloc_prep() is called with an over-estimate of allocation size, and sampling doesn't end up being triggered, the tctx must be discarded.
* Move slabs out of chunks.Jason Evans2016-06-061-16/+28
|
* Use huge size class infrastructure for large size classes.Jason Evans2016-06-061-3/+3
|
* Use rtree-based chunk lookups rather than pointer bit twiddling.Jason Evans2016-06-031-36/+45
| | | | | | | | | | | Look up chunk metadata via the radix tree, rather than using CHUNK_ADDR2BASE(). Propagate pointer's containing extent. Minimize extent lookups by doing a single lookup (e.g. in free()) and propagating the pointer's extent into nearly all the functions that may need it.
* Resolve bootstrapping issues when embedded in FreeBSD libc.Jason Evans2016-05-111-37/+37
| | | | | | | | | | | | | b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online locking validator.) caused a broad propagation of tsd throughout the internal API, but tsd_fetch() was designed to fail prior to tsd bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and nullable tsdn_t, and modifying all internal APIs that do not critically rely on tsd to take nullable pointers. Furthermore, add the tsd_booted_get() function so that tsdn_fetch() can probe whether tsd bootstrapping is complete and return NULL if not. All dangerous conversions of nullable pointers are tsdn_tsd() calls that assert-fail on invalid conversion.
* Fix fork()-related lock rank ordering reversals.Jason Evans2016-04-261-1/+2
|
* Add witness, a simple online locking validator.Jason Evans2016-04-141-39/+39
| | | | This resolves #358.
* Fast-path improvement: reduce # of branches and unnecessary operations.Qi Wang2015-11-101-3/+3
| | | | | | - Combine multiple runtime branches into a single malloc_slow check. - Avoid calling arena_choose / size2index / index2size on fast path. - A few micro optimizations.
* Resolve an unsupported special case in arena_prof_tctx_set().Jason Evans2015-09-151-3/+17
| | | | | | | | | | | Add arena_prof_tctx_reset() and use it instead of arena_prof_tctx_set() when resetting the tctx pointer during reallocation, which happens whenever an originally sampled reallocated object is not sampled during reallocation. This regression was introduced by 594c759f37c301d0245dc2accf4d4aaf9d202819 (Optimize arena_prof_tctx_set().)
* Fix prof_{malloc,free}_sample_object() call order in prof_realloc().Jason Evans2015-09-151-3/+8
| | | | | | Fix prof_realloc() to call prof_free_sampled_object() after calling prof_malloc_sample_object(). Prior to this fix, if tctx and old_tctx were the same, the tctx could have been prematurely destroyed.
* Make one call to prof_active_get_unlocked() per allocation event.Jason Evans2015-09-151-8/+10
| | | | | | | Make one call to prof_active_get_unlocked() per allocation event, and use the result throughout the relevant functions that handle an allocation event. Also add a missing check in prof_realloc(). These fixes protect allocation events against concurrent prof_active changes.
* Fix "prof.reset" mallctl-related corruption.Jason Evans2015-09-101-2/+3
| | | | | | | Fix heap profiling to distinguish among otherwise identical sample sites with interposed resets (triggered via the "prof.reset" mallctl). This bug could cause data structure corruption that would most likely result in a segfault.
* Optimize arena_prof_tctx_set().Jason Evans2015-09-021-5/+5
| | | | | Optimize arena_prof_tctx_set() to avoid reading run metadata when deciding whether it's actually necessary to write.
* Fix heap profiling regressions.Jason Evans2015-03-161-1/+22
| | | | | | | | | | | | | | | Remove the prof_tctx_state_destroying transitory state and instead add the tctx_uid field, so that the tuple <thr_uid, tctx_uid> uniquely identifies a tctx. This assures that tctx's are well ordered even when more than two with the same thr_uid coexist. A previous attempted fix based on prof_tctx_state_destroying was only sufficient for protecting against two coexisting tctx's, but it also introduced a new dumping race. These regressions were introduced by 602c8e0971160e4b85b08b16cf8a2375aa24bc04 (Implement per thread heap profiling.) and 764b00023f2bc97f240c3a758ed23ce9c0ad8526 (Fix a heap profiling regression.).
* Fix a heap profiling regression.Jason Evans2015-03-141-0/+1
| | | | | | | | | | Add the prof_tctx_state_destroying transitionary state to fix a race between a thread destroying a tctx and another thread creating a new equivalent tctx. This regression was introduced by 602c8e0971160e4b85b08b16cf8a2375aa24bc04 (Implement per thread heap profiling.).
* Refactor huge_*() calls into arena internals.Jason Evans2015-02-121-15/+2
| | | | | Make redirects to the huge_*() API the arena code's responsibility, since arenas now take responsibility for all allocation sizes.
* Implement the prof.gdump mallctl.Jason Evans2015-01-261-0/+18
| | | | | | | | This feature makes it possible to toggle the gdump feature on/off during program execution, whereas the the opt.prof_dump mallctl value can only be set during program startup. This resolves #72.
* Miscellaneous cleanups.Jason Evans2014-10-311-6/+4
|
* mark huge allocations as unlikelyDaniel Micay2014-10-311-2/+2
| | | | This cleans up the fast path a bit more by moving away more code.
* Fix a prof_tctx_t/prof_tdata_t cleanup race.Jason Evans2014-10-121-0/+6
| | | | | | Fix a prof_tctx_t/prof_tdata_t cleanup race by storing a copy of thr_uid in prof_tctx_t, so that the associated tdata need not be present during tctx teardown.
* Make prof-related inline functions always-inline.Jason Evans2014-10-041-9/+9
|
* Fix tsd cleanup regressions.Jason Evans2014-10-041-2/+4
| | | | | | | | | | | | | | | | Fix tsd cleanup regressions that were introduced in 5460aa6f6676c7f253bfcb75c028dfd38cae8aaf (Convert all tsd variables to reside in a single tsd structure.). These regressions were twofold: 1) tsd_tryget() should never (and need never) return NULL. Rename it to tsd_fetch() and simplify all callers. 2) tsd_*_set() must only be called when tsd is in the nominal state, because cleanup happens during the nominal-->purgatory transition, and re-initialization must not happen while in the purgatory state. Add tsd_nominal() and use it as needed. Note that tsd_*{p,}_get() can still be used as long as no re-initialization that would require cleanup occurs. This means that e.g. the thread_allocated counter can be updated unconditionally.
* Implement/test/fix prof-related mallctl's.Jason Evans2014-10-041-9/+25
| | | | | | | | | | | Implement/test/fix the opt.prof_thread_active_init, prof.thread_active_init, and thread.prof.active mallctl's. Test/fix the thread.prof.name mallctl. Refactor opt_prof_active to be read-only and move mutable state into the prof_active variable. Stop leaning on ctl-related locking for protection.
* Convert to uniform style: cond == false --> !condJason Evans2014-10-031-1/+1
|
* Test prof.reset mallctl and fix numerous discovered bugs.Jason Evans2014-10-031-10/+14
|
* Fix profile dumping race.Jason Evans2014-09-251-0/+1
| | | | | | | | | | | | | | Fix a race that caused a non-critical assertion failure. To trigger the race, a thread had to be part way through initializing a new sample, such that it was discoverable by the dumping thread, but not yet linked into its gctx by the time a later dump phase would normally have reset its state to 'nominal'. Additionally, lock access to the state field during modification to transition to the dumping state. It's not apparent that this oversight could have caused an actual problem due to outer locking that protects the dumping machinery, but the added locking pedantically follows the stated locking protocol for the state field.
* Convert all tsd variables to reside in a single tsd structure.Jason Evans2014-09-231-41/+37
|
* Apply likely()/unlikely() to allocation/deallocation fast paths.Jason Evans2014-09-121-5/+6
|
* Fix a profile sampling race.Jason Evans2014-09-101-20/+17
| | | | | | | | | | | Fix a profile sampling race that was due to preparing to sample, yet doing nothing to assure that the context remains valid until the stats are updated. These regressions were caused by 602c8e0971160e4b85b08b16cf8a2375aa24bc04 (Implement per thread heap profiling.), which did not make it into any releases prior to these fixes.
* Fix prof_tdata_get()-related regressions.Jason Evans2014-09-091-5/+6
| | | | | | | | | | | | | Fix prof_tdata_get() to avoid dereferencing an invalid tdata pointer (when it's PROF_TDATA_STATE_{REINCARNATED,PURGATORY}). Fix prof_tdata_get() callers to check for invalid results besides NULL (PROF_TDATA_STATE_{REINCARNATED,PURGATORY}). These regressions were caused by 602c8e0971160e4b85b08b16cf8a2375aa24bc04 (Implement per thread heap profiling.), which did not make it into any releases prior to these fixes.
* Implement per thread heap profiling.Jason Evans2014-08-201-245/+195
| | | | | | | | | | | | | | | | | | | | | | | | Rename data structures (prof_thr_cnt_t-->prof_tctx_t, prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for sampled objects. Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace depth within jemalloc functions is no longer an issue (pprof prunes irrelevant frames). Implement mallctl's: - prof.reset implements full sample data reset, and optional change of sample interval. - prof.lg_sample reads the current sample interval (opt.lg_prof_sample was the permanent source of truth prior to prof.reset). - thread.prof.name provides naming capability for threads within heap profile dumps. - thread.prof.active makes it possible to activate/deactivate heap profiling for individual threads. Modify the heap dump files to contain per thread heap profile data. This change is incompatible with the existing pprof, which will require enhancements to read and process the enriched data.
* Dump heap profile backtraces in a stable order.Jason Evans2014-08-201-10/+14
| | | | | Also iterate over per thread stats in a stable order, which prepares the way for stable ordering of per thread heap profile dumps.
* Directly embed prof_ctx_t's bt.Jason Evans2014-08-201-5/+8
|
* Convert prof_tdata_t's bt2cnt to a comprehensive map.Jason Evans2014-08-201-16/+8
| | | | | | Treat prof_tdata_t's bt2cnt as a comprehensive map of the thread's extant allocation samples (do not limit the total number of entries). This helps prepare the way for per thread heap profiling.
* Simplify backtracing.Jason Evans2014-04-231-4/+3
| | | | | | | | | | | Simplify backtracing to not ignore any frames, and compensate for this in pprof in order to increase flexibility with respect to function-based refactoring even in the presence of non-deterministic inlining. Modify pprof to blacklist all jemalloc allocation entry points including non-standard ones like mallocx(), and ignore all allocator-internal frames. Prior to this change, pprof excluded the specifically blacklisted functions from backtraces, but it left allocator-internal frames intact.
* Fix debug-only compilation failures.Jason Evans2014-04-161-3/+2
| | | | | | | | Fix debug-only compilation failures introduced by changes to prof_sample_accum_update() in: 6c39f9e059d0825f4c29d8cec9f318b798912c3c refactor profiling. only use a bytes till next sample variable.
* refactor profiling. only use a bytes till next sample variable.Ben Maurer2014-04-161-149/+69
|
* Remove support for non-prof-promote heap profiling metadata.Jason Evans2014-04-111-13/+7
| | | | | | | | | | | | | | | Make promotion of sampled small objects to large objects mandatory, so that profiling metadata can always be stored in the chunk map, rather than requiring one pointer per small region in each small-region page run. In practice the non-prof-promote code was only useful when using jemalloc to track all objects and report them as leaks at program exit. However, Valgrind is at least as good a tool for this particular use case. Furthermore, the non-prof-promote code is getting in the way of some optimizations that will make heap profiling much cheaper for the predominant use case (sampling a small representative proportion of all allocations).
* Avoid a compiler warning.Jason Evans2014-01-291-1/+5
| | | | | | | | Avoid copying "jeprof" to a 1-byte buffer within prof_boot0() when heap profiling is disabled. Although this is dead code under such conditions, the compiler doesn't figure that part out. Reported by Eduardo Silva.
* Add heap profiling tests.Jason Evans2014-01-171-0/+5
| | | | | | Fix a regression in prof_dump_ctx() due to an uninitized variable. This was caused by revision 4f37ef693e3d5903ce07dc0b61c0da320b35e3d9, so no releases are affected.
* Fix a variable prototype/definition mismatch.Jason Evans2014-01-171-1/+6
|
* Refactor prof_dump() to reduce contention.Jason Evans2014-01-161-0/+5
| | | | | | | | | | | | | | Refactor prof_dump() to use a two pass algorithm, and prof_leave() prior to the second pass. This avoids write(2) system calls while holding critical prof resources. Fix prof_dump() to close the dump file descriptor for all relevant error paths. Minimize the size of prof-related static buffers when prof is disabled. This saves roughly 65 KiB of application memory for non-prof builds. Refactor prof_ctx_init() out of prof_lookup_global().
* Optimize arena_prof_ctx_set().Jason Evans2013-12-161-26/+26
| | | | | | Refactor such that arena_prof_ctx_set() receives usize as an argument, and use it to determine whether to handle ptr as a small region, rather than reading the chunk page map.
* Add probabability distribution utility code.Jason Evans2013-12-101-1/+1
| | | | | | | | | | | | | | | Add probabability distribution utility code that enables generation of random deviates drawn from normal, Chi-square, and Gamma distributions. Fix format strings in several of the assert_* macros (remove a %s). Clean up header issues; it's critical that system headers are not included after internal definitions potentially do things like: #define inline Fix the build system to incorporate header dependencies for the test library C files.
* Disable floating point code/linking when possible.Jason Evans2013-12-061-0/+15
| | | | | | | | | | | Unless heap profiling is enabled, disable floating point code and don't link with libm. This, in combination with e.g. EXTRA_CFLAGS=-mno-sse on x64 systems, makes it possible to completely disable floating point register use. Some versions of glibc neglect to save/restore caller-saved floating point registers during dynamic lazy symbol loading, and the symbol loading code uses whatever malloc the application happens to have linked/loaded with, the result being potential floating point register corruption.
* Fix potential TLS-related memory corruption.Jason Evans2013-01-311-8/+6
| | | | | | | | | | Avoid writing to uninitialized TLS as a side effect of deallocation. Initializing TLS during deallocation is unsafe because it is possible that a thread never did any allocation, and that TLS has already been deallocated by the threads library, resulting in write-after-free corruption. These fixes affect prof_tdata and quarantine; all other uses of TLS are already safe, whether intentionally (as for tcache) or unintentionally (as for arenas).
* Fix fork(2)-related deadlocks.Jason Evans2012-10-091-0/+3
| | | | | | | | | | | | | | | | | Add a library constructor for jemalloc that initializes the allocator. This fixes a race that could occur if threads were created by the main thread prior to any memory allocation, followed by fork(2), and then memory allocation in the child process. Fix the prefork/postfork functions to acquire/release the ctl, prof, and rtree mutexes. This fixes various fork() child process deadlocks, but one possible deadlock remains (intentionally) unaddressed: prof backtracing can acquire runtime library mutexes, so deadlock is still possible if heap profiling is enabled during fork(). This deadlock is known to be a real issue in at least the case of libgcc-based backtracing. Reported by tfengjun.
* Fix heap profiling crash for realloc(p, 0) case.Jason Evans2012-05-151-1/+1
| | | | | Fix prof_realloc() to not call prof_ctx_set() if a sampled object is being freed via realloc(p, 0).
generated by cgit v0.12 at 2025-08-29 03:57:51 (GMT)