diff options
Diffstat (limited to 'Doc/library/tracemalloc.rst')
| -rw-r--r-- | Doc/library/tracemalloc.rst | 608 |
1 files changed, 608 insertions, 0 deletions
diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst new file mode 100644 index 0000000..2edf7bb --- /dev/null +++ b/Doc/library/tracemalloc.rst @@ -0,0 +1,608 @@ +:mod:`tracemalloc` --- Trace memory allocations +=============================================== + +.. module:: tracemalloc + :synopsis: Trace memory allocations. + +The tracemalloc module is a debug tool to trace memory blocks allocated by +Python. It provides the following information: + +* Traceback where an object was allocated +* Statistics on allocated memory blocks per filename and per line number: + total size, number and average size of allocated memory blocks +* Compute the differences between two snapshots to detect memory leaks + +To trace most memory blocks allocated by Python, the module should be started +as early as possible by setting the :envvar:`PYTHONTRACEMALLOC` environment +variable to ``1``, or by using :option:`-X` ``tracemalloc`` command line +option. The :func:`tracemalloc.start` function can be called at runtime to +start tracing Python memory allocations. + +By default, a trace of an allocated memory block only stores the most recent +frame (1 frame). To store 25 frames at startup: set the +:envvar:`PYTHONTRACEMALLOC` environment variable to ``25``, or use the +:option:`-X` ``tracemalloc=25`` command line option. The +:func:`set_traceback_limit` function can be used at runtime to set the limit. + +.. versionadded:: 3.4 + + +Examples +======== + +Display the top 10 +------------------ + +Display the 10 files allocating the most memory:: + + import tracemalloc + + tracemalloc.start() + + # ... run your application ... + + snapshot = tracemalloc.take_snapshot() + top_stats = snapshot.statistics('lineno') + + print("[ Top 10 ]") + for stat in top_stats[:10]: + print(stat) + + +Example of output of the Python test suite:: + + [ Top 10 ] + <frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B + <frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 B + /usr/lib/python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B + /usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B + /usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B + /usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B + <frozen importlib._bootstrap>:1446: size=70.4 KiB, count=911, average=79 B + <frozen importlib._bootstrap>:1454: size=52.0 KiB, count=25, average=2131 B + <string>:5: size=49.7 KiB, count=148, average=344 B + /usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB + +We can see that Python loaded ``4.8 MiB`` data (bytecode and constants) from +modules and that the :mod:`collections` module allocated ``244 KiB`` to build +:class:`~collections.namedtuple` types. + +See :meth:`Snapshot.statistics` for more options. + + +Compute differences +------------------- + +Take two snapshots and display the differences:: + + import tracemalloc + tracemalloc.start() + # ... start your application ... + + snapshot1 = tracemalloc.take_snapshot() + # ... call the function leaking memory ... + snapshot2 = tracemalloc.take_snapshot() + + top_stats = snapshot2.compare_to(snapshot1, 'lineno') + + print("[ Top 10 differences ]") + for stat in top_stats[:10]: + print(stat) + +Example of output before/after running some tests of the Python test suite:: + + [ Top 10 differences ] + <frozen importlib._bootstrap>:716: size=8173 KiB (+4428 KiB), count=71332 (+39369), average=117 B + /usr/lib/python3.4/linecache.py:127: size=940 KiB (+940 KiB), count=8106 (+8106), average=119 B + /usr/lib/python3.4/unittest/case.py:571: size=298 KiB (+298 KiB), count=589 (+589), average=519 B + <frozen importlib._bootstrap>:284: size=1005 KiB (+166 KiB), count=7423 (+1526), average=139 B + /usr/lib/python3.4/mimetypes.py:217: size=112 KiB (+112 KiB), count=1334 (+1334), average=86 B + /usr/lib/python3.4/http/server.py:848: size=96.0 KiB (+96.0 KiB), count=1 (+1), average=96.0 KiB + /usr/lib/python3.4/inspect.py:1465: size=83.5 KiB (+83.5 KiB), count=109 (+109), average=784 B + /usr/lib/python3.4/unittest/mock.py:491: size=77.7 KiB (+77.7 KiB), count=143 (+143), average=557 B + /usr/lib/python3.4/urllib/parse.py:476: size=71.8 KiB (+71.8 KiB), count=969 (+969), average=76 B + /usr/lib/python3.4/contextlib.py:38: size=67.2 KiB (+67.2 KiB), count=126 (+126), average=546 B + +We can see that Python loaded ``4.4 MiB`` of new data (bytecode and constants) +from modules (on of total of ``8.2 MiB``) and that the :mod:`linecache` module +cached ``940 KiB`` of Python source code to format tracebacks. + +If the system has little free memory, snapshots can be written on disk using +the :meth:`Snapshot.dump` method to analyze the snapshot offline. Then use the +:meth:`Snapshot.load` method reload the snapshot. + + +Get the traceback of a memory block +----------------------------------- + +Code to display the traceback of the biggest memory block:: + + import linecache + import tracemalloc + + tracemalloc.set_traceback_limit(25) + tracemalloc.start() + + # ... run your application ... + + snapshot = tracemalloc.take_snapshot() + top_stats = snapshot.statistics('traceback') + + # pick the biggest memory block + stat = top_stats[0] + print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024)) + for frame in stat.traceback: + print(' File "%s", line %s' % (frame.filename, frame.lineno)) + line = linecache.getline(frame.filename, frame.lineno) + line = line.strip() + if line: + print(' ' + line) + +Example of output of the Python test suite (traceback limited to 25 frames):: + + 903 memory blocks: 870.1 KiB + File "<frozen importlib._bootstrap>", line 716 + File "<frozen importlib._bootstrap>", line 1036 + File "<frozen importlib._bootstrap>", line 934 + File "<frozen importlib._bootstrap>", line 1068 + File "<frozen importlib._bootstrap>", line 619 + File "<frozen importlib._bootstrap>", line 1581 + File "<frozen importlib._bootstrap>", line 1614 + File "/usr/lib/python3.4/doctest.py", line 101 + import pdb + File "<frozen importlib._bootstrap>", line 284 + File "<frozen importlib._bootstrap>", line 938 + File "<frozen importlib._bootstrap>", line 1068 + File "<frozen importlib._bootstrap>", line 619 + File "<frozen importlib._bootstrap>", line 1581 + File "<frozen importlib._bootstrap>", line 1614 + File "/usr/lib/python3.4/test/support/__init__.py", line 1728 + import doctest + File "/usr/lib/python3.4/test/test_pickletools.py", line 21 + support.run_doctest(pickletools) + File "/usr/lib/python3.4/test/regrtest.py", line 1276 + test_runner() + File "/usr/lib/python3.4/test/regrtest.py", line 976 + display_failure=not verbose) + File "/usr/lib/python3.4/test/regrtest.py", line 761 + match_tests=ns.match_tests) + File "/usr/lib/python3.4/test/regrtest.py", line 1563 + main() + File "/usr/lib/python3.4/test/__main__.py", line 3 + regrtest.main_in_temp_cwd() + File "/usr/lib/python3.4/runpy.py", line 73 + exec(code, run_globals) + File "/usr/lib/python3.4/runpy.py", line 160 + "__main__", fname, loader, pkg_name) + +We can see that most memory was allocated in the :mod:`importlib` module to +load data (bytecode and constants) from modules: ``870 KiB``. The traceback is +where the :mod:`importlib` loaded data for the the last time: on the ``import +pdb`` line of the :mod:`doctest` module. The traceback may change if a new +module is loaded. + + +Pretty top +---------- + +Code to display the 10 lines allocating the most memory with a pretty output, +ignoring ``<frozen importlib._bootstrap>`` and ``<unknown>`` files:: + + import os + import tracemalloc + + def display_top(snapshot, group_by='lineno', limit=10): + snapshot = snapshot.filter_traces(( + tracemalloc.Filter(False, "<frozen importlib._bootstrap>"), + tracemalloc.Filter(False, "<unknown>"), + )) + top_stats = snapshot.statistics(group_by) + + print("Top %s lines" % limit) + for index, stat in enumerate(top_stats[:limit], 1): + frame = stat.traceback[0] + # replace "/path/to/module/file.py" with "module/file.py" + filename = os.sep.join(frame.filename.split(os.sep)[-2:]) + print("#%s: %s:%s: %.1f KiB" + % (index, filename, frame.lineno, + stat.size / 1024)) + + other = top_stats[limit:] + if other: + size = sum(stat.size for stat in other) + print("%s other: %.1f KiB" % (len(other), size / 1024)) + total = sum(stat.size for stat in top_stats) + print("Total allocated size: %.1f KiB" % (total / 1024)) + + tracemalloc.start() + + # ... run your application ... + + snapshot = tracemalloc.take_snapshot() + display_top(snapshot, 10) + +Example of output of the Python test suite:: + + 2013-11-08 14:16:58.149320: Top 10 lines + #1: collections/__init__.py:368: 291.9 KiB + #2: Lib/doctest.py:1291: 200.2 KiB + #3: unittest/case.py:571: 160.3 KiB + #4: Lib/abc.py:133: 99.8 KiB + #5: urllib/parse.py:476: 71.8 KiB + #6: <string>:5: 62.7 KiB + #7: Lib/base64.py:140: 59.8 KiB + #8: Lib/_weakrefset.py:37: 51.8 KiB + #9: collections/__init__.py:362: 50.6 KiB + #10: test/test_site.py:56: 48.0 KiB + 7496 other: 4161.9 KiB + Total allocated size: 5258.8 KiB + +See :meth:`Snapshot.statistics` for more options. + + +API +=== + +Functions +--------- + +.. function:: clear_traces() + + Clear traces of memory blocks allocated by Python. + + See also :func:`stop`. + + +.. function:: get_object_traceback(obj) + + Get the traceback where the Python object *obj* was allocated. + Return a :class:`Traceback` instance, or ``None`` if the :mod:`tracemalloc` + module is not tracing memory allocations or did not trace the allocation of + the object. + + See also :func:`gc.get_referrers` and :func:`sys.getsizeof` functions. + + +.. function:: get_traceback_limit() + + Get the maximum number of frames stored in the traceback of a trace. + + By default, a trace of a memory block only stores the most recent + frame: the limit is ``1``. + + Use the :func:`set_traceback_limit` function to change the limit. + + +.. function:: get_traced_memory() + + Get the current size and maximum size of memory blocks traced by the + :mod:`tracemalloc` module as a tuple: ``(size: int, max_size: int)``. + + +.. function:: get_tracemalloc_memory() + + Get the memory usage in bytes of the :mod:`tracemalloc` module used to store + traces of memory blocks. + Return an :class:`int`. + + +.. function:: is_tracing() + + ``True`` if the :mod:`tracemalloc` module is tracing Python memory + allocations, ``False`` otherwise. + + See also :func:`start` and :func:`stop` functions. + + +.. function:: set_traceback_limit(nframe: int) + + Set the maximum number of frames stored in the traceback of a trace. + *nframe* must be greater or equal to ``1``. + + Storing more than ``1`` frame is only useful to compute statistics grouped + by ``'traceback'`` or to compute cumulative statistics: see the + :meth:`Snapshot.compare_to` and :meth:`Snapshot.statistics` methods. + + Storing more frames increases the memory and CPU overhead of the + :mod:`tracemalloc` module. Use the :func:`get_tracemalloc_memory` function + to measure how much memory is used by the :mod:`tracemalloc` module. + + The :envvar:`PYTHONTRACEMALLOC` environment variable + (``PYTHONTRACEMALLOC=NFRAME``) and the :option:`-X` ``tracemalloc=NFRAME`` + command line option can be used to set the limit at startup. + + Use the :func:`get_traceback_limit` function to get the current limit. + + +.. function:: start() + + Start tracing Python memory allocations: install hooks on Python memory + allocators. + + See also :func:`stop` and :func:`is_tracing` functions. + + +.. function:: stop() + + Stop tracing Python memory allocations: uninstall hooks on Python memory + allocators. Clear also traces of memory blocks allocated by Python + + Call :func:`take_snapshot` function to take a snapshot of traces before + clearing them. + + See also :func:`start` and :func:`is_tracing` functions. + + +.. function:: take_snapshot() + + Take a snapshot of traces of memory blocks allocated by Python. Return a new + :class:`Snapshot` instance. + + The snapshot does not include memory blocks allocated before the + :mod:`tracemalloc` module started to trace memory allocations. + + Tracebacks of traces are limited to :func:`get_traceback_limit` frames. Use + :func:`set_traceback_limit` to store more frames. + + The :mod:`tracemalloc` module must be tracing memory allocations to take a + snapshot, see the the :func:`start` function. + + See also the :func:`get_object_traceback` function. + + +Filter +------ + +.. class:: Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False) + + Filter on traces of memory blocks. + + See the :func:`fnmatch.fnmatch` function for the syntax of + *filename_pattern*. The ``'.pyc'`` and ``'.pyo'`` file extensions are + replaced with ``'.py'``. + + Examples: + + * ``Filter(True, subprocess.__file__)`` only includes traces of the + :mod:`subprocess` module + * ``Filter(False, tracemalloc.__file__)`` excludes traces of the + :mod:`tracemalloc` module + * ``Filter(False, "<unknown>")`` excludes empty tracebacks + + .. attribute:: inclusive + + If *inclusive* is ``True`` (include), only trace memory blocks allocated + in a file with a name matching :attr:`filename_pattern` at line number + :attr:`lineno`. + + If *inclusive* is ``False`` (exclude), ignore memory blocks allocated in + a file with a name matching :attr:`filename_pattern` at line number + :attr:`lineno`. + + .. attribute:: lineno + + Line number (``int``) of the filter. If *lineno* is ``None``, the filter + matches any line number. + + .. attribute:: filename_pattern + + Filename pattern of the filter (``str``). + + .. attribute:: all_frames + + If *all_frames* is ``True``, all frames of the traceback are checked. If + *all_frames* is ``False``, only the most recent frame is checked. + + This attribute is ignored if the traceback limit is less than ``2``. See + the :func:`get_traceback_limit` function and + :attr:`Snapshot.traceback_limit` attribute. + + +Frame +----- + +.. class:: Frame + + Frame of a traceback. + + The :class:`Traceback` class is a sequence of :class:`Frame` instances. + + .. attribute:: filename + + Filename (``str``). + + .. attribute:: lineno + + Line number (``int``). + + +Snapshot +-------- + +.. class:: Snapshot + + Snapshot of traces of memory blocks allocated by Python. + + The :func:`take_snapshot` function creates a snapshot instance. + + .. method:: compare_to(old_snapshot: Snapshot, group_by: str, cumulative: bool=False) + + Compute the differences with an old snapshot. Get statistics as a sorted + list of :class:`StatisticDiff` instances grouped by *group_by*. + + See the :meth:`statistics` method for *group_by* and *cumulative* + parameters. + + The result is sorted from the biggest to the smallest by: absolute value + of :attr:`StatisticDiff.size_diff`, :attr:`StatisticDiff.size`, absolute + value of :attr:`StatisticDiff.count_diff`, :attr:`Statistic.count` and + then by :attr:`StatisticDiff.traceback`. + + + .. method:: dump(filename) + + Write the snapshot into a file. + + Use :meth:`load` to reload the snapshot. + + + .. method:: filter_traces(filters) + + Create a new :class:`Snapshot` instance with a filtered :attr:`traces` + sequence, *filters* is a list of :class:`Filter` instances. If *filters* + is an empty list, return a new :class:`Snapshot` instance with a copy of + the traces. + + All inclusive filters are applied at once, a trace is ignored if no + inclusive filters match it. A trace is ignored if at least one exclusive + filter matchs it. + + + .. classmethod:: load(filename) + + Load a snapshot from a file. + + See also :meth:`dump`. + + + .. method:: statistics(group_by: str, cumulative: bool=False) + + Get statistics as a sorted list of :class:`Statistic` instances grouped + by *group_by*: + + ===================== ======================== + group_by description + ===================== ======================== + ``'filename'`` filename + ``'lineno'`` filename and line number + ``'traceback'`` traceback + ===================== ======================== + + If *cumulative* is ``True``, cumulate size and count of memory blocks of + all frames of the traceback of a trace, not only the most recent frame. + The cumulative mode can only be used with *group_by* equals to + ``'filename'`` and ``'lineno'`` and :attr:`traceback_limit` greater than + ``1``. + + The result is sorted from the biggest to the smallest by: + :attr:`Statistic.size`, :attr:`Statistic.count` and then by + :attr:`Statistic.traceback`. + + + .. attribute:: traceback_limit + + Maximum number of frames stored in the traceback of :attr:`traces`: + result of the :func:`get_traceback_limit` when the snapshot was taken. + + .. attribute:: traces + + Traces of all memory blocks allocated by Python: sequence of + :class:`Trace` instances. + + The sequence has an undefined order. Use the :meth:`Snapshot.statistics` + method to get a sorted list of statistics. + + +Statistic +--------- + +.. class:: Statistic + + Statistic on memory allocations. + + :func:`Snapshot.statistics` returns a list of :class:`Statistic` instances. + + See also the :class:`StatisticDiff` class. + + .. attribute:: count + + Number of memory blocks (``int``). + + .. attribute:: size + + Total size of memory blocks in bytes (``int``). + + .. attribute:: traceback + + Traceback where the memory block was allocated, :class:`Traceback` + instance. + + +StatisticDiff +------------- + +.. class:: StatisticDiff + + Statistic difference on memory allocations between an old and a new + :class:`Snapshot` instance. + + :func:`Snapshot.compare_to` returns a list of :class:`StatisticDiff` + instances. See also the :class:`Statistic` class. + + .. attribute:: count + + Number of memory blocks in the new snapshot (``int``): ``0`` if + the memory blocks have been released in the new snapshot. + + .. attribute:: count_diff + + Difference of number of memory blocks between the old and the new + snapshots (``int``): ``0`` if the memory blocks have been allocated in + the new snapshot. + + .. attribute:: size + + Total size of memory blocks in bytes in the new snapshot (``int``): + ``0`` if the memory blocks have been released in the new snapshot. + + .. attribute:: size_diff + + Difference of total size of memory blocks in bytes between the old and + the new snapshots (``int``): ``0`` if the memory blocks have been + allocated in the new snapshot. + + .. attribute:: traceback + + Traceback where the memory blocks were allocated, :class:`Traceback` + instance. + + +Trace +----- + +.. class:: Trace + + Trace of a memory block. + + The :attr:`Snapshot.traces` attribute is a sequence of :class:`Trace` + instances. + + .. attribute:: size + + Size of the memory block in bytes (``int``). + + .. attribute:: traceback + + Traceback where the memory block was allocated, :class:`Traceback` + instance. + + +Traceback +--------- + +.. class:: Traceback + + Sequence of :class:`Frame` instances sorted from the most recent frame to + the oldest frame. + + A traceback contains at least ``1`` frame. If the ``tracemalloc`` module + failed to get a frame, the filename ``"<unknown>"`` at line number ``0`` is + used. + + When a snapshot is taken, tracebacks of traces are limited to + :func:`get_traceback_limit` frames. See the :func:`take_snapshot` function. + + The :attr:`Trace.traceback` attribute is an instance of :class:`Traceback` + instance. + + |