From e6ec6f5b50e8793172e83a9afbb05fe01f236b37 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 5 Jul 2022 16:37:28 +0200 Subject: Docs: Convert PEP 630 (Isolating Extension Modules) to a HOWTO (GH-94489) Co-authored-by: Stanley <46876382+slateny@users.noreply.github.com> Co-authored-by: Alex Waygood --- Doc/howto/index.rst | 1 + Doc/howto/isolating-extensions.rst | 536 +++++++++++++++++++++++++++++++++++++ 2 files changed, 537 insertions(+) create mode 100644 Doc/howto/isolating-extensions.rst diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst index eae8f14..8a378e6 100644 --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -31,4 +31,5 @@ Currently, the HOWTOs are: clinic.rst instrumentation.rst annotations.rst + isolating-extensions.rst diff --git a/Doc/howto/isolating-extensions.rst b/Doc/howto/isolating-extensions.rst new file mode 100644 index 0000000..8ee7e5e --- /dev/null +++ b/Doc/howto/isolating-extensions.rst @@ -0,0 +1,536 @@ +.. highlight:: c + +*************************** +Isolating Extension Modules +*************************** + +.. topic:: Abstract + + Traditionally, state belonging to Python extension modules was kept in C + ``static`` variables, which have process-wide scope. This document + describes problems of such per-process state and shows a safer way: + per-module state. + + The document also describes how to switch to per-module state where + possible. This transition involves allocating space for that state, potentially + switching from static types to heap types, and—perhaps most + importantly—accessing per-module state from code. + + +Who should read this +==================== + +This guide is written for maintainers of :ref:`C-API ` extensions +who would like to make that extension safer to use in applications where +Python itself is used as a library. + + +Background +========== + +An *interpreter* is the context in which Python code runs. It contains +configuration (e.g. the import path) and runtime state (e.g. the set of +imported modules). + +Python supports running multiple interpreters in one process. There are +two cases to think about—users may run interpreters: + +- in sequence, with several :c:func:`Py_InitializeEx`/:c:func:`Py_FinalizeEx` + cycles, and +- in parallel, managing "sub-interpreters" using + :c:func:`Py_NewInterpreter`/:c:func:`Py_EndInterpreter`. + +Both cases (and combinations of them) would be most useful when +embedding Python within a library. Libraries generally shouldn't make +assumptions about the application that uses them, which include +assuming a process-wide "main Python interpreter". + +Historically, Python extension modules don't handle this use case well. +Many extension modules (and even some stdlib modules) use *per-process* +global state, because C ``static`` variables are extremely easy to use. +Thus, data that should be specific to an interpreter ends up being shared +between interpreters. Unless the extension developer is careful, it is very +easy to introduce edge cases that lead to crashes when a module is loaded in +more than one interpreter in the same process. + +Unfortunately, *per-interpreter* state is not easy to achieve. Extension +authors tend to not keep multiple interpreters in mind when developing, +and it is currently cumbersome to test the behavior. + +Enter Per-Module State +---------------------- + +Instead of focusing on per-interpreter state, Python's C API is evolving +to better support the more granular *per-module* state. +This means that C-level data is be attached to a *module object*. +Each interpreter creates its own module object, keeping the data separate. +For testing the isolation, multiple module objects corresponding to a single +extension can even be loaded in a single interpreter. + +Per-module state provides an easy way to think about lifetime and +resource ownership: the extension module will initialize when a +module object is created, and clean up when it's freed. In this regard, +a module is just like any other :c:expr:`PyObject *`; there are no "on +interpreter shutdown" hooks to think—or forget—about. + +Note that there are use cases for different kinds of "globals": +per-process, per-interpreter, per-thread or per-task state. +With per-module state as the default, these are still possible, +but you should treat them as exceptional cases: +if you need them, you should give them additional care and testing. +(Note that this guide does not cover them.) + + +Isolated Module Objects +----------------------- + +The key point to keep in mind when developing an extension module is +that several module objects can be created from a single shared library. +For example: + +.. code-block:: pycon + + >>> import sys + >>> import binascii + >>> old_binascii = binascii + >>> del sys.modules['binascii'] + >>> import binascii # create a new module object + >>> old_binascii == binascii + False + +As a rule of thumb, the two modules should be completely independent. +All objects and state specific to the module should be encapsulated +within the module object, not shared with other module objects, and +cleaned up when the module object is deallocated. +Since this just is a rule of thumb, exceptions are possible +(see `Managing Global State`_), but they will need more +thought and attention to edge cases. + +While some modules could do with less stringent restrictions, isolated +modules make it easier to set clear expectations and guidelines that +work across a variety of use cases. + + +Surprising Edge Cases +--------------------- + +Note that isolated modules do create some surprising edge cases. Most +notably, each module object will typically not share its classes and +exceptions with other similar modules. Continuing from the +`example above `__, +note that ``old_binascii.Error`` and ``binascii.Error`` are +separate objects. In the following code, the exception is *not* caught: + +.. code-block:: pycon + + >>> old_binascii.Error == binascii.Error + False + >>> try: + ... old_binascii.unhexlify(b'qwertyuiop') + ... except binascii.Error: + ... print('boo') + ... + Traceback (most recent call last): + File "", line 2, in + binascii.Error: Non-hexadecimal digit found + +This is expected. Notice that pure-Python modules behave the same way: +it is a part of how Python works. + +The goal is to make extension modules safe at the C level, not to make +hacks behave intuitively. Mutating ``sys.modules`` "manually" counts +as a hack. + + +Making Modules Safe with Multiple Interpreters +============================================== + + +Managing Global State +--------------------- + +Sometimes, the state associated with a Python module is not specific to that module, but +to the entire process (or something else "more global" than a module). +For example: + +- The ``readline`` module manages *the* terminal. +- A module running on a circuit board wants to control *the* on-board + LED. + +In these cases, the Python module should provide *access* to the global +state, rather than *own* it. If possible, write the module so that +multiple copies of it can access the state independently (along with +other libraries, whether for Python or other languages). If that is not +possible, consider explicit locking. + +If it is necessary to use process-global state, the simplest way to +avoid issues with multiple interpreters is to explicitly prevent a +module from being loaded more than once per process—see +`Opt-Out: Limiting to One Module Object per Process`_. + + +Managing Per-Module State +------------------------- + +To use per-module state, use +:ref:`multi-phase extension module initialization `. +This signals that your module supports multiple interpreters correctly. + +Set ``PyModuleDef.m_size`` to a positive number to request that many +bytes of storage local to the module. Usually, this will be set to the +size of some module-specific ``struct``, which can store all of the +module's C-level state. In particular, it is where you should put +pointers to classes (including exceptions, but excluding static types) +and settings (e.g. ``csv``'s :py:data:`~csv.field_size_limit`) +which the C code needs to function. + +.. note:: + Another option is to store state in the module's ``__dict__``, + but you must avoid crashing when users modify ``__dict__`` from + Python code. This usually means error- and type-checking at the C level, + which is easy to get wrong and hard to test sufficiently. + + However, if module state is not needed in C code, storing it in + ``__dict__`` only is a good idea. + +If the module state includes ``PyObject`` pointers, the module object +must hold references to those objects and implement the module-level hooks +``m_traverse``, ``m_clear`` and ``m_free``. These work like +``tp_traverse``, ``tp_clear`` and ``tp_free`` of a class. Adding them will +require some work and make the code longer; this is the price for +modules which can be unloaded cleanly. + +An example of a module with per-module state is currently available as +`xxlimited `__; +example module initialization shown at the bottom of the file. + + +Opt-Out: Limiting to One Module Object per Process +-------------------------------------------------- + +A non-negative ``PyModuleDef.m_size`` signals that a module supports +multiple interpreters correctly. If this is not yet the case for your +module, you can explicitly make your module loadable only once per +process. For example:: + + static int loaded = 0; + + static int + exec_module(PyObject* module) + { + if (loaded) { + PyErr_SetString(PyExc_ImportError, + "cannot load module more than once per process"); + return -1; + } + loaded = 1; + // ... rest of initialization + } + + +Module State Access from Functions +---------------------------------- + +Accessing the state from module-level functions is straightforward. +Functions get the module object as their first argument; for extracting +the state, you can use ``PyModule_GetState``:: + + static PyObject * + func(PyObject *module, PyObject *args) + { + my_struct *state = (my_struct*)PyModule_GetState(module); + if (state == NULL) { + return NULL; + } + // ... rest of logic + } + +.. note:: + ``PyModule_GetState`` may return ``NULL`` without setting an + exception if there is no module state, i.e. ``PyModuleDef.m_size`` was + zero. In your own module, you're in control of ``m_size``, so this is + easy to prevent. + + +Heap Types +========== + +Traditionally, types defined in C code are *static*; that is, +``static PyTypeObject`` structures defined directly in code and +initialized using ``PyType_Ready()``. + +Such types are necessarily shared across the process. Sharing them +between module objects requires paying attention to any state they own +or access. To limit the possible issues, static types are immutable at +the Python level: for example, you can't set ``str.myattribute = 123``. + +.. impl-detail:: + Sharing truly immutable objects between interpreters is fine, + as long as they don't provide access to mutable objects. + However, in CPython, every Python object has a mutable implementation + detail: the reference count. Changes to the refcount are guarded by the GIL. + Thus, code that shares any Python objects across interpreters implicitly + depends on CPython's current, process-wide GIL. + +Because they are immutable and process-global, static types cannot access +"their" module state. +If any method of such a type requires access to module state, +the type must be converted to a *heap-allocated type*, or *heap type* +for short. These correspond more closely to classes created by Python's +``class`` statement. + +For new modules, using heap types by default is a good rule of thumb. + + +Changing Static Types to Heap Types +----------------------------------- + +Static types can be converted to heap types, but note that +the heap type API was not designed for "lossless" conversion +from static types—that is, creating a type that works exactly like a given +static type. +So, when rewriting the class definition in a new API, +you are likely to unintentionally change a few details (e.g. pickleability +or inherited slots). +Always test the details that are important to you. + +Watch out for the following two points in particular (but note that this is not +a comprehensive list): + +* Unlike static types, heap type objects are mutable by default. + Use the :c:data:`Py_TPFLAGS_IMMUTABLETYPE` flag to prevent mutability. +* Heap types inherit :c:member:`~PyTypeObject.tp_new` by default, + so it may become possible to instantiate them from Python code. + You can prevent this with the :c:data:`Py_TPFLAGS_DISALLOW_INSTANTIATION` flag. + + +Defining Heap Types +------------------- + +Heap types can be created by filling a :c:struct:`PyType_Spec` structure, a +description or "blueprint" of a class, and calling +:c:func:`PyType_FromModuleAndSpec` to construct a new class object. + +.. note:: + Other functions, like :c:func:`PyType_FromSpec`, can also create + heap types, but :c:func:`PyType_FromModuleAndSpec` associates the module + with the class, allowing access to the module state from methods. + +The class should generally be stored in *both* the module state (for +safe access from C) and the module's ``__dict__`` (for access from +Python code). + + +Garbage-Collection Protocol +--------------------------- + +Instances of heap types hold a reference to their type. +This ensures that the type isn't destroyed before all its instances are, +but may result in reference cycles that need to be broken by the +garbage collector. + +To avoid memory leaks, instances of heap types must implement the +garbage collection protocol. +That is, heap types should: + +- Have the :c:data:`Py_TPFLAGS_HAVE_GC` flag. +- Define a traverse function using ``Py_tp_traverse``, which + visits the type (e.g. using :c:expr:`Py_VISIT(Py_TYPE(self))`). + +Please refer to the :ref:`the documentation ` of +:c:data:`Py_TPFLAGS_HAVE_GC` and :c:member:`~PyTypeObject.tp_traverse` +for additional considerations. + +If your traverse function delegates to the ``tp_traverse`` of its base class +(or another type), ensure that ``Py_TYPE(self)`` is visited only once. +Note that only heap type are expected to visit the type in ``tp_traverse``. + +For example, if your traverse function includes:: + + base->tp_traverse(self, visit, arg) + +...and ``base`` may be a static type, then it should also include:: + + if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) { + // a heap type's tp_traverse already visited Py_TYPE(self) + } else { + Py_VISIT(Py_TYPE(self)); + } + +It is not necessary to handle the type's reference count in ``tp_new`` +and ``tp_clear``. + + +Module State Access from Classes +-------------------------------- + +If you have a type object defined with :c:func:`PyType_FromModuleAndSpec`, +you can call :c:func:`PyType_GetModule` to get the associated module, and then +:c:func:`PyModule_GetState` to get the module's state. + +To save a some tedious error-handling boilerplate code, you can combine +these two steps with :c:func:`PyType_GetModuleState`, resulting in:: + + my_struct *state = (my_struct*)PyType_GetModuleState(type); + if (state === NULL) { + return NULL; + } + + +Module State Access from Regular Methods +---------------------------------------- + +Accessing the module-level state from methods of a class is somewhat more +complicated, but is possible thanks to API introduced in Python 3.9. +To get the state, you need to first get the *defining class*, and then +get the module state from it. + +The largest roadblock is getting *the class a method was defined in*, or +that method's "defining class" for short. The defining class can have a +reference to the module it is part of. + +Do not confuse the defining class with :c:expr:`Py_TYPE(self)`. If the method +is called on a *subclass* of your type, ``Py_TYPE(self)`` will refer to +that subclass, which may be defined in different module than yours. + +.. note:: + The following Python code can illustrate the concept. + ``Base.get_defining_class`` returns ``Base`` even + if ``type(self) == Sub``: + + .. code-block:: python + + class Base: + def get_type_of_self(self): + return type(self) + + def get_defining_class(self): + return __class__ + + class Sub(Base): + pass + +For a method to get its "defining class", it must use the +:c:data:`METH_METHOD | METH_FASTCALL | METH_KEYWORDS` +:c:type:`calling convention ` +and the corresponding :c:type:`PyCMethod` signature:: + + PyObject *PyCMethod( + PyObject *self, // object the method was called on + PyTypeObject *defining_class, // defining class + PyObject *const *args, // C array of arguments + Py_ssize_t nargs, // length of "args" + PyObject *kwnames) // NULL, or dict of keyword arguments + +Once you have the defining class, call :c:func:`PyType_GetModuleState` to get +the state of its associated module. + +For example:: + + static PyObject * + example_method(PyObject *self, + PyTypeObject *defining_class, + PyObject *const *args, + Py_ssize_t nargs, + PyObject *kwnames) + { + my_struct *state = (my_struct*)PyType_GetModuleState(defining_class); + if (state === NULL) { + return NULL; + } + ... // rest of logic + } + + PyDoc_STRVAR(example_method_doc, "..."); + + static PyMethodDef my_methods[] = { + {"example_method", + (PyCFunction)(void(*)(void))example_method, + METH_METHOD|METH_FASTCALL|METH_KEYWORDS, + example_method_doc} + {NULL}, + } + + +Module State Access from Slot Methods, Getters and Setters +---------------------------------------------------------- + +.. note:: + + This is new in Python 3.11. + + .. After adding to limited API: + + If you use the `limited API __, + you must update ``Py_LIMITED_API`` to ``0x030b0000``, losing ABI + compatibility with earlier versions. + +Slot methods—the fast C equivalents for special methods, such as +:c:member:`~PyNumberMethods.nb_add` for :py:attr:`~object.__add__` or +:c:member:`~PyType.tp_new` for initialization—have a very simple API that +doesn't allow passing in the defining class, unlike with :c:type:`PyCMethod`. +The same goes for getters and setters defined with +:c:type:`PyGetSetDef`. + +To access the module state in these cases, use the +:c:func:`PyType_GetModuleByDef` function, and pass in the module definition. +Once you have the module, call :c:func:`PyModule_GetState` +to get the state:: + + PyObject *module = PyType_GetModuleByDef(Py_TYPE(self), &module_def); + my_struct *state = (my_struct*)PyModule_GetState(module); + if (state === NULL) { + return NULL; + } + +``PyType_GetModuleByDef`` works by searching the +:term:`method resolution order` (i.e. all superclasses) for the first +superclass that has a corresponding module. + +.. note:: + + In very exotic cases (inheritance chains spanning multiple modules + created from the same definition), ``PyType_GetModuleByDef`` might not + return the module of the true defining class. However, it will always + return a module with the same definition, ensuring a compatible + C memory layout. + + +Lifetime of the Module State +---------------------------- + +When a module object is garbage-collected, its module state is freed. +For each pointer to (a part of) the module state, you must hold a reference +to the module object. + +Usually this is not an issue, because types created with +:c:func:`PyType_FromModuleAndSpec`, and their instances, hold a reference +to the module. +However, you must be careful in reference counting when you reference +module state from other places, such as callbacks for external +libraries. + + +Open Issues +=========== + +Several issues around per-module state and heap types are still open. + +Discussions about improving the situation are best held on the `capi-sig +mailing list `__. + + +Per-Class Scope +--------------- + +It is currently (as of Python 3.11) not possible to attach state to individual +*types* without relying on CPython implementation details (which may change +in the future—perhaps, ironically, to allow a proper solution for +per-class scope). + + +Lossless Conversion to Heap Types +--------------------------------- + +The heap type API was not designed for "lossless" conversion from static types; +that is, creating a type that works exactly like a given static type. -- cgit v0.12