summaryrefslogtreecommitdiffstats
path: root/Python/pystate.c
Commit message (Collapse)AuthorAgeFilesLines
* gh-110079: Remove extern "C" { ...} in C code (#110080)Victor Stinner2023-09-291-15/+6
|
* gh-109793: Allow Switching Interpreters During Finalization (gh-109794)Eric Snow2023-09-271-1/+16
| | | Essentially, we should check the thread ID rather than the thread state pointer.
* gh-76785: Use Pending Calls When Releasing Cross-Interpreter Data (gh-109556)Eric Snow2023-09-191-33/+58
| | | This fixes some crashes in the _xxinterpchannels module, due to a race between interpreters.
* gh-108724: Add PyMutex and _PyParkingLot APIs (gh-109344)Sam Gross2023-09-191-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | PyMutex is a one byte lock with fast, inlineable lock and unlock functions for the common uncontended case. The design is based on WebKit's WTF::Lock. PyMutex is built using the _PyParkingLot APIs, which provides a cross-platform futex-like API (based on WebKit's WTF::ParkingLot). This internal API will be used for building other synchronization primitives used to implement PEP 703, such as one-time initialization and events. This also includes tests and a mini benchmark in Tools/lockbench/lockbench.py to compare with the existing PyThread_type_lock. Uncontended acquisition + release: * Linux (x86-64): PyMutex: 11 ns, PyThread_type_lock: 44 ns * macOS (arm64): PyMutex: 13 ns, PyThread_type_lock: 18 ns * Windows (x86-64): PyMutex: 13 ns, PyThread_type_lock: 38 ns PR Overview: The primary purpose of this PR is to implement PyMutex, but there are a number of support pieces (described below). * PyMutex: A 1-byte lock that doesn't require memory allocation to initialize and is generally faster than the existing PyThread_type_lock. The API is internal only for now. * _PyParking_Lot: A futex-like API based on the API of the same name in WebKit. Used to implement PyMutex. * _PyRawMutex: A word sized lock used to implement _PyParking_Lot. * PyEvent: A one time event. This was used a bunch in the "nogil" fork and is useful for testing the PyMutex implementation, so I've included it as part of the PR. * pycore_llist.h: Defines common operations on doubly-linked list. Not strictly necessary (could do the list operations manually), but they come up frequently in the "nogil" fork. ( Similar to https://man.freebsd.org/cgi/man.cgi?queue) --------- Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
* gh-106213: Make Emscripten trampolines work with JSPI (GH-106219)Hood Chatham2023-09-151-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | There is a WIP proposal to enable webassembly stack switching which have been implemented in v8: https://github.com/WebAssembly/js-promise-integration It is not possible to switch stacks that contain JS frames so the Emscripten JS trampolines that allow calling functions with the wrong number of arguments don't work in this case. However, the js-promise-integration proposal requires the [type reflection for Wasm/JS API](https://github.com/WebAssembly/js-types) proposal, which allows us to actually count the number of arguments a function expects. For better compatibility with stack switching, this PR checks if type reflection is available, and if so we use a switch block to decide the appropriate signature. If type reflection is unavailable, we should use the current EMJS trampoline. We cache the function argument counts since when I didn't cache them performance was negatively affected. Co-authored-by: T. Wouters <thomas@python.org> Co-authored-by: Brett Cannon <brett@python.org>
* gh-108987: Fix _thread.start_new_thread() race condition (#109135)Victor Stinner2023-09-111-0/+29
| | | | | | | | | | | | | Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time. Move _PyThreadState_CheckConsistency() from thread_run() to _PyThreadState_Bind().
* gh-104690: thread_run() checks for tstate dangling pointer (#109056)Victor Stinner2023-09-081-0/+18
| | | | | | | | thread_run() of _threadmodule.c now calls _PyThreadState_CheckConsistency() to check if tstate is a dangling pointer when Python is built in debug mode. Rename ceval_gil.c is_tstate_valid() to _PyThreadState_CheckConsistency() to reuse it in _threadmodule.c.
* GH-91079: Rename C_RECURSION_LIMIT to Py_C_RECURSION_LIMIT (#108507)Victor Stinner2023-09-081-1/+1
| | | | | | | Symbols of the C API should be prefixed by "Py_" to avoid conflict with existing names in 3rd party C extensions on "#include <Python.h>". test.pythoninfo now logs Py_C_RECURSION_LIMIT constant and other _testcapi and _testinternalcapi constants.
* GH-108716: Turn off deep-freezing of code objects. (GH-108722)Mark Shannon2023-09-081-0/+1
|
* gh-108765: Cleanup #include in Python/*.c files (#108977)Victor Stinner2023-09-061-3/+3
| | | Mention one symbol imported by each #include.
* gh-108634: PyInterpreterState_New() no longer calls Py_FatalError() (#108748)Victor Stinner2023-09-011-35/+62
| | | | | | | | | | | | pycore_create_interpreter() now returns a status, rather than calling Py_FatalError(). * PyInterpreterState_New() now calls Py_ExitStatusException() instead of calling Py_FatalError() directly. * Replace Py_FatalError() with PyStatus in init_interpreter() and _PyObject_InitState(). * _PyErr_SetFromPyStatus() now raises RuntimeError, instead of ValueError. It can now call PyErr_NoMemory(), raise MemoryError, if it detects _PyStatus_NO_MEMORY() error message.
* gh-108634: Py_TRACE_REFS uses a hash table (#108663)Victor Stinner2023-08-311-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Python built with "configure --with-trace-refs" (tracing references) is now ABI compatible with Python release build and debug build. Moreover, it now also supports the Limited API. Change Py_TRACE_REFS build: * Remove _PyObject_EXTRA_INIT macro. * The PyObject structure no longer has two extra members (_ob_prev and _ob_next). * Use a hash table (_Py_hashtable_t) to trace references (all objects): PyInterpreterState.object_state.refchain. * Py_TRACE_REFS build is now ABI compatible with release build and debug build. * Limited C API extensions can now be built with Py_TRACE_REFS: xxlimited, xxlimited_35, _testclinic_limited. * No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2() functions to PyModule_Create2TraceRefs() and PyModule_FromDefAndSpec2TraceRefs(). * _Py_PrintReferenceAddresses() is now called before finalize_interp_delete() which deletes the refchain hash table. * test_tracemalloc find_trace() now also filters by size to ignore the memory allocated by _PyRefchain_Trace(). Test changes for Py_TRACE_REFS: * Add test.support.Py_TRACE_REFS constant. * Add test_sys.test_getobjects() to test sys.getobjects() function. * test_exceptions skips test_recursion_normalizing_with_no_memory() and test_memory_error_in_PyErr_PrintEx() if Python is built with Py_TRACE_REFS. * test_repl skips test_no_memory(). * test_capi skisp test_set_nomemory().
* GH-108035: Remove the `_PyCFrame` struct as it is no longer needed for ↵Mark Shannon2023-08-171-3/+3
| | | | performance. (GH-108036)
* gh-107630: Initialize Each Interpreter's refchain Properly (gh-107733)Eric Snow2023-08-071-0/+1
| | | This finishes fixing the crashes in Py_TRACE_REFS builds. We missed this part in gh-107567.
* gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974)Eric Snow2023-07-281-69/+0
| | | | | | | This fixes a crasher due to a race condition, triggered infrequently when two isolated (own GIL) subinterpreters simultaneously initialize their sys or builtins modules. The crash happened due the combination of the "detached" thread state we were using and the "last holder" logic we use for the GIL. It turns out it's tricky to use the same thread state for different threads. Who could have guessed? We solve the problem by eliminating the one object we were still sharing between interpreters. We replace it with a low-level hashtable, using the "raw" allocator to avoid tying it to the main interpreter. We also remove the accommodations for "detached" thread states, which were a dubious idea to start with.
* gh-101524: Only Use Public C-API in the _xxsubinterpreters Module (gh-107359)Eric Snow2023-07-271-1/+1
| | | The _xxsubinterpreters module should not rely on internal API. Some of the functions it uses were recently moved there however. Here we move them back (and expose them properly).
* GH-103082: Rename PY_MONITORING_EVENTS to _PY_MONITORING_EVENTS (#107069)Victor Stinner2023-07-221-4/+4
| | | | | | Rename private C API constants: * Rename PY_MONITORING_UNGROUPED_EVENTS to _PY_MONITORING_UNGROUPED_EVENTS * Rename PY_MONITORING_EVENTS to _PY_MONITORING_EVENTS
* gh-106320: Use _PyInterpreterState_GET() (#106336)Victor Stinner2023-07-021-1/+1
| | | | Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().
* gh-106320: Remove private _PyInterpreterState functions (#106325)Victor Stinner2023-07-021-3/+3
| | | | | | | | | Remove private _PyThreadState and _PyInterpreterState C API functions: move them to the internal C API (pycore_pystate.h and pycore_interp.h). Don't export most of these functions anymore, but still export functions used by tests. Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C API, but keep it in the stable API.
* gh-105927: Avoid calling PyWeakref_GET_OBJECT() (#105997)Victor Stinner2023-06-221-3/+6
| | | | | | | * Replace PyWeakref_GET_OBJECT() with _PyWeakref_GET_REF(). * _sqlite/blob.c now holds a strong reference to the blob object while calling close_blob(). * _xidregistry_find_type() now holds a strong reference to registered while using it.
* GH-100987: Allow objects other than code objects as the "executable" of an ↵Mark Shannon2023-06-141-1/+0
| | | | | | | | | | internal frame. (GH-105727) * Add table describing possible executable classes for out-of-process debuggers. * Remove shim code object creation code as it is no longer needed. * Make lltrace a bit more robust w.r.t. non-standard frames.
* gh-104812: Run Pending Calls in any Thread (gh-104813)Eric Snow2023-06-131-1/+2
| | | For a while now, pending calls only run in the main thread (in the main interpreter). This PR changes things to allow any thread run a pending call, unless the pending call was explicitly added for the main thread to run.
* gh-100227: Lock Around Modification of the Global Allocators State (gh-105516)Eric Snow2023-06-081-1/+2
| | | The risk of a race with this state is relatively low, but we play it safe anyway. We do avoid using the lock in performance-sensitive cases where the risk of a race is very, very low.
* gh-100227: Lock Around Adding Global Audit Hooks (gh-105515)Eric Snow2023-06-081-3/+4
| | | The risk of a race with this state is relatively low, but we play it safe anyway.
* gh-100227: Lock Around Use of the Global "atexit" State (gh-105514)Eric Snow2023-06-081-22/+13
| | | The risk of a race with this state is relatively low, but we play it safe anyway.
* GH-104584: Plugin optimizer API (GH-105100)Mark Shannon2023-06-021-0/+8
|
* gh-104341: Call _PyEval_ReleaseLock() with NULL When Finalizing the Current ↵Eric Snow2023-06-011-3/+18
| | | | | | | Thread (gh-105109) This avoids the problematic race in drop_gil() by skipping the FORCE_SWITCHING code there for finalizing threads. (The idea for this approach came out of discussions with @markshannon.)
* gh-104341: Adjust tstate_must_exit() to Respect Interpreter Finalization ↵Eric Snow2023-05-151-5/+7
| | | | | (gh-104437) With the move to a per-interpreter GIL, this check slipped through the cracks.
* gh-99113: A Per-Interpreter GIL! (gh-104210)Eric Snow2023-05-081-3/+1
| | | | | This is the culmination of PEP 684 (and of my 8-year long multi-core Python project)! Each subinterpreter may now be created with its own GIL (via Py_NewInterpreterFromConfig()). If not so configured then the interpreter will share with the main interpreter--the status quo since subinterpreters were added decades ago. The main interpreter always has its own GIL and subinterpreters from Py_NewInterpreter() will always share with the main interpreter.
* gh-99113: Make Sure the GIL is Acquired at the Right Places (gh-104208)Eric Snow2023-05-061-10/+32
| | | This is a pre-requisite for a per-interpreter GIL. Without it this change isn't strictly necessary. However, there is no real downside otherwise.
* gh-99113: Share the GIL via PyInterpreterState.ceval.gil (gh-104203)Eric Snow2023-05-051-1/+1
| | | In preparation for a per-interpreter GIL, we add PyInterpreterState.ceval.gil, set it to the shared GIL for each interpreter, and use that rather than using _PyRuntime.ceval.gil directly. Note that _PyRuntime.ceval.gil is still the actual GIL.
* gh-103323: Remove PyRuntimeState_GetThreadState() (#104171)Victor Stinner2023-05-041-1/+1
| | | | | This function no longer makes sense, since its runtime parameter is no longer used. Use directly _PyThreadState_GET() and _PyInterpreterState_GET() instead.
* GH-103082: Code cleanup in instrumentation code (#103474)Mark Shannon2023-04-291-4/+4
|
* gh-100227: Add a Granular Lock for _PyRuntime.imports.extensions.dict ↵Eric Snow2023-04-251-1/+4
| | | | | | (gh-103460) The lock is unnecessary as long as there's a GIL, but completely necessary with a per-interpreter GIL.
* gh-101659: Isolate "obmalloc" State to Each Interpreter (gh-101660)Eric Snow2023-04-241-4/+13
| | | | | | | | | | | | | | This is strictly about moving the "obmalloc" runtime state from `_PyRuntimeState` to `PyInterpreterState`. Doing so improves isolation between interpreters, specifically most of the memory (incl. objects) allocated for each interpreter's use. This is important for a per-interpreter GIL, but such isolation is valuable even without it. FWIW, a per-interpreter obmalloc is the proverbial canary-in-the-coalmine when it comes to the isolation of objects between interpreters. Any object that leaks (unintentionally) to another interpreter is highly likely to cause a crash (on debug builds at least). That's a useful thing to know, relative to interpreter isolation.
* gh-103323: Get the "Current" Thread State from a Thread-Local Variable ↵Eric Snow2023-04-241-6/+32
| | | | | | | | | | | (gh-103324) We replace _PyRuntime.tstate_current with a thread-local variable. As part of this change, we add a _Py_thread_local macro in pyport.h (only for the core runtime) to smooth out the compiler differences. The main motivation here is in support of a per-interpreter GIL, but this change also provides some performance improvement opportunities. Note that we do not provide a fallback to the thread-local, either falling back to the old tstate_current or to thread-specific storage (PyThread_tss_*()). If that proves problematic then we can circle back. I consider it unlikely, but will run the buildbots to double-check. Also note that this does not change any of the code related to the GILState API, where it uses a thread state stored in thread-specific storage. I suspect we can combine that with _Py_tss_tstate (from here). However, that can be addressed separately and is not urgent (nor critical). (While this change was mostly done independently, I did take some inspiration from earlier (~2020) work by @markshannon (main...markshannon:threadstate_in_tls) and @vstinner (#23976).)
* GH-103082: Implementation of PEP 669: Low Impact Monitoring for CPython ↵Mark Shannon2023-04-121-5/+35
| | | | | | | | | | (GH-103083) * The majority of the monitoring code is in instrumentation.c * The new instrumentation bytecodes are in bytecodes.c * legacy_tracing.c adapts the new API to the old sys.setrace and sys.setprofile APIs
* gh-103176: sys._current_exceptions() returns mapping to exception instances ↵Irit Katriel2023-04-111-7/+6
| | | | instead of exc_info tuples (#103177)
* gh-100227: Use an Array for _PyRuntime's Set of Locks During Init (gh-103315)Eric Snow2023-04-061-60/+48
| | | This cleans things up a bit and simplifies adding new granular global locks.
* gh-100227: Make the Global PyModuleDef Cache Safe for Isolated Interpreters ↵Eric Snow2023-03-291-22/+106
| | | | | | | | | | | | | | | | | | | | | | | (gh-103084) Sharing mutable (or non-immortal) objects between interpreters is generally not safe. We can work around that but not easily. There are two restrictions that are critical for objects that break interpreter isolation. The first is that the object's state be guarded by a global lock. For now the GIL meets this requirement, but a granular global lock is needed once we have a per-interpreter GIL. The second restriction is that the object (and, for a container, its items) be deallocated/resized only when the interpreter in which it was allocated is the current one. This is because every interpreter has (or will have, see gh-101660) its own object allocator. Deallocating an object with a different allocator can cause crashes. The dict for the cache of module defs is completely internal, which simplifies what we have to do to meet those requirements. To do so, we do the following: * add a mechanism for re-using a temporary thread state tied to the main interpreter in an arbitrary thread * add _PyRuntime.imports.extensions.main_tstate` * add _PyThreadState_InitDetached() and _PyThreadState_ClearDetached() (pystate.c) * add _PyThreadState_BindDetached() and _PyThreadState_UnbindDetached() (pystate.c) * make sure the cache dict (_PyRuntime.imports.extensions.dict) and its items are all owned by the main interpreter) * add a placeholder using for a granular global lock Note that the cache is only used for legacy extension modules and not for multi-phase init modules. https://github.com/python/cpython/issues/100227
* gh-100227: Revert gh-102925 "gh-100227: Make the Global Interned Dict Safe ↵Eric Snow2023-03-271-183/+21
| | | | | | | for Isolated Interpreters" (gh-103063) This reverts commit 87be8d9. This approach to keeping the interned strings safe is turning out to be too complex for my taste (due to obmalloc isolation). For now I'm going with the simpler solution, making the dict per-interpreter. We can revisit that later if we want a sharing solution.
* gh-100227: Make the Global Interned Dict Safe for Isolated Interpreters ↵Eric Snow2023-03-231-21/+183
| | | | | | | | | (gh-102925) This is effectively two changes. The first (the bulk of the change) is where we add _Py_AddToGlobalDict() (and _PyRuntime.cached_objects.main_tstate, etc.). The second (much smaller) change is where we update PyUnicode_InternInPlace() to use _Py_AddToGlobalDict() instead of calling PyDict_SetDefault() directly. Basically, _Py_AddToGlobalDict() is a wrapper around PyDict_SetDefault() that should be used whenever we need to add a value to a runtime-global dict object (in the few cases where we are leaving the container global rather than moving it to PyInterpreterState, e.g. the interned strings dict). _Py_AddToGlobalDict() does all the necessary work to make sure the target global dict is shared safely between isolated interpreters. This is especially important as we move the obmalloc state to each interpreter (gh-101660), as well as, potentially, the GIL (PEP 684). https://github.com/python/cpython/issues/100227
* gh-102304: Move the Total Refcount to PyInterpreterState (gh-102545)Eric Snow2023-03-211-2/+8
| | | | | Moving it valuable with a per-interpreter GIL. However, it is also useful without one, since it allows us to identify refleaks within a single interpreter or where references are escaping an interpreter. This becomes more important as we move the obmalloc state to PyInterpreterState. https://github.com/python/cpython/issues/102304
* gh-102304: Fix Non-Debug Builds (gh-102846)Eric Snow2023-03-201-0/+2
| | | | | Some debug-only code slipped in with gh-102543. https://github.com/python/cpython/issues/102304
* gh-102304: Move _Py_RefTotal to _PyRuntimeState (gh-102543)Eric Snow2023-03-201-0/+3
| | | | | | | The essentially eliminates the global variable, with the associated benefits. This is also a precursor to isolating this bit of state to PyInterpreterState. Folks that currently read _Py_RefTotal directly would have to start using _Py_GetGlobalRefTotal() instead. https://github.com/python/cpython/issues/102304
* gh-102660: Handle m_copy Specially for the sys and builtins Modules (gh-102661)Eric Snow2023-03-141-0/+1
| | | | | | | It doesn't make sense to use multi-phase init for these modules. Using a per-interpreter "m_copy" (instead of PyModuleDef.m_base.m_copy) makes this work okay. (This came up while working on gh-101660.) Note that we might instead end up disallowing re-load for sys/builtins since they are so special. https://github.com/python/cpython/issues/102660
* gh-100227: Move the dtoa State to PyInterpreterState (gh-102331)Eric Snow2023-02-281-1/+19
| | | https://github.com/python/cpython/issues/100227
* GH-102126: fix deadlock at shutdown when clearing thread states (#102222)Kumar Aditya2023-02-251-3/+10
|
* gh-101758: Fix the wasm Buildbots (gh-101943)Eric Snow2023-02-161-0/+4
| | | | | They were broken by gh-101920. https://github.com/python/cpython/issues/101758
* gh-101758: Clean Up Uses of Import State (gh-101919)Eric Snow2023-02-151-140/+11
| | | | | | | | | | | | | | | This change is almost entirely moving code around and hiding import state behind internal API. We introduce no changes to behavior, nor to non-internal API. (Since there was already going to be a lot of churn, I took this as an opportunity to re-organize import.c into topically-grouped sections of code.) The motivation is to simplify a number of upcoming changes. Specific changes: * move existing import-related code to import.c, wherever possible * add internal API for interacting with import state (both global and per-interpreter) * use only API outside of import.c (to limit churn there when changing the location, etc.) * consolidate the import-related state of PyInterpreterState into a single struct field (this changes layout slightly) * add macros for import state in import.c (to simplify changing the location) * group code in import.c into sections *remove _PyState_AddModule() https://github.com/python/cpython/issues/101758
generated by cgit v0.12 at 2025-08-10 13:31:26 (GMT)