summaryrefslogtreecommitdiffstats
path: root/Python/pystate.c
Commit message (Collapse)AuthorAgeFilesLines
* [3.12] gh-110310: Add a Per-Interpreter XID Registry for Heap Types ↵Eric Snow2023-11-281-38/+120
| | | | | | | | | | | | | | | (gh-110311) (gh-110714) We do the following: * add a per-interpreter XID registry (PyInterpreterState.xidregistry) * put heap types there (keep static types in _PyRuntimeState.xidregistry) * clear the registries during interpreter/runtime finalization * avoid duplicate entries in the registry (when _PyCrossInterpreterData_RegisterClass() is called more than once for a type) * use Py_TYPE() instead of PyObject_Type() in _PyCrossInterpreterData_Lookup() The per-interpreter registry helps preserve isolation between interpreters. This is important when heap types are registered, which is something we haven't been doing yet but I will likely do soon. (cherry-picked from commit 80dc39e1dc2abc809f448cba5d2c5b9c1c631e11)
* [3.12] gh-105716: Support Background Threads in Subinterpreters Consistently ↵Eric Snow2023-11-281-0/+37
| | | | | | | | | (gh-109921) (gh-110707) The existence of background threads running on a subinterpreter was preventing interpreters from getting properly destroyed, as well as impacting the ability to run the interpreter again. It also affected how we wait for non-daemon threads to finish. We add PyInterpreterState.threads.main, with some internal C-API functions. (cherry-picked from commit 1dd9dee45d2591b4e701039d1673282380696849)
* [3.12] gh-109793: Allow Switching Interpreters During Finalization ↵Eric Snow2023-11-281-1/+16
| | | | | (gh-109794) (gh-110705) Essentially, we should check the thread ID rather than the thread state pointer.
* [3.12] gh-109853: Fix sys.path[0] For Subinterpreters (gh-109994) (gh-110701)Eric Snow2023-11-271-0/+4
| | | | | | | This change makes sure sys.path[0] is set properly for subinterpreters. Before, it wasn't getting set at all. This change does not address the broader concerns from gh-109853. (cherry-picked from commit a040a32ea2f13f16172394d3e3e3f80f47f25a68)
* [3.12] gh-76785: Use Pending Calls When Releasing Cross-Interpreter Data ↵Eric Snow2023-11-271-33/+58
| | | | | | (gh-109556) (gh-112288) This fixes some crashes in the _xxinterpchannels module, due to a race between interpreters. (cherry picked from commit fd7e08a6f35581e1189b9bf12feb51f7167a86c5)
* [3.12] GH-110455: Guard `assert(tstate->thread_id > 0)` with `GH-ifndef ↵Miss Islington (bot)2023-10-061-2/+2
| | | | | | | | HAVE_PTHREAD_STUBS` (GH-110487) (GH-110491) GH-110455: Guard `assert(tstate->thread_id > 0)` with `GH-ifndef HAVE_PTHREAD_STUBS` (GH-110487) (cherry picked from commit 5fd8821cf8eb1fe2e8575f8c7cc747cf78855a88) Co-authored-by: Brett Cannon <brett@python.org>
* [3.12] gh-108987: Fix _thread.start_new_thread() race condition (#109135) ↵Victor Stinner2023-10-041-0/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | (#110342) * gh-108987: Fix _thread.start_new_thread() race condition (#109135) Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time. Move _PyThreadState_CheckConsistency() from thread_run() to _PyThreadState_Bind(). (cherry picked from commit 517cd82ea7d01b344804413ef05610934a43a241) * gh-109795: `_thread.start_new_thread`: allocate thread bootstate using raw memory allocator (#109808) (cherry picked from commit 1b8f2366b38c87b0450d9c15bdfdd4c4a2fc3a01) --------- Co-authored-by: Radislav Chugunov <52372310+chgnrdv@users.noreply.github.com>
* [3.12] gh-104690: thread_run() checks for tstate dangling pointer (#109056) ↵Victor Stinner2023-10-021-0/+18
| | | | | | | | | | | | | | (#109133) gh-104690: thread_run() checks for tstate dangling pointer (#109056) thread_run() of _threadmodule.c now calls _PyThreadState_CheckConsistency() to check if tstate is a dangling pointer when Python is built in debug mode. Rename ceval_gil.c is_tstate_valid() to _PyThreadState_CheckConsistency() to reuse it in _threadmodule.c. (cherry picked from commit f63d37877ad166041489a968233b57540f8456e8)
* [3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters ↵Eric Snow2023-08-161-0/+1
| | | | | | | | | | | (#107751) * Unrevert "[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters (gh-107567) (#107599)". This reverts commit 6e4eec760648a71e1cd8f8f551997b1823b4bb9f (gh-107648). * Initialize each interpreter's refchain properly. * Skip test_basic_multiple_interpreters_deleted_no_reset on tracerefs builds.
* [3.12] gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache ↵Eric Snow2023-07-281-69/+0
| | | | | | | | | | | | | (gh-106974) (gh-107412) gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974) This fixes a crasher due to a race condition, triggered infrequently when two isolated (own GIL) subinterpreters simultaneously initialize their sys or builtins modules. The crash happened due the combination of the "detached" thread state we were using and the "last holder" logic we use for the GIL. It turns out it's tricky to use the same thread state for different threads. Who could have guessed? We solve the problem by eliminating the one object we were still sharing between interpreters. We replace it with a low-level hashtable, using the "raw" allocator to avoid tying it to the main interpreter. We also remove the accommodations for "detached" thread states, which were a dubious idea to start with. (cherry picked from commit 8ba4df91ae60833723d8d3b9afeb2b642f7176d5)
* [3.12] GH-103082: Rename PY_MONITORING_EVENTS to _PY_MONITORING_EVENTS ↵Victor Stinner2023-07-221-4/+4
| | | | | | | | | | | | (#107069) (#107075) GH-103082: Rename PY_MONITORING_EVENTS to _PY_MONITORING_EVENTS (#107069) Rename private C API constants: * Rename PY_MONITORING_UNGROUPED_EVENTS to _PY_MONITORING_UNGROUPED_EVENTS * Rename PY_MONITORING_EVENTS to _PY_MONITORING_EVENTS (cherry picked from commit 0927a2b25c059988e237108605ed8ab0c5459c53)
* [3.12] gh-104812: Run Pending Calls in any Thread (gh-104813) (gh-105752)Eric Snow2023-06-141-1/+2
| | | | For a while now, pending calls only run in the main thread (in the main interpreter). This PR changes things to allow any thread run a pending call, unless the pending call was explicitly added for the main thread to run. (cherry picked from commit 757b402)
* [3.12] gh-100227: Lock Around Modification of the Global Allocators State ↵Miss Islington (bot)2023-06-081-1/+2
| | | | | | | | (gh-105516) (gh-105532) The risk of a race with this state is relatively low, but we play it safe anyway. We do avoid using the lock in performance-sensitive cases where the risk of a race is very, very low. (cherry picked from commit 68dfa496278aa21585eb4654d5f7ef13ef76cb50) Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
* [3.12] gh-100227: Lock Around Adding Global Audit Hooks (gh-105515) (gh-105525)Eric Snow2023-06-081-3/+4
| | | | The risk of a race with this state is relatively low, but we play it safe anyway. (cherry picked from commit e822a676f1f3bef6c5413e9b856db481c08ac2a5)
* [3.12] gh-100227: Lock Around Use of the Global "atexit" State (gh-105514) ↵Miss Islington (bot)2023-06-081-22/+13
| | | | | | | | (gh-105517) The risk of a race with this state is relatively low, but we play it safe anyway. (cherry picked from commit 7799c8e678f759c7787785c6287140abe641d1b9) Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
* [3.12] gh-104341: Call _PyEval_ReleaseLock() with NULL When Finalizing the ↵Miss Islington (bot)2023-06-011-3/+18
| | | | | | | | | | Current Thread (gh-105109) (gh-105209) This avoids the problematic race in drop_gil() by skipping the FORCE_SWITCHING code there for finalizing threads. (The idea for this approach came out of discussions with @markshannon.) (cherry picked from commit 3698fda) Co-authored-by: Eric Snow ericsnowcurrently@gmail.com
* gh-104341: Adjust tstate_must_exit() to Respect Interpreter Finalization ↵Eric Snow2023-05-151-5/+7
| | | | | (gh-104437) With the move to a per-interpreter GIL, this check slipped through the cracks.
* gh-99113: A Per-Interpreter GIL! (gh-104210)Eric Snow2023-05-081-3/+1
| | | | | This is the culmination of PEP 684 (and of my 8-year long multi-core Python project)! Each subinterpreter may now be created with its own GIL (via Py_NewInterpreterFromConfig()). If not so configured then the interpreter will share with the main interpreter--the status quo since subinterpreters were added decades ago. The main interpreter always has its own GIL and subinterpreters from Py_NewInterpreter() will always share with the main interpreter.
* gh-99113: Make Sure the GIL is Acquired at the Right Places (gh-104208)Eric Snow2023-05-061-10/+32
| | | This is a pre-requisite for a per-interpreter GIL. Without it this change isn't strictly necessary. However, there is no real downside otherwise.
* gh-99113: Share the GIL via PyInterpreterState.ceval.gil (gh-104203)Eric Snow2023-05-051-1/+1
| | | In preparation for a per-interpreter GIL, we add PyInterpreterState.ceval.gil, set it to the shared GIL for each interpreter, and use that rather than using _PyRuntime.ceval.gil directly. Note that _PyRuntime.ceval.gil is still the actual GIL.
* gh-103323: Remove PyRuntimeState_GetThreadState() (#104171)Victor Stinner2023-05-041-1/+1
| | | | | This function no longer makes sense, since its runtime parameter is no longer used. Use directly _PyThreadState_GET() and _PyInterpreterState_GET() instead.
* GH-103082: Code cleanup in instrumentation code (#103474)Mark Shannon2023-04-291-4/+4
|
* gh-100227: Add a Granular Lock for _PyRuntime.imports.extensions.dict ↵Eric Snow2023-04-251-1/+4
| | | | | | (gh-103460) The lock is unnecessary as long as there's a GIL, but completely necessary with a per-interpreter GIL.
* gh-101659: Isolate "obmalloc" State to Each Interpreter (gh-101660)Eric Snow2023-04-241-4/+13
| | | | | | | | | | | | | | This is strictly about moving the "obmalloc" runtime state from `_PyRuntimeState` to `PyInterpreterState`. Doing so improves isolation between interpreters, specifically most of the memory (incl. objects) allocated for each interpreter's use. This is important for a per-interpreter GIL, but such isolation is valuable even without it. FWIW, a per-interpreter obmalloc is the proverbial canary-in-the-coalmine when it comes to the isolation of objects between interpreters. Any object that leaks (unintentionally) to another interpreter is highly likely to cause a crash (on debug builds at least). That's a useful thing to know, relative to interpreter isolation.
* gh-103323: Get the "Current" Thread State from a Thread-Local Variable ↵Eric Snow2023-04-241-6/+32
| | | | | | | | | | | (gh-103324) We replace _PyRuntime.tstate_current with a thread-local variable. As part of this change, we add a _Py_thread_local macro in pyport.h (only for the core runtime) to smooth out the compiler differences. The main motivation here is in support of a per-interpreter GIL, but this change also provides some performance improvement opportunities. Note that we do not provide a fallback to the thread-local, either falling back to the old tstate_current or to thread-specific storage (PyThread_tss_*()). If that proves problematic then we can circle back. I consider it unlikely, but will run the buildbots to double-check. Also note that this does not change any of the code related to the GILState API, where it uses a thread state stored in thread-specific storage. I suspect we can combine that with _Py_tss_tstate (from here). However, that can be addressed separately and is not urgent (nor critical). (While this change was mostly done independently, I did take some inspiration from earlier (~2020) work by @markshannon (main...markshannon:threadstate_in_tls) and @vstinner (#23976).)
* GH-103082: Implementation of PEP 669: Low Impact Monitoring for CPython ↵Mark Shannon2023-04-121-5/+35
| | | | | | | | | | (GH-103083) * The majority of the monitoring code is in instrumentation.c * The new instrumentation bytecodes are in bytecodes.c * legacy_tracing.c adapts the new API to the old sys.setrace and sys.setprofile APIs
* gh-103176: sys._current_exceptions() returns mapping to exception instances ↵Irit Katriel2023-04-111-7/+6
| | | | instead of exc_info tuples (#103177)
* gh-100227: Use an Array for _PyRuntime's Set of Locks During Init (gh-103315)Eric Snow2023-04-061-60/+48
| | | This cleans things up a bit and simplifies adding new granular global locks.
* gh-100227: Make the Global PyModuleDef Cache Safe for Isolated Interpreters ↵Eric Snow2023-03-291-22/+106
| | | | | | | | | | | | | | | | | | | | | | | (gh-103084) Sharing mutable (or non-immortal) objects between interpreters is generally not safe. We can work around that but not easily. There are two restrictions that are critical for objects that break interpreter isolation. The first is that the object's state be guarded by a global lock. For now the GIL meets this requirement, but a granular global lock is needed once we have a per-interpreter GIL. The second restriction is that the object (and, for a container, its items) be deallocated/resized only when the interpreter in which it was allocated is the current one. This is because every interpreter has (or will have, see gh-101660) its own object allocator. Deallocating an object with a different allocator can cause crashes. The dict for the cache of module defs is completely internal, which simplifies what we have to do to meet those requirements. To do so, we do the following: * add a mechanism for re-using a temporary thread state tied to the main interpreter in an arbitrary thread * add _PyRuntime.imports.extensions.main_tstate` * add _PyThreadState_InitDetached() and _PyThreadState_ClearDetached() (pystate.c) * add _PyThreadState_BindDetached() and _PyThreadState_UnbindDetached() (pystate.c) * make sure the cache dict (_PyRuntime.imports.extensions.dict) and its items are all owned by the main interpreter) * add a placeholder using for a granular global lock Note that the cache is only used for legacy extension modules and not for multi-phase init modules. https://github.com/python/cpython/issues/100227
* gh-100227: Revert gh-102925 "gh-100227: Make the Global Interned Dict Safe ↵Eric Snow2023-03-271-183/+21
| | | | | | | for Isolated Interpreters" (gh-103063) This reverts commit 87be8d9. This approach to keeping the interned strings safe is turning out to be too complex for my taste (due to obmalloc isolation). For now I'm going with the simpler solution, making the dict per-interpreter. We can revisit that later if we want a sharing solution.
* gh-100227: Make the Global Interned Dict Safe for Isolated Interpreters ↵Eric Snow2023-03-231-21/+183
| | | | | | | | | (gh-102925) This is effectively two changes. The first (the bulk of the change) is where we add _Py_AddToGlobalDict() (and _PyRuntime.cached_objects.main_tstate, etc.). The second (much smaller) change is where we update PyUnicode_InternInPlace() to use _Py_AddToGlobalDict() instead of calling PyDict_SetDefault() directly. Basically, _Py_AddToGlobalDict() is a wrapper around PyDict_SetDefault() that should be used whenever we need to add a value to a runtime-global dict object (in the few cases where we are leaving the container global rather than moving it to PyInterpreterState, e.g. the interned strings dict). _Py_AddToGlobalDict() does all the necessary work to make sure the target global dict is shared safely between isolated interpreters. This is especially important as we move the obmalloc state to each interpreter (gh-101660), as well as, potentially, the GIL (PEP 684). https://github.com/python/cpython/issues/100227
* gh-102304: Move the Total Refcount to PyInterpreterState (gh-102545)Eric Snow2023-03-211-2/+8
| | | | | Moving it valuable with a per-interpreter GIL. However, it is also useful without one, since it allows us to identify refleaks within a single interpreter or where references are escaping an interpreter. This becomes more important as we move the obmalloc state to PyInterpreterState. https://github.com/python/cpython/issues/102304
* gh-102304: Fix Non-Debug Builds (gh-102846)Eric Snow2023-03-201-0/+2
| | | | | Some debug-only code slipped in with gh-102543. https://github.com/python/cpython/issues/102304
* gh-102304: Move _Py_RefTotal to _PyRuntimeState (gh-102543)Eric Snow2023-03-201-0/+3
| | | | | | | The essentially eliminates the global variable, with the associated benefits. This is also a precursor to isolating this bit of state to PyInterpreterState. Folks that currently read _Py_RefTotal directly would have to start using _Py_GetGlobalRefTotal() instead. https://github.com/python/cpython/issues/102304
* gh-102660: Handle m_copy Specially for the sys and builtins Modules (gh-102661)Eric Snow2023-03-141-0/+1
| | | | | | | It doesn't make sense to use multi-phase init for these modules. Using a per-interpreter "m_copy" (instead of PyModuleDef.m_base.m_copy) makes this work okay. (This came up while working on gh-101660.) Note that we might instead end up disallowing re-load for sys/builtins since they are so special. https://github.com/python/cpython/issues/102660
* gh-100227: Move the dtoa State to PyInterpreterState (gh-102331)Eric Snow2023-02-281-1/+19
| | | https://github.com/python/cpython/issues/100227
* GH-102126: fix deadlock at shutdown when clearing thread states (#102222)Kumar Aditya2023-02-251-3/+10
|
* gh-101758: Fix the wasm Buildbots (gh-101943)Eric Snow2023-02-161-0/+4
| | | | | They were broken by gh-101920. https://github.com/python/cpython/issues/101758
* gh-101758: Clean Up Uses of Import State (gh-101919)Eric Snow2023-02-151-140/+11
| | | | | | | | | | | | | | | This change is almost entirely moving code around and hiding import state behind internal API. We introduce no changes to behavior, nor to non-internal API. (Since there was already going to be a lot of churn, I took this as an opportunity to re-organize import.c into topically-grouped sections of code.) The motivation is to simplify a number of upcoming changes. Specific changes: * move existing import-related code to import.c, wherever possible * add internal API for interacting with import state (both global and per-interpreter) * use only API outside of import.c (to limit churn there when changing the location, etc.) * consolidate the import-related state of PyInterpreterState into a single struct field (this changes layout slightly) * add macros for import state in import.c (to simplify changing the location) * group code in import.c into sections *remove _PyState_AddModule() https://github.com/python/cpython/issues/101758
* GH-101578: Normalize the current exception (GH-101607)Mark Shannon2023-02-081-3/+1
| | | | | | | | | | * Make sure that the current exception is always normalized. * Remove redundant type and traceback fields for the current exception. * Add new API functions: PyErr_GetRaisedException, PyErr_SetRaisedException * Add new API functions: PyException_GetArgs, PyException_SetArgs
* gh-59956: Partial Fix for GILState API Compatibility with Subinterpreters ↵Eric Snow2023-02-061-21/+4
| | | | | | | | | | | (gh-101431) The GILState API (PEP 311) implementation from 2003 made the assumption that only one thread state would ever be used for any given OS thread, explicitly disregarding the case of subinterpreters. However, PyThreadState_Swap() still facilitated switching between subinterpreters, meaning the "current" thread state (holding the GIL), and the GILState thread state could end up out of sync, causing problems (including crashes). This change addresses the issue by keeping the two in sync in PyThreadState_Swap(). I verified the fix against gh-99040. Note that the other GILState-subinterpreter incompatibility (with autoInterpreterState) is not resolved here. https://github.com/python/cpython/issues/59956
* gh-59956: Clarify Runtime State Status Expectations (gh-101308)Eric Snow2023-01-301-145/+346
| | | | | | | | | | | | | A PyThreadState can be in one of many states in its lifecycle, represented by some status value. Those statuses haven't been particularly clear, so we're addressing that here. Specifically: * made the distinct lifecycle statuses clear on PyThreadState * identified expectations of how various lifecycle-related functions relate to status * noted the various places where those expectations don't match the actual behavior At some point we'll need to address the mismatches. (This change also includes some cleanup.) https://github.com/python/cpython/issues/59956
* Fix typos in pystate.c file (#101348)Виталий Дмитриев2023-01-261-5/+5
|
* gh-59956: Allow the "Trashcan" Mechanism to Work Without a Thread State ↵Eric Snow2023-01-231-0/+16
| | | | | | | | | | | | (gh-101209) We've factored out a struct from the two PyThreadState fields. This accomplishes two things: * make it clear that the trashcan-related code doesn't need any other parts of PyThreadState * allows us to use the trashcan mechanism even when there isn't a "current" thread state We still expect the caller to hold the GIL. https://github.com/python/cpython/issues/59956
* gh-101181: Fix `unused-variable` warning in `pystate.c` (#101188)Nikita Sobolev2023-01-201-0/+3
| | | Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
* gh-59956: Fix Function Groupings in pystate.c (gh-101172)Eric Snow2023-01-201-237/+352
| | | | | This is a follow-up to gh-101161. The objective is to make it easier to read Python/pystate.c by grouping the functions there in a consistent way. This exclusively involves moving code around and adding various kinds of comments. https://github.com/python/cpython/issues/59956
* gh-59956: Clarify GILState-related Code (gh-101161)Eric Snow2023-01-191-201/+329
| | | | | | | | | | The objective of this change is to help make the GILState-related code easier to understand. This mostly involves moving code around and some semantically equivalent refactors. However, there are a also a small number of slight changes in structure and behavior: * tstate_current is moved out of _PyRuntimeState.gilstate * autoTSSkey is moved out of _PyRuntimeState.gilstate * autoTSSkey is initialized earlier * autoTSSkey is re-initialized (after fork) earlier https://github.com/python/cpython/issues/59956
* GH-100892: consolidate `HEAD_LOCK/HEAD_UNLOCK` macros (#100953)Kumar Aditya2023-01-151-4/+0
|
* GH-100126: Skip incomplete frames in more places (GH-100613)Brandt Bucher2023-01-091-7/+2
|
* GH-100000: Cleanup and polish various watchers code (GH-99998)Itamar Ostricher2022-12-141-0/+4
| | | | | | * Initialize `type_watchers` array to `NULL`s * Optimize code watchers notification * Optimize func watchers notification
generated by cgit v0.12 at 2025-07-19 19:35:38 (GMT)