summaryrefslogtreecommitdiffstats
path: root/Modules/_threadmodule.c
Commit message (Collapse)AuthorAgeFilesLines
* gh-116322: Add Py_mod_gil module slot (#116882)Brett Simmers2024-05-031-0/+1
| | | | | | | | | | | | | | This PR adds the ability to enable the GIL if it was disabled at interpreter startup, and modifies the multi-phase module initialization path to enable the GIL when loading a module, unless that module's spec includes a slot indicating it can run safely without the GIL. PEP 703 called the constant for the slot `Py_mod_gil_not_used`; I went with `Py_MOD_GIL_NOT_USED` for consistency with gh-104148. A warning will be issued up to once per interpreter for the first GIL-using module that is loaded. If `-v` is given, a shorter message will be printed to stderr every time a GIL-using module is loaded (including the first one that issues a warning).
* gh-118332: Fix deadlock involving stop the world (#118412)Sam Gross2024-04-301-1/+2
| | | | | | Avoid detaching thread state when stopping the world. When re-attaching the thread state, the thread would attempt to resume the top-most critical section, which might now be held by a thread paused for our stop-the-world request.
* gh-117764: Add signatures and improve docstrings in the _thread module ↵Serhiy Storchaka2024-04-121-46/+126
| | | | (GH-117772)
* gh-113964: Don't prevent new threads until all non-daemon threads exit (#116677)Sam Gross2024-03-191-1/+1
| | | | | | | | | | Starting in Python 3.12, we prevented calling fork() and starting new threads during interpreter finalization (shutdown). This has led to a number of regressions and flaky tests. We should not prevent starting new threads (or `fork()`) until all non-daemon threads exit and finalization starts in earnest. This changes the checks to use `_PyInterpreterState_GetFinalizing(interp)`, which is set immediately before terminating non-daemon threads.
* gh-116915: Make `_thread._ThreadHandle` support GC (#116934)mpage2024-03-181-3/+13
| | | | | | Even though it has no internal references to Python objects it still has a reference to its type by virtue of being a heap type. We need to provide a traverse function that visits the type, but we do not need to provide a clear function.
* gh-114271: Fix race in `Thread.join()` (#114839)mpage2024-03-161-305/+674
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is a race between when `Thread._tstate_lock` is released[^1] in `Thread._wait_for_tstate_lock()` and when `Thread._stop()` asserts[^2] that it is unlocked. Consider the following execution involving threads A, B, and C: 1. A starts. 2. B joins A, blocking on its `_tstate_lock`. 3. C joins A, blocking on its `_tstate_lock`. 4. A finishes and releases its `_tstate_lock`. 5. B acquires A's `_tstate_lock` in `_wait_for_tstate_lock()`, releases it, but is swapped out before calling `_stop()`. 6. C is scheduled, acquires A's `_tstate_lock` in `_wait_for_tstate_lock()` but is swapped out before releasing it. 7. B is scheduled, calls `_stop()`, which asserts that A's `_tstate_lock` is not held. However, C holds it, so the assertion fails. The race can be reproduced[^3] by inserting sleeps at the appropriate points in the threading code. To do so, run the `repro_join_race.py` from the linked repo. There are two main parts to this PR: 1. `_tstate_lock` is replaced with an event that is attached to `PyThreadState`. The event is set by the runtime prior to the thread being cleared (in the same place that `_tstate_lock` was released). `Thread.join()` blocks waiting for the event to be set. 2. `_PyInterpreterState_WaitForThreads()` provides the ability to wait for all non-daemon threads to exit. To do so, an `is_daemon` predicate was added to `PyThreadState`. This field is set each time a thread is created. `threading._shutdown()` now calls into `_PyInterpreterState_WaitForThreads()` instead of waiting on `_tstate_lock`s. [^1]: https://github.com/python/cpython/blob/441affc9e7f419ef0b68f734505fa2f79fe653c7/Lib/threading.py#L1201 [^2]: https://github.com/python/cpython/blob/441affc9e7f419ef0b68f734505fa2f79fe653c7/Lib/threading.py#L1115 [^3]: https://github.com/mpage/cpython/commit/81946532792f938cd6f6ab4c4ff92a4edf61314f --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <antoine@python.org>
* gh-116437: Use new C API PyDict_Pop() to simplify the code (GH-116438)Serhiy Storchaka2024-03-071-6/+2
|
* gh-114271: Make `_thread.lock` thread-safe in free-threaded builds (#116433)mpage2024-03-061-2/+2
| | | | | | | | Previously, the `locked` field was set after releasing the lock. This reverses the order so that the `locked` field is set while the lock is still held. There is still one thread-safety issue where `locked` is checked prior to releasing the lock, however, in practice that will only be an issue when unlocking the lock is contended, which should be rare.
* gh-114271: Make `_thread.ThreadHandle` thread-safe in free-threaded builds ↵mpage2024-03-011-40/+127
| | | | | | | | | | | | | | | | | (GH-115190) Make `_thread.ThreadHandle` thread-safe in free-threaded builds We protect the mutable state of `ThreadHandle` using a `_PyOnceFlag`. Concurrent operations (i.e. `join` or `detach`) on `ThreadHandle` block until it is their turn to execute or an earlier operation succeeds. Once an operation has been applied successfully all future operations complete immediately. The `join()` method is now idempotent. It may be called multiple times but the underlying OS thread will only be joined once. After `join()` succeeds, any future calls to `join()` will succeed immediately. The internal thread handle `detach()` method has been removed.
* gh-110850: Replace private _PyTime_MAX with public PyTime_MAX (#115751)Victor Stinner2024-02-211-1/+1
| | | | | | | Remove references to the old names _PyTime_MIN and _PyTime_MAX, now that PyTime_MIN and PyTime_MAX are public. Replace also _PyTime_MIN with PyTime_MIN.
* gh-110850: Use public PyTime functions (#115746)Victor Stinner2024-02-201-1/+1
| | | | | Replace private _PyTime functions with public PyTime functions. random_seed_time_pid() now reports errors to its caller.
* gh-110850: Cleanup pycore_time.h includes (#115724)Victor Stinner2024-02-201-0/+1
| | | | | <pycore_time.h> include is no longer needed to get the PyTime_t type in internal header files. This type is now provided by <Python.h> include. Add <pycore_time.h> includes to C files instead.
* gh-110850: Replace _PyTime_t with PyTime_t (#115719)Victor Stinner2024-02-201-6/+6
| | | | | Run command: sed -i -e 's!\<_PyTime_t\>!PyTime_t!g' $(find -name "*.c" -o -name "*.h")
* gh-114271: Make `thread._rlock` thread-safe in free-threaded builds (#115102)mpage2024-02-161-10/+22
| | | | | | | | | The ID of the owning thread (`rlock_owner`) may be accessed by multiple threads without holding the underlying lock; relaxed atomics are used in place of the previous loads/stores. The number of times that the lock has been acquired (`rlock_count`) is only ever accessed by the thread that holds the lock; we do not need to use atomics to access it.
* gh-114570: Add PythonFinalizationError exception (#115352)Victor Stinner2024-02-141-1/+1
| | | | | | | | | | | | | | | | | Add PythonFinalizationError exception. This exception derived from RuntimeError is raised when an operation is blocked during the Python finalization. The following functions now raise PythonFinalizationError, instead of RuntimeError: * _thread.start_new_thread() * subprocess.Popen * os.fork() * os.fork1() * os.forkpty() Morever, _winapi.Overlapped finalizer now logs an unraisable PythonFinalizationError, instead of an unraisable RuntimeError.
* gh-114271: Make `PyInterpreterState.threads.count` thread-safe in ↵mpage2024-02-121-3/+3
| | | | | free-threaded builds (gh-115093) Use atomics to mutate PyInterpreterState.threads.count.
* gh-115035: Mark ThreadHandles as non-joinable earlier after forking (#115042)Sam Gross2024-02-061-17/+36
| | | | | | This marks dead ThreadHandles as non-joinable earlier in `PyOS_AfterFork_Child()` before we execute any Python code. The handles are stored in a global linked list in `_PyRuntimeState` because `fork()` affects the entire process.
* gh-114315: Make `threading.Lock` a real class, not a factory function (#114479)Nikita Sobolev2024-01-251-4/+29
| | | | | | | | | | `threading.Lock` is now the underlying class and is constructable rather than the old factory function. This allows for type annotations to refer to it which had no non-ugly way to be expressed prior to this. --------- Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-114414: Assert PyType_GetModuleByDef result in _threadmodule (#114415)Nikita Sobolev2024-01-221-0/+3
|
* gh-111789: Use PyDict_GetItemRef() in Modules/_threadmodule.c (gh-112077)Serhiy Storchaka2023-11-271-6/+4
|
* gh-111262: Add PyDict_Pop() function (#112028)Victor Stinner2023-11-141-5/+2
| | | | | | | _PyDict_Pop_KnownHash(): remove the default value and the return type becomes an int. Co-authored-by: Stefan Behnel <stefan_ml@behnel.de> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* GH-110829: Ensure Thread.join() joins the OS thread (#110848)Antoine Pitrou2023-11-041-55/+282
| | | | | | | Joining a thread now ensures the underlying OS thread has exited. This is required for safer fork() in multi-threaded processes. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* gh-108082: Remove _PyErr_WriteUnraisableMsg() (GH-111643)Serhiy Storchaka2023-11-031-2/+2
| | | | Replace the remaining calls with PyErr_FormatUnraisable().
* gh-106320: Re-add some PyLong/PyDict C-API functions (GH-#111162)scoder2023-10-251-1/+0
| | | | | | | | * gh-106320: Re-add _PyLong_FromByteArray(), _PyLong_AsByteArray() and _PyLong_GCD() to the public header files since they are used by third-party packages and there is no efficient replacement. See https://github.com/python/cpython/issues/111140 See https://github.com/python/cpython/issues/111139 * gh-111262: Re-add _PyDict_Pop() to have a C-API until a new public one is designed.
* gh-84570: Add Timeouts to SendChannel.send() and RecvChannel.recv() (gh-110567)Eric Snow2023-10-171-5/+6
|
* gh-84570: Send-Wait Fixes for _xxinterpchannels (gh-111006)Eric Snow2023-10-171-50/+2
| | | | | There were a few things I did in gh-110565 that need to be fixed. I also forgot to add tests in that PR. (Note that this PR exposes a refleak introduced by gh-110246. I'll take care of that separately.)
* gh-109860: Use a New Thread State When Switching Interpreters, When ↵Eric Snow2023-10-031-1/+1
| | | | | | | | | Necessary (gh-110245) In a few places we switch to another interpreter without knowing if it has a thread state associated with the current thread. For the main interpreter there wasn't much of a problem, but for subinterpreters we were *mostly* okay re-using the tstate created with the interpreter (located via PyInterpreterState_ThreadHead()). There was a good chance that tstate wasn't actually in use by another thread. However, there are no guarantees of that. Furthermore, re-using an already used tstate is currently fragile. To address this, now we create a new thread state in each of those places and use it. One consequence of this change is that PyInterpreterState_ThreadHead() may not return NULL (though that won't happen for the main interpreter).
* gh-105716: Support Background Threads in Subinterpreters Consistently ↵Eric Snow2023-10-021-1/+15
| | | | | | | (gh-109921) The existence of background threads running on a subinterpreter was preventing interpreters from getting properly destroyed, as well as impacting the ability to run the interpreter again. It also affected how we wait for non-daemon threads to finish. We add PyInterpreterState.threads.main, with some internal C-API functions.
* gh-109593: Fix reentrancy issue in multiprocessing resource_tracker (#109629)Antoine Pitrou2023-09-261-0/+14
| | | | | --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* gh-109795: `_thread.start_new_thread`: allocate thread bootstate using raw ↵Radislav Chugunov2023-09-251-3/+6
| | | | memory allocator (#109808)
* gh-109611: Add convenient C API function _PyFile_Flush() (GH-109612)Serhiy Storchaka2023-09-231-3/+1
|
* gh-108987: Fix _thread.start_new_thread() race condition (#109135)Victor Stinner2023-09-111-16/+31
| | | | | | | | | | | | | Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time. Move _PyThreadState_CheckConsistency() from thread_run() to _PyThreadState_Bind().
* gh-104690: thread_run() checks for tstate dangling pointer (#109056)Victor Stinner2023-09-081-2/+5
| | | | | | | | thread_run() of _threadmodule.c now calls _PyThreadState_CheckConsistency() to check if tstate is a dangling pointer when Python is built in debug mode. Rename ceval_gil.c is_tstate_valid() to _PyThreadState_CheckConsistency() to reuse it in _threadmodule.c.
* gh-106320: Remove private _PyErr_WriteUnraisableMsg() (#108863)Victor Stinner2023-09-041-0/+1
| | | | | | Move the private _PyErr_WriteUnraisableMsg() functions to the internal C API (pycore_pyerrors.h). Move write_unraisable_exc() from _testcapi to _testinternalcapi.
* gh-106320: Remove private _PySys functions (#108452)Victor Stinner2023-08-241-0/+1
| | | | | | | | | | Move private functions to the internal C API (pycore_sysmodule.h): * _PySys_GetAttr() * _PySys_GetSizeOf() No longer export most of these functions. Fix also a typo in Include/cpython/optimizer.h: add a missing space.
* gh-106320: Remove private _PyDict functions (#108449)Victor Stinner2023-08-241-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Move private functions to the internal C API (pycore_dict.h): * _PyDictView_Intersect() * _PyDictView_New() * _PyDict_ContainsId() * _PyDict_DelItemId() * _PyDict_DelItem_KnownHash() * _PyDict_GetItemIdWithError() * _PyDict_GetItem_KnownHash() * _PyDict_HasSplitTable() * _PyDict_NewPresized() * _PyDict_Next() * _PyDict_Pop() * _PyDict_SetItemId() * _PyDict_SetItem_KnownHash() * _PyDict_SizeOf() No longer export most of these functions. Move also the _PyDictViewObject structure to the internal C API. Move dict_getitem_knownhash() function from _testcapi to the _testinternalcapi extension. Update test_capi.test_dict for this change.
* gh-106320: Remove private _PyEval function (#108433)Victor Stinner2023-08-241-0/+1
| | | | | | | | | | | | | | Move private _PyEval functions to the internal C API (pycore_ceval.h): * _PyEval_GetBuiltin() * _PyEval_GetBuiltinId() * _PyEval_GetSwitchInterval() * _PyEval_MakePendingCalls() * _PyEval_SetProfile() * _PyEval_SetSwitchInterval() * _PyEval_SetTrace() No longer export most of these functions.
* gh-106869: Use new PyMemberDef constant names (#106871)Victor Stinner2023-07-251-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove '#include "structmember.h"'. * If needed, add <stddef.h> to get offsetof() function. * Update Parser/asdl_c.py to regenerate Python/Python-ast.c. * Replace: * T_SHORT => Py_T_SHORT * T_INT => Py_T_INT * T_LONG => Py_T_LONG * T_FLOAT => Py_T_FLOAT * T_DOUBLE => Py_T_DOUBLE * T_STRING => Py_T_STRING * T_OBJECT => _Py_T_OBJECT * T_CHAR => Py_T_CHAR * T_BYTE => Py_T_BYTE * T_UBYTE => Py_T_UBYTE * T_USHORT => Py_T_USHORT * T_UINT => Py_T_UINT * T_ULONG => Py_T_ULONG * T_STRING_INPLACE => Py_T_STRING_INPLACE * T_BOOL => Py_T_BOOL * T_OBJECT_EX => Py_T_OBJECT_EX * T_LONGLONG => Py_T_LONGLONG * T_ULONGLONG => Py_T_ULONGLONG * T_PYSSIZET => Py_T_PYSSIZET * T_NONE => _Py_T_NONE * READONLY => Py_READONLY * PY_AUDIT_READ => Py_AUDIT_READ * READ_RESTRICTED => Py_AUDIT_READ * PY_WRITE_RESTRICTED => _Py_WRITE_RESTRICTED * RESTRICTED => (READ_RESTRICTED | _Py_WRITE_RESTRICTED)
* gh-86493: Fix possible leaks in some modules initialization (GH-106768)Serhiy Storchaka2023-07-181-2/+2
| | | | Fix _ssl, _stat, _testinternalcapi, _threadmodule, cmath, math, posix, time.
* gh-106521: Remove _PyObject_LookupAttr() function (GH-106642)Serhiy Storchaka2023-07-121-1/+1
|
* gh-106320: Remove _PyInterpreterState_Get() alias (#106321)Victor Stinner2023-07-011-1/+1
| | | | Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.
* gh-105927: _abc and _thread use PyWeakref_GetRef() (#105961)Victor Stinner2023-06-211-13/+13
| | | | | | | | Hold a strong reference on the object, rather than using a borrowed reference: replace PyWeakref_GET_OBJECT() with PyWeakref_GetRef() and _PyWeakref_GET_REF(). Remove assert(PyWeakref_CheckRef(localweakref)) since it's already tested by _PyWeakref_GET_REF().
* gh-104812: Run Pending Calls in any Thread (gh-104813)Eric Snow2023-06-131-1/+2
| | | For a while now, pending calls only run in the main thread (in the main interpreter). This PR changes things to allow any thread run a pending call, unless the pending call was explicitly added for the main thread to run.
* gh-104690 Disallow thread creation and fork at interpreter finalization ↵chgnrdv2023-06-041-0/+5
| | | | | | | | | | | | | | | | | (#104826) Disallow thread creation and fork at interpreter finalization. in the following functions, check if interpreter is finalizing and raise `RuntimeError` with appropriate message: * `_thread.start_new_thread` and thus `threading` * `posix.fork` * `posix.fork1` * `posix.forkpty` * `_posixsubprocess.fork_exec` when a `preexec_fn=` is supplied. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-104205)Eric Snow2023-05-051-0/+1
| | | Here we are doing no more than adding the value for Py_mod_multiple_interpreters and using it for stdlib modules. We will start checking for it in gh-104206 (once PyInterpreterState.ceval.own_gil is added in gh-104204).
* gh-103712: Increase the length of the type name in AttributeError messages ↵Alex Gaynor2023-04-241-1/+1
| | | | (#103713)
* gh-59956: Clarify Runtime State Status Expectations (gh-101308)Eric Snow2023-01-301-1/+1
| | | | | | | | | | | | | A PyThreadState can be in one of many states in its lifecycle, represented by some status value. Those statuses haven't been particularly clear, so we're addressing that here. Specifically: * made the distinct lifecycle statuses clear on PyThreadState * identified expectations of how various lifecycle-related functions relate to status * noted the various places where those expectations don't match the actual behavior At some point we'll need to address the mismatches. (This change also includes some cleanup.) https://github.com/python/cpython/issues/59956
* gh-59956: Clarify GILState-related Code (gh-101161)Eric Snow2023-01-191-7/+1
| | | | | | | | | | The objective of this change is to help make the GILState-related code easier to understand. This mostly involves moving code around and some semantically equivalent refactors. However, there are a also a small number of slight changes in structure and behavior: * tstate_current is moved out of _PyRuntimeState.gilstate * autoTSSkey is moved out of _PyRuntimeState.gilstate * autoTSSkey is initialized earlier * autoTSSkey is re-initialized (after fork) earlier https://github.com/python/cpython/issues/59956
* GH-100892: consolidate `HEAD_LOCK/HEAD_UNLOCK` macros (#100953)Kumar Aditya2023-01-151-5/+0
|
* GH-100892: Fix race in clearing `threading.local` (#100922)Kumar Aditya2023-01-111-10/+20
|
generated by cgit v0.12 at 2025-02-08 03:59:27 (GMT)