summaryrefslogtreecommitdiffstats
path: root/Lib/concurrent
Commit message (Collapse)AuthorAgeFilesLines
* gh-88110: Clear concurrent.futures.thread._threads_queues after fork to ↵Andrei Bodrov2024-11-221-0/+1
| | | | | | | | | avoid joining parent process' threads (GH-126098) Threads are gone after fork, so clear the queues too. Otherwise the child process (here created via multiprocessing.Process) crashes on interpreter exit. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* gh-125864: Propagate `pickle.loads()` failures in `InterpreterPoolExecutor` ↵Peter Bierma2024-10-241-1/+2
| | | | | (gh-125898) Authored-by: Peter Bierma <zintensitydev@gmail.com>
* gh-124694: Add concurrent.futures.InterpreterPoolExecutor (gh-124548)Eric Snow2024-10-163-29/+314
| | | | | | | | | | | | This is an implementation of InterpreterPoolExecutor that builds on ThreadPoolExecutor. (Note that this is not tied to PEP 734, which is strictly about adding a new stdlib module.) Possible future improvements: * support passing a script for the initializer or to submit() * support passing (most) arbitrary functions without pickling * support passing closures * optionally exec functions against __main__ instead of the their original module
* gh-125451: Fix deadlock in ProcessPoolExecutor shutdown (#125492)Sam Gross2024-10-161-29/+21
| | | | | | | | | | There was a deadlock when `ProcessPoolExecutor` shuts down at the same time that a queueing thread handles an error processing a task. Don't use `_shutdown_lock` to protect the `_ThreadWakeup` pipes -- use an internal lock instead. This fixes the ordering deadlock where the `ExecutorManagerThread` holds the `_shutdown_lock` and joins the queueing thread, while the queueing thread is attempting to acquire the `_shutdown_lock` while closing the `_ThreadWakeup`.
* gh-120417: Add #noqa to used imports in the stdlib (#120421)Victor Stinner2024-06-131-1/+1
| | | | | Tools such as ruff can ignore "imported but unused" warnings if a line ends with "# noqa: F401". It avoids the temptation to remove an import which is used effectively.
* gh-120121: Add InvalidStateError to concurrent.futures.__all__ (#120123)AN Long2024-06-081-0/+1
| | | Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
* gh-120157: Remove unused code in concurrent.future (gh-120187)Clinton2024-06-071-8/+0
|
* gh-117783: Immortalize objects that use deferred reference counting (#118112)Sam Gross2024-04-291-2/+3
| | | | | | | | | Deferred reference counting is not fully implemented yet. As a temporary measure, we immortalize objects that would use deferred reference counting to avoid multi-threaded scaling bottlenecks. This is only performed in the free-threaded build once the first non-main thread is started. Additionally, some tests, including refleak tests, suppress this behavior.
* Simplify concurrent.futures.process code by using itertools.batched() ↵NewUserHa2024-01-271-11/+1
| | | | (GH-114221)
* gh-109649: Use os.process_cpu_count() (#110165)Victor Stinner2023-10-012-3/+3
| | | | | | | | | | | | | | | | | | | | Replace os.cpu_count() with os.process_cpu_count() in modules: * compileall * concurrent.futures * multiprocessing Replace os.cpu_count() with os.process_cpu_count() in programs: * _decimal deccheck.py test * freeze.py * multissltests.py * python -m test (regrtest) * wasm_build.py Other changes: * test.pythoninfo logs os.process_cpu_count(). * regrtest gets os.process_cpu_count() / os.cpu_count() in headers.
* gh-109047: concurrent.futures catches PythonFinalizationError (#109810)Victor Stinner2023-09-291-17/+30
| | | | | | | | | | | | | | | | | | | | | | | | | concurrent.futures: The *executor manager thread* now catches exceptions when adding an item to the *call queue*. During Python finalization, creating a new thread can now raise RuntimeError. Catch the exception and call terminate_broken() in this case. Add test_python_finalization_error() to test_concurrent_futures. concurrent.futures._ExecutorManagerThread changes: * terminate_broken() no longer calls shutdown_workers() since the call queue is no longer working anymore (read and write ends of the queue pipe are closed). * terminate_broken() now terminates child processes, not only wait until they complete. * _ExecutorManagerThread.terminate_broken() now holds shutdown_lock to prevent race conditons with ProcessPoolExecutor.submit(). multiprocessing.Queue changes: * Add _terminate_broken() method. * _start_thread() sets _thread to None on exception to prevent leaking "dangling threads" even if the thread was not started yet.
* Remove concurrent.futures deadcode: process_result_item() (#109906)Victor Stinner2023-09-261-18/+8
| | | | process_result_item() cannot be called with an int anymore, the protocol changed.
* gh-109370: Fix unexpected traceback output in test_concurrent_futures ↵Serhiy Storchaka2023-09-261-1/+2
| | | | | | | | | (GH-109780) Follow-up of gh-107219. * Only close the connection writer on Windows. * Also use existing constant _winapi.ERROR_OPERATION_ABORTED instead of WSA_OPERATION_ABORTED.
* gh-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock (#108513)elfstrom2023-09-221-3/+15
| | | | | | | | This fixes issue #105829, https://github.com/python/cpython/issues/105829 Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Chris Withers <chris@withers.org> Co-authored-by: Thomas Moreau <thomas.moreau.2010@gmail.com>
* Fix typos in docs and comments (#109619)Heinz-Alexander Fuetterer2023-09-201-1/+1
|
* gh-107219: Fix concurrent.futures terminate_broken() (#109244)Victor Stinner2023-09-111-0/+4
| | | | | | | | | | | | | | | Fix a race condition in concurrent.futures. When a process in the process pool was terminated abruptly (while the future was running or pending), close the connection write end. If the call queue is blocked on sending bytes to a worker process, closing the connection write end interrupts the send, so the queue can be closed. Changes: * _ExecutorManagerThread.terminate_broken() now closes call_queue._writer. * multiprocessing PipeConnection.close() now interrupts WaitForMultipleObjects() in _send_bytes() by cancelling the overlapped operation.
* gh-107219: Fix concurrent.futures terminate_broken() (#108974)Victor Stinner2023-09-061-1/+8
| | | | | | | | Fix a race condition in _ExecutorManagerThread.terminate_broken(): ignore the InvalidStateError on future.set_exception(). It can happen if the future is cancelled before the caller. Moreover, test_crash_big_data() now waits explicitly until the executor completes.
* gh-94777: Fix deadlock in ProcessPoolExecutor (#94784)Louis Paulot2023-07-101-0/+4
| | | Fixes a hang in multiprocessing process pool executor when a child process crashes and code could otherwise block on writing to the pipe. See GH-94777 for more details.
* gh-102024: Reduced _idle_semaphore.release calls (#102025)Andrii Kuzmin2023-05-261-8/+11
| | | | | | | | | Reduced _idle_semaphore.release calls in concurrent.futures.thread._worker _idle_semaphore.release() is now only called if only work_queue is empty. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
* gh-94440: Fix issue of ProcessPoolExecutor shutdown hanging (#94468)yonatanp2023-03-161-0/+5
| | | | | Fix an issue of concurrent.futures ProcessPoolExecutor shutdown hanging. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* gh-102515: Remove unused imports in the `Lib/` directory (#102516)Alex Waygood2023-03-081-0/+2
|
* gh-84559: Remove the new multiprocessing warning, too disruptive. (#101551)Gregory P. Smith2023-02-031-17/+0
| | | | This reverts the core of #100618 while leaving relevant documentation improvements and minor refactorings in place.
* GH-84559: Deprecate fork being the multiprocessing default. (#100618)Gregory P. Smith2023-02-021-3/+20
| | | | | | | | This starts the process. Users who don't specify their own start method and use the default on platforms where it is 'fork' will see a DeprecationWarning upon multiprocessing.Pool() construction or upon multiprocessing.Process.start() or concurrent.futures.ProcessPool use. See the related issue and documentation within this change for details.
* gh-95166: cancel map waited on future on timeout (GH-95169)Thomas Grainger2022-07-281-2/+14
| | | Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
* gh-90622: Do not spawn ProcessPool workers on demand via fork method. (#91598)Gregory P. Smith2022-05-081-10/+34
| | | | | | Do not spawn ProcessPool workers on demand when they spawn via fork. This avoids potential deadlocks in the child processes due to forking from a multithreaded process.
* gh-90622: Prevent max_tasks_per_child use with a fork mp_context. (#91587)Gregory P. Smith2022-05-061-7/+17
| | | | | Prevent `max_tasks_per_child` use with a "fork" mp_context to avoid deadlocks. Also defaults to "spawn" when no mp_context is supplied for safe convenience.
* bpo-46787: Fix `ProcessPoolExecutor exception` memory leak (GH-31408) (#31408)themylogin2022-05-021-0/+3
| | | | Do not store `ProcessPoolExecutor` work item exception traceback that prevents exception frame locals from being garbage collected.
* concurrent.futures: Fix typo in docstring (#92121)Yiannis Hadjicharalambous2022-05-021-1/+1
|
* Remove trailing spaces (GH-31695)Serhiy Storchaka2022-03-051-1/+1
|
* bpo-46522: fix concurrent.futures and io AttributeError messages (GH-30887)Thomas Grainger2022-02-231-1/+1
| | | | Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
* bpo-20369: concurrent.futures.wait() now deduplicates futures given a… ↵Kumar Aditya2022-01-041-6/+7
| | | | | | | | | (GH-30168) * bpo-20369: concurrent.futures.wait() now deduplicates futures given as arg. * 📜🤖 Added by blurb_it. Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* bpo-42413: Replace `concurrent.futures.TimeoutError` and ↵Kumar Aditya2021-12-191-3/+1
| | | | | `asyncio.TimeoutError` with builtin `TimeoutError` (GH-30197) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
* bpo-43498: Fix dictionary iteration error in _ExecutorManagerThread (GH-24868)Jakub Kulík2021-11-291-1/+1
|
* bpo-44733: Add max_tasks_per_child to ProcessPoolExecutor (GH-27373)Logan Jones2021-11-201-14/+55
| | | Co-authored-by: Antoine Pitrou <antoine@python.org>
* bpo-45021: Fix a hang in forked children (GH-28007)nullptr2021-09-201-0/+6
| | | _global_shutdown_lock should be reinitialized in forked children
* Use `from` imports (GH-26594)Machinexa22021-06-081-5/+4
| | | from imports
* bpo-35930: Raising an exception raised in a "future" instance will create ↵Jesús Cea2021-03-291-15/+23
| | | | | | | reference cycles (#24995) Before: https://lists.es.python.org/pipermail/general/attachments/20201229/0c14bc58/attachment-0002.png After: https://lists.es.python.org/pipermail/general/attachments/20201229/0c14bc58/attachment-0003.png
* bpo-40692: Run more test_concurrent_futures tests (GH-20239)Asheesh Laroia2021-02-081-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | In the case of multiprocessing.synchronize() being missing, the test_concurrent_futures test suite now skips only the tests that require multiprocessing.synchronize(). Validate that multiprocessing.synchronize exists as part of _check_system_limits(), allowing ProcessPoolExecutor to raise NotImplementedError during __init__, rather than crashing with ImportError during __init__ when creating a lock imported from multiprocessing.synchronize. Use _check_system_limits() to disable tests of ProcessPoolExecutor on systems without multiprocessing.synchronize. Running the test suite without multiprocessing.synchronize reveals that Lib/compileall.py crashes when it uses a ProcessPoolExecutor. Therefore, change Lib/compileall.py to call _check_system_limits() before creating the ProcessPoolExecutor. Note that both Lib/compileall.py and Lib/test/test_compileall.py were attempting to sanity-check ProcessPoolExecutor by expecting ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent on platforms where POSIX semaphores aren't available. Avoid using sem_unlink() if it, too, does not exist. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-39349: Add cancel_futures to Executor.shutdown base class (GH-22023)Shantanu2020-09-011-1/+4
| | | | * Add cancel_futures parameter to the Executor base class, since it was missed in the original PR (https://github.com/python/cpython/pull/18057) that added cancel_futures.
* bpo-40443: Remove unused imports in the stdlib (GH-19803)Victor Stinner2020-04-302-2/+0
|
* bpo-39995: Fix concurrent.futures _ThreadWakeup (GH-19760)Victor Stinner2020-04-291-14/+27
| | | | Fix a race condition in concurrent.futures._ThreadWakeup: access to _ThreadWakeup is now protected with the shutdown lock.
* bpo-30966: concurrent.futures.Process.shutdown() closes queue (GH-19738)Victor Stinner2020-04-271-0/+2
| | | | Process.shutdown(wait=True) of concurrent.futures now closes explicitly the result queue.
* bpo-39207: Spawn workers on demand in ProcessPoolExecutor (GH-19453)Kyle Stanley2020-04-191-2/+14
| | | | | Roughly based on https://github.com/python/cpython/commit/904e34d4e6b6007986dcc585d5c553ee8ae06f95, but with a few substantial differences. /cc @pitrou @brianquinlan
* bpo-39481: PEP 585 for a variety of modules (GH-19423)Batuhan Taşkaya2020-04-102-0/+6
| | | | | | | | | | - concurrent.futures - ctypes - http.cookies - multiprocessing - queue - tempfile - unittest.case - urllib.parse
* bpo-39812: Remove daemon threads in concurrent.futures (GH-19149)Kyle Stanley2020-03-272-32/+11
| | | | | | Remove daemon threads from :mod:`concurrent.futures` by adding an internal `threading._register_atexit()`, which calls registered functions prior to joining all non-daemon threads. This allows for compatibility with subinterpreters, which don't support daemon threads.
* bpo-39678: refactor queue manager thread (GH-18551)Thomas Moreau2020-03-011-207/+229
|
* bpo-39104: Fix hanging ProcessPoolExecutor on shutdown nowait with pickling ↵Thomas Moreau2020-02-161-21/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | failure (GH-17670) As reported initially by @rad-pat in #6084, the following script causes a deadlock. ``` from concurrent.futures import ProcessPoolExecutor class ObjectWithPickleError(): """Triggers a RuntimeError when sending job to the workers""" def __reduce__(self): raise RuntimeError() if __name__ == "__main__": e = ProcessPoolExecutor() f = e.submit(id, ObjectWithPickleError()) e.shutdown(wait=False) f.result() # Deadlock on get ``` This is caused by the fact that the main process is closing communication channels that might be necessary to the `queue_management_thread` later. To avoid this, this PR let the `queue_management_thread` manage all the closing. https://bugs.python.org/issue39104 Automerge-Triggered-By: @pitrou
* bpo-39349: Add *cancel_futures* to Executor.shutdown() (GH-18057)Kyle Stanley2020-02-022-2/+36
|
* bpo-31783: Fix a race condition creating workers during shutdown (#13171)Brian Quinlan2019-06-281-2/+6
| | | | | | * bpo-31783: Fix a race condition while creating workers during interpreter shutdown * 📜🤖 Added by blurb_it.
* Fix typo in Lib/concurrent/futures/thread.py (GH-13953)ubordignon2019-06-152-2/+2
|
generated by cgit v0.12 at 2025-09-07 23:07:56 (GMT)