summaryrefslogtreecommitdiffstats
path: root/Lib/concurrent
Commit message (Collapse)AuthorAgeFilesLines
* gh-109649: Use os.process_cpu_count() (#110165)Victor Stinner2023-10-012-3/+3
| | | | | | | | | | | | | | | | | | | | Replace os.cpu_count() with os.process_cpu_count() in modules: * compileall * concurrent.futures * multiprocessing Replace os.cpu_count() with os.process_cpu_count() in programs: * _decimal deccheck.py test * freeze.py * multissltests.py * python -m test (regrtest) * wasm_build.py Other changes: * test.pythoninfo logs os.process_cpu_count(). * regrtest gets os.process_cpu_count() / os.cpu_count() in headers.
* gh-109047: concurrent.futures catches PythonFinalizationError (#109810)Victor Stinner2023-09-291-17/+30
| | | | | | | | | | | | | | | | | | | | | | | | | concurrent.futures: The *executor manager thread* now catches exceptions when adding an item to the *call queue*. During Python finalization, creating a new thread can now raise RuntimeError. Catch the exception and call terminate_broken() in this case. Add test_python_finalization_error() to test_concurrent_futures. concurrent.futures._ExecutorManagerThread changes: * terminate_broken() no longer calls shutdown_workers() since the call queue is no longer working anymore (read and write ends of the queue pipe are closed). * terminate_broken() now terminates child processes, not only wait until they complete. * _ExecutorManagerThread.terminate_broken() now holds shutdown_lock to prevent race conditons with ProcessPoolExecutor.submit(). multiprocessing.Queue changes: * Add _terminate_broken() method. * _start_thread() sets _thread to None on exception to prevent leaking "dangling threads" even if the thread was not started yet.
* Remove concurrent.futures deadcode: process_result_item() (#109906)Victor Stinner2023-09-261-18/+8
| | | | process_result_item() cannot be called with an int anymore, the protocol changed.
* gh-109370: Fix unexpected traceback output in test_concurrent_futures ↵Serhiy Storchaka2023-09-261-1/+2
| | | | | | | | | (GH-109780) Follow-up of gh-107219. * Only close the connection writer on Windows. * Also use existing constant _winapi.ERROR_OPERATION_ABORTED instead of WSA_OPERATION_ABORTED.
* gh-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock (#108513)elfstrom2023-09-221-3/+15
| | | | | | | | This fixes issue #105829, https://github.com/python/cpython/issues/105829 Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Chris Withers <chris@withers.org> Co-authored-by: Thomas Moreau <thomas.moreau.2010@gmail.com>
* Fix typos in docs and comments (#109619)Heinz-Alexander Fuetterer2023-09-201-1/+1
|
* gh-107219: Fix concurrent.futures terminate_broken() (#109244)Victor Stinner2023-09-111-0/+4
| | | | | | | | | | | | | | | Fix a race condition in concurrent.futures. When a process in the process pool was terminated abruptly (while the future was running or pending), close the connection write end. If the call queue is blocked on sending bytes to a worker process, closing the connection write end interrupts the send, so the queue can be closed. Changes: * _ExecutorManagerThread.terminate_broken() now closes call_queue._writer. * multiprocessing PipeConnection.close() now interrupts WaitForMultipleObjects() in _send_bytes() by cancelling the overlapped operation.
* gh-107219: Fix concurrent.futures terminate_broken() (#108974)Victor Stinner2023-09-061-1/+8
| | | | | | | | Fix a race condition in _ExecutorManagerThread.terminate_broken(): ignore the InvalidStateError on future.set_exception(). It can happen if the future is cancelled before the caller. Moreover, test_crash_big_data() now waits explicitly until the executor completes.
* gh-94777: Fix deadlock in ProcessPoolExecutor (#94784)Louis Paulot2023-07-101-0/+4
| | | Fixes a hang in multiprocessing process pool executor when a child process crashes and code could otherwise block on writing to the pipe. See GH-94777 for more details.
* gh-102024: Reduced _idle_semaphore.release calls (#102025)Andrii Kuzmin2023-05-261-8/+11
| | | | | | | | | Reduced _idle_semaphore.release calls in concurrent.futures.thread._worker _idle_semaphore.release() is now only called if only work_queue is empty. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
* gh-94440: Fix issue of ProcessPoolExecutor shutdown hanging (#94468)yonatanp2023-03-161-0/+5
| | | | | Fix an issue of concurrent.futures ProcessPoolExecutor shutdown hanging. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* gh-102515: Remove unused imports in the `Lib/` directory (#102516)Alex Waygood2023-03-081-0/+2
|
* gh-84559: Remove the new multiprocessing warning, too disruptive. (#101551)Gregory P. Smith2023-02-031-17/+0
| | | | This reverts the core of #100618 while leaving relevant documentation improvements and minor refactorings in place.
* GH-84559: Deprecate fork being the multiprocessing default. (#100618)Gregory P. Smith2023-02-021-3/+20
| | | | | | | | This starts the process. Users who don't specify their own start method and use the default on platforms where it is 'fork' will see a DeprecationWarning upon multiprocessing.Pool() construction or upon multiprocessing.Process.start() or concurrent.futures.ProcessPool use. See the related issue and documentation within this change for details.
* gh-95166: cancel map waited on future on timeout (GH-95169)Thomas Grainger2022-07-281-2/+14
| | | Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
* gh-90622: Do not spawn ProcessPool workers on demand via fork method. (#91598)Gregory P. Smith2022-05-081-10/+34
| | | | | | Do not spawn ProcessPool workers on demand when they spawn via fork. This avoids potential deadlocks in the child processes due to forking from a multithreaded process.
* gh-90622: Prevent max_tasks_per_child use with a fork mp_context. (#91587)Gregory P. Smith2022-05-061-7/+17
| | | | | Prevent `max_tasks_per_child` use with a "fork" mp_context to avoid deadlocks. Also defaults to "spawn" when no mp_context is supplied for safe convenience.
* bpo-46787: Fix `ProcessPoolExecutor exception` memory leak (GH-31408) (#31408)themylogin2022-05-021-0/+3
| | | | Do not store `ProcessPoolExecutor` work item exception traceback that prevents exception frame locals from being garbage collected.
* concurrent.futures: Fix typo in docstring (#92121)Yiannis Hadjicharalambous2022-05-021-1/+1
|
* Remove trailing spaces (GH-31695)Serhiy Storchaka2022-03-051-1/+1
|
* bpo-46522: fix concurrent.futures and io AttributeError messages (GH-30887)Thomas Grainger2022-02-231-1/+1
| | | | Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
* bpo-20369: concurrent.futures.wait() now deduplicates futures given a… ↵Kumar Aditya2022-01-041-6/+7
| | | | | | | | | (GH-30168) * bpo-20369: concurrent.futures.wait() now deduplicates futures given as arg. * 📜🤖 Added by blurb_it. Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* bpo-42413: Replace `concurrent.futures.TimeoutError` and ↵Kumar Aditya2021-12-191-3/+1
| | | | | `asyncio.TimeoutError` with builtin `TimeoutError` (GH-30197) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
* bpo-43498: Fix dictionary iteration error in _ExecutorManagerThread (GH-24868)Jakub Kulík2021-11-291-1/+1
|
* bpo-44733: Add max_tasks_per_child to ProcessPoolExecutor (GH-27373)Logan Jones2021-11-201-14/+55
| | | Co-authored-by: Antoine Pitrou <antoine@python.org>
* bpo-45021: Fix a hang in forked children (GH-28007)nullptr2021-09-201-0/+6
| | | _global_shutdown_lock should be reinitialized in forked children
* Use `from` imports (GH-26594)Machinexa22021-06-081-5/+4
| | | from imports
* bpo-35930: Raising an exception raised in a "future" instance will create ↵Jesús Cea2021-03-291-15/+23
| | | | | | | reference cycles (#24995) Before: https://lists.es.python.org/pipermail/general/attachments/20201229/0c14bc58/attachment-0002.png After: https://lists.es.python.org/pipermail/general/attachments/20201229/0c14bc58/attachment-0003.png
* bpo-40692: Run more test_concurrent_futures tests (GH-20239)Asheesh Laroia2021-02-081-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | In the case of multiprocessing.synchronize() being missing, the test_concurrent_futures test suite now skips only the tests that require multiprocessing.synchronize(). Validate that multiprocessing.synchronize exists as part of _check_system_limits(), allowing ProcessPoolExecutor to raise NotImplementedError during __init__, rather than crashing with ImportError during __init__ when creating a lock imported from multiprocessing.synchronize. Use _check_system_limits() to disable tests of ProcessPoolExecutor on systems without multiprocessing.synchronize. Running the test suite without multiprocessing.synchronize reveals that Lib/compileall.py crashes when it uses a ProcessPoolExecutor. Therefore, change Lib/compileall.py to call _check_system_limits() before creating the ProcessPoolExecutor. Note that both Lib/compileall.py and Lib/test/test_compileall.py were attempting to sanity-check ProcessPoolExecutor by expecting ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent on platforms where POSIX semaphores aren't available. Avoid using sem_unlink() if it, too, does not exist. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-39349: Add cancel_futures to Executor.shutdown base class (GH-22023)Shantanu2020-09-011-1/+4
| | | | * Add cancel_futures parameter to the Executor base class, since it was missed in the original PR (https://github.com/python/cpython/pull/18057) that added cancel_futures.
* bpo-40443: Remove unused imports in the stdlib (GH-19803)Victor Stinner2020-04-302-2/+0
|
* bpo-39995: Fix concurrent.futures _ThreadWakeup (GH-19760)Victor Stinner2020-04-291-14/+27
| | | | Fix a race condition in concurrent.futures._ThreadWakeup: access to _ThreadWakeup is now protected with the shutdown lock.
* bpo-30966: concurrent.futures.Process.shutdown() closes queue (GH-19738)Victor Stinner2020-04-271-0/+2
| | | | Process.shutdown(wait=True) of concurrent.futures now closes explicitly the result queue.
* bpo-39207: Spawn workers on demand in ProcessPoolExecutor (GH-19453)Kyle Stanley2020-04-191-2/+14
| | | | | Roughly based on https://github.com/python/cpython/commit/904e34d4e6b6007986dcc585d5c553ee8ae06f95, but with a few substantial differences. /cc @pitrou @brianquinlan
* bpo-39481: PEP 585 for a variety of modules (GH-19423)Batuhan Taşkaya2020-04-102-0/+6
| | | | | | | | | | - concurrent.futures - ctypes - http.cookies - multiprocessing - queue - tempfile - unittest.case - urllib.parse
* bpo-39812: Remove daemon threads in concurrent.futures (GH-19149)Kyle Stanley2020-03-272-32/+11
| | | | | | Remove daemon threads from :mod:`concurrent.futures` by adding an internal `threading._register_atexit()`, which calls registered functions prior to joining all non-daemon threads. This allows for compatibility with subinterpreters, which don't support daemon threads.
* bpo-39678: refactor queue manager thread (GH-18551)Thomas Moreau2020-03-011-207/+229
|
* bpo-39104: Fix hanging ProcessPoolExecutor on shutdown nowait with pickling ↵Thomas Moreau2020-02-161-21/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | failure (GH-17670) As reported initially by @rad-pat in #6084, the following script causes a deadlock. ``` from concurrent.futures import ProcessPoolExecutor class ObjectWithPickleError(): """Triggers a RuntimeError when sending job to the workers""" def __reduce__(self): raise RuntimeError() if __name__ == "__main__": e = ProcessPoolExecutor() f = e.submit(id, ObjectWithPickleError()) e.shutdown(wait=False) f.result() # Deadlock on get ``` This is caused by the fact that the main process is closing communication channels that might be necessary to the `queue_management_thread` later. To avoid this, this PR let the `queue_management_thread` manage all the closing. https://bugs.python.org/issue39104 Automerge-Triggered-By: @pitrou
* bpo-39349: Add *cancel_futures* to Executor.shutdown() (GH-18057)Kyle Stanley2020-02-022-2/+36
|
* bpo-31783: Fix a race condition creating workers during shutdown (#13171)Brian Quinlan2019-06-281-2/+6
| | | | | | * bpo-31783: Fix a race condition while creating workers during interpreter shutdown * 📜🤖 Added by blurb_it.
* Fix typo in Lib/concurrent/futures/thread.py (GH-13953)ubordignon2019-06-152-2/+2
|
* [3.9] bpo-37116: Use PEP 570 syntax for positional-only parameters. (GH-12620)Serhiy Storchaka2019-06-053-49/+3
| | | Turn deprecation warnings added in 3.8 into TypeError.
* bpo-35279: reduce default max_workers of ThreadPoolExecutor (GH-13618)Inada Naoki2019-05-281-3/+8
|
* bpo-24882: Let ThreadPoolExecutor reuse idle threads before creating new ↵Sean2019-05-221-2/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | thread (#6375) * Fixes issue 24882 * Add news file entry for change. * Change test_concurrent_futures.ThreadPoolShutdownTest Adjust the shutdown test so that, after submitting three jobs to the executor, the test checks for less than three threads, instead of looking for exactly three threads. If idle threads are being recycled properly, then we should have less than three threads. * Switched idle count to semaphor, Updated tests As suggested by reviewer tomMoral, swapped lock-protected counter with a semaphore to track the number of unused threads. Adjusted test_threads_terminate to wait for completiton of the previous future before submitting a new one (and checking the number of threads used). Also added a new test to confirm the thread pool can be saturated. * Updates tests as requested by pitrou. * Correct minor whitespace error. * Make test_saturation faster
* bpo-33110: Catch errors raised when running add_done_callback on already ↵Sam Martin2019-05-221-1/+4
| | | | | | completed futures (GH-13141) Wrap the callback call within the `add_done_callback` function within concurrent.futures, in order to behave in an identical manner to callbacks added to a running future are triggered once it has completed.
* bpo-26903: Limit ProcessPoolExecutor to 61 workers on Windows (GH-13132)Brian Quinlan2019-05-081-0/+14
| | | | Co-Authored-By: brianquinlan <brian@sweetapp.com>
* Don't import wait from connection, it shadows a name (GH-13112)Brian Quinlan2019-05-071-2/+2
| | | (lint cleanup) This import causes an argument parameter to shadow the global import name.
* bpo-36542: Allow to overwrite the signature for Python functions. (GH-12705)Serhiy Storchaka2019-05-063-0/+3
|
* Correct grammar in concurrent/futures/process.py docstring (GH-12761)Thomas Grainger2019-04-121-1/+1
|
* bpo-36492: Deprecate passing some arguments as keyword arguments. (GH-12637)Serhiy Storchaka2019-04-013-3/+46
| | | | | | | | | | | | | | | | | | | | | | Deprecated passing the following arguments as keyword arguments: - "func" in functools.partialmethod(), weakref.finalize(), profile.Profile.runcall(), cProfile.Profile.runcall(), bdb.Bdb.runcall(), trace.Trace.runfunc() and curses.wrapper(). - "function" in unittest.addModuleCleanup() and unittest.TestCase.addCleanup(). - "fn" in the submit() method of concurrent.futures.ThreadPoolExecutor and concurrent.futures.ProcessPoolExecutor. - "callback" in contextlib.ExitStack.callback(), contextlib.AsyncExitStack.callback() and contextlib.AsyncExitStack.push_async_callback(). - "c" and "typeid" in the create() method of multiprocessing.managers.Server and multiprocessing.managers.SharedMemoryServer. - "obj" in weakref.finalize(). Also allowed to pass arbitrary keyword arguments (even "self" and "func") if the above arguments are passed as positional argument.
generated by cgit v0.12 at 2024-11-21 17:03:00 (GMT)