summaryrefslogtreecommitdiffstats
path: root/Lib/multiprocessing
Commit message (Collapse)AuthorAgeFilesLines
* gh-101225: Increase the socket backlog when creating a ↵Ronald Oussoren2024-01-132-2/+2
| | | | | | | | multiprocessing.connection.Listener (#113567) Increase the backlog for multiprocessing.connection.Listener` objects created by `multiprocessing.manager` and `multiprocessing.resource_sharer` to significantly reduce the risk of getting a connection refused error when creating a `multiprocessing.connection.Connection` to them.
* gh-113421: Fix multiprocessing logger for "%(filename)s" (GH-113423)Xu Song2023-12-241-4/+4
|
* gh-113009: Fix multiprocessing Process.terminate() on Windows (#113128)Victor Stinner2023-12-151-24/+30
| | | | | | On Windows, Process.terminate() no longer sets the returncode attribute to always call WaitForSingleObject() in Process.wait(). Previously, sometimes the process was still running after TerminateProcess() even if GetExitCodeProcess() is not STILL_ACTIVE.
* gh-82300: Add track parameter to multiprocessing.shared_memory (#110778)pan3242023-12-051-7/+17
| | | | | | | | | | Add a track parameter to shared memory to allow resource tracking via the side-launched resource tracker process to be disabled on platforms that use it (POSIX). This allows people who do not want automated cleanup at process exit because they are using the shared memory with processes not participating in Python's resource tracking to use the shared_memory API. Co-authored-by: Łukasz Langa <lukasz@langa.pl> Co-authored-by: Guido van Rossum <gvanrossum@gmail.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-107431: Make `multiprocessing.managers.{DictProxy,ListProxy}` generic ↵Nikita Sobolev2023-11-101-2/+6
| | | | | | | | (#107433) Make `multiprocessing.managers.{DictProxy,ListProxy}` generic for type annotation use. `ListProxy[str]` for example. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-109649: Use os.process_cpu_count() (#110165)Victor Stinner2023-10-011-1/+1
| | | | | | | | | | | | | | | | | | | | Replace os.cpu_count() with os.process_cpu_count() in modules: * compileall * concurrent.futures * multiprocessing Replace os.cpu_count() with os.process_cpu_count() in programs: * _decimal deccheck.py test * freeze.py * multissltests.py * python -m test (regrtest) * wasm_build.py Other changes: * test.pythoninfo logs os.process_cpu_count(). * regrtest gets os.process_cpu_count() / os.cpu_count() in headers.
* gh-109047: concurrent.futures catches PythonFinalizationError (#109810)Victor Stinner2023-09-291-5/+25
| | | | | | | | | | | | | | | | | | | | | | | | | concurrent.futures: The *executor manager thread* now catches exceptions when adding an item to the *call queue*. During Python finalization, creating a new thread can now raise RuntimeError. Catch the exception and call terminate_broken() in this case. Add test_python_finalization_error() to test_concurrent_futures. concurrent.futures._ExecutorManagerThread changes: * terminate_broken() no longer calls shutdown_workers() since the call queue is no longer working anymore (read and write ends of the queue pipe are closed). * terminate_broken() now terminates child processes, not only wait until they complete. * _ExecutorManagerThread.terminate_broken() now holds shutdown_lock to prevent race conditons with ProcessPoolExecutor.submit(). multiprocessing.Queue changes: * Add _terminate_broken() method. * _start_thread() sets _thread to None on exception to prevent leaking "dangling threads" even if the thread was not started yet.
* gh-110036: multiprocessing Popen.terminate() catches PermissionError (#110037)Victor Stinner2023-09-291-2/+9
| | | | | | On Windows, multiprocessing Popen.terminate() now catchs PermissionError and get the process exit code. If the process is still running, raise again the PermissionError. Otherwise, the process terminated as expected: store its exit code.
* gh-109461: Update logging module lock to use context manager (#109462)Dale Collison2023-09-271-5/+1
| | | Co-authored-by: Victor Stinner <vstinner@python.org>
* gh-109593: Fix reentrancy issue in multiprocessing resource_tracker (#109629)Antoine Pitrou2023-09-261-2/+32
| | | | | --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* gh-109370: Fix unexpected traceback output in test_concurrent_futures ↵Serhiy Storchaka2023-09-261-2/+1
| | | | | | | | | (GH-109780) Follow-up of gh-107219. * Only close the connection writer on Windows. * Also use existing constant _winapi.ERROR_OPERATION_ABORTED instead of WSA_OPERATION_ABORTED.
* gh-107219: Fix concurrent.futures terminate_broken() (#109244)Victor Stinner2023-09-111-0/+18
| | | | | | | | | | | | | | | Fix a race condition in concurrent.futures. When a process in the process pool was terminated abruptly (while the future was running or pending), close the connection write end. If the call queue is blocked on sending bytes to a worker process, closing the connection write end interrupts the send, so the queue can be closed. Changes: * _ExecutorManagerThread.terminate_broken() now closes call_queue._writer. * multiprocessing PipeConnection.close() now interrupts WaitForMultipleObjects() in _send_bytes() by cancelling the overlapped operation.
* gh-108520: Fix bad fork detection in nested multiprocessing use case (#108568)albanD2023-08-301-3/+5
| | | | | | | | gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process ↵albanD2023-08-231-2/+7
| | | | | | | | | | | | before serializing it (#107275) Ensure multiprocessing SemLock is valid for spawn Process before serializing it. Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* More actionable error message when spawn is incorrectly used. (#102203)Yuxin Wu2023-08-161-1/+5
| | | | Co-authored-by: Yuxin Wu <ppwwyyxx@users.noreply.github.com> Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
* gh-107963: Fix set_forkserver_preload to check the type of given list (#107965)Dong-hee Na2023-08-151-1/+1
| | | | gh-107963: Fix set_forkserver_preload to check the type of given list
* gh-106558: break ref cycles through exceptions in multiprocessing manager ↵Andrew Geng2023-08-111-2/+8
| | | | (#106559)
* gh-106739: Add `rtype_cache` to `warnings.warn` message when leaked objects ↵shailshouryya2023-07-251-3/+4
| | | | | | | | | | | | | | | | | | | | found (#106740) Adding the `rtype_cache` to the `warnings.warn` message improves the previous, somewhat vague message from ``` /Users/username/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown ``` to ``` /Users/username/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown: {'/mp-yor5cvj8', '/mp-10jx8eqr', '/mp-eobsx9tt', '/mp-0lml23vl', '/mp-9dgtsa_m', '/mp-frntyv4s'} ``` --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* gh-90876: Restore the ability to import multiprocessing when ↵Gregory P. Smith2023-07-061-2/+4
| | | | | | | | | `sys.executable` is `None` (#106464) Prevent `multiprocessing.spawn` from failing to *import* in environments where `sys.executable` is `None`. This regressed in 3.11 with the addition of support for path-like objects in multiprocessing. Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need.
* gh-104536: Improve `multiprocessing.process._cleanup` logic (#104537)Luccccifer2023-05-221-1/+1
| | | | | | | | | | Fix a race condition in the internal `multiprocessing.process` cleanup logic that could manifest as an unintended `AttributeError` when calling `BaseProcess.close()`. --------- Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-61460: Stronger HMAC in multiprocessing (#20380)Christian Heimes2023-05-201-31/+151
| | | | | | | | | | | | | | | | | | | | | | bpo-17258: `multiprocessing` now supports stronger HMAC algorithms for inter-process connection authentication rather than only HMAC-MD5. Signed-off-by: Christian Heimes <christian@python.org> gpshead: I Reworked to be more robust while keeping the idea. The protocol modification idea remains, but we now take advantage of the message length as an indicator of legacy vs modern protocol version. No more regular expression usage. We now default to HMAC-SHA256, but do so in a way that will be compatible when communicating with older clients or older servers. No protocol transition period is needed. More integration tests to verify these claims remain true are required. I'm unaware of anyone depending on multiprocessing connections between different Python versions. --------- Signed-off-by: Christian Heimes <christian@python.org> Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
* gh-84559: Remove the new multiprocessing warning, too disruptive. (#101551)Gregory P. Smith2023-02-031-27/+1
| | | | This reverts the core of #100618 while leaving relevant documentation improvements and minor refactorings in place.
* GH-84559: Deprecate fork being the multiprocessing default. (#100618)Gregory P. Smith2023-02-021-1/+28
| | | | | | | | This starts the process. Users who don't specify their own start method and use the default on platforms where it is 'fork' will see a DeprecationWarning upon multiprocessing.Pool() construction or upon multiprocessing.Process.start() or concurrent.futures.ProcessPool use. See the related issue and documentation within this change for details.
* gh-99509: Add `__class_getitem__` to `multiprocessing.queues.Queue` (#99511)Nikita Sobolev2022-12-271-0/+2
|
* Fix typo in exception message in `multiprocessing.pool` (#99900)Arne de Laat2022-11-301-1/+1
|
* bpo-40882: Fix a memory leak in SharedMemory on Windows (GH-20684)Zackery Spytz2022-11-251-1/+4
| | | | | In multiprocessing.shared_memory.SharedMemory(), the temporary view returned by MapViewOfFile() should be unmapped when it is no longer needed.
* gh-61460: Add a comment describing the multiprocessing.connection protocol ↵Gregory P. Smith2022-11-201-0/+68
| | | | | | | | (gh-99623) Describe the multiprocessing connection protocol. It isn't a good protocol, but it is what it is. This way we can more easily reason about making changes to it in a backwards compatible way.
* gh-97514: Don't use Linux abstract sockets for multiprocessing (#98501)Gregory P. Smith2022-10-201-5/+0
| | | | | | | | | | | | | | | Linux abstract sockets are insecure as they lack any form of filesystem permissions so their use allows anyone on the system to inject code into the process. This removes the default preference for abstract sockets in multiprocessing introduced in Python 3.9+ via https://github.com/python/cpython/pull/18866 while fixing https://github.com/python/cpython/issues/84031. Explicit use of an abstract socket by a user now generates a RuntimeWarning. If we choose to keep this warning, it should be backported to the 3.7 and 3.8 branches.
* gh-98360: multiprocessing now spawns children on Windows with correct ↵Steve Dower2022-10-201-2/+3
| | | | argv[0] in virtual environments (GH-98462)
* gh-97816: Remove unused variables in `mutliprocessing.managers.Server` (#97817)Koki Saito2022-10-041-1/+0
| | | Remove unused local variables.
* gh-96819: multiprocessing.resource_tracker: check if length of pipe write <= ↵Koki Saito2022-10-031-2/+2
| | | | | 512 (#96890) Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
* GH-83658: make multiprocessing.Pool raise an exception if maxtasksperchild ↵Irit Katriel2022-06-171-0/+3
| | | | | is not None or a positive int (GH-93364) Closes #83658.
* gh-91577: SharedMemory move imports out of methods (#91579)samtygier2022-06-161-4/+3
| | | | | | | SharedMemory.unlink() uses the unregister() function from resource_tracker. Previously it was imported in the method, but this can fail if the method is called during interpreter shutdown, for example when unlink is part of a __del__() method. Moving the import to the top of the file, means that the unregister() method is available during interpreter shutdown. The register call in SharedMemory.__init__() can also use this imported resource_tracker.
* gh-90549: Fix leak of global named resources using multiprocessing spawn ↵Leo Trol2022-06-092-2/+22
| | | | | | (#30617) Co-authored-by: XD Trol <milestonejxd@gmail.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* gh-82616: Add process_group support to subprocess.Popen (#23930)Gregory P. Smith2022-05-051-1/+1
| | | | | | | | One more thing that can help prevent people from using `preexec_fn`. Also adds conditional skips to two tests exposing ASAN flakiness on the Ubuntu 20.04 Address Sanitizer Github CI system. When that build is run on more modern systems the "problem" does not show up. It seems ASAN implementation related. Co-authored-by: Zackery Spytz <zspytz@gmail.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* bpo-47029: Fix BrokenPipeError in multiprocessing.Queue at garbage ↵Géry Ogam2022-05-031-12/+11
| | | | collection and explicit close (#31913)
* GH-91355: micro-optimize Connection.send_bytes() method (gh-32247)Ma Lin2022-05-031-3/+2
|
* Fix missing `f` prefix on f-strings (GH-91910)Alexander Shadchin2022-04-271-1/+1
|
* gh-91401: Add a failsafe way to disable vfork. (#91490)Gregory P. Smith2022-04-251-1/+3
| | | | | | | | | | | | | | Just in case there is ever an issue with _posixsubprocess's use of vfork() due to the complexity of using it properly and potential directions that Linux platforms where it defaults to on could take, this adds a failsafe so that users can disable its use entirely by setting a global flag. No known reason to disable it exists. But it'd be a shame to encounter one and not be able to use CPython without patching and rebuilding it. See the linked issue for some discussion on reasoning. Also documents the existing way to disable posix_spawn.
* bpo-46720: Add support for path-like objects to ↵Géry Ogam2022-04-222-8/+12
| | | | | multiprocessing.set_executable for Windows (GH-31279) This bring the API to be on a par with Unix-like systems.
* gh-91231: Add shutdown_timeout to multiprocessing BaseManager (#32112)Victor Stinner2022-04-191-6/+10
| | | | | | | | Add an optional keyword 'shutdown_timeout' parameter to the multiprocessing.BaseManager constructor. Kill the process if terminate() takes longer than the timeout. Multiprocessing tests pass test.support.SHORT_TIMEOUT to BaseManager.shutdown_timeout.
* bpo-46565: `del` loop vars that are leaking into module namespaces (GH-30993)Nikita Sobolev2022-02-032-5/+6
|
* bpo-25066: Added repr for multiprocessing.Event (GH-29749)Kumar Aditya2021-12-091-0/+3
| | | Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* bpo-38840: Incorrect __all__ in multiprocessing.managers (GH-18034)Zackery Spytz2021-08-091-3/+4
| | | | | | | This was causing test___all__ to fail on platforms lacking a shared memory implementation. Co-Authored-By: Xavier de Gaye <xdegaye@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* Remove unnecessary pass statements (GH-27103)Elisha Hollander2021-07-131-1/+0
|
* bpo-30256: Add manager_owned keyword arg to AutoProxy (GH-16341)finefoot2021-07-021-2/+2
| | | | Co-authored-by: Jordan Speicher <jordan@jspeicher.com>
* Fix typos in multiple files (GH-26689)Binbin2021-06-131-1/+1
| | | Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
* bpo-43867: multiprocessing Server catchs SystemExit (GH-25441)Victor Stinner2021-04-161-6/+14
| | | | | The multiprocessing Server class now explicitly catchs SystemExit and closes the client connection in this case. It happens when the Server.serve_client() method reachs the end of file (EOF).
* bpo-43651: Fix test_compileall with PEP 597 (GH-25128)Inada Naoki2021-04-021-1/+1
|
* bpo-40692: Run more test_concurrent_futures tests (GH-20239)Asheesh Laroia2021-02-081-1/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | In the case of multiprocessing.synchronize() being missing, the test_concurrent_futures test suite now skips only the tests that require multiprocessing.synchronize(). Validate that multiprocessing.synchronize exists as part of _check_system_limits(), allowing ProcessPoolExecutor to raise NotImplementedError during __init__, rather than crashing with ImportError during __init__ when creating a lock imported from multiprocessing.synchronize. Use _check_system_limits() to disable tests of ProcessPoolExecutor on systems without multiprocessing.synchronize. Running the test suite without multiprocessing.synchronize reveals that Lib/compileall.py crashes when it uses a ProcessPoolExecutor. Therefore, change Lib/compileall.py to call _check_system_limits() before creating the ProcessPoolExecutor. Note that both Lib/compileall.py and Lib/test/test_compileall.py were attempting to sanity-check ProcessPoolExecutor by expecting ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent on platforms where POSIX semaphores aren't available. Avoid using sem_unlink() if it, too, does not exist. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
generated by cgit v0.12 at 2025-09-26 08:39:36 (GMT)