summaryrefslogtreecommitdiffstats
path: root/Lib/multiprocessing
Commit message (Collapse)AuthorAgeFilesLines
* [3.12] gh-109370: Fix unexpected traceback output in test_concurrent_futures ↵Serhiy Storchaka2023-11-101-2/+1
| | | | | | | | | | (GH-109780) (GH-111934) Follow-up of gh-107219. * Only close the connection writer on Windows. * Also use existing constant _winapi.ERROR_OPERATION_ABORTED instead of WSA_OPERATION_ABORTED. (cherry picked from commit 0b4e090422db5f959184353d53552d1675f74212)
* [3.12] gh-109047: concurrent.futures catches RuntimeError (#109810) (#110126)Victor Stinner2023-10-021-5/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | gh-109047: concurrent.futures catches PythonFinalizationError (#109810) concurrent.futures: The *executor manager thread* now catches exceptions when adding an item to the *call queue*. During Python finalization, creating a new thread can now raise RuntimeError. Catch the exception and call terminate_broken() in this case. Add test_python_finalization_error() to test_concurrent_futures. concurrent.futures._ExecutorManagerThread changes: * terminate_broken() no longer calls shutdown_workers() since the call queue is no longer working anymore (read and write ends of the queue pipe are closed). * terminate_broken() now terminates child processes, not only wait until they complete. * _ExecutorManagerThread.terminate_broken() now holds shutdown_lock to prevent race conditons with ProcessPoolExecutor.submit(). multiprocessing.Queue changes: * Add _terminate_broken() method. * _start_thread() sets _thread to None on exception to prevent leaking "dangling threads" even if the thread was not started yet. (cherry picked from commit 635184212179b0511768ea1cd57256e134ba2d75)
* [3.12] gh-110036: multiprocessing Popen.terminate() catches PermissionError ↵Miss Islington (bot)2023-10-021-2/+9
| | | | | | | | | | | | | (GH-110037) (#110064) gh-110036: multiprocessing Popen.terminate() catches PermissionError (GH-110037) On Windows, multiprocessing Popen.terminate() now catchs PermissionError and get the process exit code. If the process is still running, raise again the PermissionError. Otherwise, the process terminated as expected: store its exit code. (cherry picked from commit bd4518c60c9df356cf5e05b81305e3644ebb5e70) Co-authored-by: Victor Stinner <vstinner@python.org>
* [3.12] gh-109593: Fix reentrancy issue in multiprocessing resource_tracker ↵Miss Islington (bot)2023-10-021-2/+32
| | | | | | | | | | | | (GH-109629) (#109898) gh-109593: Fix reentrancy issue in multiprocessing resource_tracker (GH-109629) --------- (cherry picked from commit 0eb98837b60bc58e57ad3e2b35c6b0e9ab634678) Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* [3.12] gh-107219: Fix concurrent.futures terminate_broken() (GH-109244) ↵Miss Islington (bot)2023-10-021-0/+18
| | | | | | | | | | | | | | | | | | | | | | (#109254) gh-107219: Fix concurrent.futures terminate_broken() (GH-109244) Fix a race condition in concurrent.futures. When a process in the process pool was terminated abruptly (while the future was running or pending), close the connection write end. If the call queue is blocked on sending bytes to a worker process, closing the connection write end interrupts the send, so the queue can be closed. Changes: * _ExecutorManagerThread.terminate_broken() now closes call_queue._writer. * multiprocessing PipeConnection.close() now interrupts WaitForMultipleObjects() in _send_bytes() by cancelling the overlapped operation. (cherry picked from commit a9b1f84790e977fb09f75b148c4c4f5924a6ef99) Co-authored-by: Victor Stinner <vstinner@python.org>
* [3.12] gh-108520: Fix bad fork detection in nested multiprocessing use case ↵Miss Islington (bot)2023-08-301-3/+5
| | | | | | | | | | | | | | | (GH-108568) (#108691) gh-108520: Fix bad fork detection in nested multiprocessing use case (GH-108568) gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- (cherry picked from commit add8d45cbe46581b9748909fbbf60fdc8ee8f71e) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* [3.12] gh-77377: Ensure multiprocessing SemLock is valid for spawn-based ↵Miss Islington (bot)2023-08-231-2/+7
| | | | | | | | | | | | | | | | | Process before serializing it (GH-107275) (#108377) gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it (GH-107275) Ensure multiprocessing SemLock is valid for spawn Process before serializing it. Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early. --------- (cherry picked from commit 1700d34d314f5304a7a75363bda295a8c15c371f) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* [3.12] More actionable error message when spawn is incorrectly used. ↵Miss Islington (bot)2023-08-161-1/+5
| | | | | | | | | | (GH-102203) (#107990) More actionable error message when spawn is incorrectly used. (GH-102203) (cherry picked from commit a794ebeb028f7ef287c780d3890f816db9c21c51) Co-authored-by: Yuxin Wu <ppwwyyxxc@gmail.com> Co-authored-by: Yuxin Wu <ppwwyyxx@users.noreply.github.com> Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
* [3.12] gh-107963: Fix set_forkserver_preload to check the type of given list ↵Miss Islington (bot)2023-08-151-1/+1
| | | | | | | | | | | (GH-107965) (#107975) gh-107963: Fix set_forkserver_preload to check the type of given list (GH-107965) (cherry picked from commit 6515ec3d3d5acd3d0b99c88794bdec09f0831e5b) gh-107963: Fix set_forkserver_preload to check the type of given list Co-authored-by: Dong-hee Na <donghee.na@python.org>
* [3.12] gh-90876: Restore the ability to import multiprocessing when ↵Miss Islington (bot)2023-07-061-2/+4
| | | | | | | | | | | | | | `sys.executable` is `None` (GH-106464) (#106494) gh-90876: Restore the ability to import multiprocessing when `sys.executable` is `None` (GH-106464) Prevent `multiprocessing.spawn` from failing to *import* in environments where `sys.executable` is `None`. This regressed in 3.11 with the addition of support for path-like objects in multiprocessing. Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need. (cherry picked from commit c60df361ce2d734148d503f4a711e67c110fe223) Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-104536: Improve `multiprocessing.process._cleanup` logic (#104537)Luccccifer2023-05-221-1/+1
| | | | | | | | | | Fix a race condition in the internal `multiprocessing.process` cleanup logic that could manifest as an unintended `AttributeError` when calling `BaseProcess.close()`. --------- Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-61460: Stronger HMAC in multiprocessing (#20380)Christian Heimes2023-05-201-31/+151
| | | | | | | | | | | | | | | | | | | | | | bpo-17258: `multiprocessing` now supports stronger HMAC algorithms for inter-process connection authentication rather than only HMAC-MD5. Signed-off-by: Christian Heimes <christian@python.org> gpshead: I Reworked to be more robust while keeping the idea. The protocol modification idea remains, but we now take advantage of the message length as an indicator of legacy vs modern protocol version. No more regular expression usage. We now default to HMAC-SHA256, but do so in a way that will be compatible when communicating with older clients or older servers. No protocol transition period is needed. More integration tests to verify these claims remain true are required. I'm unaware of anyone depending on multiprocessing connections between different Python versions. --------- Signed-off-by: Christian Heimes <christian@python.org> Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
* gh-84559: Remove the new multiprocessing warning, too disruptive. (#101551)Gregory P. Smith2023-02-031-27/+1
| | | | This reverts the core of #100618 while leaving relevant documentation improvements and minor refactorings in place.
* GH-84559: Deprecate fork being the multiprocessing default. (#100618)Gregory P. Smith2023-02-021-1/+28
| | | | | | | | This starts the process. Users who don't specify their own start method and use the default on platforms where it is 'fork' will see a DeprecationWarning upon multiprocessing.Pool() construction or upon multiprocessing.Process.start() or concurrent.futures.ProcessPool use. See the related issue and documentation within this change for details.
* gh-99509: Add `__class_getitem__` to `multiprocessing.queues.Queue` (#99511)Nikita Sobolev2022-12-271-0/+2
|
* Fix typo in exception message in `multiprocessing.pool` (#99900)Arne de Laat2022-11-301-1/+1
|
* bpo-40882: Fix a memory leak in SharedMemory on Windows (GH-20684)Zackery Spytz2022-11-251-1/+4
| | | | | In multiprocessing.shared_memory.SharedMemory(), the temporary view returned by MapViewOfFile() should be unmapped when it is no longer needed.
* gh-61460: Add a comment describing the multiprocessing.connection protocol ↵Gregory P. Smith2022-11-201-0/+68
| | | | | | | | (gh-99623) Describe the multiprocessing connection protocol. It isn't a good protocol, but it is what it is. This way we can more easily reason about making changes to it in a backwards compatible way.
* gh-97514: Don't use Linux abstract sockets for multiprocessing (#98501)Gregory P. Smith2022-10-201-5/+0
| | | | | | | | | | | | | | | Linux abstract sockets are insecure as they lack any form of filesystem permissions so their use allows anyone on the system to inject code into the process. This removes the default preference for abstract sockets in multiprocessing introduced in Python 3.9+ via https://github.com/python/cpython/pull/18866 while fixing https://github.com/python/cpython/issues/84031. Explicit use of an abstract socket by a user now generates a RuntimeWarning. If we choose to keep this warning, it should be backported to the 3.7 and 3.8 branches.
* gh-98360: multiprocessing now spawns children on Windows with correct ↵Steve Dower2022-10-201-2/+3
| | | | argv[0] in virtual environments (GH-98462)
* gh-97816: Remove unused variables in `mutliprocessing.managers.Server` (#97817)Koki Saito2022-10-041-1/+0
| | | Remove unused local variables.
* gh-96819: multiprocessing.resource_tracker: check if length of pipe write <= ↵Koki Saito2022-10-031-2/+2
| | | | | 512 (#96890) Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
* GH-83658: make multiprocessing.Pool raise an exception if maxtasksperchild ↵Irit Katriel2022-06-171-0/+3
| | | | | is not None or a positive int (GH-93364) Closes #83658.
* gh-91577: SharedMemory move imports out of methods (#91579)samtygier2022-06-161-4/+3
| | | | | | | SharedMemory.unlink() uses the unregister() function from resource_tracker. Previously it was imported in the method, but this can fail if the method is called during interpreter shutdown, for example when unlink is part of a __del__() method. Moving the import to the top of the file, means that the unregister() method is available during interpreter shutdown. The register call in SharedMemory.__init__() can also use this imported resource_tracker.
* gh-90549: Fix leak of global named resources using multiprocessing spawn ↵Leo Trol2022-06-092-2/+22
| | | | | | (#30617) Co-authored-by: XD Trol <milestonejxd@gmail.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* gh-82616: Add process_group support to subprocess.Popen (#23930)Gregory P. Smith2022-05-051-1/+1
| | | | | | | | One more thing that can help prevent people from using `preexec_fn`. Also adds conditional skips to two tests exposing ASAN flakiness on the Ubuntu 20.04 Address Sanitizer Github CI system. When that build is run on more modern systems the "problem" does not show up. It seems ASAN implementation related. Co-authored-by: Zackery Spytz <zspytz@gmail.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* bpo-47029: Fix BrokenPipeError in multiprocessing.Queue at garbage ↵Géry Ogam2022-05-031-12/+11
| | | | collection and explicit close (#31913)
* GH-91355: micro-optimize Connection.send_bytes() method (gh-32247)Ma Lin2022-05-031-3/+2
|
* Fix missing `f` prefix on f-strings (GH-91910)Alexander Shadchin2022-04-271-1/+1
|
* gh-91401: Add a failsafe way to disable vfork. (#91490)Gregory P. Smith2022-04-251-1/+3
| | | | | | | | | | | | | | Just in case there is ever an issue with _posixsubprocess's use of vfork() due to the complexity of using it properly and potential directions that Linux platforms where it defaults to on could take, this adds a failsafe so that users can disable its use entirely by setting a global flag. No known reason to disable it exists. But it'd be a shame to encounter one and not be able to use CPython without patching and rebuilding it. See the linked issue for some discussion on reasoning. Also documents the existing way to disable posix_spawn.
* bpo-46720: Add support for path-like objects to ↵Géry Ogam2022-04-222-8/+12
| | | | | multiprocessing.set_executable for Windows (GH-31279) This bring the API to be on a par with Unix-like systems.
* gh-91231: Add shutdown_timeout to multiprocessing BaseManager (#32112)Victor Stinner2022-04-191-6/+10
| | | | | | | | Add an optional keyword 'shutdown_timeout' parameter to the multiprocessing.BaseManager constructor. Kill the process if terminate() takes longer than the timeout. Multiprocessing tests pass test.support.SHORT_TIMEOUT to BaseManager.shutdown_timeout.
* bpo-46565: `del` loop vars that are leaking into module namespaces (GH-30993)Nikita Sobolev2022-02-032-5/+6
|
* bpo-25066: Added repr for multiprocessing.Event (GH-29749)Kumar Aditya2021-12-091-0/+3
| | | Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* bpo-38840: Incorrect __all__ in multiprocessing.managers (GH-18034)Zackery Spytz2021-08-091-3/+4
| | | | | | | This was causing test___all__ to fail on platforms lacking a shared memory implementation. Co-Authored-By: Xavier de Gaye <xdegaye@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* Remove unnecessary pass statements (GH-27103)Elisha Hollander2021-07-131-1/+0
|
* bpo-30256: Add manager_owned keyword arg to AutoProxy (GH-16341)finefoot2021-07-021-2/+2
| | | | Co-authored-by: Jordan Speicher <jordan@jspeicher.com>
* Fix typos in multiple files (GH-26689)Binbin2021-06-131-1/+1
| | | Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
* bpo-43867: multiprocessing Server catchs SystemExit (GH-25441)Victor Stinner2021-04-161-6/+14
| | | | | The multiprocessing Server class now explicitly catchs SystemExit and closes the client connection in this case. It happens when the Server.serve_client() method reachs the end of file (EOF).
* bpo-43651: Fix test_compileall with PEP 597 (GH-25128)Inada Naoki2021-04-021-1/+1
|
* bpo-40692: Run more test_concurrent_futures tests (GH-20239)Asheesh Laroia2021-02-081-1/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | In the case of multiprocessing.synchronize() being missing, the test_concurrent_futures test suite now skips only the tests that require multiprocessing.synchronize(). Validate that multiprocessing.synchronize exists as part of _check_system_limits(), allowing ProcessPoolExecutor to raise NotImplementedError during __init__, rather than crashing with ImportError during __init__ when creating a lock imported from multiprocessing.synchronize. Use _check_system_limits() to disable tests of ProcessPoolExecutor on systems without multiprocessing.synchronize. Running the test suite without multiprocessing.synchronize reveals that Lib/compileall.py crashes when it uses a ProcessPoolExecutor. Therefore, change Lib/compileall.py to call _check_system_limits() before creating the ProcessPoolExecutor. Note that both Lib/compileall.py and Lib/test/test_compileall.py were attempting to sanity-check ProcessPoolExecutor by expecting ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent on platforms where POSIX semaphores aren't available. Avoid using sem_unlink() if it, too, does not exist. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-41344: Raise ValueError when creating shared memory of size 0 (GH-21556)Vinay Sharma2020-08-301-0/+2
|
* Fix typo in message from assert statement (GH-21283)Allen2020-08-171-1/+1
| | | The error message was missing space between the action "acquire" and "_wait_semaphore" which is an attribute for instances of Condition.
* bpo-39244: multiprocessing return default start method first on macOS (GH-18625)idomic2020-05-261-3/+4
|
* bpo-40443: Remove unused imports in stdlib (GH-19815)Victor Stinner2020-05-011-1/+0
|
* bpo-30966: Add multiprocessing.SimpleQueue.close() (GH-19735)Victor Stinner2020-04-271-0/+4
| | | | | | Add a new close() method to multiprocessing.SimpleQueue to explicitly close the queue. Automerge-Triggered-By: @pitrou
* bpo-40330: Fix utf-8 size check in ShareableList (GH-19606)Antoine Pitrou2020-04-201-4/+6
| | | | | The item size must be checked after encoding to bytes, not before. Automerge-Triggered-By: @pitrou
* bpo-38891: avoid quadratic item access performance of ShareableList (GH-18996)Thomas Krennwallner2020-04-191-30/+45
| | | | | | | | Avoid linear runtime of ShareableList.__getitem__ and ShareableList.__setitem__ by storing running allocated bytes in ShareableList._allocated_bytes instead of the number of bytes for a particular stored item. Co-authored-by: Antoine Pitrou <antoine@python.org>
* Fix two typos in multiprocessing (GH-19571)Galden2020-04-181-2/+2
|
* bpo-40221: Update multiprocessing to use _at_fork_reinit (GH-19511)Dong-hee Na2020-04-142-9/+10
|
generated by cgit v0.12 at 2025-02-02 03:42:31 (GMT)