summaryrefslogtreecommitdiffstats
path: root/Lib/multiprocessing
Commit message (Collapse)AuthorAgeFilesLines
* [3.12] gh-108520: Fix bad fork detection in nested multiprocessing use case ↵Miss Islington (bot)2023-08-301-3/+5
| | | | | | | | | | | | | | | (GH-108568) (#108691) gh-108520: Fix bad fork detection in nested multiprocessing use case (GH-108568) gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- (cherry picked from commit add8d45cbe46581b9748909fbbf60fdc8ee8f71e) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* [3.12] gh-77377: Ensure multiprocessing SemLock is valid for spawn-based ↵Miss Islington (bot)2023-08-231-2/+7
| | | | | | | | | | | | | | | | | Process before serializing it (GH-107275) (#108377) gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it (GH-107275) Ensure multiprocessing SemLock is valid for spawn Process before serializing it. Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early. --------- (cherry picked from commit 1700d34d314f5304a7a75363bda295a8c15c371f) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* [3.12] More actionable error message when spawn is incorrectly used. ↵Miss Islington (bot)2023-08-161-1/+5
| | | | | | | | | | (GH-102203) (#107990) More actionable error message when spawn is incorrectly used. (GH-102203) (cherry picked from commit a794ebeb028f7ef287c780d3890f816db9c21c51) Co-authored-by: Yuxin Wu <ppwwyyxxc@gmail.com> Co-authored-by: Yuxin Wu <ppwwyyxx@users.noreply.github.com> Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
* [3.12] gh-107963: Fix set_forkserver_preload to check the type of given list ↵Miss Islington (bot)2023-08-151-1/+1
| | | | | | | | | | | (GH-107965) (#107975) gh-107963: Fix set_forkserver_preload to check the type of given list (GH-107965) (cherry picked from commit 6515ec3d3d5acd3d0b99c88794bdec09f0831e5b) gh-107963: Fix set_forkserver_preload to check the type of given list Co-authored-by: Dong-hee Na <donghee.na@python.org>
* [3.12] gh-90876: Restore the ability to import multiprocessing when ↵Miss Islington (bot)2023-07-061-2/+4
| | | | | | | | | | | | | | `sys.executable` is `None` (GH-106464) (#106494) gh-90876: Restore the ability to import multiprocessing when `sys.executable` is `None` (GH-106464) Prevent `multiprocessing.spawn` from failing to *import* in environments where `sys.executable` is `None`. This regressed in 3.11 with the addition of support for path-like objects in multiprocessing. Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need. (cherry picked from commit c60df361ce2d734148d503f4a711e67c110fe223) Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-104536: Improve `multiprocessing.process._cleanup` logic (#104537)Luccccifer2023-05-221-1/+1
| | | | | | | | | | Fix a race condition in the internal `multiprocessing.process` cleanup logic that could manifest as an unintended `AttributeError` when calling `BaseProcess.close()`. --------- Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-61460: Stronger HMAC in multiprocessing (#20380)Christian Heimes2023-05-201-31/+151
| | | | | | | | | | | | | | | | | | | | | | bpo-17258: `multiprocessing` now supports stronger HMAC algorithms for inter-process connection authentication rather than only HMAC-MD5. Signed-off-by: Christian Heimes <christian@python.org> gpshead: I Reworked to be more robust while keeping the idea. The protocol modification idea remains, but we now take advantage of the message length as an indicator of legacy vs modern protocol version. No more regular expression usage. We now default to HMAC-SHA256, but do so in a way that will be compatible when communicating with older clients or older servers. No protocol transition period is needed. More integration tests to verify these claims remain true are required. I'm unaware of anyone depending on multiprocessing connections between different Python versions. --------- Signed-off-by: Christian Heimes <christian@python.org> Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
* gh-84559: Remove the new multiprocessing warning, too disruptive. (#101551)Gregory P. Smith2023-02-031-27/+1
| | | | This reverts the core of #100618 while leaving relevant documentation improvements and minor refactorings in place.
* GH-84559: Deprecate fork being the multiprocessing default. (#100618)Gregory P. Smith2023-02-021-1/+28
| | | | | | | | This starts the process. Users who don't specify their own start method and use the default on platforms where it is 'fork' will see a DeprecationWarning upon multiprocessing.Pool() construction or upon multiprocessing.Process.start() or concurrent.futures.ProcessPool use. See the related issue and documentation within this change for details.
* gh-99509: Add `__class_getitem__` to `multiprocessing.queues.Queue` (#99511)Nikita Sobolev2022-12-271-0/+2
|
* Fix typo in exception message in `multiprocessing.pool` (#99900)Arne de Laat2022-11-301-1/+1
|
* bpo-40882: Fix a memory leak in SharedMemory on Windows (GH-20684)Zackery Spytz2022-11-251-1/+4
| | | | | In multiprocessing.shared_memory.SharedMemory(), the temporary view returned by MapViewOfFile() should be unmapped when it is no longer needed.
* gh-61460: Add a comment describing the multiprocessing.connection protocol ↵Gregory P. Smith2022-11-201-0/+68
| | | | | | | | (gh-99623) Describe the multiprocessing connection protocol. It isn't a good protocol, but it is what it is. This way we can more easily reason about making changes to it in a backwards compatible way.
* gh-97514: Don't use Linux abstract sockets for multiprocessing (#98501)Gregory P. Smith2022-10-201-5/+0
| | | | | | | | | | | | | | | Linux abstract sockets are insecure as they lack any form of filesystem permissions so their use allows anyone on the system to inject code into the process. This removes the default preference for abstract sockets in multiprocessing introduced in Python 3.9+ via https://github.com/python/cpython/pull/18866 while fixing https://github.com/python/cpython/issues/84031. Explicit use of an abstract socket by a user now generates a RuntimeWarning. If we choose to keep this warning, it should be backported to the 3.7 and 3.8 branches.
* gh-98360: multiprocessing now spawns children on Windows with correct ↵Steve Dower2022-10-201-2/+3
| | | | argv[0] in virtual environments (GH-98462)
* gh-97816: Remove unused variables in `mutliprocessing.managers.Server` (#97817)Koki Saito2022-10-041-1/+0
| | | Remove unused local variables.
* gh-96819: multiprocessing.resource_tracker: check if length of pipe write <= ↵Koki Saito2022-10-031-2/+2
| | | | | 512 (#96890) Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
* GH-83658: make multiprocessing.Pool raise an exception if maxtasksperchild ↵Irit Katriel2022-06-171-0/+3
| | | | | is not None or a positive int (GH-93364) Closes #83658.
* gh-91577: SharedMemory move imports out of methods (#91579)samtygier2022-06-161-4/+3
| | | | | | | SharedMemory.unlink() uses the unregister() function from resource_tracker. Previously it was imported in the method, but this can fail if the method is called during interpreter shutdown, for example when unlink is part of a __del__() method. Moving the import to the top of the file, means that the unregister() method is available during interpreter shutdown. The register call in SharedMemory.__init__() can also use this imported resource_tracker.
* gh-90549: Fix leak of global named resources using multiprocessing spawn ↵Leo Trol2022-06-092-2/+22
| | | | | | (#30617) Co-authored-by: XD Trol <milestonejxd@gmail.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* gh-82616: Add process_group support to subprocess.Popen (#23930)Gregory P. Smith2022-05-051-1/+1
| | | | | | | | One more thing that can help prevent people from using `preexec_fn`. Also adds conditional skips to two tests exposing ASAN flakiness on the Ubuntu 20.04 Address Sanitizer Github CI system. When that build is run on more modern systems the "problem" does not show up. It seems ASAN implementation related. Co-authored-by: Zackery Spytz <zspytz@gmail.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* bpo-47029: Fix BrokenPipeError in multiprocessing.Queue at garbage ↵Géry Ogam2022-05-031-12/+11
| | | | collection and explicit close (#31913)
* GH-91355: micro-optimize Connection.send_bytes() method (gh-32247)Ma Lin2022-05-031-3/+2
|
* Fix missing `f` prefix on f-strings (GH-91910)Alexander Shadchin2022-04-271-1/+1
|
* gh-91401: Add a failsafe way to disable vfork. (#91490)Gregory P. Smith2022-04-251-1/+3
| | | | | | | | | | | | | | Just in case there is ever an issue with _posixsubprocess's use of vfork() due to the complexity of using it properly and potential directions that Linux platforms where it defaults to on could take, this adds a failsafe so that users can disable its use entirely by setting a global flag. No known reason to disable it exists. But it'd be a shame to encounter one and not be able to use CPython without patching and rebuilding it. See the linked issue for some discussion on reasoning. Also documents the existing way to disable posix_spawn.
* bpo-46720: Add support for path-like objects to ↵Géry Ogam2022-04-222-8/+12
| | | | | multiprocessing.set_executable for Windows (GH-31279) This bring the API to be on a par with Unix-like systems.
* gh-91231: Add shutdown_timeout to multiprocessing BaseManager (#32112)Victor Stinner2022-04-191-6/+10
| | | | | | | | Add an optional keyword 'shutdown_timeout' parameter to the multiprocessing.BaseManager constructor. Kill the process if terminate() takes longer than the timeout. Multiprocessing tests pass test.support.SHORT_TIMEOUT to BaseManager.shutdown_timeout.
* bpo-46565: `del` loop vars that are leaking into module namespaces (GH-30993)Nikita Sobolev2022-02-032-5/+6
|
* bpo-25066: Added repr for multiprocessing.Event (GH-29749)Kumar Aditya2021-12-091-0/+3
| | | Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* bpo-38840: Incorrect __all__ in multiprocessing.managers (GH-18034)Zackery Spytz2021-08-091-3/+4
| | | | | | | This was causing test___all__ to fail on platforms lacking a shared memory implementation. Co-Authored-By: Xavier de Gaye <xdegaye@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* Remove unnecessary pass statements (GH-27103)Elisha Hollander2021-07-131-1/+0
|
* bpo-30256: Add manager_owned keyword arg to AutoProxy (GH-16341)finefoot2021-07-021-2/+2
| | | | Co-authored-by: Jordan Speicher <jordan@jspeicher.com>
* Fix typos in multiple files (GH-26689)Binbin2021-06-131-1/+1
| | | Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
* bpo-43867: multiprocessing Server catchs SystemExit (GH-25441)Victor Stinner2021-04-161-6/+14
| | | | | The multiprocessing Server class now explicitly catchs SystemExit and closes the client connection in this case. It happens when the Server.serve_client() method reachs the end of file (EOF).
* bpo-43651: Fix test_compileall with PEP 597 (GH-25128)Inada Naoki2021-04-021-1/+1
|
* bpo-40692: Run more test_concurrent_futures tests (GH-20239)Asheesh Laroia2021-02-081-1/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | In the case of multiprocessing.synchronize() being missing, the test_concurrent_futures test suite now skips only the tests that require multiprocessing.synchronize(). Validate that multiprocessing.synchronize exists as part of _check_system_limits(), allowing ProcessPoolExecutor to raise NotImplementedError during __init__, rather than crashing with ImportError during __init__ when creating a lock imported from multiprocessing.synchronize. Use _check_system_limits() to disable tests of ProcessPoolExecutor on systems without multiprocessing.synchronize. Running the test suite without multiprocessing.synchronize reveals that Lib/compileall.py crashes when it uses a ProcessPoolExecutor. Therefore, change Lib/compileall.py to call _check_system_limits() before creating the ProcessPoolExecutor. Note that both Lib/compileall.py and Lib/test/test_compileall.py were attempting to sanity-check ProcessPoolExecutor by expecting ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent on platforms where POSIX semaphores aren't available. Avoid using sem_unlink() if it, too, does not exist. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-41344: Raise ValueError when creating shared memory of size 0 (GH-21556)Vinay Sharma2020-08-301-0/+2
|
* Fix typo in message from assert statement (GH-21283)Allen2020-08-171-1/+1
| | | The error message was missing space between the action "acquire" and "_wait_semaphore" which is an attribute for instances of Condition.
* bpo-39244: multiprocessing return default start method first on macOS (GH-18625)idomic2020-05-261-3/+4
|
* bpo-40443: Remove unused imports in stdlib (GH-19815)Victor Stinner2020-05-011-1/+0
|
* bpo-30966: Add multiprocessing.SimpleQueue.close() (GH-19735)Victor Stinner2020-04-271-0/+4
| | | | | | Add a new close() method to multiprocessing.SimpleQueue to explicitly close the queue. Automerge-Triggered-By: @pitrou
* bpo-40330: Fix utf-8 size check in ShareableList (GH-19606)Antoine Pitrou2020-04-201-4/+6
| | | | | The item size must be checked after encoding to bytes, not before. Automerge-Triggered-By: @pitrou
* bpo-38891: avoid quadratic item access performance of ShareableList (GH-18996)Thomas Krennwallner2020-04-191-30/+45
| | | | | | | | Avoid linear runtime of ShareableList.__getitem__ and ShareableList.__setitem__ by storing running allocated bytes in ShareableList._allocated_bytes instead of the number of bytes for a particular stored item. Co-authored-by: Antoine Pitrou <antoine@python.org>
* Fix two typos in multiprocessing (GH-19571)Galden2020-04-181-2/+2
|
* bpo-40221: Update multiprocessing to use _at_fork_reinit (GH-19511)Dong-hee Na2020-04-142-9/+10
|
* bpo-40221: Update multiprocessing to use _at_fork_reinit (GH-19477)Dong-hee Na2020-04-141-4/+4
|
* bpo-39481: PEP 585 for a variety of modules (GH-19423)Batuhan Taşkaya2020-04-104-0/+12
| | | | | | | | | | - concurrent.futures - ctypes - http.cookies - multiprocessing - queue - tempfile - unittest.case - urllib.parse
* bpo-40094: Add os.waitstatus_to_exitcode() (GH-19201)Victor Stinner2020-04-012-13/+3
| | | | | | | | | | | | | | Add os.waitstatus_to_exitcode() function to convert a wait status to an exitcode. Suggest waitstatus_to_exitcode() usage in the documentation when appropriate. Use waitstatus_to_exitcode() in: * multiprocessing, os, subprocess and _bootsubprocess modules; * test.support.wait_process(); * setup.py: run_command(); * and many tests.
* bpo-39360: Ensure all workers exit when finalizing a multiprocessing Pool ↵Batuhan Taşkaya2020-03-151-2/+5
| | | | | | | | | (GH-19009) When the pull is not used via the context manager or terminate() is called, there is a system in multiprocessing.util that handles finalization of all pools via an atexit handler (the Finalize) class. This class registers the _terminate_pool handler in the registry of finalizers of the module, and that registry is called on interpreter exit via _exit_function. The problem is that the "happy" path with the context manager or manual call to finalize() does some extra steps that _terminate_pool does not. The step that is not executed when the atexit() handler calls _terminate_pool is pinging the _change_notifier queue to unblock the maintenance threads. This commit moves the notification to the _terminate_pool function so is called from both code paths. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-39850: Add support for abstract sockets in multiprocessing (GH-18866)Pablo Galindo2020-03-094-5/+40
|
generated by cgit v0.12 at 2025-04-12 05:55:11 (GMT)