summaryrefslogtreecommitdiffstats
path: root/Lib/multiprocessing/connection.py
Commit message (Collapse)AuthorAgeFilesLines
* Add the missing `f` on an f-string error message in multiprocessing. (GH-127462)Gregory P. Smith2024-12-011-1/+1
|
* gh-97514: Authenticate the forkserver control socket. (GH-99309)Gregory P. Smith2024-11-201-0/+4
| | | | | | | | | | | | | | | | | | | This adds authentication to the forkserver control socket. In the past only filesystem permissions protected this socket from code injection into the forkserver process by limiting access to the same UID, which didn't exist when Linux abstract namespace sockets were used (see issue) meaning that any process in the same system network namespace could inject code. We've since stopped using abstract namespace sockets by default, but protecting our control sockets regardless of type is a good idea. This reuses the HMAC based shared key auth already used by `multiprocessing.connection` sockets for other purposes. Doing this is useful so that filesystem permissions are not relied upon and trust isn't implied by default between all processes running as the same UID with access to the unix socket. ### pyperformance benchmarks No significant changes. Including `concurrent_imap` which exercises `multiprocessing.Pool.imap` in that suite. ### Microbenchmarks This does _slightly_ slow down forkserver use. How much so appears to depend on the platform. Modern platforms and simple platforms are less impacted. This PR adds additional IPC round trips to the control socket to tell forkserver to spawn a new process. Systems with potentially high latency IPC are naturally impacted more. Typically a 1-4% slowdown on a very targeted process creation microbenchmark, with a worst case overloaded system slowdown of 20%. No evidence that these slowdowns appear in practical sense. See the PR for details.
* gh-121313: multiprocessing: simplify by increasing the connection buffer ↵Inada Naoki2024-09-031-16/+4
| | | | | | | | size to 64KiB (GH-123559) Increases the multiprocessing connection buffer size from 8k to 64k for efficiency, without overallocating. Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Victor Stinner <vstinner@python.org>
* gh-121313: Limit the reading size from pipes to their default buffer size on ↵Alexander P.2024-08-311-3/+18
| | | | | POSIX systems (GH-121315) See https://github.com/python/cpython/issues/121313 for analysis, but this greatly reduces memory overallocation and overhead when multiprocessing is sending non-small data over its pipes between processes.
* bpo-43952: Fix multiprocessing Listener authkey bug (GH-25845)Miguel Brito2024-02-271-1/+2
| | | | Listener.accept() no longer hangs when authkey is an empty bytes object.
* gh-89240: Enable multiprocessing on Windows to use large process pools ↵Steve Dower2024-02-131-1/+13
| | | | | | | (GH-107873) We add _winapi.BatchedWaitForMultipleObjects to wait for larger numbers of handles. This is an internal module, hence undocumented, and should be used with caution. Check the docstring for info before using BatchedWaitForMultipleObjects.
* gh-114807: multiprocessing: don't raise ImportError if _multiprocessing is ↵Hood Chatham2024-02-111-1/+1
| | | | | missing (#114808) `_multiprocessing` is only used under the `if _winapi:` block, this moves the import to be within the `_winapi` ImportError handling try/except for equivalent treatment.
* gh-109370: Fix unexpected traceback output in test_concurrent_futures ↵Serhiy Storchaka2023-09-261-2/+1
| | | | | | | | | (GH-109780) Follow-up of gh-107219. * Only close the connection writer on Windows. * Also use existing constant _winapi.ERROR_OPERATION_ABORTED instead of WSA_OPERATION_ABORTED.
* gh-107219: Fix concurrent.futures terminate_broken() (#109244)Victor Stinner2023-09-111-0/+18
| | | | | | | | | | | | | | | Fix a race condition in concurrent.futures. When a process in the process pool was terminated abruptly (while the future was running or pending), close the connection write end. If the call queue is blocked on sending bytes to a worker process, closing the connection write end interrupts the send, so the queue can be closed. Changes: * _ExecutorManagerThread.terminate_broken() now closes call_queue._writer. * multiprocessing PipeConnection.close() now interrupts WaitForMultipleObjects() in _send_bytes() by cancelling the overlapped operation.
* gh-61460: Stronger HMAC in multiprocessing (#20380)Christian Heimes2023-05-201-31/+151
| | | | | | | | | | | | | | | | | | | | | | bpo-17258: `multiprocessing` now supports stronger HMAC algorithms for inter-process connection authentication rather than only HMAC-MD5. Signed-off-by: Christian Heimes <christian@python.org> gpshead: I Reworked to be more robust while keeping the idea. The protocol modification idea remains, but we now take advantage of the message length as an indicator of legacy vs modern protocol version. No more regular expression usage. We now default to HMAC-SHA256, but do so in a way that will be compatible when communicating with older clients or older servers. No protocol transition period is needed. More integration tests to verify these claims remain true are required. I'm unaware of anyone depending on multiprocessing connections between different Python versions. --------- Signed-off-by: Christian Heimes <christian@python.org> Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
* gh-61460: Add a comment describing the multiprocessing.connection protocol ↵Gregory P. Smith2022-11-201-0/+68
| | | | | | | | (gh-99623) Describe the multiprocessing connection protocol. It isn't a good protocol, but it is what it is. This way we can more easily reason about making changes to it in a backwards compatible way.
* gh-97514: Don't use Linux abstract sockets for multiprocessing (#98501)Gregory P. Smith2022-10-201-5/+0
| | | | | | | | | | | | | | | Linux abstract sockets are insecure as they lack any form of filesystem permissions so their use allows anyone on the system to inject code into the process. This removes the default preference for abstract sockets in multiprocessing introduced in Python 3.9+ via https://github.com/python/cpython/pull/18866 while fixing https://github.com/python/cpython/issues/84031. Explicit use of an abstract socket by a user now generates a RuntimeWarning. If we choose to keep this warning, it should be backported to the 3.7 and 3.8 branches.
* GH-91355: micro-optimize Connection.send_bytes() method (gh-32247)Ma Lin2022-05-031-3/+2
|
* Fix typos in multiple files (GH-26689)Binbin2021-06-131-1/+1
| | | Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
* bpo-39850: Add support for abstract sockets in multiprocessing (GH-18866)Pablo Galindo2020-03-091-2/+8
|
* bpo-17560: Too small type for struct.pack/unpack in ↵Alexander Buchkovsky2018-11-061-10/+20
| | | | | mutliprocessing.Connection (GH-10305) Allow sending more than 2 GB at once on a multiprocessing connection on non-Windows systems.
* bpo-34054: multiprocessing uses time.monotonic() (GH-8118)Victor Stinner2018-07-061-4/+4
| | | | | The multiprocessing module now uses the monotonic clock time.monotonic() instead of the system clock time.time() to implement timeouts.
* bpo-5001: More-informative multiprocessing error messages (#3079)Allen W. Smith, Ph.D2017-08-291-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make error message more informative Replace assertions in error-reporting code with more-informative version that doesn't cause confusion over where and what the error is. * Additional clarification + get travis to check * Change from SystemError to TypeError As suggested in PR comment by @pitrou, changing from SystemError; TypeError appears appropriate. * NEWS file installation; ACKS addition (will do my best to justify it by additional work) * Making current AssertionErrors in multiprocessing more informative * Blurb added re multiprocessing managers.py, queues.py cleanup * Further multiprocessing cleanup - went through pool.py * Fix two asserts in multiprocessing/util.py * Most asserts in multiprocessing more informative * Didn't save right version * Further work on multiprocessing error messages * Correct typo * Correct typo v2 * Blasted colon... serves me right for trying to work on two things at once * Simplify NEWS entry * Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst * Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst OK, never mind. * Corrected (thanks to pitrou) error messages for notify * Remove extraneous backslash in docstring.
* bpo-29776: Use decorator syntax for properties. (#585)Serhiy Storchaka2017-03-191-2/+7
|
* Issue #28053: Applying refactorings, docs and other cleanup to follow.Davin Potts2016-09-091-4/+4
|
* Issue #25899: Converted non-ASCII characters in docstrings and manpageSerhiy Storchaka2015-12-181-1/+1
| | | | to ASCII replacements. Original patch by Chris Angelico.
* Issue #23865: close() methods in multiple modules now are idempotent and moreSerhiy Storchaka2015-04-101-5/+10
|\ | | | | | | | | robust at shutdown. If needs to release multiple resources, they are released even if errors are occured.
| * Issue #23865: close() methods in multiple modules now are idempotent and moreSerhiy Storchaka2015-04-101-5/+10
| | | | | | | | | | robust at shutdown. If needs to release multiple resources, they are released even if errors are occured.
* | Issue #18382: Zero-length messages are consumed by ReadFile on Windows 8 and ↵Steve Dower2015-03-021-2/+11
|\ \ | |/ | | | | later
| * Issue #18382: Zero-length messages are consumed by ReadFile on Windows 8 and ↵Steve Dower2015-03-021-2/+11
| | | | | | | | later
* | Issue #23285: PEP 475 -- Retry system calls failing with EINTR.Charles-François Natali2015-02-071-15/+3
| |
* | Issue #22581: Use more "bytes-like object" throughout the docs and comments.Serhiy Storchaka2014-12-051-1/+1
|\ \ | |/
| * Issue #22581: Use more "bytes-like object" throughout the docs and comments.Serhiy Storchaka2014-12-051-1/+1
| |
* | Simplify code in multiprocessing.Connection.send_bytes().Antoine Pitrou2014-07-311-9/+6
|/ | | | Followup to issue #20540; patch by Serhiy.
* do not generate pipe names in the temporary dirBenjamin Peterson2014-04-141-1/+1
|
* Issue #20976: pyflakes: Remove unused importsVictor Stinner2014-03-201-1/+0
|
* Issue #20978: pyflakes: fix undefined namesVictor Stinner2014-03-201-1/+1
|
* Issue #20540: Fix a performance regression (vs. Python 3.2) when layering a ↵Antoine Pitrou2014-02-081-6/+16
|\ | | | | | | | | | | multiprocessing Connection over a TCP socket. For small payloads, Nagle's algorithm would introduce idle delays before the entire transmission of a message.
| * Issue #20540: Fix a performance regression (vs. Python 3.2) when layering a ↵Antoine Pitrou2014-02-081-6/+16
| | | | | | | | | | | | multiprocessing Connection over a TCP socket. For small payloads, Nagle's algorithm would introduce idle delays before the entire transmission of a message.
* | Issue #17276: MD5 as default digestmod for HMAC is deprecated. The HMACChristian Heimes2013-11-201-2/+2
| | | | | | | | module supports digestmod names, e.g. hmac.HMAC('sha1').
* | Issue #18934: Use poll/select-based selectors for multiprocessing.Connection,Charles-François Natali2013-09-081-1/+9
| | | | | | | | to avoid one extra FD per Connection.
* | Issue #18934: multiprocessing: use selectors module.Charles-François Natali2013-09-051-35/+16
| |
* | Issue #18571: Implementation of the PEP 446: file descriptors and file handlesVictor Stinner2013-08-271-2/+4
| | | | | | | | | | are now created non-inheritable; add functions os.get/set_inheritable(), os.get/set_handle_inheritable() and socket.socket.get/set_inheritable().
* | Issue #8713: Support alternative start methods in multiprocessing on Unix.Richard Oudkerk2013-08-141-20/+41
| | | | | | | | See http://hg.python.org/sandbox/sbt#spawn
* | Issue #18200: Back out usage of ModuleNotFoundError (8d28d44f3a9a)Brett Cannon2013-07-041-1/+1
| |
* | Issue #17097: Merge.Richard Oudkerk2013-07-011-3/+15
|\ \ | |/
| * Issue #17097: Make multiprocessing ignore EINTR.Richard Oudkerk2013-07-011-3/+15
| |
| * Fix issue #17707: multiprocessing.Queue's get() method does not block for ↵Giampaolo Rodola'2013-04-171-1/+1
| | | | | | | | short timeouts.
| * fix for previous commit related to issue 10527 which didn't have the ↵Giampaolo Rodola'2013-01-141-23/+25
| | | | | | | | intended effect as per http://bugs.python.org/issue10527#msg179895
* | Issue #18200: Update the stdlib (except tests) to useBrett Cannon2013-06-141-1/+1
| | | | | | | | ModuleNotFoundError.
* | Fix issue #17707: multiprocessing.Queue's get() method does not block for ↵Giampaolo Rodola'2013-04-171-1/+1
| | | | | | | | short timeouts.
* | Issue #17025: Add dumps() and loads() to ForkingPickler.Charles-François Natali2013-03-241-5/+2
| |
* | fix for previous commit related to issue 10527 which didn't have the ↵Giampaolo Rodola'2013-01-141-23/+25
| | | | | | | | intended effect as per http://bugs.python.org/issue10527#msg179895
* | Issue #16955: Fix the poll() method for multiprocessing's socketRichard Oudkerk2013-01-131-1/+1
|\ \ | |/ | | | | connections on Windows.
| * Issue #16955: Fix the poll() method for multiprocessing's socketRichard Oudkerk2013-01-131-1/+1
| | | | | | | | connections on Windows.
generated by cgit v0.12 at 2025-08-29 12:29:43 (GMT)