summaryrefslogtreecommitdiffstats
path: root/Modules/_posixsubprocess.c
Commit message (Collapse)AuthorAgeFilesLines
* [3.10] gh-91401: Conservative backport of `subprocess._USE_VFORK` (#91932)Gregory P. Smith2022-05-011-2/+25
| | | | | | | | | | | | This does not alter the `_posixsubprocess.fork_exec()` private API to avoid issues for anyone relying on that (bad idea) or for anyone who's `subprocess.py` and `_posixsubprocess.so` upgrades may not become visible to existing Python 3.10 processes at the same time. Backports the concept of cd5726fe674eaff442510eeb6c75628858be9e9f. Provides a fail-safe way to disable vfork for #91401. I didn't backport the documentation as I don't actually expect this to be used and `.. versionadded: 3.10.5` always looks weird in docs. It's being done more to have a fail-safe in place for people just in case.
* bpo-47151: Fallback to fork when vfork fails in subprocess. (GH-32186)Miss Islington (bot)2022-03-311-0/+6
| | | | | | bpo-47151: Fallback to fork when vfork fails in subprocess. An OS kernel can specifically decide to disallow vfork() in a process. No need for that to prevent us from launching subprocesses. (cherry picked from commit 4a08c4c469d36f99d3a5e0f17ad82ab35dcf2835) Co-authored-by: Gregory P. Smith <greg@krypto.org>
* bpo-28254: _posixsubprocess uses PyGC_Enable/PyGC_Disable (GH-25693)Victor Stinner2021-04-281-111/+5
|
* bpo-42655: Fix subprocess extra_groups gid conversion (GH-23762)Jakub Kulík2020-12-291-5/+1
|
* bpo-1635741: Port _posixsubprocess module to multiphase init (GH-23406)Christian Heimes2020-11-211-35/+42
|
* bpo-42375: subprocess DragonFlyBSD build update. (GH-23320)David CARLIER2020-11-191-4/+4
| | | Same as FreeBSD, file descriptors in /dev/fd id from 0 to 63.
* bpo-42146: Unify cleanup in subprocess_fork_exec() (GH-22970)Alexey Izbyshev2020-11-011-35/+18
| | | | | | | | | | | * bpo-42146: Unify cleanup in subprocess_fork_exec() Also ignore errors from _enable_gc(): * They are always suppressed by the current code due to a bug. * _enable_gc() is only used if `preexec_fn != None`, which is unsafe. * We don't have a good way to handle errors in case we successfully created a child process. Co-authored-by: Gregory P. Smith <greg@krypto.org>
* bpo-42146: Fix memory leak in subprocess.Popen() in case of uid/gid overflow ↵Alexey Izbyshev2020-10-261-2/+2
| | | | | | | | | | | (GH-22966) Fix memory leak in subprocess.Popen() in case of uid/gid overflow Also add a test that would catch this leak with `--huntrleaks`. Alas, the test for `extra_groups` also exposes an inconsistency in our error reporting: we use a custom ValueError for `extra_groups`, but propagate OverflowError for `user` and `group`.
* bpo-35823: Allow setsid() after vfork() on Linux. (GH-22945)Gregory P. Smith2020-10-241-2/+3
| | | | | | | | | | | It should just be a syscall updating a couple of fields in the kernel side process info. Confirming, in glibc is appears to be a shim for the setsid syscall (based on not finding any code implementing anything special for it) and in uclibc (*much* easier to read) it is clearly just a setsid syscall shim. A breadcrumb _suggesting_ that it is not allowed on Darwin/macOS comes from a commit in emacs: https://lists.gnu.org/archive/html/bug-gnu-emacs/2017-04/msg00297.html but I don't have a way to verify if that is true or not. As we are not supporting vfork on macOS today I just left a note in a comment.
* bpo-35823: subprocess: Fix handling of pthread_sigmask() errors (GH-22944)Alexey Izbyshev2020-10-241-4/+15
| | | | | | | | Using POSIX_CALL() is incorrect since pthread_sigmask() returns the error number instead of setting errno. Also handle failure of the first call to pthread_sigmask() in the parent process, and explain why we don't handle failure of the second call in a comment.
* bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe ↵Alexey Izbyshev2020-10-241-27/+197
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (GH-11671) * bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe When used to run a new executable image, fork() is not a good choice for process creation, especially if the parent has a large working set: fork() needs to copy page tables, which is slow, and may fail on systems where overcommit is disabled, despite that the child is not going to touch most of its address space. Currently, subprocess is capable of using posix_spawn() instead, which normally provides much better performance. However, posix_spawn() does not support many of child setup operations exposed by subprocess.Popen(). Most notably, it's not possible to express `close_fds=True`, which happens to be the default, via posix_spawn(). As a result, most users can't benefit from faster process creation, at least not without changing their code. However, Linux provides vfork() system call, which creates a new process without copying the address space of the parent, and which is actually used by C libraries to efficiently implement posix_spawn(). Due to sharing of the address space and even the stack with the parent, extreme care is required to use vfork(). At least the following restrictions must hold: * No signal handlers must execute in the child process. Otherwise, they might clobber memory shared with the parent, potentially confusing it. * Any library function called after vfork() in the child must be async-signal-safe (as for fork()), but it must also not interact with any library state in a way that might break due to address space sharing and/or lack of any preparations performed by libraries on normal fork(). POSIX.1 permits to call only execve() and _exit(), and later revisions remove vfork() specification entirely. In practice, however, almost all operations needed by subprocess.Popen() can be safely implemented on Linux. * Due to sharing of the stack with the parent, the child must be careful not to clobber local variables that are alive across vfork() call. Compilers are normally aware of this and take extra care with vfork() (and setjmp(), which has a similar problem). * In case the parent is privileged, special attention must be paid to vfork() use, because sharing an address space across different privilege domains is insecure[1]. This patch adds support for using vfork() instead of fork() on Linux when it's possible to do safely given the above. In particular: * vfork() is not used if credential switch is requested. The reverse case (simple subprocess.Popen() but another application thread switches credentials concurrently) is not possible for pure-Python apps because subprocess.Popen() and functions like os.setuid() are mutually excluded via GIL. We might also consider to add a way to opt-out of vfork() (and posix_spawn() on platforms where it might be implemented via vfork()) in a future PR. * vfork() is not used if `preexec_fn != None`. With this change, subprocess will still use posix_spawn() if possible, but will fallback to vfork() on Linux in most cases, and, failing that, to fork(). [1] https://ewontfix.com/7 Co-authored-by: Gregory P. Smith [Google LLC] <gps@google.com>
* bpo-40422: Move _Py_closerange to fileutils.c (GH-22680)Kyle Evans2020-10-131-0/+1
| | | | | | | This API is relatively lightweight and organizationally, given that it's used by multiple modules, it makes sense to move it to fileutils. Requires making sure that _posixsubprocess is compiled with the appropriate Py_BUIILD_CORE_BUILTIN macro.
* bpo-40422: create a common _Py_closerange API (GH-19754)Kyle Evans2020-10-111-13/+2
| | | | | | | Such an API can be used both for os.closerange and subprocess. For the latter, this yields potential improvement for platforms that have fdwalk but wouldn't have used it there. This will prove even more beneficial later for platforms that have close_range(2), as the new API will prefer that over all else if it's available. The new API is structured to look more like close_range(2), closing from [start, end] rather than the [low, high) of os.closerange(). Automerge-Triggered-By: @gpshead
* bpo-40955: Fix memory leak in subprocess module (GH-20825)Christian Heimes2020-06-121-0/+1
| | | | | | | | | | ``` Direct leak of 8 byte(s) in 1 object(s) allocated from: #0 0x7f008bf19667 in __interceptor_malloc (/lib64/libasan.so.6+0xb0667) #1 0x7f007a0bee4a in subprocess_fork_exec /home/heimes/dev/python/cpython/Modules/_posixsubprocess.c:774 #2 0xe0305b in cfunction_call Objects/methodobject.c:546 ``` Signed-off-by: Christian Heimes <christian@python.org>
* bpo-40453: Add PyConfig._isolated_subinterpreter (GH-19820)Victor Stinner2020-05-011-0/+8
| | | | | | | | | An isolated subinterpreter cannot spawn threads, spawn a child process or call os.fork(). * Add private _Py_NewInterpreter(isolated_subinterpreter) function. * Add isolated=True keyword-only parameter to _xxsubinterpreters.create(). * Allow again os.fork() in "non-isolated" subinterpreters.
* bpo-38061: subprocess uses closefrom() on FreeBSD (GH-19697)Victor Stinner2020-04-241-1/+7
| | | | | | | | | | | | | Optimize the subprocess module on FreeBSD using closefrom(). A single close(fd) syscall is cheap, but when sysconf(_SC_OPEN_MAX) is high, the loop calling close(fd) on each file descriptor can take several milliseconds. The workaround on FreeBSD to improve performance was to load and mount the fdescfs kernel module, but this is not enabled by default. Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans (kevans) and Kubilay Kocak (koobs): https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
* bpo-39968: Convert extension modules' macros of get_module_state() to inline ↵Hai Shi2020-03-161-11/+18
| | | | functions (GH-19017)
* bpo-39947: Add PyInterpreterState_Get() function (GH-18979)Victor Stinner2020-03-131-1/+1
| | | | | | * Rename _PyInterpreterState_Get() to PyInterpreterState_Get() and move it the limited C API. * Add _PyInterpreterState_Get() alias to PyInterpreterState_Get() for backward compatibility with Python 3.8.
* bpo-39245: Switch to public API for Vectorcall (GH-18460)Petr Viktorin2020-02-111-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The bulk of this patch was generated automatically with: for name in \ PyObject_Vectorcall \ Py_TPFLAGS_HAVE_VECTORCALL \ PyObject_VectorcallMethod \ PyVectorcall_Function \ PyObject_CallOneArg \ PyObject_CallMethodNoArgs \ PyObject_CallMethodOneArg \ ; do echo $name git grep -lwz _$name | xargs -0 sed -i "s/\b_$name\b/$name/g" done old=_PyObject_FastCallDict new=PyObject_VectorcallDict git grep -lwz $old | xargs -0 sed -i "s/\b$old\b/$new/g" and then cleaned up: - Revert changes to in docs & news - Revert changes to backcompat defines in headers - Nudge misaligned comments
* bpo-38417: Add umask support to subprocess (GH-16726)Gregory P. Smith2019-10-121-5/+9
| | | | On POSIX systems, allow the umask to be set in the child process before we exec.
* bpo-6559: Update _posixsubprocess.fork_exec doc (GH-16283)Orivej Desh2019-09-201-4/+7
| | | | | | | | | | It did not list the argument added in d4cc7bf993eda4149a05ed25f2f33e18e43fd7c1. https://bugs.python.org/issue6559 Automerge-Triggered-By: @gpshead
* bpo-36046: Add user and group parameters to subprocess (GH-11950)Patrick McLean2019-09-121-4/+132
| | | | | | | | | | | | | | * subprocess: Add user, group and extra_groups paremeters to subprocess.Popen This adds a `user` parameter to the Popen constructor that will call setreuid() in the child before calling exec(). This allows processes running as root to safely drop privileges before running the subprocess without having to use a preexec_fn. This also adds a `group` parameter that will call setregid() in the child process before calling exec(). Finally an `extra_groups` parameter was added that will call setgroups() to set the supplimental groups.
* bpo-38069: Convert _posixsubprocess to PEP-384 (GH-15780)Dino Viehland2019-09-101-8/+58
| | | | | | | | | | Summary: Eliminate uses of `_Py_IDENTIFIER` from `_posixsubprocess`, replacing them with interned strings. Also tries to find an existing version of the module, which will allow subinterpreters. https://bugs.python.org/issue38069
* bpo-37951: Lift subprocess's fork() restriction (GH-15544)Christian Heimes2019-08-271-2/+4
|
* bpo-37337: Add _PyObject_CallMethodNoArgs() (GH-14267)Jeroen Demeyer2019-07-081-3/+3
|
* bpo-35550: Fix incorrect Solaris define guards (GH-11275)Jakub Kulík2018-12-311-1/+1
| | | | | | | Python source code uses on several places ifdef sun or defined(sun) without the underscores, which is not standard compliant and shouldn't be used. Defines should check for __sun instead. Reference: http://nadeausoftware.com/articles/2012/01/c_c_tip_how_use_compiler_predefined_macros_detect_operating_system#Solaris https://bugs.python.org/issue35550
* bpo-35214: Add _Py_ prefix to MEMORY_SANITIZER def. (GH-10503)Gregory P. Smith2018-11-131-2/+2
| | | | Rename our new MEMORY_SANITIZER define to _Py_MEMORY_SANITIZER. Project based C Preprocessor namespacing at its finest. :P
* bpo-35214: Initial clang MemorySanitizer support (GH-10479)Gregory P. Smith2018-11-121-0/+7
| | | | | | | | | | Adds configure flags for msan and ubsan builds to make it easier to enable. These also encode the detail that address sanitizer and memory sanitizer should disable pymalloc. Define MEMORY_SANITIZER when appropriate at build time and adds workarounds to existing code to mark things as initialized where the sanitizer is otherwise unable to determine that. This lets our build succeed under the memory sanitizer. not all tests pass without sanitizer failures yet but we're in pretty good shape after this.
* bpo-34651: Only allow the main interpreter to fork. (gh-9279)Eric Snow2018-09-141-0/+5
| | | | | When os.fork() is called (on platforms that support it) all threads but the current one are destroyed in the child process. Consequently we must ensure that all but the associated interpreter are likewise destroyed. The main interpreter is critical for runtime operation, so we must ensure that fork only happens in the main interpreter. https://bugs.python.org/issue34651
* bpo-34658: Fix rare subprocess prexec_fn fork error. (GH-9255)Gregory P. Smith2018-09-131-7/+13
| | | | | | | | | [bpo-34658](https://www.bugs.python.org/issue34658): Fix a rare interpreter unhandled exception state SystemError only seen when using subprocess with a preexec_fn while an after_parent handler has been registered with os.register_at_fork and the fork system call fails. https://bugs.python.org/issue34658
* bpo-32270: Don't close stdin/out/err in pass_fds (GH-6242)Gregory P. Smith2018-09-111-10/+14
| | | | | | | | | | | | | | When subprocess.Popen() stdin= stdout= or stderr= handles are specified and appear in pass_fds=, don't close the original fds after dup'ing them. This implementation and unittest primarily came from @izbyshev (see the PR) See also https://github.com/izbyshev/cpython/commit/b89b52f28490b69142d5c061604b3a3989cec66c This also removes the old manual p2cread, c2pwrite, and errwrite closing logic as inheritable flags and _close_open_fds takes care of that properly today without special treatment. This code is within child_exec() where it is the only thread so there is no race condition between the dup and _Py_set_inheritable_async_safe call.
* bpo-32844: Fix a subprocess misredirection of a low fd (GH5689)Alexey Izbyshev2018-03-261-1/+1
| | | | | | | | | | bpo-32844: subprocess: Fix a potential misredirection of a low fd to stderr. When redirecting, subprocess attempts to achieve the following state: each fd to be redirected to is less than or equal to the fd it is redirected from, which is necessary because redirection occurs in the ascending order of destination descriptors. It fails to do so in a couple of corner cases, for example, if 1 is redirected to 2 and 0 is closed in the parent.
* bpo-32777: Fix _Py_set_inheritable async-safety in subprocess (GH-5560)Alexey Izbyshev2018-02-061-4/+4
| | | | | | | Fix a rare but potential pre-exec child process deadlock in subprocess on POSIX systems when marking file descriptors inheritable on exec in the child process. This bug appears to have been introduced in 3.4 with the inheritable file descriptors support. This also changes Python/fileutils.c `set_inheritable` to use the "slow" two `fcntl` syscall path instead of the "fast" single `ioctl` syscall path when asked to be async signal safe (by way of being asked not to raise exceptions). `ioctl` is not a POSIX async-signal-safe approved function. ref: http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html
* Fix miscellaneous typos (#4275)luzpaz2017-11-051-6/+6
|
* bpo-16500: Don't use string constants for os.register_at_fork() behavior (#1834)Gregory P. Smith2017-05-291-10/+11
| | | | Instead use keyword only arguments to os.register_at_fork for each of the scenarios. Updates the documentation for clarity.
* bpo-16500: Allow registering at-fork handlers (#1715)Antoine Pitrou2017-05-271-21/+8
| | | | | | | | | | | | * bpo-16500: Allow registering at-fork handlers * Address Serhiy's comments * Add doc for new C API * Add doc for new Python-facing function * Add NEWS entry + doc nit
* bpo-30065: Fixed arguments validation in _posixsubprocess.fork_exec(). (#1110)Serhiy Storchaka2017-04-191-20/+23
|
* Issue #20947: Fixed a gcc warning with -Wstrict-overflow.Serhiy Storchaka2016-09-271-1/+1
|\
| * Issue #20947: Fixed a gcc warning with -Wstrict-overflow.Serhiy Storchaka2016-09-271-1/+1
| |
* | Issue #26862: SYS_getdents64 does not need to be defined on android API 21.Xavier de Gaye2016-06-151-2/+1
| |
* | Issue #25923: Added more const qualifiers to signatures of static and ↵Serhiy Storchaka2015-12-251-1/+1
|/ | | | private functions.
* Issue #25764: Merge subprocess fix from 3.4 into 3.5Martin Panter2015-12-051-17/+18
|\
| * Issue #25764: Preserve subprocess fork exception when preexec_fn usedMartin Panter2015-11-301-17/+18
| | | | | | | | Also fix handling of failure to release the import lock.
* | Fixes #23564: Fix a partially broken sanity check in the _posixsubprocessGregory P. Smith2015-11-161-1/+2
| | | | | | | | | | internals regarding how fds_to_pass were passed to the child. The bug had no actual impact as subprocess.py already avoided it.
* | Close #24784: Fix compilation without thread supportVictor Stinner2015-10-111-2/+10
| | | | | | | | | | | | | | | | Add "#ifdef WITH_THREAD" around cals to: * PyGILState_Check() * _PyImport_AcquireLock() * _PyImport_ReleaseLock()
* | Fix computation of max_fd on OpenBSD. Issue #23852.Gregory P. Smith2015-04-261-0/+10
|\ \ | |/
| * Fix computation of max_fd on OpenBSD. Issue #23852.Gregory P. Smith2015-04-261-0/+10
| |
* | Issue9951: update _hashopenssl and md5module to use _Py_strhex().Gregory P. Smith2015-04-251-1/+1
| | | | | | | | Also update _posixsubprocess to use Py_hexdigits instead of its own constant.
* | Merge 3.4 (_posixsubprocess)Victor Stinner2015-04-021-6/+6
|\ \ | |/
| * Issue #23851: close() must not be retried when it fails with EINTRVictor Stinner2015-04-021-6/+6
| | | | | | | | See the PEP 475 for the rationale.
generated by cgit v0.12 at 2024-12-12 17:20:03 (GMT)