summaryrefslogtreecommitdiffstats
path: root/Modules/_posixsubprocess.c
Commit message (Collapse)AuthorAgeFilesLines
* gh-116322: Add Py_mod_gil module slot (#116882)Brett Simmers2024-05-031-0/+1
| | | | | | | | | | | | | | This PR adds the ability to enable the GIL if it was disabled at interpreter startup, and modifies the multi-phase module initialization path to enable the GIL when loading a module, unless that module's spec includes a slot indicating it can run safely without the GIL. PEP 703 called the constant for the slot `Py_mod_gil_not_used`; I went with `Py_MOD_GIL_NOT_USED` for consistency with gh-104148. A warning will be issued up to once per interpreter for the first GIL-using module that is loaded. If `-v` is given, a shorter message will be printed to stderr every time a GIL-using module is loaded (including the first one that issues a warning).
* gh-113964: Don't prevent new threads until all non-daemon threads exit (#116677)Sam Gross2024-03-191-1/+3
| | | | | | | | | | Starting in Python 3.12, we prevented calling fork() and starting new threads during interpreter finalization (shutdown). This has led to a number of regressions and flaky tests. We should not prevent starting new threads (or `fork()`) until all non-daemon threads exit and finalization starts in earnest. This changes the checks to use `_PyInterpreterState_GetFinalizing(interp)`, which is set immediately before terminating non-daemon threads.
* gh-114570: Add PythonFinalizationError exception (#115352)Victor Stinner2024-02-141-1/+1
| | | | | | | | | | | | | | | | | Add PythonFinalizationError exception. This exception derived from RuntimeError is raised when an operation is blocked during the Python finalization. The following functions now raise PythonFinalizationError, instead of RuntimeError: * _thread.start_new_thread() * subprocess.Popen * os.fork() * os.fork1() * os.forkpty() Morever, _winapi.Overlapped finalizer now logs an unraisable PythonFinalizationError, instead of an unraisable RuntimeError.
* gh-104522: Fix OSError raised when run a subprocess (#114195)Serhiy Storchaka2024-01-181-10/+11
| | | | | | | Only set filename to cwd if it was caused by failed chdir(cwd). _fork_exec() now returns "noexec:chdir" for failed chdir(cwd). Co-authored-by: Robert O'Shea <PurityLake@users.noreply.github.com>
* gh-112334: Restore subprocess's use of `vfork()` & fix `extra_groups=[]` ↵Gregory P. Smith2023-12-041-2/+10
| | | | | | | | | | | | | | | | | | | | | behavior (#112617) Restore `subprocess`'s intended use of `vfork()` by default for performance on Linux; also fixes the behavior of `extra_groups=[]` which was unintentionally broken in 3.12.0: Fixed a performance regression in 3.12's :mod:`subprocess` on Linux where it would no longer use the fast-path ``vfork()`` system call when it could have due to a logic bug, instead falling back to the safe but slower ``fork()``. Also fixed a security bug introduced in 3.12.0. If a value of ``extra_groups=[]`` was passed to :mod:`subprocess.Popen` or related APIs, the underlying ``setgroups(0, NULL)`` system call to clear the groups list would not be made in the child process prior to ``exec()``. The security issue was identified via code inspection in the process of fixing the first bug. Thanks to @vain for the detailed report and analysis in the initial bug on Github. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* gh-108494: Argument Clinic: inline parsing code for positional-only ↵Serhiy Storchaka2023-09-031-4/+5
| | | | parameters in the limited C API (GH-108622)
* gh-108765: Python.h no longer includes <unistd.h> (#108783)Victor Stinner2023-09-021-11/+11
|
* gh-106320: Remove private pylifecycle.h functions (#106400)Victor Stinner2023-07-041-1/+3
| | | | | | | Remove private pylifecycle.h functions: move them to the internal C API ( pycore_atexit.h, pycore_pylifecycle.h and pycore_signal.h). No longer export most of these functions. Move _testcapi.test_atexit() to _testinternalcapi.
* gh-106320: Use _PyInterpreterState_GET() (#106336)Victor Stinner2023-07-021-1/+1
| | | | Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().
* gh-106084: Remove _PySequence_BytesToCharpArray() function (#106088)Victor Stinner2023-06-261-0/+82
| | | | | Remove private _PySequence_BytesToCharpArray() and _Py_FreeCharPArray() functions from the public C API: move these functions from Objects/abstract.c to Modules/_posixsubprocess.c.
* gh-104690 Disallow thread creation and fork at interpreter finalization ↵chgnrdv2023-06-041-0/+6
| | | | | | | | | | | | | | | | | (#104826) Disallow thread creation and fork at interpreter finalization. in the following functions, check if interpreter is finalizing and raise `RuntimeError` with appropriate message: * `_thread.start_new_thread` and thus `threading` * `posix.fork` * `posix.fork1` * `posix.forkpty` * `_posixsubprocess.fork_exec` when a `preexec_fn=` is supplied. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-104372: Drop the GIL around the vfork() call. (#104782)Gregory P. Smith2023-05-251-1/+18
| | | | | | | | | On Linux where the `subprocess` module can use the `vfork` syscall for faster spawning, prevent the parent process from blocking other threads by dropping the GIL while it waits for the vfork'ed child process `exec` outcome. This prevents spawning a binary from a slow filesystem from blocking the rest of the application. Fixes #104372.
* gh-104372: use == -1 before PyErr_Occurred (#104831)Gregory P. Smith2023-05-241-1/+1
| | | The ideal pattern for this. (already in the 3.11 backport)
* gh-104372: Use non-Raw malloc for c_fds_to_keep in _posixsubprocess (#104697)Gregory P. Smith2023-05-201-2/+2
| | | Use non-Raw malloc for c_fds_to_keep as the code could ask for 0 length.
* gh-104372: Cleanup _posixsubprocess `make_inheritable` for async signal ↵Gregory P. Smith2023-05-171-34/+91
| | | | | | | | | | safety and no GIL requirement (#104518) Move all of the Python C API calls into the parent process up front instead of doing PyLong_AsLong and PyErr_Occurred and PyTuple_GET from the post-fork/vfork child process. Much of this was long overdue. We shouldn't have been using PyTuple and PyLong APIs within all of these low level functions anyways.
* gh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-104205)Eric Snow2023-05-051-0/+1
| | | Here we are doing no more than adding the value for Py_mod_multiple_interpreters and using it for stdlib modules. We will start checking for it in gh-104206 (once PyInterpreterState.ceval.own_gil is added in gh-104204).
* gh-94518: Port 23-argument `_posixsubprocess.fork_exec` to Argument Clinic ↵Oleg Iarygin2023-04-241-66/+93
| | | | | | | (#94519) Convert fork_exec to pre-inlined-argparser Argument Clinic Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-94518: Rename `group*` to `extra_group*` to avoid confusion (#101054)Oleg Iarygin2023-01-261-28/+28
| | | | | * Rename `group*` to `extra_group*` to avoid confusion * Rename `num_groups` into `extra_group_size` * Rename `groups_list` to `extra_groups_packed`
* gh-94518: [_posixsubprocess] Replace variable validity flags with reserved ↵Oleg Iarygin2023-01-141-33/+30
| | | | | | | | | | | | | values (#94687) Have _posixsubprocess.c stop using boolean flags to say if gid and uid values were supplied and action is required. Such an implicit "either initialized or look somewhere else" confused both the reader (another mental connection to constantly track between functions) and a compiler (warnings on potentially uninitialized variables being passed). Instead, we can utilize a special group/user id as a flag value -1 defined by POSIX but used nowhere else. Namely: gid: call_setgid = False → gid = -1 uid: call_setuid = False → uid = -1 groups: call_setgroups = False → groups = NULL (obtained with (groups_list != Py_None) ? groups : NULL) This PR is required for #94519.
* bpo-15999: Accept arbitrary values for boolean parameters. (#15609)Serhiy Storchaka2022-12-031-1/+1
| | | builtins and extension module functions and methods that expect boolean values for parameters now accept any Python object rather than just a bool or int type. This is more consistent with how native Python code itself behaves.
* gh-98610: Adjust the Optional Restrictions on Subinterpreters (GH-98618)Eric Snow2022-10-311-9/+2
| | | | | | | Previously, the optional restrictions on subinterpreters were: disallow fork, subprocess, and threads. By default, we were disallowing all three for "isolated" interpreters. We always allowed all three for the main interpreter and those created through the legacy `Py_NewInterpreter()` API. Those settings were a bit conservative, so here we've adjusted the optional restrictions to: fork, exec, threads, and daemon threads. The default for "isolated" interpreters disables fork, exec, and daemon threads. Regular threads are allowed by default. We continue always allowing everything For the main interpreter and the legacy API. In the code, we add `_PyInterpreterConfig.allow_exec` and `_PyInterpreterConfig.allow_daemon_threads`. We also add `Py_RTFLAGS_DAEMON_THREADS` and `Py_RTFLAGS_EXEC`.
* gh-98608: Change _Py_NewInterpreter() to _Py_NewInterpreterFromConfig() ↵Eric Snow2022-10-261-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | (gh-98609) (see https://github.com/python/cpython/issues/98608) This change does the following: 1. change the argument to a new `_PyInterpreterConfig` struct 2. rename the function to `_Py_NewInterpreterFromConfig()`, inspired by `Py_InitializeFromConfig()` (takes a `_PyInterpreterConfig` instead of `isolated_subinterpreter`) 3. split up the boolean `isolated_subinterpreter` into the corresponding multiple granular settings * allow_fork * allow_subprocess * allow_threads 4. add `PyInterpreterState.feature_flags` to store those settings 5. add a function for checking if a feature is enabled on an opaque `PyInterpreterState *` 6. drop `PyConfig._isolated_interpreter` The existing default (see `Py_NewInterpeter()` and `Py_Initialize*()`) allows fork, subprocess, and threads and the optional "isolated" interpreter (see the `_xxsubinterpreters` module) disables all three. None of that changes here; the defaults are preserved. Note that the given `_PyInterpreterConfig` will not be used outside `_Py_NewInterpreterFromConfig()`, nor preserved. This contrasts with how `PyConfig` is currently preserved, used, and even modified outside `Py_InitializeFromConfig()`. I'd rather just avoid that mess from the start for `_PyInterpreterConfig`. We can preserve it later if we find an actual need. This change allows us to follow up with a number of improvements (e.g. stop disallowing subprocess and support disallowing exec instead). (Note that this PR adds "private" symbols. We'll probably make them public, and add docs, in a separate change.)
* gh-82616: Add Py_IS_TYPE_SIGNED() macro (#93178)Victor Stinner2022-05-271-1/+3
| | | | | | _posixsubprocess: add a static assertion to ensure that the pid_t type is signed. Replace _Py_IntegralTypeSigned() with _Py_IS_TYPE_SIGNED().
* gh-82616: Add process_group support to subprocess.Popen (#23930)Gregory P. Smith2022-05-051-7/+13
| | | | | | | | One more thing that can help prevent people from using `preexec_fn`. Also adds conditional skips to two tests exposing ASAN flakiness on the Ubuntu 20.04 Address Sanitizer Github CI system. When that build is run on more modern systems the "problem" does not show up. It seems ASAN implementation related. Co-authored-by: Zackery Spytz <zspytz@gmail.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* gh-92301: subprocess: Prefer close_range() to procfs-based fd closing (#92303)Alexey Izbyshev2022-05-051-18/+70
| | | | | | | | #92301: subprocess: Prefer `close_range()` to procfs-based fd closing. `close_range()` is much faster for large number of file descriptors, e.g. 4 times faster for 1000 descriptors in a Linux 5.16-based environment. We prefer close_range() only if it's known to be async-signal-safe.
* gh-91401: Add a failsafe way to disable vfork. (#91490)Gregory P. Smith2022-04-251-3/+4
| | | | | | | | | | | | | | Just in case there is ever an issue with _posixsubprocess's use of vfork() due to the complexity of using it properly and potential directions that Linux platforms where it defaults to on could take, this adds a failsafe so that users can disable its use entirely by setting a global flag. No known reason to disable it exists. But it'd be a shame to encounter one and not be able to use CPython without patching and rebuilding it. See the linked issue for some discussion on reasoning. Also documents the existing way to disable posix_spawn.
* bpo-47151: Fallback to fork when vfork fails in subprocess. (GH-32186)Gregory P. Smith2022-03-311-0/+6
| | | bpo-47151: Fallback to fork when vfork fails in subprocess. An OS kernel can specifically decide to disallow vfork() in a process. No need for that to prevent us from launching subprocesses.
* bpo-43974: Move Py_BUILD_CORE_MODULE into module code (GH-29157)Christian Heimes2021-10-221-0/+4
| | | | | | | | | | | | | | setup.py no longer defines Py_BUILD_CORE_MODULE. Instead every module defines the macro before #include "Python.h" unless Py_BUILD_CORE_BUILTIN is already defined. Py_BUILD_CORE_BUILTIN is defined for every module that is built by Modules/Setup. The PR also simplifies Modules/Setup. Makefile and makesetup already define Py_BUILD_CORE_BUILTIN and include Modules/internal for us. Signed-off-by: Christian Heimes <christian@python.org>
* bpo-45094: Add Py_NO_INLINE macro (GH-28140)Victor Stinner2021-09-031-2/+2
| | | | | * Rename _Py_NO_INLINE macro to Py_NO_INLINE: make it public and document it. * Sort macros in the C API documentation.
* bpo-28254: _posixsubprocess uses PyGC_Enable/PyGC_Disable (GH-25693)Victor Stinner2021-04-281-111/+5
|
* bpo-42655: Fix subprocess extra_groups gid conversion (GH-23762)Jakub Kulík2020-12-291-5/+1
|
* bpo-1635741: Port _posixsubprocess module to multiphase init (GH-23406)Christian Heimes2020-11-211-35/+42
|
* bpo-42375: subprocess DragonFlyBSD build update. (GH-23320)David CARLIER2020-11-191-4/+4
| | | Same as FreeBSD, file descriptors in /dev/fd id from 0 to 63.
* bpo-42146: Unify cleanup in subprocess_fork_exec() (GH-22970)Alexey Izbyshev2020-11-011-35/+18
| | | | | | | | | | | * bpo-42146: Unify cleanup in subprocess_fork_exec() Also ignore errors from _enable_gc(): * They are always suppressed by the current code due to a bug. * _enable_gc() is only used if `preexec_fn != None`, which is unsafe. * We don't have a good way to handle errors in case we successfully created a child process. Co-authored-by: Gregory P. Smith <greg@krypto.org>
* bpo-42146: Fix memory leak in subprocess.Popen() in case of uid/gid overflow ↵Alexey Izbyshev2020-10-261-2/+2
| | | | | | | | | | | (GH-22966) Fix memory leak in subprocess.Popen() in case of uid/gid overflow Also add a test that would catch this leak with `--huntrleaks`. Alas, the test for `extra_groups` also exposes an inconsistency in our error reporting: we use a custom ValueError for `extra_groups`, but propagate OverflowError for `user` and `group`.
* bpo-35823: Allow setsid() after vfork() on Linux. (GH-22945)Gregory P. Smith2020-10-241-2/+3
| | | | | | | | | | | It should just be a syscall updating a couple of fields in the kernel side process info. Confirming, in glibc is appears to be a shim for the setsid syscall (based on not finding any code implementing anything special for it) and in uclibc (*much* easier to read) it is clearly just a setsid syscall shim. A breadcrumb _suggesting_ that it is not allowed on Darwin/macOS comes from a commit in emacs: https://lists.gnu.org/archive/html/bug-gnu-emacs/2017-04/msg00297.html but I don't have a way to verify if that is true or not. As we are not supporting vfork on macOS today I just left a note in a comment.
* bpo-35823: subprocess: Fix handling of pthread_sigmask() errors (GH-22944)Alexey Izbyshev2020-10-241-4/+15
| | | | | | | | Using POSIX_CALL() is incorrect since pthread_sigmask() returns the error number instead of setting errno. Also handle failure of the first call to pthread_sigmask() in the parent process, and explain why we don't handle failure of the second call in a comment.
* bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe ↵Alexey Izbyshev2020-10-241-27/+197
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (GH-11671) * bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe When used to run a new executable image, fork() is not a good choice for process creation, especially if the parent has a large working set: fork() needs to copy page tables, which is slow, and may fail on systems where overcommit is disabled, despite that the child is not going to touch most of its address space. Currently, subprocess is capable of using posix_spawn() instead, which normally provides much better performance. However, posix_spawn() does not support many of child setup operations exposed by subprocess.Popen(). Most notably, it's not possible to express `close_fds=True`, which happens to be the default, via posix_spawn(). As a result, most users can't benefit from faster process creation, at least not without changing their code. However, Linux provides vfork() system call, which creates a new process without copying the address space of the parent, and which is actually used by C libraries to efficiently implement posix_spawn(). Due to sharing of the address space and even the stack with the parent, extreme care is required to use vfork(). At least the following restrictions must hold: * No signal handlers must execute in the child process. Otherwise, they might clobber memory shared with the parent, potentially confusing it. * Any library function called after vfork() in the child must be async-signal-safe (as for fork()), but it must also not interact with any library state in a way that might break due to address space sharing and/or lack of any preparations performed by libraries on normal fork(). POSIX.1 permits to call only execve() and _exit(), and later revisions remove vfork() specification entirely. In practice, however, almost all operations needed by subprocess.Popen() can be safely implemented on Linux. * Due to sharing of the stack with the parent, the child must be careful not to clobber local variables that are alive across vfork() call. Compilers are normally aware of this and take extra care with vfork() (and setjmp(), which has a similar problem). * In case the parent is privileged, special attention must be paid to vfork() use, because sharing an address space across different privilege domains is insecure[1]. This patch adds support for using vfork() instead of fork() on Linux when it's possible to do safely given the above. In particular: * vfork() is not used if credential switch is requested. The reverse case (simple subprocess.Popen() but another application thread switches credentials concurrently) is not possible for pure-Python apps because subprocess.Popen() and functions like os.setuid() are mutually excluded via GIL. We might also consider to add a way to opt-out of vfork() (and posix_spawn() on platforms where it might be implemented via vfork()) in a future PR. * vfork() is not used if `preexec_fn != None`. With this change, subprocess will still use posix_spawn() if possible, but will fallback to vfork() on Linux in most cases, and, failing that, to fork(). [1] https://ewontfix.com/7 Co-authored-by: Gregory P. Smith [Google LLC] <gps@google.com>
* bpo-40422: Move _Py_closerange to fileutils.c (GH-22680)Kyle Evans2020-10-131-0/+1
| | | | | | | This API is relatively lightweight and organizationally, given that it's used by multiple modules, it makes sense to move it to fileutils. Requires making sure that _posixsubprocess is compiled with the appropriate Py_BUIILD_CORE_BUILTIN macro.
* bpo-40422: create a common _Py_closerange API (GH-19754)Kyle Evans2020-10-111-13/+2
| | | | | | | Such an API can be used both for os.closerange and subprocess. For the latter, this yields potential improvement for platforms that have fdwalk but wouldn't have used it there. This will prove even more beneficial later for platforms that have close_range(2), as the new API will prefer that over all else if it's available. The new API is structured to look more like close_range(2), closing from [start, end] rather than the [low, high) of os.closerange(). Automerge-Triggered-By: @gpshead
* bpo-40955: Fix memory leak in subprocess module (GH-20825)Christian Heimes2020-06-121-0/+1
| | | | | | | | | | ``` Direct leak of 8 byte(s) in 1 object(s) allocated from: #0 0x7f008bf19667 in __interceptor_malloc (/lib64/libasan.so.6+0xb0667) #1 0x7f007a0bee4a in subprocess_fork_exec /home/heimes/dev/python/cpython/Modules/_posixsubprocess.c:774 #2 0xe0305b in cfunction_call Objects/methodobject.c:546 ``` Signed-off-by: Christian Heimes <christian@python.org>
* bpo-40453: Add PyConfig._isolated_subinterpreter (GH-19820)Victor Stinner2020-05-011-0/+8
| | | | | | | | | An isolated subinterpreter cannot spawn threads, spawn a child process or call os.fork(). * Add private _Py_NewInterpreter(isolated_subinterpreter) function. * Add isolated=True keyword-only parameter to _xxsubinterpreters.create(). * Allow again os.fork() in "non-isolated" subinterpreters.
* bpo-38061: subprocess uses closefrom() on FreeBSD (GH-19697)Victor Stinner2020-04-241-1/+7
| | | | | | | | | | | | | Optimize the subprocess module on FreeBSD using closefrom(). A single close(fd) syscall is cheap, but when sysconf(_SC_OPEN_MAX) is high, the loop calling close(fd) on each file descriptor can take several milliseconds. The workaround on FreeBSD to improve performance was to load and mount the fdescfs kernel module, but this is not enabled by default. Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans (kevans) and Kubilay Kocak (koobs): https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
* bpo-39968: Convert extension modules' macros of get_module_state() to inline ↵Hai Shi2020-03-161-11/+18
| | | | functions (GH-19017)
* bpo-39947: Add PyInterpreterState_Get() function (GH-18979)Victor Stinner2020-03-131-1/+1
| | | | | | * Rename _PyInterpreterState_Get() to PyInterpreterState_Get() and move it the limited C API. * Add _PyInterpreterState_Get() alias to PyInterpreterState_Get() for backward compatibility with Python 3.8.
* bpo-39245: Switch to public API for Vectorcall (GH-18460)Petr Viktorin2020-02-111-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The bulk of this patch was generated automatically with: for name in \ PyObject_Vectorcall \ Py_TPFLAGS_HAVE_VECTORCALL \ PyObject_VectorcallMethod \ PyVectorcall_Function \ PyObject_CallOneArg \ PyObject_CallMethodNoArgs \ PyObject_CallMethodOneArg \ ; do echo $name git grep -lwz _$name | xargs -0 sed -i "s/\b_$name\b/$name/g" done old=_PyObject_FastCallDict new=PyObject_VectorcallDict git grep -lwz $old | xargs -0 sed -i "s/\b$old\b/$new/g" and then cleaned up: - Revert changes to in docs & news - Revert changes to backcompat defines in headers - Nudge misaligned comments
* bpo-38417: Add umask support to subprocess (GH-16726)Gregory P. Smith2019-10-121-5/+9
| | | | On POSIX systems, allow the umask to be set in the child process before we exec.
* bpo-6559: Update _posixsubprocess.fork_exec doc (GH-16283)Orivej Desh2019-09-201-4/+7
| | | | | | | | | | It did not list the argument added in d4cc7bf993eda4149a05ed25f2f33e18e43fd7c1. https://bugs.python.org/issue6559 Automerge-Triggered-By: @gpshead
* bpo-36046: Add user and group parameters to subprocess (GH-11950)Patrick McLean2019-09-121-4/+132
| | | | | | | | | | | | | | * subprocess: Add user, group and extra_groups paremeters to subprocess.Popen This adds a `user` parameter to the Popen constructor that will call setreuid() in the child before calling exec(). This allows processes running as root to safely drop privileges before running the subprocess without having to use a preexec_fn. This also adds a `group` parameter that will call setregid() in the child process before calling exec(). Finally an `extra_groups` parameter was added that will call setgroups() to set the supplimental groups.
* bpo-38069: Convert _posixsubprocess to PEP-384 (GH-15780)Dino Viehland2019-09-101-8/+58
| | | | | | | | | | Summary: Eliminate uses of `_Py_IDENTIFIER` from `_posixsubprocess`, replacing them with interned strings. Also tries to find an existing version of the module, which will allow subinterpreters. https://bugs.python.org/issue38069
generated by cgit v0.12 at 2025-09-16 17:40:42 (GMT)