summaryrefslogtreecommitdiffstats
path: root/Lib/test/libregrtest
Commit message (Collapse)AuthorAgeFilesLines
* gh-108416: Mark slow but not CPU bound test methods with ↵Serhiy Storchaka2023-09-051-1/+3
| | | | requires_resource('walltime') (GH-108480)
* Add missed "f" in an f-string (GH-108906)Serhiy Storchaka2023-09-051-1/+1
|
* gh-89392: Remove support of test_main() in libregrtest (GH-108876)Serhiy Storchaka2023-09-051-8/+5
|
* gh-108834: regrtest --fail-rerun exits with code 5 (#108896)Victor Stinner2023-09-051-2/+3
| | | | When the --fail-rerun option is used and a test fails and then pass, regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
* gh-108834: regrtest reruns failed tests in subprocesses (#108839)Victor Stinner2023-09-035-389/+585
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When using --rerun option, regrtest now re-runs failed tests in verbose mode in fresh worker processes to have more deterministic behavior. So it can write its final report even if a test killed a worker progress. Add --fail-rerun option to regrtest: exit with non-zero exit code if a test failed pass passed when re-run in verbose mode (in a fresh process). That's now more useful since tests can pass when re-run in a fresh worker progress, whereas they failed when run after other tests when tests are run sequentially. Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a deprecated alias. Changes: * Fix and enhance statistics in regrtest summary. Add "(filtered)" when --match and/or --ignore options are used. * Add RunTests class. * Add TestResult.get_rerun_match_tests() method * Rewrite code to serialize/deserialize worker arguments as JSON using a new WorkerJob class. * Fix stats when a test is run with --forever --rerun. * If failed test names cannot be parsed, log a warning and don't filter tests. * test_regrtest.test_rerun_success() now uses a marker file, since the test is re-run in a separated process. * Add tests on normalize_test_name() function. * Add test_success() and test_skip() tests to test_regrtest.
* regrtest computes statistics (#108793)Victor Stinner2023-09-025-241/+306
| | | | | | | | | | | | | | | | | | | | | | test_netrc, test_pep646_syntax and test_xml_etree now return results in the test_main() function. Changes: * Rewrite TestResult as a dataclass with a new State class. * Add test.support.TestStats class and Regrtest.stats_dict attribute. * libregrtest.runtest functions now modify a TestResult instance in-place. * libregrtest summary lists the number of run tests and skipped tests, and denied resources. * Add TestResult.has_meaningful_duration() method. * Compute TestResult duration in the upper function. * Use time.perf_counter() instead of time.monotonic(). * Regrtest: rename 'resource_denieds' attribute to 'resource_denied'. * Rename CHILD_ERROR to MULTIPROCESSING_ERROR. * Use match/case syntadx to have different code depending on the test state. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* gh-108388: Convert test_concurrent_futures to package (#108401)Victor Stinner2023-08-241-0/+1
| | | Convert test_concurrent_futures to a package of sub-tests.
* gh-108388: Split test_multiprocessing_spawn (#108396)Victor Stinner2023-08-241-0/+3
| | | | | | Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration.
* gh-108388: regrtest splits test_asyncio package (#108393)Victor Stinner2023-08-242-44/+39
| | | | | | | | | | Currently, test_asyncio package is only splitted into sub-tests when using command "./python -m test". With this change, it's also splitted when passing it on the command line: "./python -m test test_asyncio". Remove the concept of "STDTESTS". Python is now mature enough to not have to bother with that anymore. Removing STDTESTS simplify the code.
* gh-90791: test.pythoninfo logs ASAN_OPTIONS env var (#108289)Victor Stinner2023-08-221-17/+24
| | | | * Cleanup libregrtest code logging ASAN_OPTIONS. * Fix a typo on "ASAN_OPTIONS" vs "MSAN_OPTIONS".
* gh-108223: test.pythoninfo and libregrtest log Py_NOGIL (#108238)Victor Stinner2023-08-211-0/+5
| | | | | | | | | | | Enable with --disable-gil --without-pydebug: $ make pythoninfo|grep NOGIL sysconfig[Py_NOGIL]: 1 $ ./python -m test ... == Python build: nogil debug ...
* gh-101634: regrtest reports decoding error as failed test (#106169)Victor Stinner2023-06-281-1/+11
| | | | | When running the Python test suite with -jN option, if a worker stdout cannot be decoded from the locale encoding report a failed testn so the exitcode is non-zero.
* Display the sanitizer config in the regrtest header. (#105301)Gregory P. Smith2023-06-061-0/+20
| | | | | | | Display the sanitizers present in libregrtest. Having this in the CI output for tests with the relevant environment variable displayed will help make it easier to do what we need to create an equivalent local test run.
* gh-91896: Revert some very noisy DeprecationWarnings for `ByteString` (#104424)Alex Waygood2023-05-131-9/+6
|
* GH-86275: Implementation of hypothesis stubs for property-based tests, with ↵Paul Ganssle2023-05-121-1/+3
| | | | | | | | | | | | | zoneinfo tests (#22863) These are stubs to be used for adding hypothesis (https://hypothesis.readthedocs.io/en/latest/) tests to the standard library. When the tests are run in an environment where `hypothesis` and its various dependencies are not installed, the stubs will turn any tests with examples into simple parameterized tests and any tests without examples are skipped. It also adds hypothesis tests for the `zoneinfo` module, and a Github Actions workflow to run the hypothesis tests as a non-required CI job. The full hypothesis interface is not stubbed out — missing stubs can be added as necessary. Co-authored-by: Zac Hatfield-Dodds <zac.hatfield.dodds@gmail.com>
* gh-91896: Improve visibility of `ByteString` deprecation warnings (#104294)Alex Waygood2023-05-121-6/+10
|
* gh-103193: Fix refleaks in `test_inspect` and `test_typing` (#104320)Alex Waygood2023-05-091-0/+7
|
* gh-82054: allow test runner to split test_asyncio to execute in parallel by ↵Joshua Herman2023-04-301-3/+16
| | | | | | | | | | | | | | | | | | | | | | | sharding. (#103927) This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel. After porting we can see the direct impact on a multicore system. Without this change: Running make test is 5 min 26 seconds With this change: Running make test takes 3 min 39 seconds That'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%. The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant. Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests. --------- Co-authored-by: Carl Meyer <carl@oddbird.net> Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC]
* gh-84436: Implement Immortal Objects (gh-19474)Eddie Elizondo2023-04-221-4/+10
| | | | | | | | | This is the implementation of PEP683 Motivation: The PR introduces the ability to immortalize instances in CPython which bypasses reference counting. Tagging objects as immortal allows up to skip certain operations when we know that the object will be around for the entire execution of the runtime. Note that this by itself will bring a performance regression to the runtime due to the extra reference count checks. However, this brings the ability of having truly immutable objects that are useful in other contexts such as immutable data sharing between sub-interpreters.
* bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (#30895)Nikita Sobolev2023-04-071-2/+29
| | | | Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* gh-100086: Add build info to test.libregrtest (#100093)Victor Stinner2022-12-082-1/+88
| | | | The Python test runner (libregrtest) now logs Python build information like "debug" vs "release" build, or LTO and PGO optimizations.
* gh-72719: Remove asyncore and asynchat modules (#96580)Nikita Sobolev2022-11-081-2/+2
| | | | | Remove modules asyncore and asynchat, which were deprecated by PEP 594. Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
* gh-92584: Remove the distutils package (#99061)Victor Stinner2022-11-031-9/+0
| | | | | | | | | | | | Remove the distutils package. It was deprecated in Python 3.10 by PEP 632 "Deprecate distutils module". For projects still using distutils and cannot be updated to something else, the setuptools project can be installed: it still provides distutils. * Remove Lib/distutils/ directory * Remove test_distutils * Remove references to distutils * Skip test_check_c_globals and test_peg_generator since they use distutils
* GH-98686: Quicken everything (GH-98687)Brandt Bucher2022-11-021-2/+1
|
* gh-98903: Test suite fails with exit code 4 if no tests ran (#98904)Victor Stinner2022-11-021-6/+16
| | | | | | | The Python test suite now fails wit exit code 4 if no tests ran. It should help detecting typos in test names and test methods. * Add "EXITCODE_" constants to Lib/test/libregrtest/main.py. * Fix a typo: "NO TEST RUN" becomes "NO TESTS RAN"
* gh-95027: Fix regrtest stdout encoding on Windows (#98492)Victor Stinner2022-10-211-3/+11
| | | | | | On Windows, when the Python test suite is run with the -jN option, the ANSI code page is now used as the encoding for the stdout temporary file, rather than using UTF-8 which can lead to decoding errors.
* gh-96465: Clear fractions hash lru_cache under refleak testing (GH-96689)Zachary Ware2022-09-081-0/+7
| | | Automerge-Triggered-By: GH:zware
* gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253)Christian Heimes2022-06-291-39/+38
| | | Co-authored-by: Victor Stinner <vstinner@python.org>
* Run Tools/scripts/reindent.py (#94225)Victor Stinner2022-06-261-1/+0
| | | | | Reindent files which were not properly formatted (PEP 8: 4 spaces). Remove also some trailing spaces.
* gh-94052: Don't re-run failed tests with --python option (#94054)Christian Heimes2022-06-213-6/+17
|
* gh-84461: Fix Emscripten umask and permission issues (GH-94002)Christian Heimes2022-06-191-0/+12
| | | | | | | | - Emscripten's default umask is too strict, see https://github.com/emscripten-core/emscripten/issues/17269 - getuid/getgid and geteuid/getegid are stubs that always return 0 (root). Disable effective uid/gid syscalls and fix tests that use chmod() current user. - Cannot drop X bit from directory.
* gh-93353: regrtest supports checking tmp files with -j2 (#93909)Victor Stinner2022-06-161-3/+6
| | | | | regrtest now also implements checking for leaked temporary files and directories when using -jN for N >= 2. Use tempfile.mkdtemp() to create the temporary directory. Skip this check on WASI.
* gh-93353: Fix regrtest for -jN with N >= 2 (GH-93813)Victor Stinner2022-06-141-15/+21
|
* gh-93353: regrtest checks for leaked temporary files (#93776)Victor Stinner2022-06-142-7/+35
| | | | | When running tests with -jN, create a temporary directory per process and mark a test as "environment changed" if a test leaks a temporary file or directory.
* gh-84461: Fix parallel testing on WebAssembly (GH-93768)Christian Heimes2022-06-131-3/+8
|
* gh-84623: Remove unused imports in tests (#93772)Victor Stinner2022-06-131-1/+1
|
* gh-84461: Add ability for multiprocessed libregrtest to use a different ↵Ethan Smith2022-05-022-2/+11
| | | | Python executable (GH-91930)
* gh-91904: Fix setting envvar PYTHONREGRTEST_UNICODE_GUARD (GH-91905)Serhiy Storchaka2022-04-251-4/+5
| | | | It always failed on non-UTF-8 locale and prevented running regrtests.
* gh-91156: Use `locale.getencoding()` instead of getpreferredencoding (GH-91732)Inada Naoki2022-04-221-2/+1
| | | Co-authored-by: Victor Stinner <vstinner@python.org>
* bpo-40280: Detect missing threading on WASM platforms (GH-32352)Christian Heimes2022-04-071-1/+3
| | | Co-authored-by: Brett Cannon <brett@python.org>
* bpo-40280: Detect if WASM platform supports threading (GH-32243)Christian Heimes2022-04-021-1/+4
| | | Automerge-Triggered-By: GH:tiran
* bpo-44336: Prevent tests hanging on child process handles on Windows (GH-26578)Jeremy Kloth2022-03-222-185/+117
| | | | | Replace the child process `typeperf.exe` with a daemon thread that reads the performance counters directly. This prevents the issues that arise from inherited handles in grandchild processes (see issue37531 for discussion). We only use the load tracker when running tests in multiprocess mode. This prevents inadvertent interactions with tests expecting a single threaded environment. Displaying load is really only helpful for buildbots running in multiprocess mode anyway.
* Fix EncodingWarning in libregrtest (GH-31654)Inada Naoki2022-03-031-1/+1
|
* bpo-46205: exit if no workers are alive in runtest_mp (GH-30470)Sam Gross2022-01-111-8/+10
|
* Revert "bpo-28533: Remove asyncore, asynchat, smtpd modules (GH-29521)" ↵Victor Stinner2021-12-071-1/+11
| | | | | (GH-29951) This reverts commit 9bf2cbc4c498812e14f20d86acb61c53928a5a57.
* bpo-28533: Remove asyncore, asynchat, smtpd modules (GH-29521)Victor Stinner2021-11-151-11/+1
| | | | | | | | | | | | | | | | Remove the asyncore and asynchat modules, deprecated in Python 3.6: use the asyncio module instead. Remove the smtpd module, deprecated in Python 3.6: the aiosmtpd module can be used instead, it is based on asyncio. * Remove asyncore, asynchat and smtpd documentation * Remove test_asyncore, test_asynchat and test_smtpd * Rename Lib/asynchat.py to Lib/test/support/_asynchat.py * Rename Lib/asyncore.py to Lib/test/support/_asyncore.py * Rename Lib/smtpd.py to Lib/test/support/_smtpd.py * Remove DeprecationWarning from private _asyncore, _asynchat and _smtpd modules * _smtpd: remove deprecated properties
* bpo-45745: Remove regrtest --findleaks options (GH-29514)Victor Stinner2021-11-123-10/+3
| | | | | Remove the --findleaks command line option of regrtest: use the --fail-env-changed option instead. Since Python 3.7, it was a deprecated alias to the --fail-env-changed option.
* bpo-10572: Fixup Lib/test/libregrtest/pgo.py (GH-29327)Erlend Egeberg Aasland2021-11-011-1/+1
|
* bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)Gregory P. Smith2021-10-221-0/+24
| | | | | | | Raise RLIMIT_NOFILE in test.libregrtest. On macOS the default is often too low for our testsuite to succeed. Co-authored by reviewer: Victor Stinner
* bpo-45410: regrtest replaces print_warning.orig_stderr (GH-28926)Victor Stinner2021-10-132-3/+13
| | | | | | When running Python tests with -W, runtest() now replaces support.print_warning.orig_stderr to preserve the messages order. Add an unit test.
generated by cgit v0.12 at 2025-03-02 08:25:34 (GMT)