summaryrefslogtreecommitdiffstats
path: root/Lib/test/test_regrtest.py
Commit message (Collapse)AuthorAgeFilesLines
* gh-127718: Add colour to `test.regrtest` output (#127719)Hugo van Kemenade2024-12-101-0/+46
|
* gh-112192: Increase the trace module coverage precision to one decimal (#126972)RUANG (James Roy)2024-12-031-1/+1
|
* Move get_signal_name() to test.support (#121251)Victor Stinner2024-07-021-10/+0
| | | | | | * Move get_signal_name() from test.libregrtest to test.support. * Use get_signal_name() in support.script_helper. * support.script_helper now decodes stdout and stderr from UTF-8, instead of ASCII, if a command failed.
* gh-121188: Sanitize invalid XML characters in regrtest (#121195)Victor Stinner2024-07-011-0/+59
| | | | | When creating the JUnit XML file, regrtest now escapes characters which are invalid in XML, such as the chr(27) control character used in ANSI escape sequences.
* gh-120417: Use import_helper() in test_regrtest (#120680)Victor Stinner2024-06-181-14/+12
|
* gh-119727: Add --single-process option to regrtest (#119728)Victor Stinner2024-06-031-0/+13
|
* gh-119050: Add XML support to libregrtest refleak checker (#119148)Victor Stinner2024-05-201-9/+0
| | | | | | | | | regrtest test runner: Add XML support to the refleak checker (-R option). * run_unittest() now stores XML elements as string, rather than objects, in support.junit_xml_list. * runtest_refleak() now saves/restores XML strings before/after checking for reference leaks. Save XML into a temporary file.
* Enhance regrtest get_signal_name(): support shell exit code (#117647)Victor Stinner2024-04-081-0/+1
|
* gh-116303: Skip test module dependent tests if test modules are unavailable ↵Erlend E. Aasland2024-04-031-0/+8
| | | | (#117341)
* gh-83434: Disable XML in regrtest when -R option is used (#117232)Victor Stinner2024-03-261-0/+18
|
* gh-110918: Fix side effects of regrtest test_match_tests() (#116718)Victor Stinner2024-03-131-1/+5
| | | | | | | | test_match_tests now saves and restores patterns. Add get_match_tests() function to libregrtest.filter. Previously, running test_regrtest multiple times in a row only ran tests once: "./python -m test test_regrtest -R 3:3.
* gh-115720: Show number of leaks in huntrleaks progress reports (GH-115726)Petr Viktorin2024-02-271-2/+2
| | | | | | | | | | | | | | | | | Instead of showing a dot for each iteration, show: - '.' for zero (on negative) leaks - number of leaks for 1-9 - 'X' if there are more leaks This allows more rapid iteration: when bisecting, I don't need to wait for the final report to see if the test still leaks. Also, show the full result if there are any non-zero entries. This shows negative entries, for the unfortunate cases where a reference is created and cleaned up in different runs. Test *failure* is still determined by the existing heuristic.
* gh-115122: Add --bisect option to regrtest (#115123)Victor Stinner2024-02-181-1/+51
| | | | | | | | * test.bisect_cmd now exit with code 0 on success, and code 1 on failure. Before, it was the opposite. * test.bisect_cmd now runs the test worker process with -X faulthandler. * regrtest RunTests: Add create_python_cmd() and bisect_cmd() methods.
* gh-112984 Update Windows build and installer for free-threaded builds ↵Steve Dower2024-01-171-0/+4
| | | | (GH-113129)
* gh-108927: Fix removing testing modules from sys.modules (GH-108952)Serhiy Storchaka2023-12-041-0/+19
| | | | | It breaks import machinery if the test module has submodules used in other tests.
* gh-110722: Make `-m test -T -j` use sys.monitoring (GH-111710)Łukasz Langa2023-11-101-4/+14
| | | | | | | | | | | Now all results from worker processes are aggregated and displayed together as a summary at the end of a regrtest run. The traditional trace is left in place for use with sequential in-process test runs but now raises a warning that those numbers are not precise. `-T -j` requires `--with-pydebug` as it relies on `-Xpresite=`.
* gh-110367: Make regrtest --verbose3 compatible with --huntrleaks -jN (#111577)Victor Stinner2023-11-011-0/+23
| | | | | "./python -m test -j1 -R 3:3 --verbose3" now works as expected, since run_single_test() does not replace sys.stdout with StringIO in this case.
* gh-111165: Move test running code from test.support to libregrtest (GH-111166)Serhiy Storchaka2023-10-251-1/+117
| | | | Remove no longer used functions run_unittest() and run_doctest() from the test.support module.
* gh-110918: regrtest: allow to intermix --match and --ignore options (GH-110919)Serhiy Storchaka2023-10-211-23/+19
| | | | | | | Test case matching patterns specified by options --match, --ignore, --matchfile and --ignorefile are now tested in the order of specification, and the last match determines whether the test case be run or ignored.
* gh-110932: Fix regrtest for SOURCE_DATE_EPOCH (#111143)Victor Stinner2023-10-211-13/+59
| | | | If the SOURCE_DATE_EPOCH environment variable is defined, use its value as the random seed.
* regrtest: Prepend 'use' options in --{fast,slow}-ci (GH-110363)Zachary Ware2023-10-151-3/+5
| | | This allows individual resources to be disabled without having to explicitly re-enable all others.
* regrtest: --fast-ci/slow-ci no longer imply --rerun-fail (#110849)Victor Stinner2023-10-141-1/+0
|
* Add support.MS_WINDOWS constant (#110446)Victor Stinner2023-10-061-3/+1
|
* gh-110367: Fix regrtest test_worker_output_on_failure() on ASAN build (#110387)Victor Stinner2023-10-051-4/+11
| | | Set ASAN_OPTIONS="handle_segv=0" env var to run the test.
* gh-110367: Enhance regrtest -jN --verbose3 (#110368)Victor Stinner2023-10-051-0/+34
| | | | | | When using worker processes (-jN) with --verbose3 option, regrtest can now display the worker output even if a worker process does crash. Previously, sys.stdout and sys.stderr were replaced and so the worker output was lost on a crash.
* gh-109276: regrtest: shorter list of resources (#110326)Victor Stinner2023-10-041-0/+20
|
* gh-110171: `libregrtest` always sets `random.seed` (#110172)Nikita Sobolev2023-10-041-2/+6
|
* gh-110164: regrtest disables random if SOURCE_DATE_EPOCH (#110168)Victor Stinner2023-10-011-0/+8
| | | | If the SOURCE_DATE_EPOCH environment variable is defined, regrtest now disables randomization of tests.
* gh-110152: regrtest handles cross compilation and HOSTRUNNER (#110156)Victor Stinner2023-09-301-8/+0
| | | | | | | | * _add_python_opts() now handles cross compilation and HOSTRUNNER. * display_header() now tells if Python is cross-compiled, display HOSTRUNNER, and get the host platform. * Remove Tools/scripts/run_tests.py script. * Remove "make hostrunnertest": use "make buildbottest" or "make test" instead.
* gh-109276: regrtest: add WORKER_FAILED state (#110148)Victor Stinner2023-09-301-0/+10
| | | | | | | | | Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does not stop the manager, whereas WORKER_BUG does. Change also TestResults.display_result() order: display failed tests at the end, the important important information. WorkerThread now tries to get the signal name for negative exit code.
* gh-109566: regrtest --fast-ci no longer enables --nowindows (#110121)Victor Stinner2023-09-291-2/+0
| | | | The --nowindows option is deprecated and does nothing but logs a warning.
* gh-109566: regrtest doesn't enable --rerun if --python is used (#109969)Victor Stinner2023-09-271-2/+10
| | | | regrtest: --fast-ci and --slow-ci options no longer enable --rerun if the --python option is used.
* gh-109566: Fix regrtest Python options for WASM/WASI (#109954)Victor Stinner2023-09-271-4/+9
| | | | | WASM and WASI buildbots use multiple PYTHON environment variables such as PYTHONPATH and _PYTHON_HOSTRUNNER. Don't use -E if the --python=COMMAND option is used.
* gh-109566: Fix regrtest code adding Python options (#109926)Victor Stinner2023-09-261-11/+15
| | | | | | * On Windows, use subprocess.run() instead of os.execv(). * Only add needed options * Rename reexec parameter to _add_python_opts. * Rename --no-reexec option to --dont-add-python-opts.
* gh-109566: regrtest reexecutes the process (#109909)Victor Stinner2023-09-261-1/+57
| | | | | | | | | | | | | When --fast-ci or --slow-ci option is used, regrtest now replaces the current process with a new process to add "-u -W default -bb -E" options to Python. Changes: * PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add "-u -W default -bb -E" options to Python: it's now done by regrtest. * Fix Tools/scripts/run_tests.py: flush stdout before replacing the process. Previously, buffered messages were lost.
* gh-109566, regrtest: Add --fast-ci and --slow-ci options (#109570)Victor Stinner2023-09-261-2/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add --fast-ci and --slow-ci options to libregrtest: * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu" (skip slowest tests). * --slow-ci uses a default timeout of 20 minues and "-u all" (run all tests). * regrtest header now lists test resources. * Makefile changes: * "make test", "make hostrunnertest" and "make coverage-report" now use --fast-ci option and TESTTIMEOUT variable. * "make buildbottest" now uses "--slow-ci". Remove options which became redundant with "--slow-ci". * "make testall" and "make testuniversal" now use --slow-ci option and TESTTIMEOUT variable. * "make testall" now uses "find -exec rm ..." instead of "find ... -print|xargs rm ...", same as "make clean". * GitHub Actions workflow: * Ubuntu and Address Sanitizer jobs now use "make test". Remove options which became redundant with "--fast-ci". * Windows jobs now use --fast-ci option. * Use -j0 to detect the number of CPUs. * Set Makefile TESTTIMEOUT default to an empty string, since --slow-ci and --fast-ci use different default timeout. It's now accepted to pass "--timeout=" to regrtest: treated as not timeout. * Tools/scripts/run_tests.py now uses --fast-ci option. * Tools/buildbot/test.bat now uses --slow-ci option. Remove --timeout=1200 option, redundant with --slow-ci.
* gh-109276: regrtest re-runs "env changed" tests (#109831)Victor Stinner2023-09-251-2/+11
| | | | When a test fails with "env changed" and --rerun option is used, the test is now re-run in verbose mode in a fresh process.
* gh-109276: Enhance libregrtest results (#109828)Victor Stinner2023-09-251-1/+2
| | | | | | * Factorize code listing "bad / env changed / ..." tests. * Add TestResults.is_all_good() method. * Move "All 400 tests OK." to the end * Move "Test suite interrupted by signal SIGINT." to the end.
* gh-109425: regrtest decodes worker stdout with backslashreplace (#109428)Victor Stinner2023-09-141-15/+28
| | | | | libregrtest now decodes stdout of test worker processes with the "backslashreplace" error handler to log corrupted stdout, instead of failing with an error and not logging the stdout.
* gh-109276: libregrtest: WASM use filename for JSON (#109340)Victor Stinner2023-09-121-7/+32
| | | | | | | | | | | | | | | | On Emscripten and WASI platforms, or if --python command line option is used, libregrtest now uses a filename for the JSON file. Emscripten and WASI buildbot workers run the main test process with a different Python (Linux) which spawns Emscripten/WASI processes using the command specified in --python command line option. Passing a file descriptor from the parent process to the child process doesn't work in this case. * Add JsonFile and JsonFileType classes * Add RunTests.json_file_use_filename() method. * Add a test in test_regrtest on the --python command line option. * test_regrtest: add parallel=False parameter. * Split long RunWorkers._runtest() function into sub-functions.
* gh-109276: libregrtest calls random.seed() before each test (#109279)Victor Stinner2023-09-121-11/+50
| | | | | | | | | | | | | libregrtest now calls random.seed() before running each test file when -r/--randomize command line option is used. Moreover, it's also called in worker processes. It should help to make tests more deterministic. Previously, it was only called once in the main process before running all test files and it was not called in worker processes. * Convert some f-strings to regular strings in test_regrtest when f-string is not needed. * Remove unused all_methods variable from test_regrtest. * Add RunTests members are now mandatory.
* gh-109039: Branch prediction for Tier 2 interpreter (#109038)Guido van Rossum2023-09-111-3/+4
| | | | | | | | | | | This adds a 16-bit inline cache entry to the conditional branch instructions POP_JUMP_IF_{FALSE,TRUE,NONE,NOT_NONE} and their instrumented variants, which is used to keep track of the branch direction. Each time we encounter these instructions we shift the cache entry left by one and set the bottom bit to whether we jumped. Then when it's time to translate such a branch to Tier 2 uops, we use the bit count from the cache entry to decided whether to continue translating the "didn't jump" branch or the "jumped" branch. The counter is initialized to a pattern of alternating ones and zeros to avoid bias. The .pyc file magic number is updated. There's a new test, some fixes for existing tests, and a few miscellaneous cleanups.
* gh-109162: libregrtest: use relative imports (#109250)Victor Stinner2023-09-111-57/+62
| | | | libregrtest.__init__ no longer exposes any symbol, so "python -m test.libregrtest.worker" imports less modules.
* gh-109162: libregrtest: add single.py and result.py (#109243)Victor Stinner2023-09-111-1/+1
| | | | | | | | | | | * Add single.py and result.py files. * Rename runtest.py to runtests.py. * Move run_single_test() function and its helper functions to single.py. * Move remove_testfn(), abs_module_name() and normalize_test_name() to utils.py. * Move setup_support() to setup.py. * Move type hints like TestName to utils.py. * Rename runtest.py to runtests.py.
* gh-109162: libregrtest: add worker.py (#109229)Victor Stinner2023-09-101-5/+0
| | | | | | | | | | | | | | | | | | | | | | | | Add new worker.py file: * Move create_worker_process() and worker_process() to this file. * Add main() function to worker.py. create_worker_process() now runs the command: "python -m test.libregrtest.worker JSON". * create_worker_process() now starts the worker process in the current working directory. Regrtest now gets the absolute path of the reflog.txt filename: -R command line option filename. * Remove --worker-json command line option. Remove test_regrtest.test_worker_json(). Related changes: * Add write_json() and from_json() methods to TestResult. * Rename select_temp_dir() to get_temp_dir() and move it to utils. * Rename make_temp_dir() to get_work_dir() and move it to utils. It no longer calls os.makedirs(): Regrtest.main() now calls it. * Move fix_umask() to utils. The function is now called by setup_tests(). * Move StrPath to utils. * Add exit_timeout() context manager to utils. * RunTests: Replace junit_filename (StrPath) with use_junit (bool).
* gh-109162: libregrtest: fix _decode_worker_job() (#109202)Victor Stinner2023-09-091-4/+14
| | | | | Decode also HuntRefleak() object inside the RunTests object. Add an unit test on huntrleaks with multiprocessing (-R -jN).
* gh-109162: Refactor libregrtest WorkerJob (#109171)Victor Stinner2023-09-091-4/+4
| | | | | | | | | | | | | | | * Rename --worker-args command line option to --worker-json. * Rename _parse_worker_args() to _parse_worker_json(). * WorkerJob: * Add runtests attribute * Remove test_name and rerun attribute * Rename run_test_in_subprocess() to create_worker_process(). * Rename run_tests_worker() to worker_process(). * create_worker_process() uses json.dump(): write directly JSON to stdout. * Convert MultiprocessResult to a frozen dataclass. * Rename RunTests.match_tests to RunTests.match_tests_dict.
* gh-109162: Regrtest copies 'ns' attributes (#109168)Victor Stinner2023-09-081-1/+1
| | | | | | | | | | * Regrtest.__init__() now copies 'ns' namespace attributes to Regrtest attributes. Regrtest match_tests and ignore_tests attributes have type FilterTuple (tuple), instead of a list. * Add RunTests.copy(). Regrtest._rerun_failed_tests() now uses RunTests.copy(). * Replace Regrtest.all_tests (list) with Regrtest.first_runtests (RunTests). * Make random_seed maximum 10x larger (9 digits, instead of 8).
* gh-89392: Remove support of test_main() in libregrtest (GH-108876)Serhiy Storchaka2023-09-051-4/+4
|
* gh-108834: regrtest --fail-rerun exits with code 5 (#108896)Victor Stinner2023-09-051-2/+3
| | | | When the --fail-rerun option is used and a test fails and then pass, regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
generated by cgit v0.12 at 2025-04-26 19:17:17 (GMT)