summaryrefslogtreecommitdiffstats
path: root/Lib/test/libregrtest/main.py
Commit message (Collapse)AuthorAgeFilesLines
* bpo-44708: Only re-run test methods that match names of previously failing ↵Miss Islington (bot)2021-07-221-31/+45
| | | | | | | | | | | | | | test methods (GH-27287) (GH-27290) * Move to a static argparse.Namespace subclass * Roughly annotate runtest.py * Refactor libregrtest to use lossless test result objects * Only re-run test methods that match names of previously failing test methods * Adopt tests to cover test method name matching Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit f1afef5e0d93d66fbf3c9aaeab8b3b8da9617583) Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* bpo-41718: libregrtest avoids importing datetime (GH-24985)Victor Stinner2021-03-231-3/+5
| | | | * libregrtest reimplements datetime.timedelta.__str__() * support.testresult only imports datetime if USE_XML is true.
* bpo-40275: Use new test.support helper submodules in tests (GH-21219)Hai Shi2020-06-301-6/+6
|
* bpo-41069: Make TESTFN and the CWD for tests containing non-ascii ↵Serhiy Storchaka2020-06-251-0/+1
| | | | characters. (GH-21035)
* bpo-40275: Use new test.support helper submodules in tests (GH-20849)Hai Shi2020-06-251-1/+2
|
* bpo-41043: Escape literal part of the path for glob(). (GH-20994)Serhiy Storchaka2020-06-201-1/+1
|
* regrtest: log timeout at startup (GH-19514)Victor Stinner2020-04-141-1/+4
| | | Reduce also worker timeout.
* bpo-37957: Allow regrtest to receive a file with test (and subtests) to ↵Pablo Galindo2019-11-191-1/+1
| | | | | | | ignore (GH-16989) When building Python in some uncommon platforms there are some known tests that will fail. Right now, the test suite has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter. The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept and he problem with -x is that is not easy to ignore just a subtests that fail and the whole test needs to be ignored. For these reasons, add a new option to allow to ignore a list of test and subtests for these situations.
* bpo-36670: Enhance regrtest (GH-16556)Victor Stinner2019-10-031-14/+21
| | | | | | | | | | | | | | * Add log() method: add timestamp and load average prefixes to main messages. * WindowsLoadTracker: * LOAD_FACTOR_1 is now computed using SAMPLING_INTERVAL * Initialize the load to the arithmetic mean of the first 5 values of the Processor Queue Length value (so over 5 seconds), rather than 0.0. * Handle BrokenPipeError and when typeperf exit. * format_duration(1.5) now returns '1.5 sec', rather than '1 sec 500 ms'
* bpo-36670: Multiple regrtest bugfixes (GH-16511)Victor Stinner2019-10-011-9/+10
| | | | | | | | | | | | | | | | * Windows: Fix counter name in WindowsLoadTracker. Counter names are localized: use the registry to get the counter name. Original change written by Lorenz Mende. * Regrtest.main() now ensures that the Windows load tracker is also killed if an exception is raised * TestWorkerProcess now ensures that worker processes are no longer running before exiting: kill also worker processes when an exception is raised. * Enhance regrtest messages and warnings: include test name, duration, add a worker identifier, etc. * Rename MultiprocessRunner to TestWorkerProcess * Use print_warning() to display warnings. Co-Authored-By: Lorenz Mende <Lorenz.mende@gmail.com>
* bpo-38203: faulthandler.dump_traceback_later() is always available (GH-16249)Victor Stinner2019-09-181-5/+0
| | | | dump_traceback_later() and cancel_dump_traceback_later() functions of the faulthandler module are always available since Python 3.7.
* bpo-38203: regrtest: put a 2 min timeout on Python exit (GH-16250)Victor Stinner2019-09-181-10/+24
|
* bpo-37531: Fix regrtest timeout for subprocesses (GH-15072)Victor Stinner2019-08-141-1/+3
| | | Co-Authored-By: Joannah Nanjekye <joannah.nanjekye@ibm.com>
* bpo-37707: Exclude expensive unit tests from PGO task (GH-15009)Neil Schemenauer2019-07-301-0/+1
| | | | | Mark some individual tests to skip when --pgo is used. The tests marked increase the PGO task time significantly and likely don't help improve optimization of the final executable.
* Only setup PGO tests when --pgo is enabled. (GH-14927)Gregory P. Smith2019-07-241-2/+3
|
* bpo-36044: Reduce number of unit tests run for PGO build (GH-14702)Neil Schemenauer2019-07-221-0/+4
| | | | | | | | | | | | Reduce the number of unit tests run for the PGO generation task. This speeds up the task by a factor of about 15x. Running the full unit test suite is slow. This change may result in a slightly less optimized build since not as many code branches will be executed. If you are willing to wait for the much slower build, the old behavior can be restored using './configure [..] PROFILE_TASK="-m test --pgo-extended"'. We make no guarantees as to which PGO task set produces a faster build. Users who care should run their own relevant benchmarks as results can depend on the environment, workload, and compiler tool chain.
* Remove unused imports in tests (GH-14518)Victor Stinner2019-07-011-1/+0
|
* bpo-37359: Fix regrtest --cleanup (GH-14336)Victor Stinner2019-06-241-1/+1
|
* bpo-37359: Add --cleanup option to python3 -m test (GH-14332)Victor Stinner2019-06-241-13/+38
| | | | | * regrtest: Add --cleanup option to remove "test_python_*" directories of previous failed test jobs. * Add "make cleantest" to run "python3 -m test --cleanup".
* bpo-36915: regrtest always remove tempdir of worker processes (GH-13312)Victor Stinner2019-05-141-36/+54
| | | | | | | | | When using multiprocessing (-jN option), worker processes now create their temporary directory inside the temporary directory of the main process. So the main process is able to remove temporary directories of worker processes even if they crash or when they are killed by regrtest on KeyboardInterrupt (CTRL+c). Rework also how multiprocessing arguments are parsed in main.py.
* bpo-36719: regrtest -jN no longer stops on crash (GH-13231)Victor Stinner2019-05-131-2/+5
| | | | | | | "python3 -m test -jN ..." now continues the execution of next tests when a worker process crash (CHILD_ERROR state). Previously, the test suite stopped immediately. Use --failfast to stop at the first error. Moreover, --forever now also implies --failfast.
* bpo-36719: regrtest closes explicitly WindowsLoadTracker (GH-12965)Victor Stinner2019-04-261-13/+22
| | | | Regrtest.finalize() now closes explicitly the WindowsLoadTracker instance.
* bpo-36719: Fix regrtest re-run (GH-12964)Victor Stinner2019-04-261-10/+13
| | | | | Properly handle a test which fail but then pass. Add test_rerun_success() unit test.
* bpo-36719: regrtest always detect uncollectable objects (GH-12951)Victor Stinner2019-04-261-38/+5
| | | | | | | regrtest now always detects uncollectable objects. Previously, the check was only enabled by --findleaks. The check now also works with -jN/--multiprocess N. --findleaks becomes a deprecated alias to --fail-env-changed.
* bpo-36725: Refactor regrtest multiprocessing code (GH-12961)Victor Stinner2019-04-261-7/+10
| | | | | | | | | | | | | | | | Rewrite run_tests_multiprocess() function as a new MultiprocessRunner class with multiple methods to better report errors and stop immediately when needed. Changes: * Worker processes are now killed immediately if tests are interrupted or if a test does crash (CHILD_ERROR): worker processes are killed. * Rewrite how errors in a worker thread are reported to the main thread. No longer ignore BaseException or parsing errors silently. * Remove 'finished' variable: use worker.is_alive() instead * Always compute omitted tests. Add Regrtest.get_executed() method.
* bpo-36725: regrtest: add TestResult type (GH-12960)Victor Stinner2019-04-261-43/+45
| | | | | | | | | | | | | | | * Add TestResult and MultiprocessResult types to ensure that results always have the same fields. * runtest() now handles KeyboardInterrupt * accumulate_result() and format_test_result() now takes a TestResult * cleanup_test_droppings() is now called by runtest() and mark the test as ENV_CHANGED if the test leaks support.TESTFN file. * runtest() now includes code "around" the test in the test timing * Add print_warning() in test.libregrtest.utils to standardize how libregrtest logs warnings to ease parsing the test output. * support.unload() is now called with abstest rather than test_name * Rename 'test' variable/parameter to 'test_name' * dash_R(): remove unused the_module parameter * Remove unused imports
* bpo-36638: Fix WindowsLoadTracker exception on some Windows versions (GH-12849)Paul Monson2019-04-181-2/+7
|
* bpo-34060: Report system load when running test suite for Windows (GH-8357)Ammar Askar2019-04-091-2/+15
| | | | | | | | | | | | | | | While Windows exposes the system processor queue length, the raw value used for load calculations on Unix systems, it does not provide an API to access the averaged value. Hence to calculate the load we must track and average it ourselves. We can't use multiprocessing or a thread to read it in the background while the tests run since using those would conflict with test_multiprocessing and test_xxsubprocess. Thus, we use Window's asynchronous IO API to run the tracker in the background with it sampling at the correct rate. When we wish to access the load we check to see if there's new data on the stream, if there is, we update our load values.
* bpo-35683: Improve Azure Pipelines steps (GH-11493)Steve Dower2019-01-221-0/+4
|
* bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10150)Pablo Galindo2018-11-291-1/+12
|
* Add --tempdir option for test run (GH-10322)Steve Dower2018-11-171-7/+5
|
* bpo-34582: Adds JUnit XML output for regression tests (GH-9210)Steve Dower2018-09-181-2/+45
|
* bpo-34605, libregrtest: Rename --slaveargs to --worker-args (GH-9099)Victor Stinner2018-09-071-3/+3
| | | Rename also run_tests_slave() to run_tests_worker().
* bpo-33873: Fix bug in `runtest.py` and add checks for invalid `-R` ↵Pablo Galindo2018-06-261-0/+9
| | | | | | | | parameters (GH-7735) Fix bug in `Lib/test/libregrtest/runtest.py` that makes running tests an extra time than the specified number of runs. Add check for invalid --huntrleaks/-R parameters.
* bpo-33873: regrtest: Add warning on -R 1:3 (GH-7736)Pablo Galindo2018-06-161-0/+7
| | | regrtest: Add warning when using less than 3 warmup runs like -R 1:3.
* bpo-33718: regrtest: use "xxx then yyy" result if re-run (GH-7521)Victor Stinner2018-06-081-1/+7
| | | | | | If tests are re-run, use "xxx then yyy" result format (ex: "FAILURE then SUCCESS") to show that some failing tests have been re-run. Add also test_regrtest.test_rerun_fail() test.
* bpo-33718: regrtest: enhance "running:" formatting (GH-7292)Victor Stinner2018-06-011-42/+1
| | | | | * "running:" progress: Format number of seconds as hours and minutes * format_duration(): count also minutes as hours * Create Lib/test/libregrtest/utils.py
* bpo-33718: regrtest keeps filters to re-run fails (GH-7291)Victor Stinner2018-05-311-17/+22
| | | | | | * No longer clear filters, like --match, to re-run failed tests in verbose mode (-w option). * Tests result: always indicate if tests have been interrupted. * Enhance tests summary
* regrtest: repeat summary after re-run (GH-7159)Victor Stinner2018-05-281-6/+19
| | | | Using -w, when failing tests are re-run in verbose mode, display again the tests results at the end.
* bpo-31324: Optimize support._match_test() (#4421)Victor Stinner2017-11-211-2/+2
| | | | | | | | | | | * Rename support._match_test() to support.match_test(): make it public * Remove support.match_tests global variable. It is replaced with a new support.set_match_tests() function, so match_test() doesn't have to check each time if patterns were modified. * Rewrite match_test(): use different code paths depending on the kind of patterns for best performances. Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
* bpo-30871: Add test.pythoninfo (#3075)Victor Stinner2017-08-171-3/+0
| | | | | | * Add Lib/test/pythoninfo.py: script collecting various informations about Python to help debugging test failures. * regrtest: remove sys.hash_info and sys.flags from header. * Travis CI, Appveyor: run pythoninfo before tests
* regrtest: count also env changed as failures in progress (#3061)Victor Stinner2017-08-101-2/+3
|
* bpo-30764: Fix regrtest --fail-env-changed --forever (#2536)Victor Stinner2017-07-031-1/+3
| | | --forever now stops if a fail changes the environment.
* bpo-30764: regrtest: add --fail-env-changed option (#2402)Victor Stinner2017-06-261-1/+9
| | | | | | | | | | | | * bpo-30764: regrtest: change exit code on failure * Exit code 2 if failed tests ("bad") * Exit code 3 if interrupted * bpo-30764: regrtest: add --fail-env-changed option If the option is set, mark a test as failed if it alters the environment, for example if it creates a file without removing it.
* bpo-30523: regrtest --list-cases --match (#2401)Victor Stinner2017-06-261-1/+5
| | | | | | | * regrtest --list-cases now supports --match and --match-file options. Example: ./python -m test --list-cases -m FileTests test_os * --list-cases now also sets support.verbose to False to prevent messages to stdout when loading test modules. * Add support._match_test() private function.
* bpo-30523: regrtest: Add --list-cases option (#2238)mlouielu2017-06-161-3/+32
| | | | | | | | | | | * bpo-30523: regrtest: Add --list-cases option * bpo-30523: Enhance --list-cases * Add get_abs_module() function, use it in list_cases() * list_cases() now logs skipped tests into stderr * Remove unused doctest
* bpo-30284: Fix regrtest for out of tree build (#1481)Victor Stinner2017-06-141-1/+7
| | | | | | | | Use a build/ directory in the build directory, not in the source directory, since the source directory may be read-only and must not be modified. Fallback on the source directory if the build directory is not available (missing "abs_builddir" sysconfig variable).
* bpo-30258: regrtest: Fix run_tests_multiprocess() (#1479)Victor Stinner2017-05-091-0/+2
| | | | | | If the child process exited with a non-zero code, don't strip the last line of stdout anymore. Add also a sanity check in accumulate_result().
* bpo-30263: regrtest: log system load (#1452)Victor Stinner2017-05-041-24/+33
| | | | | | | | * bpo-30263: regrtest: log system load * regrtest: log the number of CPUs Log the number of CPUs in the header. --verbose now imply --header.
* regrtest: don't fail immediately if a child does crashVictor Stinner2017-02-061-1/+1
| | | | | | Issue #29362: Catch a crash of a worker process as a normal failure and continue to run next tests. It allows to get the usual test summary: single line result (OK/FAIL), total duration, etc.
generated by cgit v0.12 at 2025-03-27 23:09:41 (GMT)