diff options
| -rw-r--r-- | Lib/test/README | 195 |
1 files changed, 142 insertions, 53 deletions
diff --git a/Lib/test/README b/Lib/test/README index 2cf7736..c969fcc 100644 --- a/Lib/test/README +++ b/Lib/test/README @@ -1,40 +1,73 @@ - Writing Python Test Cases - ------------------------- + Writing Python Regression Tests + ------------------------------- Skip Montanaro + (skip@mojam.com) + + +Introduction If you add a new module to Python or modify the functionality of an existing -module, it is your responsibility to write one or more test cases to test -that new functionality. The mechanics of the test system are fairly -straightforward. If you are writing test cases for module zyzzyx, you need -to create a file in .../Lib/test named test_zyzzyx.py and an expected output -file in .../Lib/test/output named test_zyzzyx ("..." represents the -top-level directory in the Python source tree, the directory containing the -configure script). Generate the initial version of the test output file by -executing: - - cd .../Lib/test - python regrtest.py -g test_zyzzyx.py - -Any time you modify test_zyzzyx.py you need to generate a new expected +module, you should write one or more test cases to exercise that new +functionality. The mechanics of how the test system operates are fairly +straightforward. When a test case is run, the output is compared with the +expected output that is stored in .../Lib/test/output. If the test runs to +completion and the actual and expected outputs match, the test succeeds, if +not, it fails. If an ImportError is raised, the test is not run. + +You will be writing unit tests (isolated tests of functions and objects +defined by the module) using white box techniques. Unlike black box +testing, where you only have the external interfaces to guide your test case +writing, in white box testing you can see the code being tested and tailor +your test cases to exercise it more completely. In particular, you will be +able to refer to the C and Python code in the CVS repository when writing +your regression test cases. + + +Executing Test Cases + +If you are writing test cases for module spam, you need to create a file +in .../Lib/test named test_spam.py and an expected output file in +.../Lib/test/output named test_spam ("..." represents the top-level +directory in the Python source tree, the directory containing the configure +script). From the top-level directory, generate the initial version of the +test output file by executing: + + ./python Lib/test/regrtest.py -g test_spam.py + +Any time you modify test_spam.py you need to generate a new expected output file. Don't forget to desk check the generated output to make sure it's really what you expected to find! To run a single test after modifying a module, simply run regrtest.py without the -g flag: - cd .../Lib/test - python regrtest.py test_zyzzyx.py + ./python Lib/test/regrtest.py test_spam.py + +While debugging a regression test, you can of course execute it +independently of the regression testing framework and see what it prints: + + ./python Lib/test/test_spam.py To run the entire test suite, make the "test" target at the top level: - cd ... make test -Test cases generate output based upon computed values and branches taken in -the code. When executed, regrtest.py compares the actual output generated -by executing the test case with the expected output and reports success or -failure. It stands to reason that if the actual and expected outputs are to -match, they must not contain any machine dependencies. This means -your test cases should not print out absolute machine addresses or floating -point numbers with large numbers of significant digits. +On non-Unix platforms where make may not be available, you can simply +execute the two runs of regrtest (optimized and non-optimized) directly: + + ./python Lib/test/regrtest.py + ./python -O Lib/test/regrtest.py + + +Test cases generate output based upon values computed by the test code. +When executed, regrtest.py compares the actual output generated by executing +the test case with the expected output and reports success or failure. It +stands to reason that if the actual and expected outputs are to match, they +must not contain any machine dependencies. This means your test cases +should not print out absolute machine addresses (e.g. the return value of +the id() builtin function) or floating point numbers with large numbers of +significant digits (unless you understand what you are doing!). + + +Test Case Writing Tips Writing good test cases is a skilled task and is too complex to discuss in detail in this short document. Many books have been written on the subject. @@ -46,32 +79,88 @@ object-oriented software revolution, so doesn't cover that subject at all. Unfortunately, it is very expensive (about $100 new). If you can borrow it or find it used (around $20), I strongly urge you to pick up a copy. -As an author of at least part of a module, you will be writing unit tests -(isolated tests of functions and objects defined by the module) using white -box techniques. (Unlike black box testing, where you only have the external -interfaces to guide your test case writing, in white box testing you can see -the code being tested and tailor your test cases to exercise it more -completely). - The most important goal when writing test cases is to break things. A test -case that doesn't uncover a bug is less valuable than one that does. In -designing test cases you should pay attention to the following: - - 1. Your test cases should exercise all the functions and objects defined - in the module, not just the ones meant to be called by users of your - module. This may require you to write test code that uses the module - in ways you don't expect (explicitly calling internal functions, for - example - see test_atexit.py). - - 2. You should consider any boundary values that may tickle exceptional - conditions (e.g. if you were testing a division module you might well - want to generate tests with numerators and denominators at the limits - of floating point and integer numbers on the machine performing the - tests as well as a denominator of zero). - - 3. You should exercise as many paths through the code as possible. This - may not always be possible, but is a goal to strive for. In - particular, when considering if statements (or their equivalent), you - want to create test cases that exercise both the true and false - branches. For while and for statements, you should create test cases - that exercise the loop zero, one and multiple times. +case that doesn't uncover a bug is much less valuable than one that does. +In designing test cases you should pay attention to the following: + + * Your test cases should exercise all the functions and objects defined + in the module, not just the ones meant to be called by users of your + module. This may require you to write test code that uses the module + in ways you don't expect (explicitly calling internal functions, for + example - see test_atexit.py). + + * You should consider any boundary values that may tickle exceptional + conditions (e.g. if you were writing regression tests for division, + you might well want to generate tests with numerators and denominators + at the limits of floating point and integer numbers on the machine + performing the tests as well as a denominator of zero). + + * You should exercise as many paths through the code as possible. This + may not always be possible, but is a goal to strive for. In + particular, when considering if statements (or their equivalent), you + want to create test cases that exercise both the true and false + branches. For loops, you should create test cases that exercise the + loop zero, one and multiple times. + + * You should test with obviously invalid input. If you know that a + function requires an integer input, try calling it with other types of + objects to see how it responds. + + * You should test with obviously out-of-range input. If the domain of a + function is only defined for positive integers, try calling it with a + negative integer. + + * If you are going to fix a bug that wasn't uncovered by an existing + test, try to write a test case that exposes the bug (preferably before + fixing it). + + +Regression Test Writing Rules + +Each test case is different. There is no "standard" form for a Python +regression test case, though there are some general rules: + + * If your test case detects a failure, raise TestFailed (found in + test_support). + + * Import everything you'll need as early as possible. + + * If you'll be importing objects from a module that is at least + partially platform-dependent, only import those objects you need for + the current test case to avoid spurious ImportError exceptions that + prevent the test from running to completion. + + * Print all your test case results using the print statement. For + non-fatal errors, print an error message (or omit a successful + completion print) to indicate the failure, but proceed instead of + raising TestFailed. + + +Miscellaneous + +There is a test_support module you can import from your test case. It +provides the following useful objects: + + * TestFailed - raise this exception when your regression test detects a + failure. + + * findfile(file) - you can call this function to locate a file somewhere + along sys.path or in the Lib/test tree - see test_linuxaudiodev.py for + an example of its use. + + * verbose - you can use this variable to control print output. Many + modules use it. Search for "verbose" in the test_*.py files to see + lots of examples. + + * fcmp(x,y) - you can call this function to compare two floating point + numbers when you expect them to only be approximately equal withing a + fuzz factor (test_support.FUZZ, which defaults to 1e-6). + +Python and C statement coverage results are currently available at + + http://www.musi-cal.com/~skip/python/Python/dist/src/ + +As of this writing (July, 2000) these results are being generated nightly. +You can refer to the summaries and the test coverage output files to see +where coverage is adequate or lacking and write test cases to beef up the +coverage. |