diff options
Diffstat (limited to 'tests/README')
| -rw-r--r-- | tests/README | 489 |
1 files changed, 408 insertions, 81 deletions
diff --git a/tests/README b/tests/README index 07915c9..fe4bb30 100644 --- a/tests/README +++ b/tests/README @@ -1,96 +1,423 @@ -Tcl Test Suite --------------- +README -- Tcl test suite design document. -RCS: @(#) $Id: README,v 1.2 1998/09/14 18:40:07 stanton Exp $ +RCS: @(#) $Id: README,v 1.3 1999/04/16 00:47:22 stanton Exp $ -This directory contains a set of validation tests for the Tcl -commands. Each of the files whose name ends in ".test" is -intended to fully exercise one or a few Tcl commands. The -commands tested by a given file are listed in the first line -of the file. +Contents: +--------- + + 1. Introduction + 2. Definitions file + 3. Writing a new test + 4. Constraints + 5. Adding a New Test File + 6. Test output + 7. Selecting tests for execution within a file + 8. Selecting files to be sourced by all.tcl + 9. Incompatibilities with prior Tcl versions + +1. Introduction: +---------------- + +This directory contains a set of validation tests for the Tcl commands +and C Library procedures for Tcl. Each of the files whose name ends +in ".test" is intended to fully exercise the functions in the C source +file that corresponds to the file prefix. The C functions and/or Tcl +commands tested by a given file are listed in the first line of the +file. + +You can run the tests in three ways: -You can run the tests in two ways: (a) type "make test" in ../unix; this will run all of the tests. - (b) start up tcltest in this directory, then "source" the test + + (b) type "tcltest <testFile> ?<option> <value>? + Command line options include: + + -verbose <level> set the level of verbosity to a substring + of "bps". See the "Test output" section + for an explanation of this option. + + -match <matchList> only run tests that match one or more of + the glob patterns in <matchList> + + -skip <skipList> do not run tests that match one or more + of the glob patterns in <skipList> + + -file <globPattern> only source test files that match + <globPattern> (relative to the "tests" + directory). This option only applies + when you run the test suite with the + "all.tcl" file. + + -constraints <list> tests with any constraints in <list> will + not be skipped. Not that elements of + <list> must exactly match the existing + constraints. + + (c) start up tcltest in this directory, then "source" the test file (for example, type "source parse.test"). To run all - of the tests, type "source all". -In either case no output will be generated if all goes well, except -for a listing of the tests.. If there are errors then additional -messages will appear in the format described below. Note: don't -run the tests as superuser, since this will cause several of the tests -to fail. - -The rest of this file provides additional information on the -features of the testing environment. - -This approach to testing was designed and initially implemented -by Mary Ann May-Pumphrey of Sun Microsystems. Many thanks to -her for donating her work back to the public Tcl release. - -Definitions file: ------------------ - -The file "defs" defines a collection of procedures and variables -used to run the tests. It is read in automatically by each of the -.test files if needed, but once it has been read once it will not -be read again by the .test files. If you change defs while running -tests you'll have to "source" it by hand to load its new contents. - -Test output: ------------- - -Normally, output only appears when there are errors. However, if -the variable VERBOSE is set to 1 then tests will be run in "verbose" -mode and output will be generated for each test regardless of -whether it succeeded or failed. Test output consists of the -following information: - - - the test identifier (which can be used to locate the test code - in the .test file) - - a brief description of the test - - the contents of the test code - - the actual results produced by the tests - - a "PASSED" or "FAILED" message - - the expected results (if the test failed) - -You can set VERBOSE either interactively (after the defs file has been -read in), or you can change the default value in "defs". - -Selecting tests for execution: ------------------------------- + of the tests, type "source all.tcl". To use the options in + interactive mode, you can set their corresponding tcltest + namespace variables after sourcing the defs.tcl file. + ::tcltest::match + ::tcltest::skip + ::tcltest::testConfig(nonPortable) + ::tcltest::testConfig(knownBug) + ::tcltest::testConfig(userInteractive) + +In all cases, no output will be generated if all goes well, except for +a listing of the test files and a statistical summary. If there are +errors, then additional messages will appear in the format described +below. Note that some tests will be skipped if you run as superuser. + +This approach to testing was designed and initially implemented by +Mary Ann May-Pumphrey of Sun Microsystems in the early 1990's. Many +thanks to her for donating her work back to the public Tcl release. + + +2. Definitions file: +-------------------- + +The file "defs.tcl" defines the "tcltest" namespace which contains a +collection of procedures and variables used to run the tests. It is +read in automatically by each of the .test files if needed, but once +it has been read once it will not be read again by the .test files. +Currently, the following procedures are exported from the "tcltest" +namespace and automatically imported: + + test Run a test script. + + cleanupTests Print stats and remove files created by tests. + + dotests Source a test file and run tests of the + specified pattern. + + makeFile Create a file--the file will automatically + be removed by cleanupTests. + + removeFile Force a file to be removed. + + makeDirectory Create a directory--the directory will + automatically be removed by cleanupTests. + + removeDirectory Force a directory to be removed. + + viewFile Returns the contents of a file. + + normalizeMsg Remove extra newlines from a string. + + bytestring Construct a string that consists of the + requested sequence of bytes, as opposed to a + string of properly formed UTF-8 characters. + + set_iso8859_1_locale Set the locale to iso8859_1. + + restore_locale Restore the locale to its original setting. + + saveState Save the procedure and global variable names. + + restoreState Restore the procedure and global variable names. + +Please refer to the defs.tcl file for more documentation on these +procedures. + + +3. Writing a new test: +---------------------- + +The test procedure runs a test script and prints an error message if +the script's result does not match the expected result. The following +is the spec for the "test" command: + + test <name> <description> ?<constraint>? <script> <expectedAnswer> + +The <name> argument should follow the pattern, +"<target>-<majorNum>.<minorNum>". For white-box (regression) tests, +the target should be the name of the c function or Tcl procedure being +tested. For black-box tests, the target should be the name of the +feature being tested. Related tests should share a major number. + +The <description> argument is a short textual description of the test, +to help humans understand what it does. + +The optional <constraints> argument is list of one or more keywords, +each of which must be the name of an element in the array +"::tcltest::testConfig". If any of these elements is false or does +not exist, the test is skipped. Add appropriate constraints (e.g., +unixOnly) to any tests that should not always be run. For example, a +test that should only be run on Unix should look like the following: + + test getAttribute-1.1 {testing file permissions} {unixOnly} { + lindex [file attributes foo.tcl] 5 + } {00644} + +See the "Constraints" section for a list of built-in +constraints and information on how to add your own constraints. + +The <script> argument contains the script to run to carry out the +test. It must return a result that can be checked for correctness. +If your script requires that a file be created on the fly, please use +the ::tcltest::makeFile procedure. If your test requires that a small +file (<50 lines) be checked in, please consider creating the file on +the fly using the ::tcltest::makeFile procedure. Files created by the +::tcltest::makeFile procedure will automatically be removed by the +::tcltest::cleanupTests call at the end of each test file. + +The <expectedAnswer> argument will be compared against the result of +evaluating the <script> argument. If they match, the test passes, +otherwise the test fails. + + +4. Constraints: +--------------- + +Constraints are used to determine whether a test should be skipped. +Each constraint is stored as an index in the array +::tcltest::testConfig. For example, the unixOnly constraint is +defined as the following: + + set ::tcltest::testConfig(unixOnly) \ + [expr {$tcl_platform(platform) == "unix"}] + +If a test is constrained by "unixOnly", then it will only be run if +the value of ::tcltest::testConfig(unixOnly) is true. + +The following is a list of constraints defined in the defs.tcl file: + +unix test can only be run on any UNIX platform +pc test can only be run on any Windows platform +nt test can only be run on any Windows NT platform +95 test can only be run on any Windows 95 platform +mac test can only be run on any Mac platform +unixOrPc test can only be run on a UNIX or PC platform +macOrPc test can only be run on a Mac or PC platform +macOrUnix test can only be run on a Mac or UNIX platform +tempNotPc test can not be run on Windows. This flag is used + to temporarily disable a test. +tempNotMac test can not be run on a Mac. This flag is used + to temporarily disable a test. +unixCrash test crashes if it's run on UNIX. This flag is used + to temporarily disable a test. +pcCrash test crashes if it's run on Windows. This flag is + used to temporarily disable a test. +macCrash test crashes if it's run on a Mac. This flag is used + to temporarily disable a test. + +emptyTest test is empty, and so not worth running, but + it remains as a place-holder for a test to be + written in the future. This constraint always + causes tests to be skipped. + +knownBug test is known to fail and the bug is not yet + fixed. This constraint always causes tests to be + skipped unless the user specifies otherwise. See the + "Introduction" section for more details. + +nonPortable test can only be run in the master Tcl/Tk + development environment. Some tests are inherently + non-portable because they depend on things like word + length, file system configuration, window manager, + etc. These tests are only run in the main Tcl + development directory where the configuration is + well known. This constraint always causes tests to be + skipped unless the user specifies otherwise. See the + "Introduction" section for more details. + +userInteraction test requires interaction from the user. This + constraint always causes tests to be skipped unless + the user specifies otherwise. See the "Introduction" + section for more details. + +interactive test can only be run in if the interpreter is in + interactive mode, that is the global tcl_interactive + variable is set to 1. + +nonBlockFiles test can only be run if platform supports setting + files into nonblocking mode + +asyncPipeClose test can only be run if platform supports async + flush and async close on a pipe + +unixExecs test can only be run if this machine has commands + such as 'cat', 'echo', etc. available. + +hasIsoLocale test can only be run if can switch to an ISO locale + +fonts test can only be run if the wish app's fonts can + be controlled by Tk. + +root test can only run if Unix user is root + +notRoot test can only run if Unix user is not root + +eformat test can only run if app has a working version of + sprintf with respect to the "e" format of + floating-point numbers. + +stdio test can only be run if the current app can be + spawned via a pipe + + +5. Adding a new test file: +-------------------------- + +Tests files should begin by sourcing the defs.tcl file: + + if {[lsearch [namespace children] ::tcltest] == -1} { + source [file join [pwd] [file dirname [info script]] defs.tcl] + } + +Test files sould end by cleaning up after themselves and calling +::tcltest::cleanupTests. The ::tcltest::cleanupTests procedure prints +statistics about the number of tests that passed, skipped, and failed, +and removes all files that were created using the ::tcltest::makeFile +and ::tcltest::makeDirectory procedures. + + # Remove files created by these tests + # Change to original working directory + # Unset global arrays + ::tcltest::cleanupTests + return + +The all.tcl file will source your new test file if the filename +matches the tests/*.test pattern (as it should). The names of test +files that contain regression (or glass-box) tests should correspond +to the Tcl or C code file that they are testing. For example, the +test file for the C file "tclCmdAH.c" is "cmdAH.test". Test files +that contain black-box tests may not correspond to any Tcl or C code +file so they should match the pattern "*_bb.test". + +Be sure your new test file can be run from any working directory. + +Be sure no temporary files are left behind by your test file. + +Be sure your tests can run cross-platform in both a build environment +as well as an installation environment. If your test file contains +tests that should not be run in one or more of those cases, please use +the constraints mechanism to skip those tests. + + +6. Test output: +--------------- + +After all specified test files are sourced, the number of tests +passed, skipped, and failed is printed to stdout. Aside from this +statistical information, output can be controlled on a per-test basis +by the ::tcltest::verbose variable. + +::tcltest::verbose can be set to any substring or permutation of "bps". +In the string "bps", the 'b' stands for a test's "body", the 'p' +stands for "passed" tests, and the 's' stands for "skipped" tests. +The default value of ::tcltest::verbose is "b". If 'b' is present, then +the entire body of the test is printed for each failed test, otherwise +only the test's name, desired output, and actual output, are printed +for each failed test. If 'p' is present, then a line is printed for +each passed test, otherwise no line is printed for passed tests. If +'s' is present, then a line (containing the consraints that cause the +test to be skipped) is printed for each skipped test, otherwise no +line is printed for skipped tests. + +You can set ::tcltest::verbose either interactively (after the defs.tcl +file has been sourced) or by the command line argument -verbose, for +example: + + tcltest socket.test -verbose bps + + +7. Selecting tests for execution within a file: +----------------------------------------------- Normally, all the tests in a file are run whenever the file is -"source"d. However, you can select a specific set of tests using -the global variable TESTS. This variable contains a pattern; any -test whose identifier matches TESTS will be run. For example, -the following interactive command causes all of the "for" tests in -groups 2 and 4 to be executed: +sourced. An individual test will be skipped if one of the following +conditions is met: + + 1) the "name" of the tests does not match (using glob style + matching) one or more elements in the ::tcltest::match + variable + + 2) the "name" of the tests matches (using glob style matching) one + or more elements in the ::tcltest::skip variable + + 3) the "constraints" argument to the "test" call, if given, + contains one or more false elements. + +You can set ::tcltest::match and/or ::tcltest::skip +either interactively (after the defs.tcl file has been sourced), or by +the command line arguments -match and -skip, for example: + + tcltest info.test -match '*-5.* *-7.*' -skip '*-7.1*' + +Be sure to use the proper quoting convention so that your shell does +not perform the glob substitution on the match or skip patterns you +specify. + +The two predefined constraints (knownBug and nonPortable) can be +overridden either interactively (after the defs.tcl file has been +sourced) by setting the ::tcltest::testConfig(<constraint>) variable, +or by using the -constraints command line option with the name of the +constraint in the argument. The following example shows how to run +tests that are constrained by the knownBug and nonPortable +restricions: + + tcltest all.tcl -constraints "knownBug nonPortable" + +See the defs.tcl file for information about each of these constraints. +Other constraints can be added at any time. See the "Writing a new +test" section below for more details about using built-in constraints +and adding new ones. + + +8. Selecting files to be sourced by all.tcl: +-------------------------------------------- + +You can specify the files you want all.tcl to source on the command +line with the -file options. For example, if you call the +following: + + tcltest all.tcl -file 'unix*.test' + +all files in "tests" directory that match the pattern unix*.test will +be sourced by the all.tcl file. Another useful example is if a +particular test hangs, say "get.test", and you just want to run the +remaining tests, then you can call the following: + + tcltest all.tcl -file '[h-z]*.test' + +Note that the argument to -file will be substituted relative to the +"tests" directory. Be sure to use the proper quoting convention so +that your shell does not perform the glob substitution. + - set TESTS {for-[24]*} +9. Incompatibilities with prior Tcl versions: +--------------------------------------------- -TESTS defaults to *, but you can change the default in "defs" if -you wish. +1) Global variables such as VERBOSE, TESTS, and testConfig are now + renamed to use the new "tcltest" namespace. -Saving keystrokes: ------------------- + old name new name + -------- -------- + VERBOSE ::tcltest::verbose + TESTS ::tcltest::match + testConfig ::tcltest::testConfig -A convenience procedure named "dotests" is included in file -"defs". It takes two arguments--the name of the test file (such -as "parse.test"), and a pattern selecting the tests you want to -execute. It sets TESTS to the second argument, calls "source" on -the file specified in the first argument, and restores TESTS to -its pre-call value at the end. + The introduction of the "tcltest" namespace is a precursor to using + a "tcltest" package. This next step will be part of a future Tcl + version. -Batch vs. interactive execution: --------------------------------- +2) VERBOSE values are no longer numeric. Please see the section above + on "Test output" for the new usage of the ::tcltest::verbose variable. -The tests can be run in either batch or interactive mode. Batch -mode refers to using I/O redirection from a UNIX shell. For example, -the following command causes the tests in the file named "parse.test" -to be executed: +3) When you run "make test", the working dir for the test suite is now + the one from which you called "make test", rather than the "tests" + directory. This change allows for both unix and windows test + suites to be run simultaneously without interference with each + other or with existing files. All tests must now run independently + of their working directory. - tclTest < parse.test > parse.test.results +4) The "all", "defs", and "visual" files are now called "all.tcl", + "defs.tcl", and "visual_bb.test", respectively. -Users who want to execute the tests in this fashion need to first -ensure that the file "defs" has proper values for the global -variables that control the testing environment (VERBOSE and TESTS). +5) Instead of creating a doAllTests file in the tests directory, to + run all nonPortable tests, just use the "-constraints nonPortable" + command line flag. If you are running interactively, you can set + the ::tcltest::testConfig(nonPortable) variable to 1 (after + sourcing the defs.tcl file). |