summaryrefslogtreecommitdiffstats
path: root/tests/README
diff options
context:
space:
mode:
Diffstat (limited to 'tests/README')
-rw-r--r--tests/README489
1 files changed, 408 insertions, 81 deletions
diff --git a/tests/README b/tests/README
index 07915c9..fe4bb30 100644
--- a/tests/README
+++ b/tests/README
@@ -1,96 +1,423 @@
-Tcl Test Suite
---------------
+README -- Tcl test suite design document.
-RCS: @(#) $Id: README,v 1.2 1998/09/14 18:40:07 stanton Exp $
+RCS: @(#) $Id: README,v 1.3 1999/04/16 00:47:22 stanton Exp $
-This directory contains a set of validation tests for the Tcl
-commands. Each of the files whose name ends in ".test" is
-intended to fully exercise one or a few Tcl commands. The
-commands tested by a given file are listed in the first line
-of the file.
+Contents:
+---------
+
+ 1. Introduction
+ 2. Definitions file
+ 3. Writing a new test
+ 4. Constraints
+ 5. Adding a New Test File
+ 6. Test output
+ 7. Selecting tests for execution within a file
+ 8. Selecting files to be sourced by all.tcl
+ 9. Incompatibilities with prior Tcl versions
+
+1. Introduction:
+----------------
+
+This directory contains a set of validation tests for the Tcl commands
+and C Library procedures for Tcl. Each of the files whose name ends
+in ".test" is intended to fully exercise the functions in the C source
+file that corresponds to the file prefix. The C functions and/or Tcl
+commands tested by a given file are listed in the first line of the
+file.
+
+You can run the tests in three ways:
-You can run the tests in two ways:
(a) type "make test" in ../unix; this will run all of the tests.
- (b) start up tcltest in this directory, then "source" the test
+
+ (b) type "tcltest <testFile> ?<option> <value>?
+ Command line options include:
+
+ -verbose <level> set the level of verbosity to a substring
+ of "bps". See the "Test output" section
+ for an explanation of this option.
+
+ -match <matchList> only run tests that match one or more of
+ the glob patterns in <matchList>
+
+ -skip <skipList> do not run tests that match one or more
+ of the glob patterns in <skipList>
+
+ -file <globPattern> only source test files that match
+ <globPattern> (relative to the "tests"
+ directory). This option only applies
+ when you run the test suite with the
+ "all.tcl" file.
+
+ -constraints <list> tests with any constraints in <list> will
+ not be skipped. Not that elements of
+ <list> must exactly match the existing
+ constraints.
+
+ (c) start up tcltest in this directory, then "source" the test
file (for example, type "source parse.test"). To run all
- of the tests, type "source all".
-In either case no output will be generated if all goes well, except
-for a listing of the tests.. If there are errors then additional
-messages will appear in the format described below. Note: don't
-run the tests as superuser, since this will cause several of the tests
-to fail.
-
-The rest of this file provides additional information on the
-features of the testing environment.
-
-This approach to testing was designed and initially implemented
-by Mary Ann May-Pumphrey of Sun Microsystems. Many thanks to
-her for donating her work back to the public Tcl release.
-
-Definitions file:
------------------
-
-The file "defs" defines a collection of procedures and variables
-used to run the tests. It is read in automatically by each of the
-.test files if needed, but once it has been read once it will not
-be read again by the .test files. If you change defs while running
-tests you'll have to "source" it by hand to load its new contents.
-
-Test output:
-------------
-
-Normally, output only appears when there are errors. However, if
-the variable VERBOSE is set to 1 then tests will be run in "verbose"
-mode and output will be generated for each test regardless of
-whether it succeeded or failed. Test output consists of the
-following information:
-
- - the test identifier (which can be used to locate the test code
- in the .test file)
- - a brief description of the test
- - the contents of the test code
- - the actual results produced by the tests
- - a "PASSED" or "FAILED" message
- - the expected results (if the test failed)
-
-You can set VERBOSE either interactively (after the defs file has been
-read in), or you can change the default value in "defs".
-
-Selecting tests for execution:
-------------------------------
+ of the tests, type "source all.tcl". To use the options in
+ interactive mode, you can set their corresponding tcltest
+ namespace variables after sourcing the defs.tcl file.
+ ::tcltest::match
+ ::tcltest::skip
+ ::tcltest::testConfig(nonPortable)
+ ::tcltest::testConfig(knownBug)
+ ::tcltest::testConfig(userInteractive)
+
+In all cases, no output will be generated if all goes well, except for
+a listing of the test files and a statistical summary. If there are
+errors, then additional messages will appear in the format described
+below. Note that some tests will be skipped if you run as superuser.
+
+This approach to testing was designed and initially implemented by
+Mary Ann May-Pumphrey of Sun Microsystems in the early 1990's. Many
+thanks to her for donating her work back to the public Tcl release.
+
+
+2. Definitions file:
+--------------------
+
+The file "defs.tcl" defines the "tcltest" namespace which contains a
+collection of procedures and variables used to run the tests. It is
+read in automatically by each of the .test files if needed, but once
+it has been read once it will not be read again by the .test files.
+Currently, the following procedures are exported from the "tcltest"
+namespace and automatically imported:
+
+ test Run a test script.
+
+ cleanupTests Print stats and remove files created by tests.
+
+ dotests Source a test file and run tests of the
+ specified pattern.
+
+ makeFile Create a file--the file will automatically
+ be removed by cleanupTests.
+
+ removeFile Force a file to be removed.
+
+ makeDirectory Create a directory--the directory will
+ automatically be removed by cleanupTests.
+
+ removeDirectory Force a directory to be removed.
+
+ viewFile Returns the contents of a file.
+
+ normalizeMsg Remove extra newlines from a string.
+
+ bytestring Construct a string that consists of the
+ requested sequence of bytes, as opposed to a
+ string of properly formed UTF-8 characters.
+
+ set_iso8859_1_locale Set the locale to iso8859_1.
+
+ restore_locale Restore the locale to its original setting.
+
+ saveState Save the procedure and global variable names.
+
+ restoreState Restore the procedure and global variable names.
+
+Please refer to the defs.tcl file for more documentation on these
+procedures.
+
+
+3. Writing a new test:
+----------------------
+
+The test procedure runs a test script and prints an error message if
+the script's result does not match the expected result. The following
+is the spec for the "test" command:
+
+ test <name> <description> ?<constraint>? <script> <expectedAnswer>
+
+The <name> argument should follow the pattern,
+"<target>-<majorNum>.<minorNum>". For white-box (regression) tests,
+the target should be the name of the c function or Tcl procedure being
+tested. For black-box tests, the target should be the name of the
+feature being tested. Related tests should share a major number.
+
+The <description> argument is a short textual description of the test,
+to help humans understand what it does.
+
+The optional <constraints> argument is list of one or more keywords,
+each of which must be the name of an element in the array
+"::tcltest::testConfig". If any of these elements is false or does
+not exist, the test is skipped. Add appropriate constraints (e.g.,
+unixOnly) to any tests that should not always be run. For example, a
+test that should only be run on Unix should look like the following:
+
+ test getAttribute-1.1 {testing file permissions} {unixOnly} {
+ lindex [file attributes foo.tcl] 5
+ } {00644}
+
+See the "Constraints" section for a list of built-in
+constraints and information on how to add your own constraints.
+
+The <script> argument contains the script to run to carry out the
+test. It must return a result that can be checked for correctness.
+If your script requires that a file be created on the fly, please use
+the ::tcltest::makeFile procedure. If your test requires that a small
+file (<50 lines) be checked in, please consider creating the file on
+the fly using the ::tcltest::makeFile procedure. Files created by the
+::tcltest::makeFile procedure will automatically be removed by the
+::tcltest::cleanupTests call at the end of each test file.
+
+The <expectedAnswer> argument will be compared against the result of
+evaluating the <script> argument. If they match, the test passes,
+otherwise the test fails.
+
+
+4. Constraints:
+---------------
+
+Constraints are used to determine whether a test should be skipped.
+Each constraint is stored as an index in the array
+::tcltest::testConfig. For example, the unixOnly constraint is
+defined as the following:
+
+ set ::tcltest::testConfig(unixOnly) \
+ [expr {$tcl_platform(platform) == "unix"}]
+
+If a test is constrained by "unixOnly", then it will only be run if
+the value of ::tcltest::testConfig(unixOnly) is true.
+
+The following is a list of constraints defined in the defs.tcl file:
+
+unix test can only be run on any UNIX platform
+pc test can only be run on any Windows platform
+nt test can only be run on any Windows NT platform
+95 test can only be run on any Windows 95 platform
+mac test can only be run on any Mac platform
+unixOrPc test can only be run on a UNIX or PC platform
+macOrPc test can only be run on a Mac or PC platform
+macOrUnix test can only be run on a Mac or UNIX platform
+tempNotPc test can not be run on Windows. This flag is used
+ to temporarily disable a test.
+tempNotMac test can not be run on a Mac. This flag is used
+ to temporarily disable a test.
+unixCrash test crashes if it's run on UNIX. This flag is used
+ to temporarily disable a test.
+pcCrash test crashes if it's run on Windows. This flag is
+ used to temporarily disable a test.
+macCrash test crashes if it's run on a Mac. This flag is used
+ to temporarily disable a test.
+
+emptyTest test is empty, and so not worth running, but
+ it remains as a place-holder for a test to be
+ written in the future. This constraint always
+ causes tests to be skipped.
+
+knownBug test is known to fail and the bug is not yet
+ fixed. This constraint always causes tests to be
+ skipped unless the user specifies otherwise. See the
+ "Introduction" section for more details.
+
+nonPortable test can only be run in the master Tcl/Tk
+ development environment. Some tests are inherently
+ non-portable because they depend on things like word
+ length, file system configuration, window manager,
+ etc. These tests are only run in the main Tcl
+ development directory where the configuration is
+ well known. This constraint always causes tests to be
+ skipped unless the user specifies otherwise. See the
+ "Introduction" section for more details.
+
+userInteraction test requires interaction from the user. This
+ constraint always causes tests to be skipped unless
+ the user specifies otherwise. See the "Introduction"
+ section for more details.
+
+interactive test can only be run in if the interpreter is in
+ interactive mode, that is the global tcl_interactive
+ variable is set to 1.
+
+nonBlockFiles test can only be run if platform supports setting
+ files into nonblocking mode
+
+asyncPipeClose test can only be run if platform supports async
+ flush and async close on a pipe
+
+unixExecs test can only be run if this machine has commands
+ such as 'cat', 'echo', etc. available.
+
+hasIsoLocale test can only be run if can switch to an ISO locale
+
+fonts test can only be run if the wish app's fonts can
+ be controlled by Tk.
+
+root test can only run if Unix user is root
+
+notRoot test can only run if Unix user is not root
+
+eformat test can only run if app has a working version of
+ sprintf with respect to the "e" format of
+ floating-point numbers.
+
+stdio test can only be run if the current app can be
+ spawned via a pipe
+
+
+5. Adding a new test file:
+--------------------------
+
+Tests files should begin by sourcing the defs.tcl file:
+
+ if {[lsearch [namespace children] ::tcltest] == -1} {
+ source [file join [pwd] [file dirname [info script]] defs.tcl]
+ }
+
+Test files sould end by cleaning up after themselves and calling
+::tcltest::cleanupTests. The ::tcltest::cleanupTests procedure prints
+statistics about the number of tests that passed, skipped, and failed,
+and removes all files that were created using the ::tcltest::makeFile
+and ::tcltest::makeDirectory procedures.
+
+ # Remove files created by these tests
+ # Change to original working directory
+ # Unset global arrays
+ ::tcltest::cleanupTests
+ return
+
+The all.tcl file will source your new test file if the filename
+matches the tests/*.test pattern (as it should). The names of test
+files that contain regression (or glass-box) tests should correspond
+to the Tcl or C code file that they are testing. For example, the
+test file for the C file "tclCmdAH.c" is "cmdAH.test". Test files
+that contain black-box tests may not correspond to any Tcl or C code
+file so they should match the pattern "*_bb.test".
+
+Be sure your new test file can be run from any working directory.
+
+Be sure no temporary files are left behind by your test file.
+
+Be sure your tests can run cross-platform in both a build environment
+as well as an installation environment. If your test file contains
+tests that should not be run in one or more of those cases, please use
+the constraints mechanism to skip those tests.
+
+
+6. Test output:
+---------------
+
+After all specified test files are sourced, the number of tests
+passed, skipped, and failed is printed to stdout. Aside from this
+statistical information, output can be controlled on a per-test basis
+by the ::tcltest::verbose variable.
+
+::tcltest::verbose can be set to any substring or permutation of "bps".
+In the string "bps", the 'b' stands for a test's "body", the 'p'
+stands for "passed" tests, and the 's' stands for "skipped" tests.
+The default value of ::tcltest::verbose is "b". If 'b' is present, then
+the entire body of the test is printed for each failed test, otherwise
+only the test's name, desired output, and actual output, are printed
+for each failed test. If 'p' is present, then a line is printed for
+each passed test, otherwise no line is printed for passed tests. If
+'s' is present, then a line (containing the consraints that cause the
+test to be skipped) is printed for each skipped test, otherwise no
+line is printed for skipped tests.
+
+You can set ::tcltest::verbose either interactively (after the defs.tcl
+file has been sourced) or by the command line argument -verbose, for
+example:
+
+ tcltest socket.test -verbose bps
+
+
+7. Selecting tests for execution within a file:
+-----------------------------------------------
Normally, all the tests in a file are run whenever the file is
-"source"d. However, you can select a specific set of tests using
-the global variable TESTS. This variable contains a pattern; any
-test whose identifier matches TESTS will be run. For example,
-the following interactive command causes all of the "for" tests in
-groups 2 and 4 to be executed:
+sourced. An individual test will be skipped if one of the following
+conditions is met:
+
+ 1) the "name" of the tests does not match (using glob style
+ matching) one or more elements in the ::tcltest::match
+ variable
+
+ 2) the "name" of the tests matches (using glob style matching) one
+ or more elements in the ::tcltest::skip variable
+
+ 3) the "constraints" argument to the "test" call, if given,
+ contains one or more false elements.
+
+You can set ::tcltest::match and/or ::tcltest::skip
+either interactively (after the defs.tcl file has been sourced), or by
+the command line arguments -match and -skip, for example:
+
+ tcltest info.test -match '*-5.* *-7.*' -skip '*-7.1*'
+
+Be sure to use the proper quoting convention so that your shell does
+not perform the glob substitution on the match or skip patterns you
+specify.
+
+The two predefined constraints (knownBug and nonPortable) can be
+overridden either interactively (after the defs.tcl file has been
+sourced) by setting the ::tcltest::testConfig(<constraint>) variable,
+or by using the -constraints command line option with the name of the
+constraint in the argument. The following example shows how to run
+tests that are constrained by the knownBug and nonPortable
+restricions:
+
+ tcltest all.tcl -constraints "knownBug nonPortable"
+
+See the defs.tcl file for information about each of these constraints.
+Other constraints can be added at any time. See the "Writing a new
+test" section below for more details about using built-in constraints
+and adding new ones.
+
+
+8. Selecting files to be sourced by all.tcl:
+--------------------------------------------
+
+You can specify the files you want all.tcl to source on the command
+line with the -file options. For example, if you call the
+following:
+
+ tcltest all.tcl -file 'unix*.test'
+
+all files in "tests" directory that match the pattern unix*.test will
+be sourced by the all.tcl file. Another useful example is if a
+particular test hangs, say "get.test", and you just want to run the
+remaining tests, then you can call the following:
+
+ tcltest all.tcl -file '[h-z]*.test'
+
+Note that the argument to -file will be substituted relative to the
+"tests" directory. Be sure to use the proper quoting convention so
+that your shell does not perform the glob substitution.
+
- set TESTS {for-[24]*}
+9. Incompatibilities with prior Tcl versions:
+---------------------------------------------
-TESTS defaults to *, but you can change the default in "defs" if
-you wish.
+1) Global variables such as VERBOSE, TESTS, and testConfig are now
+ renamed to use the new "tcltest" namespace.
-Saving keystrokes:
-------------------
+ old name new name
+ -------- --------
+ VERBOSE ::tcltest::verbose
+ TESTS ::tcltest::match
+ testConfig ::tcltest::testConfig
-A convenience procedure named "dotests" is included in file
-"defs". It takes two arguments--the name of the test file (such
-as "parse.test"), and a pattern selecting the tests you want to
-execute. It sets TESTS to the second argument, calls "source" on
-the file specified in the first argument, and restores TESTS to
-its pre-call value at the end.
+ The introduction of the "tcltest" namespace is a precursor to using
+ a "tcltest" package. This next step will be part of a future Tcl
+ version.
-Batch vs. interactive execution:
---------------------------------
+2) VERBOSE values are no longer numeric. Please see the section above
+ on "Test output" for the new usage of the ::tcltest::verbose variable.
-The tests can be run in either batch or interactive mode. Batch
-mode refers to using I/O redirection from a UNIX shell. For example,
-the following command causes the tests in the file named "parse.test"
-to be executed:
+3) When you run "make test", the working dir for the test suite is now
+ the one from which you called "make test", rather than the "tests"
+ directory. This change allows for both unix and windows test
+ suites to be run simultaneously without interference with each
+ other or with existing files. All tests must now run independently
+ of their working directory.
- tclTest < parse.test > parse.test.results
+4) The "all", "defs", and "visual" files are now called "all.tcl",
+ "defs.tcl", and "visual_bb.test", respectively.
-Users who want to execute the tests in this fashion need to first
-ensure that the file "defs" has proper values for the global
-variables that control the testing environment (VERBOSE and TESTS).
+5) Instead of creating a doAllTests file in the tests directory, to
+ run all nonPortable tests, just use the "-constraints nonPortable"
+ command line flag. If you are running interactively, you can set
+ the ::tcltest::testConfig(nonPortable) variable to 1 (after
+ sourcing the defs.tcl file).