merged tcl 8.1 branch back into the main trunk

1 files changed, 408 insertions, 81 deletions

diff --git a/tests/README b/tests/README
index 07915c9..fe4bb30 100644
--- a/tests/README
+++ b/tests/README
@@ -1,96 +1,423 @@
-Tcl Test Suite
---------------
+README -- Tcl test suite design document.
 
-RCS: @(#) $Id: README,v 1.2 1998/09/14 18:40:07 stanton Exp $
+RCS: @(#) $Id: README,v 1.3 1999/04/16 00:47:22 stanton Exp $
 
-This directory contains a set of validation tests for the Tcl
-commands.  Each of the files whose name ends in ".test" is
-intended to fully exercise one or a few Tcl commands.  The
-commands tested by a given file are listed in the first line
-of the file.
+Contents:
+---------
+
+    1. Introduction
+    2. Definitions file
+    3. Writing a new test
+    4. Constraints
+    5. Adding a New Test File
+    6. Test output
+    7. Selecting tests for execution within a file
+    8. Selecting files to be sourced by all.tcl
+    9. Incompatibilities with prior Tcl versions
+
+1. Introduction:
+----------------
+
+This directory contains a set of validation tests for the Tcl commands
+and C Library procedures for Tcl.  Each of the files whose name ends
+in ".test" is intended to fully exercise the functions in the C source
+file that corresponds to the file prefix.  The C functions and/or Tcl
+commands tested by a given file are listed in the first line of the
+file.
+
+You can run the tests in three ways:
 
-You can run the tests in two ways:
     (a) type "make test" in ../unix; this will run all of the tests.
-    (b) start up tcltest in this directory, then "source" the test
+
+    (b) type "tcltest <testFile> ?<option> <value>?
+	Command line options include:
+
+	-verbose <level>     set the level of verbosity to a substring
+			     of "bps".  See the "Test output" section
+			     for an explanation of this option.
+
+	-match <matchList>   only run tests that match one or more of
+			     the glob patterns in <matchList>
+
+	-skip <skipList>     do not run tests that match one or more
+			     of the glob patterns in <skipList>
+
+	-file <globPattern>  only source test files that match
+			     <globPattern> (relative to the "tests"
+			     directory).  This option only applies
+			     when you run the test suite with the
+			     "all.tcl" file.
+
+	-constraints <list>  tests with any constraints in <list> will
+			     not be skipped.  Not that elements of
+			     <list> must exactly match the existing
+			     constraints.
+
+    (c) start up tcltest in this directory, then "source" the test
         file (for example, type "source parse.test").  To run all
-	of the tests, type "source all".
-In either case no output will be generated if all goes well, except
-for a listing of the tests..  If there are errors then additional
-messages will appear in the format described below.  Note: don't
-run the tests as superuser, since this will cause several of the tests
-to fail.
-
-The rest of this file provides additional information on the
-features of the testing environment.
-
-This approach to testing was designed and initially implemented
-by Mary Ann May-Pumphrey of Sun Microsystems.  Many thanks to
-her for donating her work back to the public Tcl release.
-
-Definitions file:
------------------
-
-The file "defs" defines a collection of procedures and variables
-used to run the tests.  It is read in automatically by each of the
-.test files if needed, but once it has been read once it will not
-be read again by the .test files.  If you change defs while running
-tests you'll have to "source" it by hand to load its new contents.
-
-Test output:
-------------
-
-Normally, output only appears when there are errors.  However, if
-the variable VERBOSE is set to 1 then tests will be run in "verbose"
-mode and output will be generated for each test regardless of
-whether it succeeded or failed.  Test output consists of the
-following information:
-
-    - the test identifier (which can be used to locate the test code
-	    in the .test file)
-    - a brief description of the test
-    - the contents of the test code
-    - the actual results produced by the tests
-    - a "PASSED" or "FAILED" message
-    - the expected results (if the test failed)
-
-You can set VERBOSE either interactively (after the defs file has been
-read in), or you can change the default value in "defs".
-
-Selecting tests for execution:
-------------------------------
+	of the tests, type "source all.tcl".  To use the options in
+	interactive mode, you can set their corresponding tcltest
+	namespace variables after sourcing the defs.tcl file.
+		  ::tcltest::match
+		  ::tcltest::skip
+		  ::tcltest::testConfig(nonPortable)
+		  ::tcltest::testConfig(knownBug)
+		  ::tcltest::testConfig(userInteractive)
+
+In all cases, no output will be generated if all goes well, except for
+a listing of the test files and a statistical summary.  If there are
+errors, then additional messages will appear in the format described
+below.  Note that some tests will be skipped if you run as superuser.
+
+This approach to testing was designed and initially implemented by
+Mary Ann May-Pumphrey of Sun Microsystems in the early 1990's.  Many
+thanks to her for donating her work back to the public Tcl release.
+
+
+2. Definitions file:
+--------------------
+
+The file "defs.tcl" defines the "tcltest" namespace which contains a
+collection of procedures and variables used to run the tests.  It is
+read in automatically by each of the .test files if needed, but once
+it has been read once it will not be read again by the .test files.
+Currently, the following procedures are exported from the "tcltest"
+namespace and automatically imported:
+
+    test	      Run a test script.
+
+    cleanupTests      Print stats and remove files created by tests.
+
+    dotests	      Source a test file and run tests of the
+		      specified pattern.
+
+    makeFile	      Create a file--the file will automatically
+		      be removed by cleanupTests.
+
+    removeFile	      Force a file to be removed.
+
+    makeDirectory     Create a directory--the directory will
+		      automatically be removed by cleanupTests.
+
+    removeDirectory   Force a directory to be removed.
+
+    viewFile	      Returns the contents of a file.
+
+    normalizeMsg      Remove extra newlines from a string.
+
+    bytestring	      Construct a string that consists of the
+		      requested sequence of bytes, as opposed to a
+		      string of properly formed UTF-8 characters.
+
+    set_iso8859_1_locale    Set the locale to iso8859_1.
+
+    restore_locale    Restore the locale to its original setting.
+
+    saveState	      Save the procedure and global variable names.
+
+    restoreState      Restore the procedure and global variable names.
+
+Please refer to the defs.tcl file for more documentation on these
+procedures.
+
+
+3. Writing a new test:
+----------------------
+
+The test procedure runs a test script and prints an error message if
+the script's result does not match the expected result.  The following
+is the spec for the "test" command:
+
+    test <name> <description> ?<constraint>? <script> <expectedAnswer>
+
+The <name> argument should follow the pattern,
+"<target>-<majorNum>.<minorNum>".  For white-box (regression) tests,
+the target should be the name of the c function or Tcl procedure being
+tested.  For black-box tests, the target should be the name of the
+feature being tested.  Related tests should share a major number.
+
+The <description> argument is a short textual description of the test,
+to help humans understand what it does.
+
+The optional <constraints> argument is list of one or more keywords,
+each of which must be the name of an element in the array
+"::tcltest::testConfig".  If any of these elements is false or does
+not exist, the test is skipped.  Add appropriate constraints (e.g.,
+unixOnly) to any tests that should not always be run.  For example, a
+test that should only be run on Unix should look like the following:
+
+    test getAttribute-1.1 {testing file permissions} {unixOnly} {
+        lindex [file attributes foo.tcl] 5
+    } {00644}
+
+See the "Constraints" section for a list of built-in
+constraints and information on how to add your own constraints.
+
+The <script> argument contains the script to run to carry out the
+test.  It must return a result that can be checked for correctness.
+If your script requires that a file be created on the fly, please use
+the ::tcltest::makeFile procedure.  If your test requires that a small
+file (<50 lines) be checked in, please consider creating the file on
+the fly using the ::tcltest::makeFile procedure.  Files created by the
+::tcltest::makeFile procedure will automatically be removed by the
+::tcltest::cleanupTests call at the end of each test file.
+
+The <expectedAnswer> argument will be compared against the result of
+evaluating the <script> argument.  If they match, the test passes,
+otherwise the test fails.
+
+
+4. Constraints:
+---------------
+
+Constraints are used to determine whether a test should be skipped.
+Each constraint is stored as an index in the array
+::tcltest::testConfig.  For example, the unixOnly constraint is
+defined as the following:
+
+    set ::tcltest::testConfig(unixOnly) \
+	[expr {$tcl_platform(platform) == "unix"}]
+
+If a test is constrained by "unixOnly", then it will only be run if
+the value of ::tcltest::testConfig(unixOnly) is true.
+
+The following is a list of constraints defined in the defs.tcl file:
+
+unix            test can only be run on any UNIX platform
+pc	        test can only be run on any Windows platform
+nt	        test can only be run on any Windows NT platform
+95	        test can only be run on any Windows 95 platform
+mac	        test can only be run on any Mac platform
+unixOrPc        test can only be run on a UNIX or PC platform
+macOrPc	        test can only be run on a Mac or PC platform
+macOrUnix       test can only be run on a Mac or UNIX platform
+tempNotPc	test can not be run on Windows.  This flag is used
+		to temporarily disable a test.
+tempNotMac	test can not be run on a Mac.  This flag is used
+		to temporarily disable a test.
+unixCrash       test crashes if it's run on UNIX.  This flag is used
+		to temporarily disable a test.
+pcCrash 	test crashes if it's run on Windows.  This flag is
+		used to temporarily disable a test.
+macCrash 	test crashes if it's run on a Mac.  This flag is used
+		to temporarily disable a test.
+
+emptyTest	test is empty, and so not worth running, but
+                it remains as a place-holder for a test to be
+                written in the future.  This constraint always
+                causes tests to be skipped.
+
+knownBug	test is known to fail and the bug is not yet
+                fixed.  This constraint always causes tests to be
+                skipped unless the user specifies otherwise.  See the
+                "Introduction" section for more details.
+
+nonPortable	test can only be run in the master Tcl/Tk
+		development environment.  Some tests are inherently
+		non-portable because they depend on things like word
+		length, file system configuration, window manager,
+		etc.  These tests are only run in the main Tcl
+		development directory where the configuration is
+		well known.  This constraint always causes tests to be
+		skipped unless the user specifies otherwise.  See the
+		"Introduction" section for more details.
+
+userInteraction test requires interaction from the user.  This
+                constraint always causes tests to be skipped unless
+                the user specifies otherwise.  See the "Introduction"
+                section for more details. 
+
+interactive	test can only be run in if the interpreter is in
+		interactive mode, that is the global tcl_interactive
+		variable is set to 1.
+
+nonBlockFiles	test can only be run if platform supports setting
+		files into nonblocking mode
+
+asyncPipeClose	test can only be run if platform supports async
+		flush and async close on a pipe
+
+unixExecs	test can only be run if this machine has commands
+		such as 'cat', 'echo', etc. available.
+
+hasIsoLocale	test can only be run if can switch to an ISO locale
+
+fonts		test can only be run if the wish app's fonts can
+		be controlled by Tk.
+
+root		test can only run if Unix user is root
+
+notRoot		test can only run if Unix user is not root
+
+eformat		test can only run if app has a working version of
+		sprintf with respect to the "e" format of
+		floating-point numbers.
+
+stdio		test can only be run if the current app can be
+		spawned via a pipe
+
+
+5. Adding a new test file:
+--------------------------
+
+Tests files should begin by sourcing the defs.tcl file:
+
+    if {[lsearch [namespace children] ::tcltest] == -1} {
+        source [file join [pwd] [file dirname [info script]] defs.tcl]
+    }
+
+Test files sould end by cleaning up after themselves and calling
+::tcltest::cleanupTests.  The ::tcltest::cleanupTests procedure prints
+statistics about the number of tests that passed, skipped, and failed,
+and removes all files that were created using the ::tcltest::makeFile
+and ::tcltest::makeDirectory procedures.
+
+    # Remove files created by these tests
+    # Change to original working directory
+    # Unset global arrays
+    ::tcltest::cleanupTests
+    return
+
+The all.tcl file will source your new test file if the filename
+matches the tests/*.test pattern (as it should).  The names of test
+files that contain regression (or glass-box) tests should correspond
+to the Tcl or C code file that they are testing.  For example, the
+test file for the C file "tclCmdAH.c" is "cmdAH.test".  Test files
+that contain black-box tests may not correspond to any Tcl or C code
+file so they should match the pattern "*_bb.test".
+
+Be sure your new test file can be run from any working directory.
+
+Be sure no temporary files are left behind by your test file.
+
+Be sure your tests can run cross-platform in both a build environment
+as well as an installation environment.  If your test file contains
+tests that should not be run in one or more of those cases, please use
+the constraints mechanism to skip those tests.
+
+
+6. Test output:
+---------------
+
+After all specified test files are sourced, the number of tests
+passed, skipped, and failed is printed to stdout.  Aside from this
+statistical information, output can be controlled on a per-test basis
+by the ::tcltest::verbose variable.
+
+::tcltest::verbose can be set to any substring or permutation of "bps".
+In the string "bps", the 'b' stands for a test's "body", the 'p'
+stands for "passed" tests, and the 's' stands for "skipped" tests.
+The default value of ::tcltest::verbose is "b".  If 'b' is present, then
+the entire body of the test is printed for each failed test, otherwise
+only the test's name, desired output, and actual output, are printed
+for each failed test.  If 'p' is present, then a line is printed for
+each passed test, otherwise no line is printed for passed tests.  If
+'s' is present, then a line (containing the consraints that cause the
+test to be skipped) is printed for each skipped test, otherwise no
+line is printed for skipped tests.
+
+You can set ::tcltest::verbose either interactively (after the defs.tcl
+file has been sourced) or by the command line argument -verbose, for
+example:
+
+      tcltest socket.test -verbose bps
+
+
+7. Selecting tests for execution within a file:
+-----------------------------------------------
 
 Normally, all the tests in a file are run whenever the file is
-"source"d.  However, you can select a specific set of tests using
-the global variable TESTS.  This variable contains a pattern;  any
-test whose identifier matches TESTS will be run.  For example,
-the following interactive command causes all of the "for" tests in
-groups 2 and 4 to be executed:
+sourced.  An individual test will be skipped if one of the following
+conditions is met:
+
+    1) the "name" of the tests does not match (using glob style
+       matching) one or more elements in the ::tcltest::match
+       variable
+
+    2) the "name" of the tests matches (using glob style matching) one
+       or more elements in the ::tcltest::skip variable
+
+    3) the "constraints" argument to the "test" call, if given,
+       contains one or more false elements.
+
+You can set ::tcltest::match and/or ::tcltest::skip
+either interactively (after the defs.tcl file has been sourced), or by
+the command line arguments -match and -skip, for example:
+
+       tcltest info.test -match '*-5.* *-7.*' -skip '*-7.1*'
+
+Be sure to use the proper quoting convention so that your shell does
+not perform the glob substitution on the match or skip patterns you
+specify.
+
+The two predefined constraints (knownBug and nonPortable) can be
+overridden either interactively (after the defs.tcl file has been
+sourced) by setting the ::tcltest::testConfig(<constraint>) variable,
+or by using the -constraints command line option with the name of the
+constraint in the argument.  The following example shows how to run
+tests that are constrained by the knownBug and nonPortable
+restricions:
+
+	tcltest all.tcl -constraints "knownBug nonPortable"
+
+See the defs.tcl file for information about each of these constraints.
+Other constraints can be added at any time.  See the "Writing a new
+test" section below for more details about using built-in constraints
+and adding new ones.
+
+
+8. Selecting files to be sourced by all.tcl:
+--------------------------------------------
+
+You can specify the files you want all.tcl to source on the command
+line with the -file options.  For example, if you call the
+following:
+
+     tcltest all.tcl -file 'unix*.test'
+
+all files in "tests" directory that match the pattern unix*.test will
+be sourced by the all.tcl file.  Another useful example is if a
+particular test hangs, say "get.test", and you just want to run the
+remaining tests, then you can call the following:
+
+     tcltest all.tcl -file '[h-z]*.test'
+
+Note that the argument to -file will be substituted relative to the
+"tests" directory.  Be sure to use the proper quoting convention so
+that your shell does not perform the glob substitution.
+
 
-    set TESTS {for-[24]*}
+9. Incompatibilities with prior Tcl versions:
+---------------------------------------------
 
-TESTS defaults to *, but you can change the default in "defs" if
-you wish.
+1) Global variables such as VERBOSE, TESTS, and testConfig are now
+   renamed to use the new "tcltest" namespace.
 
-Saving keystrokes:
-------------------
+   old name   new name
+   --------   --------
+   VERBOSE    ::tcltest::verbose
+   TESTS      ::tcltest::match
+   testConfig ::tcltest::testConfig
 
-A convenience procedure named "dotests" is included in file
-"defs".  It takes two arguments--the name of the test file (such
-as "parse.test"), and a pattern selecting the tests you want to
-execute.  It sets TESTS to the second argument, calls "source" on
-the file specified in the first argument, and restores TESTS to
-its pre-call value at the end.
+   The introduction of the "tcltest" namespace is a precursor to using
+   a "tcltest" package.  This next step will be part of a future Tcl
+   version.
 
-Batch vs. interactive execution:
---------------------------------
+2) VERBOSE values are no longer numeric.  Please see the section above
+   on "Test output" for the new usage of the ::tcltest::verbose variable.
 
-The tests can be run in either batch or interactive mode.  Batch
-mode refers to using I/O redirection from a UNIX shell.  For example,
-the following command causes the tests in the file named "parse.test"
-to be executed:
+3) When you run "make test", the working dir for the test suite is now
+   the one from which you called "make test", rather than the "tests"
+   directory.  This change allows for both unix and windows test
+   suites to be run simultaneously without interference with each
+   other or with existing files.  All tests must now run independently
+   of their working directory.
 
-    tclTest < parse.test > parse.test.results
+4) The "all", "defs", and "visual" files are now called "all.tcl",
+   "defs.tcl", and "visual_bb.test", respectively.
 
-Users who want to execute the tests in this fashion need to first
-ensure that the file "defs" has proper values for the global
-variables that control the testing environment (VERBOSE and TESTS).
+5) Instead of creating a doAllTests file in the tests directory, to
+   run all nonPortable tests, just use the "-constraints nonPortable"
+   command line flag.  If you are running interactively, you can set
+   the ::tcltest::testConfig(nonPortable) variable to 1 (after
+   sourcing the defs.tcl file).