summaryrefslogtreecommitdiffstats
path: root/doc/tcltest2.n
diff options
context:
space:
mode:
Diffstat (limited to 'doc/tcltest2.n')
-rwxr-xr-xdoc/tcltest2.n1044
1 files changed, 1044 insertions, 0 deletions
diff --git a/doc/tcltest2.n b/doc/tcltest2.n
new file mode 100755
index 0000000..86c2f48
--- /dev/null
+++ b/doc/tcltest2.n
@@ -0,0 +1,1044 @@
+'\"
+'\" Copyright (c) 1990-1994 The Regents of the University of California
+'\" Copyright (c) 1994-1997 Sun Microsystems, Inc.
+'\" Copyright (c) 1998-1999 Scriptics Corporation
+'\" Copyright (c) 2000 Ajuba Solutions
+'\"
+'\" See the file "license.terms" for information on usage and redistribution
+'\" of this file, and for a DISCLAIMER OF ALL WARRANTIES.
+'\"
+'\" RCS: @(#) $Id: tcltest2.n,v 1.1 2000/09/20 23:09:49 jenn Exp $
+'\"
+.so man.macros
+.TH "tcltest" n 8.4 Tcl "Tcl Built-In Commands"
+.BS
+'\" Note: do not modify the .SH NAME line immediately below!
+.SH NAME
+tcltest \- Test harness support code and utilities
+.SH SYNOPSIS
+\fBpackage require tcltest ?2.0?\fP
+.sp
+\fBtcltest::test \fIname {?attribute value? ...}\fR
+.sp
+\fBtcltest::cleanupTests \fI?runningMultipleTests?\fR
+.sp
+\fBtcltest::runAllTests\fR
+.sp
+\fBtcltest::interpreter \fI?interp?\fR
+.sp
+\fBtcltest::singleProcess \fI?boolean?\fR
+.sp
+\fBtcltest::debug \fI?level?\fR
+.sp
+\fBtcltest::verbose \fI?level?\fR
+.sp
+\fBtcltest::preserveCore \fI?level?\fR
+.sp
+\fBtcltest::testConstraint \fIconstraint ?value?\fR
+.sp
+\fBtcltest::limitConstraints \fI?constraintList?\fR
+.sp
+\fBtcltest::workingDirectory \fI?dir?\fR
+.sp
+\fBtcltest::temporaryDirectory \fI?dir?\fR
+.sp
+\fBtcltest::testsDirectory \fI?dir?\fR
+.sp
+\fBtcltest::match \fI?patternList?\fR
+.sp
+\fBtcltest::matchFiles \fI?patternList?\fR
+.sp
+\fBtcltest::matchDirectories \fI?patternList?\fR
+.sp
+\fBtcltest::skip \fI?patternList?\fR
+.sp
+\fBtcltest::skipFiles \fI?patternList?\fR
+.sp
+\fBtcltest::skipDirectories \fI?patternList?\fR
+.sp
+\fBtcltest::loadTestedCommands\fR
+.sp
+\fBtcltest::loadScript \fI?script?\fR
+.sp
+\fBtcltest::loadFile \fI?filename?\fR
+.sp
+\fBtcltest::outputChannel \fI?channelID?\fR
+.sp
+\fBtcltest::outputFile \fI?filename?\fR
+.sp
+\fBtcltest::errorChannel \fI?channelID?\fR
+.sp
+\fBtcltest::errorFile \fI?filename?\fR
+.sp
+\fBtcltest::makeFile \fIcontents name\fR
+.sp
+\fBtcltest::removeFile \fIname\fR
+.sp
+\fBtcltest::makeDirectory \fIname\fR
+.sp
+\fBtcltest::removeDirectory \fIname\fR
+.sp
+\fBtcltest::viewFile \fIname\fR
+.sp
+\fBtcltest::normalizeMsg \fImsg\fR
+.sp
+\fBtcltest::normalizePath \fIpathVar\fR
+.sp
+\fBtcltest::bytestring \fIstring\fR
+.sp
+\fBtcltest::saveState\fR
+.sp
+\fBtcltest::restoreState\fR
+.sp
+\fBtcltest::threadReap\fR
+.BE
+.SH DESCRIPTION
+.PP
+The \fBtcltest\fR package provides the user with utility tools for
+writing and running tests in the Tcl test suite. It can also be used
+to create a customized test harness for an extension.
+.PP
+The Tcl test suite consists of multiple .test files, each of which
+contains multiple test cases. Each test case consists of a call to
+the test command, which specifies the name of test, a short
+description, any constraints that apply to the test case, the script
+to be run, and expected results. See the \fI"Tests"\fR section for more
+details.
+.PP
+It is also possible to add to this test harness to create your own
+customized test harness implementation. For more defails, see the
+section \fI"How to Customize the Test Harness"\fR.
+.SH COMMANDS
+.TP
+\fBtcltest::test\fP \fIname {?attribute value? ...}\fR
+The \fBtcltest::test\fR command runs the value supplied for attribute
+\fIscript\fR and compares its result to possible results.
+It prints an error message if actual results and expected results do
+not match. The \fBtcltest::test\fR command returns 0 if it completes
+successfully. Any other return value indicates that an error has
+occurred in the tcltest package. See the \fI"Tests"\fR section for
+more details on this command.
+.TP
+\fBtcltest::cleanupTests\fP \fI?runningMultipleTests?\fR
+This command should be called at the end of a test file. It prints
+statistics about the tests run and removes files that were created by
+\fBtcltest::makeDirectory\fR and \fBtcltest::makeFile\fR. Names
+of files and directories created outside of
+\fBtcltest::makeFile\fR and \fBtcltest::makeDirectory\fR and
+never deleted are printed to \fBtcltest::outputChannel\fR. This command
+also restores the original shell environment, as described by the ::env
+array. \fIcalledFromAll\fR should be specified if
+\fBtcltest::cleanupTests\fR is called explicitly from an "all.tcl"
+file. Tcl files files are generally used to run multiple tests. For
+more details on how to run multiple tests, please see the section
+\fI"Running test files"\fR. This proc has no defined return value.
+.TP
+\fBtcltest::runAllTests\fP
+This command should be used in your 'all.tcl' file. It is used to
+loop over test files and directories, determining which test files to
+run and then running them. Note that this test calls
+tcltest::cleanupTests; if using this proc in your 'all.tcl' file, you
+should not call tcltest::cleanupTests explicitly in that file. See the
+sample 'all.tcl' file in the \fI"Examples"\fR section.
+.TP
+\fBtcltest::interpreter\fR \fI?executableName?\fR
+Sets or returns the name of the executable used to invoke the test
+suite. This is the interpreter used in runAllTests to run test files
+if singleProcess is set to false. The default value for interpreter
+is the name of the interpreter in which the tests were started.
+.TP
+\fBtcltest::singleProcess\fR \fI?boolean?\fR
+Sets or returns a boolean indicating whether test files should be sourced
+into the current interpreter by runAllTests or run in their own
+processes. If \fIboolean\fR is true (1), tests are sourced into the
+current interpreter. If \fIboolean\fR is false (0), tests are run in
+the interpreter specified in tcltest::interpreter. The default value
+for tcltest::singleProcess is false.
+.TP
+\fBtcltest::debug\fR \fI?level?\fR
+Sets or returns the current debug level. The debug level determines
+how much tcltest package debugging information is printed to stdout.
+The default debug level is 0. Levels are defined as:
+.RS
+.IP 0
+Do not display any debug information.
+.IP 1
+Display information regarding whether a test is skipped because it
+doesn't match any of the tests that were specified using -match or
+tcltest::match (userSpecifiedNonMatch) or matches any of the tests
+specified by -skip or tcltest::skip (userSpecifiedSkip).
+.IP 2
+Display the flag array parsed by the command line processor, the
+contents of the ::env array, and all user-defined variables that exist
+in the current namespace as they are used.
+.IP 3
+Display information regarding what individual procs in the test
+harness are doing.
+.RE
+.TP
+\fBtcltest::verbose\fR \fI?level?\fR
+Sets or returns the current verbosity level. This level must be a
+substring of "bpst". The default verbosity level is "b". See the
+"Test output" section for a more detailed explanation of this
+option. Levels are defined as:
+.RS
+.IP b
+Display the body of failed tests
+.IP p
+Print output when a test passes
+.IP s
+Print output when a test is skipped
+.IP t
+Print output whenever a test starts
+.RE
+.TP
+\fBtcltest::preserveCore\fR \fI?level?\fR
+Sets or returns the current core preservation level. This level
+determines how stringent checks for core files are. The default core
+preservation level is 0. Levels are defined as:
+.RS
+.IP 0
+No checking - do not check for core files at the end of each test
+command, but do check for them whenever tcltest::cleanupTests is
+called from tcltest::runAllTests.
+.IP 1
+Check for core files at the end of each test command and whenever
+tcltest::cleanupTests is called from tcltest::runAllTests.
+.IP 2
+Check for core files at the end of all test commands and whenever
+tcltest::cleanupTests is called from all.tcl. Save any core files
+produced in tcltest::temporaryDirectory.
+.RE
+.TP
+\fBtcltest::testConstraint \fIconstraint ?value?\fR
+Sets or returns the value associated with the named \fIconstraint\fR.
+See the section \fI"Test constraints"\fR for more information.
+.TP
+\fBtcltest::limitConstraints \fI?constraintList?\fR
+Sets or returns a boolean indicating whether testing is being limited
+to constraints listed in \fIconstraintList\fR.
+If limitConstraints is not false, only those tests with constraints matching
+values in \fIconstraintList\fR will be run.
+.TP
+\fBtcltest::workingDirectory\fR \fI?directoryName?\fR
+Sets or returns the directory in which the test suite is being run.
+The default value for workingDirectory is the directory in which the
+test suite was launched.
+.TP
+\fBtcltest::temporaryDirectory\fR \fI?directoryName?\fR
+Sets or returns the output directory for temporary files created by
+tcltest::makeFile and tcltest::makeDirectory. This defaults to the
+directory returned by \fItcltest::workingDirectory\fR.
+.TP
+\fBtcltest::testsDirectory\fR \fI?directoryName?\fR
+Sets or returns the directory where the tests reside. This defaults
+to the directory returned by \fItcltest::workingDirectory\fR
+if the script cannot determine where the \fItests\fR directory is
+located. This variable should be explicitly set if tests are being run
+from an all.tcl file.
+.TP
+\fBtcltest::match\fR \fI?globPatternList\fR
+Sets or returns the glob pattern list that determines which tests
+should be run. Only tests which match one of the glob patterns in
+\fIglobPatternList\fR are run by the test harness. The default value
+for \fIglobPatternList\fR is '*'.
+.TP
+\fBtcltest::matchFiles\fR \fI?globPatternList\fR
+Sets or returns the glob pattern list that determines which test files
+should be run. Only test files which match one of the glob patterns in
+\fIglobPatternList\fR are run by the test harness. The default value
+for \fIglobPatternList\fR is '*.test'.
+.TP
+\fBtcltest::matchDirectories\fR \fI?globPatternList\fR
+Sets or returns the glob pattern list that determines which test
+subdirectories of the current test directory should be run. Only test
+subdirectories which match one of the glob patterns in
+\fIglobPatternList\fR are run by the test harness. The default value
+for \fIglobPatternList\fR is '*'.
+.TP
+\fBtcltest::skip\fR \fI?globPatternList\fR
+Sets or returns the glob pattern list that determines which tests (of
+those matched by tcltest::match) should be skipped. The default value
+for \fIglobPatternList\fR is {}.
+.TP
+\fBtcltest::skipFiles\fR \fI?globPatternList\fR
+Sets or returns the glob pattern list that determines which test files (of
+those matched by tcltest::matchFiles) should be skipped. The default value
+for \fIglobPatternList\fR is {}.
+.TP
+\fBtcltest::skipDirectories\fR \fI?globPatternList\fR
+Sets or returns the glob pattern list that determines which test subdirectories (of
+those matched by tcltest::matchDirectories) should be skipped. The default value
+for \fIglobPatternList\fR is {}.
+.TP
+\fBtcltest::loadTestedCommands\fP
+This command uses the script specified via the \fI-load\fR or
+\fI-loadfile\fR to load the commands checked by the test suite.
+Allowed to be empty, as the tested commands could have been compiled
+into the interpreter running the test suite.
+.TP
+\fBtcltest::loadScript\fR \fI?script\fR
+Sets or returns the script executed by \fBloadTestedCommands\fR.
+.TP
+\fBtcltest::loadFile\fR \fI?filename?\fR
+Sets ore returns the file name associated with the script executed
+\fBloadTestedCommands\fR. If setting \fIfilename\fR, this proc will
+open the file and call \fItcltest::loadScript\fR with the content.
+.TP
+\fBtcltest::outputChannel\fR \fI?channelID?\fR
+Sets or returns the output file ID. This defaults to stdout.
+Any test that prints test related output should send
+that output to \fItcltest::outputChannel\fR rather than letting
+that output default to stdout.
+.TP
+\fBtcltest::outputFile\fR \fI?filename?\fR
+Sets or returns the file name corresponding to the output file. This
+defaults to stdout. This proc calls
+outputChannel to set the output file channel.
+Any test that prints test related output should send
+that output to \fItcltest::outputChannel\fR rather than letting
+that output default to stdout.
+.TP
+\fBtcltest::errorChannel\fR \fI?channelID?\fR
+Sets or returns the error file ID. This defaults to stderr.
+Any test that prints error messages should send
+that output to \fItcltest::errorChannel\fR rather than printing
+directly to stderr.
+.TP
+\fBtcltest::errorFile\fR \fI?filename?\fR
+Sets or returns the file name corresponding to the error file. This
+defaults to stderr. This proc calls
+errorChannel to set the error file channel.
+Any test that prints test related error output should send
+that output to \fItcltest::errorChannel\fR or
+\fItcltest::outputChannel\fR rather than letting
+that output default to stdout.
+.TP
+\fBtcltest::mainThread\fR
+Sets or returns the main thread ID. This defaults to 1. This is the
+only thread that is not killed by tcltest::threadReap and is set
+according to the return value of \fItestthread names\fR at
+initialization.
+.TP
+\fBtcltest::makeFile\fP \fIcontents name\fR
+Create a file that will be automatically be removed by
+\fBtcltest::cleanupTests\fR at the end of a test file. This file is
+created relative to tcltest::temporaryDirectory.
+Returns the full path of the file created.
+.TP
+\fBtcltest::removeFile\fP \fIname\fR
+Force the file referenced by \fIname\fR to be removed. This file name
+should be relative to \fItcltest::temporaryDirectory\fR. This proc has no
+defined return values.
+.TP
+\fBtcltest::makeDirectory\fP \fIname\fR
+Create a directory named \fIname\fR that will automatically be removed
+by \fBtcltest::cleanupTests\fR at the end of a test file. This
+directory is created relative to tcltest::temporaryDirectory.
+Returns the full path of the directory created.
+.TP
+\fBtcltest::removeDirectory\fP \fIname\fR
+Force the directory referenced by \fIname\fR to be removed. This
+directory should be relative to tcltest::temporaryDirectory. This proc
+has no defined return value.
+.TP
+\fBtcltest::viewFile\fP \fIfile\fR
+Returns the contents of \fIfile\fR.
+.TP
+\fBtcltest::normalizeMsg\fP \fImsg\fR
+Remove extra newlines from \fImsg\fR.
+.TP
+\fBtcltest::normalizePath\fP \fIpathVar\fR
+Resolves symlinks in a path, thus creating a path without internal
+redirection. It is assumed that \fIpathVar\fR is absolute.
+\fIpathVar\fR is modified in place.
+.TP
+\fBtcltest::bytestring\fP \fIstring\fR
+Construct a string that consists of the requested sequence of bytes,
+as opposed to a string of properly formed UTF-8 characters using the
+value supplied in \fIstring\fR. This allows the tester to create
+denormalized or improperly formed strings to pass to C procedures that
+are supposed to accept strings with embedded NULL types and confirm
+that a string result has a certain pattern of bytes.
+.TP
+\fBtcltest::saveState\fP
+Save procedure and global variable names.
+A test file might contain calls to \fBtcltest::saveState\fR and
+\fB::tcltest:restoreState\fR if it creates or deletes global variables
+or procs.
+.TP
+\fBtcltest::restoreState\fP
+Restore procedure and global variable names.
+A test file might contain calls to \fBtcltest::saveState\fR and
+\fB::tcltest:restoreState\fR if it creates or deletes global variables
+or procs.
+.TP
+\fBtcltest::threadReap\fP
+\fBtcltest::threadReap\fR only works if \fItestthread\fR is
+defined, generally by compiling tcltest. If \fItestthread\fR is
+defined, \fBtcltest::threadReap\fR kills all threads except for the
+main thread. It gets the ID of the main thread by calling
+\fItestthread names\fR during initialization. This value is stored in
+\fItcltest::mainThread\fR. \fBtcltest::threadReap\fR returns the
+number of existing threads at completion.
+.SH TESTS
+The \fBtest\fR procedure runs a test script and prints an error
+message if the script's result does not match the expected result.
+The following is the spec for the \fBtest\fR command:
+.DS
+test \fIname\fR {
+ description \fIdescription\fR
+ ?constraints \fIkeywordList|expression\fR
+ ?setup \fIsetupScript\fR?
+ ?script \fItestScript\fR?
+ ?cleanup \fIcleanupScript\fR?
+ ?expect \fIexpectedAnswer\fR? |
+ ?expect {-exact \fIexpectedAnswer\fR?}? |
+ ?expect {-regexp \fIexpectedAnswer\fR?}? |
+ ?expect {-glob \fIexpectedAnswer\fR?}?
+ ?expect_out \fIexpectedOutput\fR? |
+ ?expect_out {-exact \fIexpectedOutput\fR?}? |
+ ?expect_out {-regexp \fIexpectedOutput\fR?}? |
+ ?expect_out {-glob \fIexpectedOutput\fR?}?
+ ?expect_err \fIexpectedError\fR? |
+ ?expect_err {-exact \fIexpectedError\fR?}? |
+ ?expect_err {-regexp \fIexpectedError\fR?}? |
+ ?expect_err {-glob \fIexpectedError\fR?}?
+ ?expect_codes \fIcodeList\fR
+}
+.DE
+The \fIname\fR argument should follow the pattern:
+.DS
+<target>-<majorNum>.<minorNum>
+.DE
+For white-box (regression) tests, the target should be the name of the
+C function or Tcl procedure being tested. For black-box tests, the
+target should be the name of the feature being tested. Related tests
+should share a major number.
+.PP
+Valid attributes and associated values are:
+.TP
+\fBdescription \fIdesc\fR\fP
+A value must be provided for the required \fIdescription\fR attribute.
+This should be a short textual description of the test used to help humans
+understand the purpose of the test. The name of a Tcl or C function
+being tested should be included for regression tests. If the test
+case exists to reproduce a bug, include the bug ID in the description.
+.TP
+\fBconstraints \fIkeywordList|expression\fR\fP
+The optional \fIconstraints\fR attribute can be list of one or more
+keywords or an expression. If the \fIconstraints\fR value consists of
+keywords, each of these keywords being the name of a constraint
+defined by a call to \fItcltest::testConstraint\fR. If any of these
+elements is false or does
+not exist, the test is skipped. If the \fIconstraints\fR argument
+consists of an expression, that expression is evaluated. If the
+expression evaluates to true, then the test is run. Appropriate
+constraints should be added to any tests that should
+not always be run. See the "Test Constraints" section for a list of built-in
+constraints and information on how to add your own constraints.
+.TP
+\fBsetup \fIscript\fR\fP
+The optional \fIsetup\fR attribute indicates a script that will be run
+before the script indicated by the \fIscript\fR attribute. If setup
+fails, the test will fail.
+.TP
+\fBscript \fIscript\fR\fP
+The \fIscript\fR attribute indicates the script to run to carry out the
+test. It must return a result that can be checked for correctness.
+.TP
+\fBcleanup \fIscript\fR\fP
+The optional \fIcleanup\fR attribute indicates a script that will be
+run after the script indicated by the \fIscript\fR attribute. If
+cleanup fails, the test will fail.
+.TP
+\fBexpect \fI{?option? expectedValue}|expectedValue\fR\fP
+The \fIexpect\fR attribute supplies the comparison value with which
+the return value from script will be compared. Valid options for the
+value supplied are "-regexp", "-glob", and "-exact". The default
+comparison option is "-exact".
+.TP
+\fBexpect_out \fI{?option? expectedValue}|expectedValue\fR\fP
+The \fIexpect_out\fR attribute supplies the comparison value with which
+any output sent to stdout or tcltest::outputChannel during the script
+run will be compared. Valid options for the
+value supplied are "-regexp", "-glob", and "-exact". The default
+comparison option is "-exact". Note that only output printed using
+puts is used for comparison.
+.TP
+\fBexpect_err \fI{?option? expectedValue}|expectedValue\fR\fP
+The \fIexpect_err\fR attribute supplies the comparison value with which
+any output sent to stderr or tcltest::errorChannel during the script
+run will be compared. Valid options for the
+value supplied are "-regexp", "-glob", and "-exact". The default
+comparison option is "-exact". Note that only output printed using
+puts is used for comparison.
+.TP
+\fBexpect_codes \fIexpectedCodes\fR\fP
+The optional \fIexpect_code\fR attribute indicates which return codes
+from the script supplied with the \fIscript\fR attribute are correct.
+Default values for \fIexpectedCodes\fR are 0 (normal return) and 2
+(return exception).
+.PP
+The <expectedAnswer> argument will be compared against the result of
+evaluating the <script> argument. If they match, the test passes,
+otherwise the test fails.
+.SH "TEST CONSTRAINTS"
+Constraints are used to determine whether or not a test should be skipped.
+If a test is constrained by "unixOnly", then it will only be run if
+the value of the constraint is true. Several
+constraints are defined in the \fBtcltest\fR package. To add
+constraints, you can call \fBtcltest::testConstraint\fR
+with the appropriate arguments in your own test file.
+.PP
+The following is a list of constraints defined in the \fBtcltest\fR package:
+.TP
+\fIsingleTestInterp\fR
+test can only be run if all test files are sourced into a single interpreter
+.TP
+\fIunix\fR
+test can only be run on any UNIX platform
+.TP
+\fIpc\fR
+test can only be run on any Windows platform
+.TP
+\fInt\fR
+test can only be run on any Windows NT platform
+.TP
+\fI95\fR
+test can only be run on any Windows 95 platform
+.TP
+\fI98\fR
+test can only be run on any Windows 98 platform
+.TP
+\fImac\fR
+test can only be run on any Mac platform
+.TP
+\fIunixOrPc\fR
+test can only be run on a UNIX or PC platform
+.TP
+\fImacOrPc\fR
+test can only be run on a Mac or PC platform
+.TP
+\fImacOrUnix\fR
+test can only be run on a Mac or UNIX platform
+.TP
+\fItempNotPc\fR
+test can not be run on Windows. This flag is used to temporarily
+disable a test.
+.TP
+\fItempNotMac\fR
+test can not be run on a Mac. This flag is used
+to temporarily disable a test.
+.TP
+\fIunixCrash\fR
+test crashes if it's run on UNIX. This flag is used to temporarily
+disable a test.
+.TP
+\fIpcCrash\fR
+test crashes if it's run on Windows. This flag is used to temporarily
+disable a test.
+.TP
+\fImacCrash\fR
+test crashes if it's run on a Mac. This flag is used to temporarily
+disable a test.
+.TP
+\fIemptyTest\fR
+test is empty, and so not worth running, but it remains as a
+place-holder for a test to be written in the future. This constraint
+always causes tests to be skipped.
+.TP
+\fIknownBug\fR
+test is known to fail and the bug is not yet fixed. This constraint
+always causes tests to be skipped unless the user specifies otherwise.
+See the "Introduction" section for more details.
+.TP
+\fInonPortable\fR
+test can only be run in the master Tcl/Tk development environment.
+Some tests are inherently non-portable because they depend on things
+like word length, file system configuration, window manager, etc.
+These tests are only run in the main Tcl development directory where
+the configuration is well known. This constraint always causes tests
+to be skipped unless the user specifies otherwise.
+.TP
+\fIuserInteraction\fR
+test requires interaction from the user. This constraint always
+causes tests to be skipped unless the user specifies otherwise.
+.TP
+\fIinteractive\fR
+test can only be run in if the interpreter is in interactive mode
+(when the global tcl_interactive variable is set to 1).
+.TP
+\fInonBlockFiles\fR
+test can only be run if platform supports setting files into
+nonblocking mode
+.TP
+\fIasyncPipeClose\fR
+test can only be run if platform supports async flush and async close
+on a pipe
+.TP
+\fIunixExecs\fR
+test can only be run if this machine has commands such as 'cat', 'echo',
+etc. available.
+.TP
+\fIhasIsoLocale\fR
+test can only be run if can switch to an ISO locale
+.TP
+\fIroot\fR
+test can only run if Unix user is root
+.TP
+\fInotRoot\fR
+test can only run if Unix user is not root
+.TP
+\fIeformat\fR
+test can only run if app has a working version of sprintf with respect
+to the "e" format of floating-point numbers.
+.TP
+\fIstdio\fR
+test can only be run if the current app can be spawned via a pipe
+.SH "RUNNING TEST FILES"
+Use the following command to run a test file that uses package
+tcltest:
+.DS
+<shell> <testFile> ?<option> ?<value>?? ...
+.DE
+Command line options include (tcltest accessor procs that
+correspond to each flag are listed at the end of each flag description
+in parenthesis):
+.RS
+.TP
+\fB-help\fR
+display usage information.
+.TP
+\fB-singleproc 0|1\fR
+if 0, run test files in multiple interpreters. if 1, source test
+files into the current intpreter.
+.TP
+\fB-verbose <level>\fR
+set the level of verbosity to a substring of "bpst". See the "Test
+output" section for an explanation of this option. (tcltest::verbose)
+.TP
+\fB-match <matchList>\fR
+only run tests that match one or more of the glob patterns in
+<matchList>. (tcltest::match)
+.TP
+\fB-skip <skipList>\fR
+do not run tests that match one or more of the glob patterns in
+<skipList>. (tcltest::skip)
+.TP
+\fB-file <globPatternList>\fR
+only source test files that match any of the items in
+<globPatternList> relative to tcltest::testsDirectory.
+This option
+only makes sense if you are running tests using "all.tcl" as the
+<testFile> instead of running single test files directly.
+(tcltest::matchFiles)
+.TP
+\fB-notfile <globPatternList>\fR
+source files except for those that match any of the items in
+<globPatternList> relative to tcltest::testsDirectory.
+This option
+only makes sense if you are running tests using "all.tcl" as the
+<testFile> instead of running single test files directly.
+(tcltest::skipFiles)
+.TP
+\fB-relateddir <globPatternList>\fR
+only run tests in directories that match any of the items in
+<globPatternList> relative to tcltest::testsDirectory.
+This option
+only makes sense if you are running tests using "all.tcl" as the
+<testFile> instead of running single test files directly.
+(tcltest::matchDirectories)
+.TP
+\fB-asidefromdir <globPatternList>\fR
+run tests in directories except for those that match any of the items in
+<globPatternList> relative to tcltest::testsDirectory.
+This option
+only makes sense if you are running tests using "all.tcl" as the
+<testFile> instead of running single test files directly.
+(tcltest::skipDirectories)
+.TP
+\fB-constraints <list>\fR
+tests with any constraints in <list> will not be skipped. Note that
+elements of <list> must exactly match the existing constraints. This
+is useful if you want to make sure that tests with a particular
+constraint are run (for example, if the tester wants to run all tests
+with the knownBug constraint).
+(tcltest::testConstraint)
+.TP
+\fB-limitconstraints <bool>\fR
+If the argument to this flag is 1, the test harness limits test runs
+to those tests that match the constraints listed by the -constraints
+flag. Use of this flag requires use of the -constraints flag. The
+default value for this flag is 0 (false). This is useful if you want
+to run \fBonly\fR those tests that match the constraints listed using
+the -constraints option. A tester might want to do this if (for
+example) he were
+interested in running only those tests that are constrained to be
+unixOnly and no other tests.
+(tcltest::limitConstraints)
+.TP
+\fB-load <script>\fR
+will use the specified script to load the commands under test
+(tcltest::loadTestedCommands). The default is the empty
+script. See -loadfile below too. (tcltest::loadScript)
+.TP
+\fB-loadfile <scriptfile>\fR
+will use the contents of the named file to load the commands under
+test (tcltest::loadTestedCommands). See -load above too. The default
+is the empty script. (tcltest::loadScript)
+.TP
+\fB-tmpdir <directoryName>\fR
+put any temporary files (created with tcltest::makeFile and
+tcltest::makeDirectory) into the named directory. The default
+location is tcltest::workingDirectory. (tcltest::temporaryDirectory)
+.TP
+\fB-testdir <directoryName>\fR
+search the test suite to execute in the named directory. The default
+location is tcltest::workingDirectory. (tcltest::testsDirectory)
+.TP
+\fB-preservecore <level>\fR
+check for core files. This flag is used to determine how much
+checking should be done for core files. (tcltest::preserveCore)
+.TP
+\fB-debug <debugLevel>\fR
+print debug information to stdout. This is used to debug code in the
+tcltest package. (tcltest::debug)
+.TP
+\fB-outfile <filename>\fR
+print output generated by the tcltest package to the named file. This
+defaults to stdout. Note that debug output always goes to stdout,
+regardless of this flag's setting. (tcltest::outputChannel)
+.TP
+\fB-errfile <filename>\fR
+print errors generated by the tcltest package to the named file. This
+defaults to stderr. (tcltest::errorChannel)
+.RE
+.PP
+You can specify any of the above options on the command line or by
+defining an environment variable named TCLTEST_OPTIONS containing a
+list of options (e.g. "-debug 3 -verbose 'ps'"). This environment
+variable is evaluated before the command line arguments. Options
+specified on the command line override those specified in
+TCLTEST_OPTIONS.
+.PP
+A second way to run tets is to start up a shell, load the
+\fBtcltest\fR package, and then source an appropriate test file or use
+the test command. To use the options in interactive mode, set
+their corresponding tcltest namespace variables after loading the
+package.
+.PP
+See \fI"Test Constraints"\fR for a list of all built-in constraint names.
+.PP
+A final way to run tests would be to specify which test files to run
+within an \fIall.tcl\fR (or otherwise named) file. This is the
+approach used by the Tcl test suite. This file loads the tcltest
+package, sets the location of
+the test directory (tcltest::testsDirectory), determines which test
+files to run, sources each of these files, calls
+tcltest::cleanupTests and then exits.
+.PP
+A more elaborate \fIall.tcl\fR file might do some pre- and
+post-processing before sourcing
+each .test file, use separate interpreters for each file, or handle
+complex directory structures.
+For an example of an all.tcl file,
+please see the "Examples" section of this document.
+.SH "TEST OUTPUT"
+After all specified test files are run, the number of tests
+passed, skipped, and failed is printed to
+\fBtcltest::outputChannel\fR. Aside from this
+statistical information, output can be controlled on a per-test basis
+by the \fBtcltest::verbose\fR variable.
+.PP
+\fBtcltest::verbose\fR can be set to any substring or permutation
+of "bpst". In the string "bpst", the 'b' stands for a test's "body",
+the 'p' stands for "passed" tests, the 's' stands for "skipped"
+tests, and the 't' indicates when a test "starts".
+The default value of \fBtcltest::verbose\fR is "b". If 'b'
+is present, then the entire body of the test is printed for each
+failed test, otherwise only the test's name, desired output, and
+actual output, are printed for each failed test. If 'p' is present,
+then a line is printed for each passed test, otherwise no line is
+printed for passed tests. If 's' is present, then a line (containing
+the consraints that cause the test to be skipped) is printed for each
+skipped test, otherwise no line is printed for skipped tests. If 't'
+is present, then a line is printed each time a new test starts.
+.PP
+You can set \fBtcltest::verbose\fR either interactively (after the
+\fBtcltest\fR package has been loaded) or by using the command line
+argument \fB-verbose\fR, for example:
+.DS
+tclsh socket.test -verbose bps
+.DE
+.SH "CONTENTS OF A TEST FILE"
+Test files should begin by loading the \fBtcltest\fR package:
+.DS
+package require tcltest
+namespace import -force tcltest::*
+.DE
+Test files should end by cleaning up after themselves and calling
+\fBtcltest::cleanupTests\fR. The \fBtcltest::cleanupTests\fR
+procedure prints statistics about the number of tests that passed,
+skipped, and failed, and removes all files that were created using the
+\fBtcltest::makeFile\fR and \fBtcltest::makeDirectory\fR procedures.
+.DS
+# Remove files created by these tests
+# Change to original working directory
+# Unset global arrays
+tcltest::cleanupTests
+return
+.DE
+When naming test files, file names should end with a .test extension.
+The names of test files that contain regression (or glass-box) tests
+should correspond to the Tcl or C code file that they are testing.
+For example, the test file for the C file "tclCmdAH.c" is "cmdAH.test".
+.SH "SELECTING TESTS FOR EXECUTION"
+.PP
+Normally, all the tests in a file are run whenever the file is
+sourced. An individual test will be skipped if one of the following
+conditions is met:
+.IP [1]
+the \fIname\fR of the tests does not match (using glob style matching)
+one or more elements in the \fBtcltest::match\fR variable
+.IP [2]
+the \fIname\fR of the tests matches (using glob style matching) one or
+more elements in the \fBtcltest::skip\fR variable
+.IP [3]
+the \fIconstraints\fR argument to the \fBtcltest::test\fR call, if
+given, contains one or more false elements.
+.PP
+You can set \fBtcltest::match\fR and/or \fBtcltest::skip\fR
+either interactively (after the \fBtcltest\fR package has been
+sourced), or by using the command line arguments \fB-match\fR and
+\fB-skip\fR, for example:
+.PP
+.CS
+tclsh info.test -match '*-5.* *-7.*' -skip '*-7.1*'
+.CE
+.PP
+Be sure to use the proper quoting convention so that your shell does
+not perform the glob substitution on the match or skip patterns you
+specify.
+.PP
+Predefined constraints (e.g. \fIknownBug\fR and \fInonPortable\fR) can be
+overridden either interactively (after the \fBtcltest\fR package has been
+sourced) by setting the proper constraint
+or by using the \fB-constraints\fR command line option with the name of the
+constraint in the argument. The following example shows how to run
+tests that are constrained by the \fIknownBug\fR and \fInonPortable\fR
+restrictions:
+.PP
+.CS
+tclsh all.tcl -constraints "knownBug nonPortable"
+.CE
+.PP
+See the \fI"Constraints"\fR section for information about using
+built-in constraints and adding new ones.
+.PP
+When tests are run from within an 'all.tcl' file, all files with a
+'.test' extension are normally run. An individual test file will be
+skipped if one of the following conditions is met:
+.IP [1]
+the \fIname\fR of the test files does not match (using glob style matching)
+one or more elements in the \fBtcltest::matchFiles\fR variable
+.IP [2]
+the \fIname\fR of the test file matches (using glob style matching) one or
+more elements in the \fBtcltest::skipFiles\fR variable
+.PP
+You can set \fBtcltest::matchFiles\fR and/or \fBtcltest::skipFiles\fR
+either interactively (after the \fBtcltest\fR package has been
+sourced), or by using the command line arguments \fB-file\fR and
+\fB-notfile\fR, for example:
+.PP
+.CS
+tclsh info.test -file 'unix*.test' -notfile 'unixNotfy.test'
+.CE
+.PP
+Additionally, if tests are run from within an 'all.tcl' containing a
+call to \fBtcltest::runAllTests\fR, any subdirectory of
+\fItcltest::testsDirectory\fR containing an 'all.tcl' file will also
+be run. Individual test subdirectories will be skipped if one of the
+following conditions is met:
+.IP [1]
+the \fIname\fR of the directory does not match (using glob style matching)
+one or more elements in the \fBtcltest::matchDirectories\fR variable
+.IP [2]
+the \fIname\fR of the directory matches (using glob style matching) one or
+more elements in the \fBtcltest::skipDirectories\fR variable
+.PP
+You can set \fBtcltest::matchDirectories\fR and/or \fBtcltest::skipDirectories\fR
+either interactively (after the \fBtcltest\fR package has been
+sourced), or by using the command line arguments \fB-relateddir\fR and
+\fB-asidefromdir\fR, for example:
+.PP
+.CS
+tclsh info.test -relateddir 'subdir*' -asidefromdir 'subdir2'
+.CE
+.SH "HOW TO CUSTOMIZE THE TEST HARNESS"
+To create your own custom test harness, create a .tcl file that contains your
+namespace. Within this file, require package \fBtcltest\fR. Commands
+that can be redefined to customize the test harness include:
+.TP
+\fBtcltest::PrintUsageInfoHook\fP
+print additional usage information specific to your situation.
+.TP
+\fBtcltest::processCmdLineArgsFlagHook\fP
+tell the test harness about additional flags that you want it to understand.
+.TP
+\fBtcltest::processCmdLineArgsHook\fR \fIflags\fP
+process the additional flags that you told the harness about in
+tcltest::processCmdLineArgsFlagHook.
+.TP
+\fBtcltest::initConstraintsHook\fP
+used to add additional built-in constraints to those already defined
+by \fBtcltest\fR.
+.TP
+\fBtcltest::cleanupTestsHook\fP
+do additional cleanup
+.PP
+.PP
+To add new flags to your customized test harness, redefine
+\fBtcltest::processCmdLineArgsAddFlagHook\fR to define additional flags to be
+parsed and \fBtcltest::processCmdLineArgsHook\fR to actually process them.
+For example:
+.DS
+proc tcltest::processCmdLineArgsAddFlagHook {} {
+ return [list -flag1 -flag2]
+}
+
+proc tcltest::processCmdLineArgsHook {flagArray} {
+ array set flag $flagArray
+
+ if {[info exists flag(-flag1)]} {
+ # Handle flag1
+ }
+
+ if {[info exists flag(-flag2)]} {
+ # Handle flag2
+ }
+
+ return
+}
+.DE
+You may also want to add usage information for these flags. This
+information would be displayed whenever the user specifies -help. To
+define additional usage information, define your own
+tcltest::PrintUsageInfoHook proc. Within this proc, you should
+print out additional usage information for any flags that you've
+implemented.
+.PP
+To add new built-in
+constraints to the test harness, define your own version of
+\fBtcltest::initConstraintsHook\fR.
+Within your proc, you can add to the \fBtcltest::testConstraints\fR array.
+For example:
+.DS
+proc tcltest::initConstraintsHook {} {
+ set tcltest::testConstraints(win95Or98) \\
+ [expr {$tcltest::testConstraints(95) || \\
+ $tcltest::testConstraints(98)}]
+}
+.DE
+.PP
+Finally, if you want to add additional cleanup code to your harness
+you can define your own \fBtcltest::cleanupTestsHook\fR. For example:
+.DS
+proc tcltest::cleanupTestsHook {} {
+ # Add your cleanup code here
+}
+.DE
+.SH EXAMPLES
+.IP [1]
+A simple test file (foo.test)
+.DS
+package require tcltest
+namespace import -force ::tcltest::*
+
+test foo-1.1 {
+ description {save 1 in variable name foo}
+ script {
+ set foo 1
+ }
+ expect {1}
+}
+
+tcltest::cleanupTests
+return
+.DE
+.IP [2]
+A simple all.tcl
+.DS
+package require tcltest
+namespace import -force ::tcltest::*
+
+tcltest::testsDirectory [file dir [info script]]
+tcltest::runAllTests
+
+return
+.DE
+.IP [3]
+Running a single test
+.DS
+tclsh foo.test
+.DE
+.IP [4]
+Running multiple tests
+.DS
+tclsh all.tcl -file 'foo*.test' -notfile 'foo2.test'
+.IP [5] A test that uses the unixOnly constraint and should only be
+run on Unix
+.DS
+test getAttribute-1.1 {
+ description {testing file permissions}
+ constraints {unixOnly}
+ script {
+ lindex [file attributes foo.tcl] 5
+ }
+ expect {00644}
+}
+.DE
+.IP [6] A constraint containing an expression that evaluates to true (a case where the test would be run) if it is being run on unix and if threads are not being tested
+.DS
+ unixOnly && !testthread
+.DE
+.SH "KNOWN ISSUES"
+There are two known issues related to nested test commands.
+The first issue relates to the stack level in which test scripts are
+executed. Tests nested within other tests may be executed at the same
+stack level as the outermost test. For example, in the following test
+code:
+.DS
+test level-1.1 {
+ description {level 1}
+ script {
+ test level-2.1 {
+ description {level 2}
+ }
+ }
+}
+.DE
+any script executed in level-2.1 may be executed at the same stack
+level as the script defined for level-1.1.
+.PP
+In addition, while two
+test commands have been run, results will only be reported for tests
+at the same level as test level-1.1. However, test results for all
+tests run prior to level-1.1 will be available when test level-2.1
+runs. What this means is that if you try to access the test results
+for test level-2.1, it will may say that 'm' tests have run, 'n' tests
+have been skipped, 'o' tests have passed and 'p' tests have failed,
+where 'm', 'n', 'o', and 'p' refer to tests that were run at the same
+test level as test level-1.1.
+.PP
+Implementation of output and error comparison in the test command
+depends on usage of puts in your application code. Output is
+intercepted by redefining the puts command while the defined test
+script is being run. Errors thrown by C procedures or printed
+directly from C applications will not be caught by the test command.
+Therefore, usage of expect_out and expect_err in the test command is
+useful only for pure Tcl applications that use the puts command for
+output.
+.SH KEYWORDS
+test, test harness, test suite
+