diff options
Diffstat (limited to 'doc')
| -rwxr-xr-x | doc/tcltest2.n | 1044 |
1 files changed, 1044 insertions, 0 deletions
diff --git a/doc/tcltest2.n b/doc/tcltest2.n new file mode 100755 index 0000000..86c2f48 --- /dev/null +++ b/doc/tcltest2.n @@ -0,0 +1,1044 @@ +'\" +'\" Copyright (c) 1990-1994 The Regents of the University of California +'\" Copyright (c) 1994-1997 Sun Microsystems, Inc. +'\" Copyright (c) 1998-1999 Scriptics Corporation +'\" Copyright (c) 2000 Ajuba Solutions +'\" +'\" See the file "license.terms" for information on usage and redistribution +'\" of this file, and for a DISCLAIMER OF ALL WARRANTIES. +'\" +'\" RCS: @(#) $Id: tcltest2.n,v 1.1 2000/09/20 23:09:49 jenn Exp $ +'\" +.so man.macros +.TH "tcltest" n 8.4 Tcl "Tcl Built-In Commands" +.BS +'\" Note: do not modify the .SH NAME line immediately below! +.SH NAME +tcltest \- Test harness support code and utilities +.SH SYNOPSIS +\fBpackage require tcltest ?2.0?\fP +.sp +\fBtcltest::test \fIname {?attribute value? ...}\fR +.sp +\fBtcltest::cleanupTests \fI?runningMultipleTests?\fR +.sp +\fBtcltest::runAllTests\fR +.sp +\fBtcltest::interpreter \fI?interp?\fR +.sp +\fBtcltest::singleProcess \fI?boolean?\fR +.sp +\fBtcltest::debug \fI?level?\fR +.sp +\fBtcltest::verbose \fI?level?\fR +.sp +\fBtcltest::preserveCore \fI?level?\fR +.sp +\fBtcltest::testConstraint \fIconstraint ?value?\fR +.sp +\fBtcltest::limitConstraints \fI?constraintList?\fR +.sp +\fBtcltest::workingDirectory \fI?dir?\fR +.sp +\fBtcltest::temporaryDirectory \fI?dir?\fR +.sp +\fBtcltest::testsDirectory \fI?dir?\fR +.sp +\fBtcltest::match \fI?patternList?\fR +.sp +\fBtcltest::matchFiles \fI?patternList?\fR +.sp +\fBtcltest::matchDirectories \fI?patternList?\fR +.sp +\fBtcltest::skip \fI?patternList?\fR +.sp +\fBtcltest::skipFiles \fI?patternList?\fR +.sp +\fBtcltest::skipDirectories \fI?patternList?\fR +.sp +\fBtcltest::loadTestedCommands\fR +.sp +\fBtcltest::loadScript \fI?script?\fR +.sp +\fBtcltest::loadFile \fI?filename?\fR +.sp +\fBtcltest::outputChannel \fI?channelID?\fR +.sp +\fBtcltest::outputFile \fI?filename?\fR +.sp +\fBtcltest::errorChannel \fI?channelID?\fR +.sp +\fBtcltest::errorFile \fI?filename?\fR +.sp +\fBtcltest::makeFile \fIcontents name\fR +.sp +\fBtcltest::removeFile \fIname\fR +.sp +\fBtcltest::makeDirectory \fIname\fR +.sp +\fBtcltest::removeDirectory \fIname\fR +.sp +\fBtcltest::viewFile \fIname\fR +.sp +\fBtcltest::normalizeMsg \fImsg\fR +.sp +\fBtcltest::normalizePath \fIpathVar\fR +.sp +\fBtcltest::bytestring \fIstring\fR +.sp +\fBtcltest::saveState\fR +.sp +\fBtcltest::restoreState\fR +.sp +\fBtcltest::threadReap\fR +.BE +.SH DESCRIPTION +.PP +The \fBtcltest\fR package provides the user with utility tools for +writing and running tests in the Tcl test suite. It can also be used +to create a customized test harness for an extension. +.PP +The Tcl test suite consists of multiple .test files, each of which +contains multiple test cases. Each test case consists of a call to +the test command, which specifies the name of test, a short +description, any constraints that apply to the test case, the script +to be run, and expected results. See the \fI"Tests"\fR section for more +details. +.PP +It is also possible to add to this test harness to create your own +customized test harness implementation. For more defails, see the +section \fI"How to Customize the Test Harness"\fR. +.SH COMMANDS +.TP +\fBtcltest::test\fP \fIname {?attribute value? ...}\fR +The \fBtcltest::test\fR command runs the value supplied for attribute +\fIscript\fR and compares its result to possible results. +It prints an error message if actual results and expected results do +not match. The \fBtcltest::test\fR command returns 0 if it completes +successfully. Any other return value indicates that an error has +occurred in the tcltest package. See the \fI"Tests"\fR section for +more details on this command. +.TP +\fBtcltest::cleanupTests\fP \fI?runningMultipleTests?\fR +This command should be called at the end of a test file. It prints +statistics about the tests run and removes files that were created by +\fBtcltest::makeDirectory\fR and \fBtcltest::makeFile\fR. Names +of files and directories created outside of +\fBtcltest::makeFile\fR and \fBtcltest::makeDirectory\fR and +never deleted are printed to \fBtcltest::outputChannel\fR. This command +also restores the original shell environment, as described by the ::env +array. \fIcalledFromAll\fR should be specified if +\fBtcltest::cleanupTests\fR is called explicitly from an "all.tcl" +file. Tcl files files are generally used to run multiple tests. For +more details on how to run multiple tests, please see the section +\fI"Running test files"\fR. This proc has no defined return value. +.TP +\fBtcltest::runAllTests\fP +This command should be used in your 'all.tcl' file. It is used to +loop over test files and directories, determining which test files to +run and then running them. Note that this test calls +tcltest::cleanupTests; if using this proc in your 'all.tcl' file, you +should not call tcltest::cleanupTests explicitly in that file. See the +sample 'all.tcl' file in the \fI"Examples"\fR section. +.TP +\fBtcltest::interpreter\fR \fI?executableName?\fR +Sets or returns the name of the executable used to invoke the test +suite. This is the interpreter used in runAllTests to run test files +if singleProcess is set to false. The default value for interpreter +is the name of the interpreter in which the tests were started. +.TP +\fBtcltest::singleProcess\fR \fI?boolean?\fR +Sets or returns a boolean indicating whether test files should be sourced +into the current interpreter by runAllTests or run in their own +processes. If \fIboolean\fR is true (1), tests are sourced into the +current interpreter. If \fIboolean\fR is false (0), tests are run in +the interpreter specified in tcltest::interpreter. The default value +for tcltest::singleProcess is false. +.TP +\fBtcltest::debug\fR \fI?level?\fR +Sets or returns the current debug level. The debug level determines +how much tcltest package debugging information is printed to stdout. +The default debug level is 0. Levels are defined as: +.RS +.IP 0 +Do not display any debug information. +.IP 1 +Display information regarding whether a test is skipped because it +doesn't match any of the tests that were specified using -match or +tcltest::match (userSpecifiedNonMatch) or matches any of the tests +specified by -skip or tcltest::skip (userSpecifiedSkip). +.IP 2 +Display the flag array parsed by the command line processor, the +contents of the ::env array, and all user-defined variables that exist +in the current namespace as they are used. +.IP 3 +Display information regarding what individual procs in the test +harness are doing. +.RE +.TP +\fBtcltest::verbose\fR \fI?level?\fR +Sets or returns the current verbosity level. This level must be a +substring of "bpst". The default verbosity level is "b". See the +"Test output" section for a more detailed explanation of this +option. Levels are defined as: +.RS +.IP b +Display the body of failed tests +.IP p +Print output when a test passes +.IP s +Print output when a test is skipped +.IP t +Print output whenever a test starts +.RE +.TP +\fBtcltest::preserveCore\fR \fI?level?\fR +Sets or returns the current core preservation level. This level +determines how stringent checks for core files are. The default core +preservation level is 0. Levels are defined as: +.RS +.IP 0 +No checking - do not check for core files at the end of each test +command, but do check for them whenever tcltest::cleanupTests is +called from tcltest::runAllTests. +.IP 1 +Check for core files at the end of each test command and whenever +tcltest::cleanupTests is called from tcltest::runAllTests. +.IP 2 +Check for core files at the end of all test commands and whenever +tcltest::cleanupTests is called from all.tcl. Save any core files +produced in tcltest::temporaryDirectory. +.RE +.TP +\fBtcltest::testConstraint \fIconstraint ?value?\fR +Sets or returns the value associated with the named \fIconstraint\fR. +See the section \fI"Test constraints"\fR for more information. +.TP +\fBtcltest::limitConstraints \fI?constraintList?\fR +Sets or returns a boolean indicating whether testing is being limited +to constraints listed in \fIconstraintList\fR. +If limitConstraints is not false, only those tests with constraints matching +values in \fIconstraintList\fR will be run. +.TP +\fBtcltest::workingDirectory\fR \fI?directoryName?\fR +Sets or returns the directory in which the test suite is being run. +The default value for workingDirectory is the directory in which the +test suite was launched. +.TP +\fBtcltest::temporaryDirectory\fR \fI?directoryName?\fR +Sets or returns the output directory for temporary files created by +tcltest::makeFile and tcltest::makeDirectory. This defaults to the +directory returned by \fItcltest::workingDirectory\fR. +.TP +\fBtcltest::testsDirectory\fR \fI?directoryName?\fR +Sets or returns the directory where the tests reside. This defaults +to the directory returned by \fItcltest::workingDirectory\fR +if the script cannot determine where the \fItests\fR directory is +located. This variable should be explicitly set if tests are being run +from an all.tcl file. +.TP +\fBtcltest::match\fR \fI?globPatternList\fR +Sets or returns the glob pattern list that determines which tests +should be run. Only tests which match one of the glob patterns in +\fIglobPatternList\fR are run by the test harness. The default value +for \fIglobPatternList\fR is '*'. +.TP +\fBtcltest::matchFiles\fR \fI?globPatternList\fR +Sets or returns the glob pattern list that determines which test files +should be run. Only test files which match one of the glob patterns in +\fIglobPatternList\fR are run by the test harness. The default value +for \fIglobPatternList\fR is '*.test'. +.TP +\fBtcltest::matchDirectories\fR \fI?globPatternList\fR +Sets or returns the glob pattern list that determines which test +subdirectories of the current test directory should be run. Only test +subdirectories which match one of the glob patterns in +\fIglobPatternList\fR are run by the test harness. The default value +for \fIglobPatternList\fR is '*'. +.TP +\fBtcltest::skip\fR \fI?globPatternList\fR +Sets or returns the glob pattern list that determines which tests (of +those matched by tcltest::match) should be skipped. The default value +for \fIglobPatternList\fR is {}. +.TP +\fBtcltest::skipFiles\fR \fI?globPatternList\fR +Sets or returns the glob pattern list that determines which test files (of +those matched by tcltest::matchFiles) should be skipped. The default value +for \fIglobPatternList\fR is {}. +.TP +\fBtcltest::skipDirectories\fR \fI?globPatternList\fR +Sets or returns the glob pattern list that determines which test subdirectories (of +those matched by tcltest::matchDirectories) should be skipped. The default value +for \fIglobPatternList\fR is {}. +.TP +\fBtcltest::loadTestedCommands\fP +This command uses the script specified via the \fI-load\fR or +\fI-loadfile\fR to load the commands checked by the test suite. +Allowed to be empty, as the tested commands could have been compiled +into the interpreter running the test suite. +.TP +\fBtcltest::loadScript\fR \fI?script\fR +Sets or returns the script executed by \fBloadTestedCommands\fR. +.TP +\fBtcltest::loadFile\fR \fI?filename?\fR +Sets ore returns the file name associated with the script executed +\fBloadTestedCommands\fR. If setting \fIfilename\fR, this proc will +open the file and call \fItcltest::loadScript\fR with the content. +.TP +\fBtcltest::outputChannel\fR \fI?channelID?\fR +Sets or returns the output file ID. This defaults to stdout. +Any test that prints test related output should send +that output to \fItcltest::outputChannel\fR rather than letting +that output default to stdout. +.TP +\fBtcltest::outputFile\fR \fI?filename?\fR +Sets or returns the file name corresponding to the output file. This +defaults to stdout. This proc calls +outputChannel to set the output file channel. +Any test that prints test related output should send +that output to \fItcltest::outputChannel\fR rather than letting +that output default to stdout. +.TP +\fBtcltest::errorChannel\fR \fI?channelID?\fR +Sets or returns the error file ID. This defaults to stderr. +Any test that prints error messages should send +that output to \fItcltest::errorChannel\fR rather than printing +directly to stderr. +.TP +\fBtcltest::errorFile\fR \fI?filename?\fR +Sets or returns the file name corresponding to the error file. This +defaults to stderr. This proc calls +errorChannel to set the error file channel. +Any test that prints test related error output should send +that output to \fItcltest::errorChannel\fR or +\fItcltest::outputChannel\fR rather than letting +that output default to stdout. +.TP +\fBtcltest::mainThread\fR +Sets or returns the main thread ID. This defaults to 1. This is the +only thread that is not killed by tcltest::threadReap and is set +according to the return value of \fItestthread names\fR at +initialization. +.TP +\fBtcltest::makeFile\fP \fIcontents name\fR +Create a file that will be automatically be removed by +\fBtcltest::cleanupTests\fR at the end of a test file. This file is +created relative to tcltest::temporaryDirectory. +Returns the full path of the file created. +.TP +\fBtcltest::removeFile\fP \fIname\fR +Force the file referenced by \fIname\fR to be removed. This file name +should be relative to \fItcltest::temporaryDirectory\fR. This proc has no +defined return values. +.TP +\fBtcltest::makeDirectory\fP \fIname\fR +Create a directory named \fIname\fR that will automatically be removed +by \fBtcltest::cleanupTests\fR at the end of a test file. This +directory is created relative to tcltest::temporaryDirectory. +Returns the full path of the directory created. +.TP +\fBtcltest::removeDirectory\fP \fIname\fR +Force the directory referenced by \fIname\fR to be removed. This +directory should be relative to tcltest::temporaryDirectory. This proc +has no defined return value. +.TP +\fBtcltest::viewFile\fP \fIfile\fR +Returns the contents of \fIfile\fR. +.TP +\fBtcltest::normalizeMsg\fP \fImsg\fR +Remove extra newlines from \fImsg\fR. +.TP +\fBtcltest::normalizePath\fP \fIpathVar\fR +Resolves symlinks in a path, thus creating a path without internal +redirection. It is assumed that \fIpathVar\fR is absolute. +\fIpathVar\fR is modified in place. +.TP +\fBtcltest::bytestring\fP \fIstring\fR +Construct a string that consists of the requested sequence of bytes, +as opposed to a string of properly formed UTF-8 characters using the +value supplied in \fIstring\fR. This allows the tester to create +denormalized or improperly formed strings to pass to C procedures that +are supposed to accept strings with embedded NULL types and confirm +that a string result has a certain pattern of bytes. +.TP +\fBtcltest::saveState\fP +Save procedure and global variable names. +A test file might contain calls to \fBtcltest::saveState\fR and +\fB::tcltest:restoreState\fR if it creates or deletes global variables +or procs. +.TP +\fBtcltest::restoreState\fP +Restore procedure and global variable names. +A test file might contain calls to \fBtcltest::saveState\fR and +\fB::tcltest:restoreState\fR if it creates or deletes global variables +or procs. +.TP +\fBtcltest::threadReap\fP +\fBtcltest::threadReap\fR only works if \fItestthread\fR is +defined, generally by compiling tcltest. If \fItestthread\fR is +defined, \fBtcltest::threadReap\fR kills all threads except for the +main thread. It gets the ID of the main thread by calling +\fItestthread names\fR during initialization. This value is stored in +\fItcltest::mainThread\fR. \fBtcltest::threadReap\fR returns the +number of existing threads at completion. +.SH TESTS +The \fBtest\fR procedure runs a test script and prints an error +message if the script's result does not match the expected result. +The following is the spec for the \fBtest\fR command: +.DS +test \fIname\fR { + description \fIdescription\fR + ?constraints \fIkeywordList|expression\fR + ?setup \fIsetupScript\fR? + ?script \fItestScript\fR? + ?cleanup \fIcleanupScript\fR? + ?expect \fIexpectedAnswer\fR? | + ?expect {-exact \fIexpectedAnswer\fR?}? | + ?expect {-regexp \fIexpectedAnswer\fR?}? | + ?expect {-glob \fIexpectedAnswer\fR?}? + ?expect_out \fIexpectedOutput\fR? | + ?expect_out {-exact \fIexpectedOutput\fR?}? | + ?expect_out {-regexp \fIexpectedOutput\fR?}? | + ?expect_out {-glob \fIexpectedOutput\fR?}? + ?expect_err \fIexpectedError\fR? | + ?expect_err {-exact \fIexpectedError\fR?}? | + ?expect_err {-regexp \fIexpectedError\fR?}? | + ?expect_err {-glob \fIexpectedError\fR?}? + ?expect_codes \fIcodeList\fR +} +.DE +The \fIname\fR argument should follow the pattern: +.DS +<target>-<majorNum>.<minorNum> +.DE +For white-box (regression) tests, the target should be the name of the +C function or Tcl procedure being tested. For black-box tests, the +target should be the name of the feature being tested. Related tests +should share a major number. +.PP +Valid attributes and associated values are: +.TP +\fBdescription \fIdesc\fR\fP +A value must be provided for the required \fIdescription\fR attribute. +This should be a short textual description of the test used to help humans +understand the purpose of the test. The name of a Tcl or C function +being tested should be included for regression tests. If the test +case exists to reproduce a bug, include the bug ID in the description. +.TP +\fBconstraints \fIkeywordList|expression\fR\fP +The optional \fIconstraints\fR attribute can be list of one or more +keywords or an expression. If the \fIconstraints\fR value consists of +keywords, each of these keywords being the name of a constraint +defined by a call to \fItcltest::testConstraint\fR. If any of these +elements is false or does +not exist, the test is skipped. If the \fIconstraints\fR argument +consists of an expression, that expression is evaluated. If the +expression evaluates to true, then the test is run. Appropriate +constraints should be added to any tests that should +not always be run. See the "Test Constraints" section for a list of built-in +constraints and information on how to add your own constraints. +.TP +\fBsetup \fIscript\fR\fP +The optional \fIsetup\fR attribute indicates a script that will be run +before the script indicated by the \fIscript\fR attribute. If setup +fails, the test will fail. +.TP +\fBscript \fIscript\fR\fP +The \fIscript\fR attribute indicates the script to run to carry out the +test. It must return a result that can be checked for correctness. +.TP +\fBcleanup \fIscript\fR\fP +The optional \fIcleanup\fR attribute indicates a script that will be +run after the script indicated by the \fIscript\fR attribute. If +cleanup fails, the test will fail. +.TP +\fBexpect \fI{?option? expectedValue}|expectedValue\fR\fP +The \fIexpect\fR attribute supplies the comparison value with which +the return value from script will be compared. Valid options for the +value supplied are "-regexp", "-glob", and "-exact". The default +comparison option is "-exact". +.TP +\fBexpect_out \fI{?option? expectedValue}|expectedValue\fR\fP +The \fIexpect_out\fR attribute supplies the comparison value with which +any output sent to stdout or tcltest::outputChannel during the script +run will be compared. Valid options for the +value supplied are "-regexp", "-glob", and "-exact". The default +comparison option is "-exact". Note that only output printed using +puts is used for comparison. +.TP +\fBexpect_err \fI{?option? expectedValue}|expectedValue\fR\fP +The \fIexpect_err\fR attribute supplies the comparison value with which +any output sent to stderr or tcltest::errorChannel during the script +run will be compared. Valid options for the +value supplied are "-regexp", "-glob", and "-exact". The default +comparison option is "-exact". Note that only output printed using +puts is used for comparison. +.TP +\fBexpect_codes \fIexpectedCodes\fR\fP +The optional \fIexpect_code\fR attribute indicates which return codes +from the script supplied with the \fIscript\fR attribute are correct. +Default values for \fIexpectedCodes\fR are 0 (normal return) and 2 +(return exception). +.PP +The <expectedAnswer> argument will be compared against the result of +evaluating the <script> argument. If they match, the test passes, +otherwise the test fails. +.SH "TEST CONSTRAINTS" +Constraints are used to determine whether or not a test should be skipped. +If a test is constrained by "unixOnly", then it will only be run if +the value of the constraint is true. Several +constraints are defined in the \fBtcltest\fR package. To add +constraints, you can call \fBtcltest::testConstraint\fR +with the appropriate arguments in your own test file. +.PP +The following is a list of constraints defined in the \fBtcltest\fR package: +.TP +\fIsingleTestInterp\fR +test can only be run if all test files are sourced into a single interpreter +.TP +\fIunix\fR +test can only be run on any UNIX platform +.TP +\fIpc\fR +test can only be run on any Windows platform +.TP +\fInt\fR +test can only be run on any Windows NT platform +.TP +\fI95\fR +test can only be run on any Windows 95 platform +.TP +\fI98\fR +test can only be run on any Windows 98 platform +.TP +\fImac\fR +test can only be run on any Mac platform +.TP +\fIunixOrPc\fR +test can only be run on a UNIX or PC platform +.TP +\fImacOrPc\fR +test can only be run on a Mac or PC platform +.TP +\fImacOrUnix\fR +test can only be run on a Mac or UNIX platform +.TP +\fItempNotPc\fR +test can not be run on Windows. This flag is used to temporarily +disable a test. +.TP +\fItempNotMac\fR +test can not be run on a Mac. This flag is used +to temporarily disable a test. +.TP +\fIunixCrash\fR +test crashes if it's run on UNIX. This flag is used to temporarily +disable a test. +.TP +\fIpcCrash\fR +test crashes if it's run on Windows. This flag is used to temporarily +disable a test. +.TP +\fImacCrash\fR +test crashes if it's run on a Mac. This flag is used to temporarily +disable a test. +.TP +\fIemptyTest\fR +test is empty, and so not worth running, but it remains as a +place-holder for a test to be written in the future. This constraint +always causes tests to be skipped. +.TP +\fIknownBug\fR +test is known to fail and the bug is not yet fixed. This constraint +always causes tests to be skipped unless the user specifies otherwise. +See the "Introduction" section for more details. +.TP +\fInonPortable\fR +test can only be run in the master Tcl/Tk development environment. +Some tests are inherently non-portable because they depend on things +like word length, file system configuration, window manager, etc. +These tests are only run in the main Tcl development directory where +the configuration is well known. This constraint always causes tests +to be skipped unless the user specifies otherwise. +.TP +\fIuserInteraction\fR +test requires interaction from the user. This constraint always +causes tests to be skipped unless the user specifies otherwise. +.TP +\fIinteractive\fR +test can only be run in if the interpreter is in interactive mode +(when the global tcl_interactive variable is set to 1). +.TP +\fInonBlockFiles\fR +test can only be run if platform supports setting files into +nonblocking mode +.TP +\fIasyncPipeClose\fR +test can only be run if platform supports async flush and async close +on a pipe +.TP +\fIunixExecs\fR +test can only be run if this machine has commands such as 'cat', 'echo', +etc. available. +.TP +\fIhasIsoLocale\fR +test can only be run if can switch to an ISO locale +.TP +\fIroot\fR +test can only run if Unix user is root +.TP +\fInotRoot\fR +test can only run if Unix user is not root +.TP +\fIeformat\fR +test can only run if app has a working version of sprintf with respect +to the "e" format of floating-point numbers. +.TP +\fIstdio\fR +test can only be run if the current app can be spawned via a pipe +.SH "RUNNING TEST FILES" +Use the following command to run a test file that uses package +tcltest: +.DS +<shell> <testFile> ?<option> ?<value>?? ... +.DE +Command line options include (tcltest accessor procs that +correspond to each flag are listed at the end of each flag description +in parenthesis): +.RS +.TP +\fB-help\fR +display usage information. +.TP +\fB-singleproc 0|1\fR +if 0, run test files in multiple interpreters. if 1, source test +files into the current intpreter. +.TP +\fB-verbose <level>\fR +set the level of verbosity to a substring of "bpst". See the "Test +output" section for an explanation of this option. (tcltest::verbose) +.TP +\fB-match <matchList>\fR +only run tests that match one or more of the glob patterns in +<matchList>. (tcltest::match) +.TP +\fB-skip <skipList>\fR +do not run tests that match one or more of the glob patterns in +<skipList>. (tcltest::skip) +.TP +\fB-file <globPatternList>\fR +only source test files that match any of the items in +<globPatternList> relative to tcltest::testsDirectory. +This option +only makes sense if you are running tests using "all.tcl" as the +<testFile> instead of running single test files directly. +(tcltest::matchFiles) +.TP +\fB-notfile <globPatternList>\fR +source files except for those that match any of the items in +<globPatternList> relative to tcltest::testsDirectory. +This option +only makes sense if you are running tests using "all.tcl" as the +<testFile> instead of running single test files directly. +(tcltest::skipFiles) +.TP +\fB-relateddir <globPatternList>\fR +only run tests in directories that match any of the items in +<globPatternList> relative to tcltest::testsDirectory. +This option +only makes sense if you are running tests using "all.tcl" as the +<testFile> instead of running single test files directly. +(tcltest::matchDirectories) +.TP +\fB-asidefromdir <globPatternList>\fR +run tests in directories except for those that match any of the items in +<globPatternList> relative to tcltest::testsDirectory. +This option +only makes sense if you are running tests using "all.tcl" as the +<testFile> instead of running single test files directly. +(tcltest::skipDirectories) +.TP +\fB-constraints <list>\fR +tests with any constraints in <list> will not be skipped. Note that +elements of <list> must exactly match the existing constraints. This +is useful if you want to make sure that tests with a particular +constraint are run (for example, if the tester wants to run all tests +with the knownBug constraint). +(tcltest::testConstraint) +.TP +\fB-limitconstraints <bool>\fR +If the argument to this flag is 1, the test harness limits test runs +to those tests that match the constraints listed by the -constraints +flag. Use of this flag requires use of the -constraints flag. The +default value for this flag is 0 (false). This is useful if you want +to run \fBonly\fR those tests that match the constraints listed using +the -constraints option. A tester might want to do this if (for +example) he were +interested in running only those tests that are constrained to be +unixOnly and no other tests. +(tcltest::limitConstraints) +.TP +\fB-load <script>\fR +will use the specified script to load the commands under test +(tcltest::loadTestedCommands). The default is the empty +script. See -loadfile below too. (tcltest::loadScript) +.TP +\fB-loadfile <scriptfile>\fR +will use the contents of the named file to load the commands under +test (tcltest::loadTestedCommands). See -load above too. The default +is the empty script. (tcltest::loadScript) +.TP +\fB-tmpdir <directoryName>\fR +put any temporary files (created with tcltest::makeFile and +tcltest::makeDirectory) into the named directory. The default +location is tcltest::workingDirectory. (tcltest::temporaryDirectory) +.TP +\fB-testdir <directoryName>\fR +search the test suite to execute in the named directory. The default +location is tcltest::workingDirectory. (tcltest::testsDirectory) +.TP +\fB-preservecore <level>\fR +check for core files. This flag is used to determine how much +checking should be done for core files. (tcltest::preserveCore) +.TP +\fB-debug <debugLevel>\fR +print debug information to stdout. This is used to debug code in the +tcltest package. (tcltest::debug) +.TP +\fB-outfile <filename>\fR +print output generated by the tcltest package to the named file. This +defaults to stdout. Note that debug output always goes to stdout, +regardless of this flag's setting. (tcltest::outputChannel) +.TP +\fB-errfile <filename>\fR +print errors generated by the tcltest package to the named file. This +defaults to stderr. (tcltest::errorChannel) +.RE +.PP +You can specify any of the above options on the command line or by +defining an environment variable named TCLTEST_OPTIONS containing a +list of options (e.g. "-debug 3 -verbose 'ps'"). This environment +variable is evaluated before the command line arguments. Options +specified on the command line override those specified in +TCLTEST_OPTIONS. +.PP +A second way to run tets is to start up a shell, load the +\fBtcltest\fR package, and then source an appropriate test file or use +the test command. To use the options in interactive mode, set +their corresponding tcltest namespace variables after loading the +package. +.PP +See \fI"Test Constraints"\fR for a list of all built-in constraint names. +.PP +A final way to run tests would be to specify which test files to run +within an \fIall.tcl\fR (or otherwise named) file. This is the +approach used by the Tcl test suite. This file loads the tcltest +package, sets the location of +the test directory (tcltest::testsDirectory), determines which test +files to run, sources each of these files, calls +tcltest::cleanupTests and then exits. +.PP +A more elaborate \fIall.tcl\fR file might do some pre- and +post-processing before sourcing +each .test file, use separate interpreters for each file, or handle +complex directory structures. +For an example of an all.tcl file, +please see the "Examples" section of this document. +.SH "TEST OUTPUT" +After all specified test files are run, the number of tests +passed, skipped, and failed is printed to +\fBtcltest::outputChannel\fR. Aside from this +statistical information, output can be controlled on a per-test basis +by the \fBtcltest::verbose\fR variable. +.PP +\fBtcltest::verbose\fR can be set to any substring or permutation +of "bpst". In the string "bpst", the 'b' stands for a test's "body", +the 'p' stands for "passed" tests, the 's' stands for "skipped" +tests, and the 't' indicates when a test "starts". +The default value of \fBtcltest::verbose\fR is "b". If 'b' +is present, then the entire body of the test is printed for each +failed test, otherwise only the test's name, desired output, and +actual output, are printed for each failed test. If 'p' is present, +then a line is printed for each passed test, otherwise no line is +printed for passed tests. If 's' is present, then a line (containing +the consraints that cause the test to be skipped) is printed for each +skipped test, otherwise no line is printed for skipped tests. If 't' +is present, then a line is printed each time a new test starts. +.PP +You can set \fBtcltest::verbose\fR either interactively (after the +\fBtcltest\fR package has been loaded) or by using the command line +argument \fB-verbose\fR, for example: +.DS +tclsh socket.test -verbose bps +.DE +.SH "CONTENTS OF A TEST FILE" +Test files should begin by loading the \fBtcltest\fR package: +.DS +package require tcltest +namespace import -force tcltest::* +.DE +Test files should end by cleaning up after themselves and calling +\fBtcltest::cleanupTests\fR. The \fBtcltest::cleanupTests\fR +procedure prints statistics about the number of tests that passed, +skipped, and failed, and removes all files that were created using the +\fBtcltest::makeFile\fR and \fBtcltest::makeDirectory\fR procedures. +.DS +# Remove files created by these tests +# Change to original working directory +# Unset global arrays +tcltest::cleanupTests +return +.DE +When naming test files, file names should end with a .test extension. +The names of test files that contain regression (or glass-box) tests +should correspond to the Tcl or C code file that they are testing. +For example, the test file for the C file "tclCmdAH.c" is "cmdAH.test". +.SH "SELECTING TESTS FOR EXECUTION" +.PP +Normally, all the tests in a file are run whenever the file is +sourced. An individual test will be skipped if one of the following +conditions is met: +.IP [1] +the \fIname\fR of the tests does not match (using glob style matching) +one or more elements in the \fBtcltest::match\fR variable +.IP [2] +the \fIname\fR of the tests matches (using glob style matching) one or +more elements in the \fBtcltest::skip\fR variable +.IP [3] +the \fIconstraints\fR argument to the \fBtcltest::test\fR call, if +given, contains one or more false elements. +.PP +You can set \fBtcltest::match\fR and/or \fBtcltest::skip\fR +either interactively (after the \fBtcltest\fR package has been +sourced), or by using the command line arguments \fB-match\fR and +\fB-skip\fR, for example: +.PP +.CS +tclsh info.test -match '*-5.* *-7.*' -skip '*-7.1*' +.CE +.PP +Be sure to use the proper quoting convention so that your shell does +not perform the glob substitution on the match or skip patterns you +specify. +.PP +Predefined constraints (e.g. \fIknownBug\fR and \fInonPortable\fR) can be +overridden either interactively (after the \fBtcltest\fR package has been +sourced) by setting the proper constraint +or by using the \fB-constraints\fR command line option with the name of the +constraint in the argument. The following example shows how to run +tests that are constrained by the \fIknownBug\fR and \fInonPortable\fR +restrictions: +.PP +.CS +tclsh all.tcl -constraints "knownBug nonPortable" +.CE +.PP +See the \fI"Constraints"\fR section for information about using +built-in constraints and adding new ones. +.PP +When tests are run from within an 'all.tcl' file, all files with a +'.test' extension are normally run. An individual test file will be +skipped if one of the following conditions is met: +.IP [1] +the \fIname\fR of the test files does not match (using glob style matching) +one or more elements in the \fBtcltest::matchFiles\fR variable +.IP [2] +the \fIname\fR of the test file matches (using glob style matching) one or +more elements in the \fBtcltest::skipFiles\fR variable +.PP +You can set \fBtcltest::matchFiles\fR and/or \fBtcltest::skipFiles\fR +either interactively (after the \fBtcltest\fR package has been +sourced), or by using the command line arguments \fB-file\fR and +\fB-notfile\fR, for example: +.PP +.CS +tclsh info.test -file 'unix*.test' -notfile 'unixNotfy.test' +.CE +.PP +Additionally, if tests are run from within an 'all.tcl' containing a +call to \fBtcltest::runAllTests\fR, any subdirectory of +\fItcltest::testsDirectory\fR containing an 'all.tcl' file will also +be run. Individual test subdirectories will be skipped if one of the +following conditions is met: +.IP [1] +the \fIname\fR of the directory does not match (using glob style matching) +one or more elements in the \fBtcltest::matchDirectories\fR variable +.IP [2] +the \fIname\fR of the directory matches (using glob style matching) one or +more elements in the \fBtcltest::skipDirectories\fR variable +.PP +You can set \fBtcltest::matchDirectories\fR and/or \fBtcltest::skipDirectories\fR +either interactively (after the \fBtcltest\fR package has been +sourced), or by using the command line arguments \fB-relateddir\fR and +\fB-asidefromdir\fR, for example: +.PP +.CS +tclsh info.test -relateddir 'subdir*' -asidefromdir 'subdir2' +.CE +.SH "HOW TO CUSTOMIZE THE TEST HARNESS" +To create your own custom test harness, create a .tcl file that contains your +namespace. Within this file, require package \fBtcltest\fR. Commands +that can be redefined to customize the test harness include: +.TP +\fBtcltest::PrintUsageInfoHook\fP +print additional usage information specific to your situation. +.TP +\fBtcltest::processCmdLineArgsFlagHook\fP +tell the test harness about additional flags that you want it to understand. +.TP +\fBtcltest::processCmdLineArgsHook\fR \fIflags\fP +process the additional flags that you told the harness about in +tcltest::processCmdLineArgsFlagHook. +.TP +\fBtcltest::initConstraintsHook\fP +used to add additional built-in constraints to those already defined +by \fBtcltest\fR. +.TP +\fBtcltest::cleanupTestsHook\fP +do additional cleanup +.PP +.PP +To add new flags to your customized test harness, redefine +\fBtcltest::processCmdLineArgsAddFlagHook\fR to define additional flags to be +parsed and \fBtcltest::processCmdLineArgsHook\fR to actually process them. +For example: +.DS +proc tcltest::processCmdLineArgsAddFlagHook {} { + return [list -flag1 -flag2] +} + +proc tcltest::processCmdLineArgsHook {flagArray} { + array set flag $flagArray + + if {[info exists flag(-flag1)]} { + # Handle flag1 + } + + if {[info exists flag(-flag2)]} { + # Handle flag2 + } + + return +} +.DE +You may also want to add usage information for these flags. This +information would be displayed whenever the user specifies -help. To +define additional usage information, define your own +tcltest::PrintUsageInfoHook proc. Within this proc, you should +print out additional usage information for any flags that you've +implemented. +.PP +To add new built-in +constraints to the test harness, define your own version of +\fBtcltest::initConstraintsHook\fR. +Within your proc, you can add to the \fBtcltest::testConstraints\fR array. +For example: +.DS +proc tcltest::initConstraintsHook {} { + set tcltest::testConstraints(win95Or98) \\ + [expr {$tcltest::testConstraints(95) || \\ + $tcltest::testConstraints(98)}] +} +.DE +.PP +Finally, if you want to add additional cleanup code to your harness +you can define your own \fBtcltest::cleanupTestsHook\fR. For example: +.DS +proc tcltest::cleanupTestsHook {} { + # Add your cleanup code here +} +.DE +.SH EXAMPLES +.IP [1] +A simple test file (foo.test) +.DS +package require tcltest +namespace import -force ::tcltest::* + +test foo-1.1 { + description {save 1 in variable name foo} + script { + set foo 1 + } + expect {1} +} + +tcltest::cleanupTests +return +.DE +.IP [2] +A simple all.tcl +.DS +package require tcltest +namespace import -force ::tcltest::* + +tcltest::testsDirectory [file dir [info script]] +tcltest::runAllTests + +return +.DE +.IP [3] +Running a single test +.DS +tclsh foo.test +.DE +.IP [4] +Running multiple tests +.DS +tclsh all.tcl -file 'foo*.test' -notfile 'foo2.test' +.IP [5] A test that uses the unixOnly constraint and should only be +run on Unix +.DS +test getAttribute-1.1 { + description {testing file permissions} + constraints {unixOnly} + script { + lindex [file attributes foo.tcl] 5 + } + expect {00644} +} +.DE +.IP [6] A constraint containing an expression that evaluates to true (a case where the test would be run) if it is being run on unix and if threads are not being tested +.DS + unixOnly && !testthread +.DE +.SH "KNOWN ISSUES" +There are two known issues related to nested test commands. +The first issue relates to the stack level in which test scripts are +executed. Tests nested within other tests may be executed at the same +stack level as the outermost test. For example, in the following test +code: +.DS +test level-1.1 { + description {level 1} + script { + test level-2.1 { + description {level 2} + } + } +} +.DE +any script executed in level-2.1 may be executed at the same stack +level as the script defined for level-1.1. +.PP +In addition, while two +test commands have been run, results will only be reported for tests +at the same level as test level-1.1. However, test results for all +tests run prior to level-1.1 will be available when test level-2.1 +runs. What this means is that if you try to access the test results +for test level-2.1, it will may say that 'm' tests have run, 'n' tests +have been skipped, 'o' tests have passed and 'p' tests have failed, +where 'm', 'n', 'o', and 'p' refer to tests that were run at the same +test level as test level-1.1. +.PP +Implementation of output and error comparison in the test command +depends on usage of puts in your application code. Output is +intercepted by redefining the puts command while the defined test +script is being run. Errors thrown by C procedures or printed +directly from C applications will not be caught by the test command. +Therefore, usage of expect_out and expect_err in the test command is +useful only for pure Tcl applications that use the puts command for +output. +.SH KEYWORDS +test, test harness, test suite + |