'\" '\" Copyright (c) 1990-1994 The Regents of the University of California '\" Copyright (c) 1994-1997 Sun Microsystems, Inc. '\" Copyright (c) 1998-1999 Scriptics Corporation '\" Copyright (c) 2000 Ajuba Solutions '\" '\" See the file "license.terms" for information on usage and redistribution '\" of this file, and for a DISCLAIMER OF ALL WARRANTIES. '\" '\" RCS: @(#) $Id: tcltest.n,v 1.12 2001/11/10 02:34:57 hobbs Exp $ '\" .so man.macros .TH "tcltest" n 8.4 Tcl "Tcl Built-In Commands" .BS '\" Note: do not modify the .SH NAME line immediately below! .SH NAME tcltest \- Test harness support code and utilities .SH SYNOPSIS \fBpackage require tcltest ?2.0?\fR .sp \fBtcltest::test \fIname desc ?option value? ?option value? ...\fR .br \fBtcltest::test \fIname desc {?option value? ?option value? ...}\fR .sp \fBtcltest::cleanupTests \fI?runningMultipleTests?\fR .sp \fBtcltest::runAllTests\fR .sp \fBtcltest::interpreter \fI?interp?\fR .sp \fBtcltest::singleProcess \fI?boolean?\fR .sp \fBtcltest::debug \fI?level?\fR .sp \fBtcltest::verbose \fI?levelList?\fR .sp \fBtcltest::preserveCore \fI?level?\fR .sp \fBtcltest::testConstraint \fIconstraint ?value?\fR .sp \fBtcltest::limitConstraints \fI?constraintList?\fR .sp \fBtcltest::workingDirectory \fI?dir?\fR .sp \fBtcltest::temporaryDirectory \fI?dir?\fR .sp \fBtcltest::testsDirectory \fI?dir?\fR .sp \fBtcltest::match \fI?patternList?\fR .sp \fBtcltest::matchFiles \fI?patternList?\fR .sp \fBtcltest::matchDirectories \fI?patternList?\fR .sp \fBtcltest::skip \fI?patternList?\fR .sp \fBtcltest::skipFiles \fI?patternList?\fR .sp \fBtcltest::skipDirectories \fI?patternList?\fR .sp \fBtcltest::loadTestedCommands\fR .sp \fBtcltest::loadScript \fI?script?\fR .sp \fBtcltest::loadFile \fI?filename?\fR .sp \fBtcltest::outputChannel \fI?channelID?\fR .sp \fBtcltest::outputFile \fI?filename?\fR .sp \fBtcltest::errorChannel \fI?channelID?\fR .sp \fBtcltest::errorFile \fI?filename?\fR .sp \fBtcltest::makeFile \fIcontents name ?directory?\fR .sp \fBtcltest::removeFile \fIname ?directory?\fR .sp \fBtcltest::makeDirectory \fIname ?directory?\fR .sp \fBtcltest::removeDirectory \fIname ?directory?\fR .sp \fBtcltest::viewFile \fIname ?directory?\fR .sp \fBtcltest::normalizeMsg \fImsg\fR .sp \fBtcltest::normalizePath \fIpathVar\fR .sp \fBtcltest::bytestring \fIstring\fR .sp \fBtcltest::saveState\fR .sp \fBtcltest::restoreState\fR .sp \fBtcltest::threadReap\fR .BE .SH DESCRIPTION .PP The \fBtcltest\fR package provides the user with utility tools for writing and running tests in the Tcl test suite. It can also be used to create a customized test harness for an extension. .PP The Tcl test suite consists of multiple .test files, each of which contains multiple test cases. Each test case consists of a call to the test command, which specifies the name of test, a short description, any constraints that apply to the test case, the script to be run, and expected results. See the \fI"Tests"\fR section for more details. .PP It is also possible to add to this test harness to create your own customized test harness implementation. For more defails, see the section \fI"How to Customize the Test Harness"\fR. .SH COMMANDS .TP \fBtcltest::test\fR \fIname desc ?option value? ?option value? ...\fR .TP \fBtcltest::test\fR \fIname desc {?option value? ?option value? ...}\fR The \fBtcltest::test\fR command runs the value supplied for attribute \fIscript\fR and compares its result to possible results. It prints an error message if actual results and expected results do not match. The \fBtcltest::test\fR command returns 0 if it completes successfully. Any other return value indicates that an error has occurred in the tcltest package. See the \fI"Tests"\fR section for more details on this command. .TP \fBtcltest::cleanupTests\fR \fI?runningMultipleTests?\fR This command should be called at the end of a test file. It prints statistics about the tests run and removes files that were created by \fBtcltest::makeDirectory\fR and \fBtcltest::makeFile\fR. Names of files and directories created outside of \fBtcltest::makeFile\fR and \fBtcltest::makeDirectory\fR and never deleted are printed to \fBtcltest::outputChannel\fR. This command also restores the original shell environment, as described by the ::env array. \fIcalledFromAll\fR should be specified if \fBtcltest::cleanupTests\fR is called explicitly from an "all.tcl" file. Tcl files files are generally used to run multiple tests. For more details on how to run multiple tests, please see the section \fI"Running test files"\fR. This proc has no defined return value. .TP \fBtcltest::runAllTests\fR This command should be used in your 'all.tcl' file. It is used to loop over test files and directories, determining which test files to run and then running them. Note that this test calls tcltest::cleanupTests; if using this proc in your 'all.tcl' file, you should not call tcltest::cleanupTests explicitly in that file. See the sample 'all.tcl' file in the \fI"Examples"\fR section. .TP \fBtcltest::interpreter\fR \fI?executableName?\fR Sets or returns the name of the executable used to invoke the test suite. This is the interpreter used in runAllTests to run test files if singleProcess is set to false. The default value for interpreter is the name of the interpreter in which the tests were started. .TP \fBtcltest::singleProcess\fR \fI?boolean?\fR Sets or returns a boolean indicating whether test files should be sourced into the current interpreter by runAllTests or run in their own processes. If \fIboolean\fR is true (1), tests are sourced into the current interpreter. If \fIboolean\fR is false (0), tests are run in the interpreter specified in tcltest::interpreter. The default value for tcltest::singleProcess is false. .TP \fBtcltest::debug\fR \fI?level?\fR Sets or returns the current debug level. The debug level determines how much tcltest package debugging information is printed to stdout. The default debug level is 0. Levels are defined as: .RS .IP 0 Do not display any debug information. .IP 1 Display information regarding whether a test is skipped because it doesn't match any of the tests that were specified using -match or tcltest::match (userSpecifiedNonMatch) or matches any of the tests specified by -skip or tcltest::skip (userSpecifiedSkip). .IP 2 Display the flag array parsed by the command line processor, the contents of the ::env array, and all user-defined variables that exist in the current namespace as they are used. .IP 3 Display information regarding what individual procs in the test harness are doing. .RE .TP \fBtcltest::verbose\fR \fI?levelList?\fR Sets or returns the current verbosity level. The default verbosity level is "body". See the "Test output" section for a more detailed explanation of this option. Levels are defined as: .RS .IP body Display the body of failed tests .IP pass Print output when a test passes .IP skip Print output when a test is skipped .IP start Print output whenever a test starts .IP error Print errorInfo and errorCode, if they exist, when a test return code does not match its expected return code .RE .TP \fBtcltest::preserveCore\fR \fI?level?\fR Sets or returns the current core preservation level. This level determines how stringent checks for core files are. The default core preservation level is 0. Levels are defined as: .RS .IP 0 No checking - do not check for core files at the end of each test command, but do check for them whenever tcltest::cleanupTests is called from tcltest::runAllTests. .IP 1 Check for core files at the end of each test command and whenever tcltest::cleanupTests is called from tcltest::runAllTests. .IP 2 Check for core files at the end of all test commands and whenever tcltest::cleanupTests is called from all.tcl. Save any core files produced in tcltest::temporaryDirectory. .RE .TP \fBtcltest::testConstraint \fIconstraint ?value?\fR Sets or returns the value associated with the named \fIconstraint\fR. See the section \fI"Test constraints"\fR for more information. .TP \fBtcltest::limitConstraints \fI?constraintList?\fR Sets or returns a boolean indicating whether testing is being limited to constraints listed in \fIconstraintList\fR. If limitConstraints is not false, only those tests with constraints matching values in \fIconstraintList\fR will be run. .TP \fBtcltest::workingDirectory\fR \fI?directoryName?\fR Sets or returns the directory in which the test suite is being run. The default value for workingDirectory is the directory in which the test suite was launched. .TP \fBtcltest::temporaryDirectory\fR \fI?directoryName?\fR Sets or returns the output directory for temporary files created by tcltest::makeFile and tcltest::makeDirectory. This defaults to the directory returned by \fItcltest::workingDirectory\fR. .TP \fBtcltest::testsDirectory\fR \fI?directoryName?\fR Sets or returns the directory where the tests reside. This defaults to the directory returned by \fItcltest::workingDirectory\fR if the script cannot determine where the \fItests\fR directory is located. This variable should be explicitly set if tests are being run from an all.tcl file. .TP \fBtcltest::match\fR \fI?globPatternList?\fR Sets or returns the glob pattern list that determines which tests should be run. Only tests which match one of the glob patterns in \fIglobPatternList\fR are run by the test harness. The default value for \fIglobPatternList\fR is '*'. .TP \fBtcltest::matchFiles\fR \fI?globPatternList?\fR Sets or returns the glob pattern list that determines which test files should be run. Only test files which match one of the glob patterns in \fIglobPatternList\fR are run by the test harness. The default value for \fIglobPatternList\fR is '*.test'. .TP \fBtcltest::matchDirectories\fR \fI?globPatternList?\fR Sets or returns the glob pattern list that determines which test subdirectories of the current test directory should be run. Only test subdirectories which match one of the glob patterns in \fIglobPatternList\fR are run by the test harness. The default value for \fIglobPatternList\fR is '*'. .TP \fBtcltest::skip\fR \fI?globPatternList?\fR Sets or returns the glob pattern list that determines which tests (of those matched by tcltest::match) should be skipped. The default value for \fIglobPatternList\fR is {}. .TP \fBtcltest::skipFiles\fR \fI?globPatternList?\fR Sets or returns the glob pattern list that determines which test files (of those matched by tcltest::matchFiles) should be skipped. The default value for \fIglobPatternList\fR is {}. .TP \fBtcltest::skipDirectories\fR \fI?globPatternList?\fR Sets or returns the glob pattern list that determines which test subdirectories (of those matched by tcltest::matchDirectories) should be skipped. The default value for \fIglobPatternList\fR is {}. .TP \fBtcltest::loadTestedCommands\fR This command uses the script specified via the \fI-load\fR or \fI-loadfile\fR options or the tcltest::loadScript or tcltest::loadFile procs to load the commands checked by the test suite. It is allowed to be empty, as the tested commands could have been compiled into the interpreter running the test suite. .TP \fBtcltest::loadScript\fR \fI?script?\fR Sets or returns the script executed by \fBloadTestedCommands\fR. .TP \fBtcltest::loadFile\fR \fI?filename?\fR Sets ore returns the file name associated with the script executed \fBloadTestedCommands\fR. If setting \fIfilename\fR, this proc will open the file and call \fItcltest::loadScript\fR with the content. .TP \fBtcltest::outputChannel\fR \fI?channelID?\fR Sets or returns the output file ID. This defaults to stdout. Any test that prints test related output should send that output to \fItcltest::outputChannel\fR rather than letting that output default to stdout. .TP \fBtcltest::outputFile\fR \fI?filename?\fR Sets or returns the file name corresponding to the output file. This defaults to stdout. This proc calls outputChannel to set the output file channel. Any test that prints test related output should send that output to \fItcltest::outputChannel\fR rather than letting that output default to stdout. .TP \fBtcltest::errorChannel\fR \fI?channelID?\fR Sets or returns the error file ID. This defaults to stderr. Any test that prints error messages should send that output to \fItcltest::errorChannel\fR rather than printing directly to stderr. .TP \fBtcltest::errorFile\fR \fI?filename?\fR Sets or returns the file name corresponding to the error file. This defaults to stderr. This proc calls errorChannel to set the error file channel. Any test that prints test related error output should send that output to \fItcltest::errorChannel\fR or \fItcltest::outputChannel\fR rather than letting that output default to stdout. .TP \fBtcltest::makeFile\fR \fIcontents name ?directory?\fR Create a file that will be automatically be removed by \fBtcltest::cleanupTests\fR at the end of a test file. This file is created relative to \fIdirectory\fR. If left unspecified, \fIdirectory\fR defaults to tcltest::temporaryDirectory. Returns the full path of the file created. .TP \fBtcltest::removeFile\fR \fIname ?directory?\fR Force the file referenced by \fIname\fR to be removed. This file name should be relative to \fIdirectory\fR. If left unspecified, \fIdirectory\fR defaults to tcltest::temporaryDirectory. This proc has no defined return values. .TP \fBtcltest::makeDirectory\fR \fIname ?directory?\fR Create a directory named \fIname\fR that will automatically be removed by \fBtcltest::cleanupTests\fR at the end of a test file. This directory is created relative to tcltest::temporaryDirectory. Returns the full path of the directory created. .TP \fBtcltest::removeDirectory\fR \fIname\fR Force the directory referenced by \fIname\fR to be removed. This directory should be relative to \fIdirectory\fR. If left unspecified, \fIdirectory\fR defaults to tcltest::temporaryDirectory. This proc has no defined return value. .TP \fBtcltest::viewFile\fR \fIfile ?directory?\fR Returns the contents of \fIfile\fR. This file name should be relative to \fIdirectory\fR. If left unspecified, \fIdirectory\fR defaults to tcltest::temporaryDirectory. .TP \fBtcltest::normalizeMsg\fR \fImsg\fR Remove extra newlines from \fImsg\fR. .TP \fBtcltest::normalizePath\fR \fIpathVar\fR Resolves symlinks in a path, thus creating a path without internal redirection. It is assumed that \fIpathVar\fR is absolute. \fIpathVar\fR is modified in place. .TP \fBtcltest::bytestring\fR \fIstring\fR Construct a string that consists of the requested sequence of bytes, as opposed to a string of properly formed UTF-8 characters using the value supplied in \fIstring\fR. This allows the tester to create denormalized or improperly formed strings to pass to C procedures that are supposed to accept strings with embedded NULL types and confirm that a string result has a certain pattern of bytes. .TP \fBtcltest::saveState\fR Save procedure and global variable names. A test file might contain calls to \fBtcltest::saveState\fR and \fB::tcltest:restoreState\fR if it creates or deletes global variables or procs. .TP \fBtcltest::restoreState\fR Restore procedure and global variable names. A test file might contain calls to \fBtcltest::saveState\fR and \fB::tcltest:restoreState\fR if it creates or deletes global variables or procs. .TP \fBtcltest::threadReap\fR \fBtcltest::threadReap\fR only works if \fItestthread\fR is defined, generally by compiling tcltest. If \fItestthread\fR is defined, \fBtcltest::threadReap\fR kills all threads except for the main thread. It gets the ID of the main thread by calling \fItestthread names\fR during initialization. This value is stored in \fItcltest::mainThread\fR. \fBtcltest::threadReap\fR returns the number of existing threads at completion. .TP \fBtcltest::mainThread\fR Sets or returns the main thread ID. This defaults to 1. This is the only thread that is not killed by tcltest::threadReap and is set according to the return value of \fItestthread names\fR at initialization. .SH TESTS The \fBtest\fR procedure runs a test script and prints an error message if the script's result does not match the expected result. Two syntaxes are provided for specifying the attributes of the tests. The first uses a separate argument for each of the attributes and values. The second form places all of the attributes and values together into a single argument; the argument must have proper list structure, with teh elements of the list being the attributes and values. The second form makes it easy to construct multi-line scripts, since the braces around the whole list make it unnecessary to include a backslash at the end of each line. In the second form, no command or variable substitutions are performed on the attribute names. This makes the behavior of the second form different from the first form in some cases. .PP The first form for the \fBtest\fR command: .DS test \fIname\fR \fIdescription\fR ?-constraints \fIkeywordList|expression\fR ?-setup \fIsetupScript\fR? ?-body \fItestScript\fR? ?-cleanup \fIcleanupScript\fR? ?-result \fIexpectedAnswer\fR? ?-output \fIexpectedOutput\fR? ?-errorOutput \fIexpectedError\fR? ?-returnCodes \fIcodeList\fR? ?-match \fIexact|glob|regexp\fR? .DE .PP The second form for the \fBtest\fR command (adds brace grouping): .DS test \fIname\fR \fIdescription\fR { ?-constraints \fIkeywordList|expression\fR ?-setup \fIsetupScript\fR? ?-body \fItestScript\fR? ?-cleanup \fIcleanupScript\fR? ?-result \fIexpectedAnswer\fR? ?-output\fIexpectedOutput\fR? ?-errorOutput \fIexpectedError\fR? ?-returnCodes \fIcodeList\fR? ?-match \fIexact|glob|regexp\fR? } .DE The \fIname\fR argument should follow the pattern: .DS -. .DE For white-box (regression) tests, the target should be the name of the C function or Tcl procedure being tested. For black-box tests, the target should be the name of the feature being tested. Related tests should share a major number. .PP The \fIdescription\fR should be a short textual description of the test. It is generally used to help humans understand the purpose of the test. The name of a Tcl or C function being tested should be included in the description for regression tests. If the test case exists to reproduce a bug, include the bug ID in the description. .PP Valid attributes and associated values are: .TP \fB-constraints \fIkeywordList|expression\fR The optional \fIconstraints\fR attribute can be list of one or more keywords or an expression. If the \fIconstraints\fR value consists of keywords, each of these keywords being the name of a constraint defined by a call to \fItcltest::testConstraint\fR. If any of these elements is false or does not exist, the test is skipped. If the \fIconstraints\fR argument consists of an expression, that expression is evaluated. If the expression evaluates to true, then the test is run. Appropriate constraints should be added to any tests that should not always be run. See the "Test Constraints" section for a list of built-in constraints and information on how to add your own constraints. .TP \fB-setup \fIscript\fR The optional \fIsetup\fR attribute indicates a script that will be run before the script indicated by the \fIscript\fR attribute. If setup fails, the test will fail. .TP \fB-body \fIscript\fR The \fIbody\fR attribute indicates the script to run to carry out the test. It must return a result that can be checked for correctness. If left unspecified, the script value will be {}. .TP \fB-cleanup \fIscript\fR The optional \fIcleanup\fR attribute indicates a script that will be run after the script indicated by the \fIscript\fR attribute. If cleanup fails, the test will fail. .TP \fB-match \fIregexp|glob|exact\fR The \fImatch\fR attribute determines how expected answers supplied in \fIresult\fR, \fIoutput\fR, and \fIerrorOutput\fR are compared. Valid options for the value supplied are ``regexp'', ``glob'', and ``exact''. If \fImatch\fR is not specified, the comparisons will be done in ``exact'' mode by default. .TP \fB-result \fIexpectedValue\fR The \fIresult\fR attribute supplies the comparison value with which the return value from script will be compared. If left unspecified, the default \fIexpectedValue\fR will be the empty list. .TP \fB-output \fIexpectedValue\fR The \fIoutput\fR attribute supplies the comparison value with which any output sent to stdout or tcltest::outputChannel during the script run will be compared. Note that only output printed using puts is used for comparison. If \fIoutput\fR is not specified, output sent to stdout and tcltest::outputChannel is not processed for comparison. .TP \fB-errorOutut \fIexpectedValue\fR The \fIerrorOutput\fR attribute supplies the comparison value with which any output sent to stderr or tcltest::errorChannel during the script run will be compared. Note that only output printed using puts is used for comparison. If \fIerrorOutut\fR is not specified, output sent to stderr and tcltest::errorChannel is not processed for comparison. .TP \fB-returnCodes \fIexpectedCodeList\fR The optional \fIreturnCodes\fR attribute indicates which return codes from the script supplied with the \fIscript\fR attribute are correct. Default values for \fIexpectedCodeList\fR are 0 (normal return) and 2 (return exception). Symbolic values \fInormal\fR (0), \fIerror\fR (1), \fIreturn\fR (2), \fIbreak\fR (3), and \fIcontinue\fR (4) can be used in the \fIexpectedCodeList\fR list. .PP To pass, a test must successfully execute its setup, script, and cleanup code. The return code of the test and its return values must match expected values, and if specified, output and error data from the test must match expected output and error values. If all of these conditions are not met, then the test fails. .SH "TEST CONSTRAINTS" Constraints are used to determine whether or not a test should be skipped. If a test is constrained by ``unixOnly'', then it will only be run if the value of the constraint is true. Several constraints are defined in the \fBtcltest\fR package. To add constraints, you can call \fBtcltest::testConstraint\fR with the appropriate arguments in your own test file. .PP The following is a list of constraints defined in the \fBtcltest\fR package: .TP \fIsingleTestInterp\fR test can only be run if all test files are sourced into a single interpreter .TP \fIunix\fR test can only be run on any UNIX platform .TP \fIwin\fR test can only be run on any Windows platform .TP \fInt\fR test can only be run on any Windows NT platform .TP \fI95\fR test can only be run on any Windows 95 platform .TP \fI98\fR test can only be run on any Windows 98 platform .TP \fImac\fR test can only be run on any Mac platform .TP \fIunixOrWin\fR test can only be run on a UNIX or Windows platform .TP \fImacOrWin\fR test can only be run on a Mac or Windows platform .TP \fImacOrUnix\fR test can only be run on a Mac or UNIX platform .TP \fItempNotWin\fR test can not be run on Windows. This flag is used to temporarily disable a test. .TP \fItempNotMac\fR test can not be run on a Mac. This flag is used to temporarily disable a test. .TP \fIunixCrash\fR test crashes if it's run on UNIX. This flag is used to temporarily disable a test. .TP \fIwinCrash\fR test crashes if it's run on Windows. This flag is used to temporarily disable a test. .TP \fImacCrash\fR test crashes if it's run on a Mac. This flag is used to temporarily disable a test. .TP \fIemptyTest\fR test is empty, and so not worth running, but it remains as a place-holder for a test to be written in the future. This constraint always causes tests to be skipped. .TP \fIknownBug\fR test is known to fail and the bug is not yet fixed. This constraint always causes tests to be skipped unless the user specifies otherwise. See the "Introduction" section for more details. .TP \fInonPortable\fR test can only be run in the master Tcl/Tk development environment. Some tests are inherently non-portable because they depend on things like word length, file system configuration, window manager, etc. These tests are only run in the main Tcl development directory where the configuration is well known. This constraint always causes tests to be skipped unless the user specifies otherwise. .TP \fIuserInteraction\fR test requires interaction from the user. This constraint always causes tests to be skipped unless the user specifies otherwise. .TP \fIinteractive\fR test can only be run in if the interpreter is in interactive mode (when the global tcl_interactive variable is set to 1). .TP \fInonBlockFiles\fR test can only be run if platform supports setting files into nonblocking mode .TP \fIasyncPipeClose\fR test can only be run if platform supports async flush and async close on a pipe .TP \fIunixExecs\fR test can only be run if this machine has Unix-style commands \fBcat\fR, \fBecho\fR, \fBsh\fR, \fBwc\fR, \fBrm\fR, \fBsleep\fR, \fBfgrep\fR, \fBps\fR, \fBchmod\fR, and \fBmkdir\fR available .TP \fIhasIsoLocale\fR test can only be run if can switch to an ISO locale .TP \fIroot\fR test can only run if Unix user is root .TP \fInotRoot\fR test can only run if Unix user is not root .TP \fIeformat\fR test can only run if app has a working version of sprintf with respect to the "e" format of floating-point numbers. .TP \fIstdio\fR test can only be run if the current app can be spawned via a pipe .SH "RUNNING TEST FILES" Use the following command to run a test file that uses package tcltest: .DS ?