From 5cbefa50780d8b1b97ad53388a8d92029eb63aff Mon Sep 17 00:00:00 2001 From: jenn Date: Tue, 24 Oct 2000 22:30:17 +0000 Subject: * tests/all.tcl: Removed support for tcltest 1.0. * tests/tcltest.test: * library/tcltest1.0/tcltest.tcl: * library/tcltest1.0/pkgIndex.tcl: * docs/tcltest.n: Moved tcltest2 code so that it's the standard version of tcltest. Removed all tcltest2 files (tests/tcltest2.test, library/tcltest1.0/tcltest2.tcl, docs/tcltest2.n). --- ChangeLog | 14 +- doc/tcltest.n | 1007 +++++++---- doc/tcltest2.n | 1088 ------------ library/tcltest/pkgIndex.tcl | 3 +- library/tcltest/tcltest.tcl | 3457 +++++++++++++++++++++++++++----------- library/tcltest/tcltest2.tcl | 3490 --------------------------------------- library/tcltest1.0/pkgIndex.tcl | 3 +- library/tcltest1.0/tcltest.tcl | 3457 +++++++++++++++++++++++++++----------- library/tcltest1.0/tcltest2.tcl | 3490 --------------------------------------- tests/all.tcl | 47 +- tests/tcltest.test | 1148 +++++++++++-- tests/tcltest2.test | 1308 --------------- 12 files changed, 6753 insertions(+), 11759 deletions(-) delete mode 100755 doc/tcltest2.n delete mode 100755 library/tcltest/tcltest2.tcl delete mode 100755 library/tcltest1.0/tcltest2.tcl delete mode 100755 tests/tcltest2.test diff --git a/ChangeLog b/ChangeLog index f52c55f..33a883b 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,15 @@ +2000-10-24 Jennifer Hom + + * tests/all.tcl: Removed support for tcltest 1.0. + + * tests/tcltest.test: + * library/tcltest1.0/tcltest.tcl: + * library/tcltest1.0/pkgIndex.tcl: + * docs/tcltest.n: Moved tcltest2 code so that it's the standard + version of tcltest. Removed all tcltest2 files + (tests/tcltest2.test, library/tcltest1.0/tcltest2.tcl, + docs/tcltest2.n). + 2000-10-20 Jeff Hobbs * win/tclWinFile.c (TclpMatchFilesTypes): made the stat call only @@ -7,7 +19,7 @@ 2000-10-19 Jennifer Hom * library/tcltest1.0/tcltest2.tcl: - * tests/tcltest2.test + * tests/tcltest2.test: * doc/tcltest2.n: Code and documentation cleanup. Modified -verbose to take list of keywords as well as string of letters. Removed Tcl version information from tcltest. Removed diff --git a/doc/tcltest.n b/doc/tcltest.n index f1d9616..b9dc2c7 100644 --- a/doc/tcltest.n +++ b/doc/tcltest.n @@ -2,48 +2,97 @@ '\" Copyright (c) 1990-1994 The Regents of the University of California '\" Copyright (c) 1994-1997 Sun Microsystems, Inc. '\" Copyright (c) 1998-1999 Scriptics Corporation +'\" Copyright (c) 2000 Ajuba Solutions '\" '\" See the file "license.terms" for information on usage and redistribution '\" of this file, and for a DISCLAIMER OF ALL WARRANTIES. '\" -'\" RCS: @(#) $Id: tcltest.n,v 1.9 1999/12/22 00:41:13 hobbs Exp $ +'\" RCS: @(#) $Id: tcltest.n,v 1.10 2000/10/24 22:30:25 jenn Exp $ '\" .so man.macros -.TH "Tcltest" n 8.2 Tcl "Tcl Built-In Commands" +.TH "tcltest" n 8.4 Tcl "Tcl Built-In Commands" .BS '\" Note: do not modify the .SH NAME line immediately below! .SH NAME -Tcltest \- Test harness support code and utilities +tcltest \- Test harness support code and utilities .SH SYNOPSIS -\fBpackage require tcltest ?1.0?\fP +\fBpackage require tcltest ?2.0?\fP .sp -\fB::tcltest::test \fIname desc ?constraint? script expectedAnswer\fR +\fBtcltest::test \fIname desc ?option value? ?option value? ...\fR +.br +\fBtcltest::test \fIname desc {?option value? ?option value? ...}\fR .sp -\fB::tcltest::cleanupTests \fI?runningMultipleTests?\fR +\fBtcltest::cleanupTests \fI?runningMultipleTests?\fR .sp -\fB::tcltest::getMatchingTestFiles\fR +\fBtcltest::runAllTests\fR .sp -\fB::tcltest::loadTestedCommands\fR +\fBtcltest::interpreter \fI?interp?\fR .sp -\fB::tcltest::makeFile \fIcontents name\fR +\fBtcltest::singleProcess \fI?boolean?\fR .sp -\fB::tcltest::removeFile \fIname\fR +\fBtcltest::debug \fI?level?\fR .sp -\fB::tcltest::makeDirectory \fIname\fR +\fBtcltest::verbose \fI?levelList?\fR .sp -\fB::tcltest::removeDirectory \fIname\fR +\fBtcltest::preserveCore \fI?level?\fR .sp -\fB::tcltest::viewFile \fIname\fR +\fBtcltest::testConstraint \fIconstraint ?value?\fR .sp -\fB::tcltest::normalizeMsg \fImsg\fR +\fBtcltest::limitConstraints \fI?constraintList?\fR .sp -\fB::tcltest::bytestring \fIstring\fR +\fBtcltest::workingDirectory \fI?dir?\fR .sp -\fB::tcltest::saveState\fR +\fBtcltest::temporaryDirectory \fI?dir?\fR .sp -\fB::tcltest::restoreState\fR +\fBtcltest::testsDirectory \fI?dir?\fR .sp -\fB::tcltest::threadReap\fR +\fBtcltest::match \fI?patternList?\fR +.sp +\fBtcltest::matchFiles \fI?patternList?\fR +.sp +\fBtcltest::matchDirectories \fI?patternList?\fR +.sp +\fBtcltest::skip \fI?patternList?\fR +.sp +\fBtcltest::skipFiles \fI?patternList?\fR +.sp +\fBtcltest::skipDirectories \fI?patternList?\fR +.sp +\fBtcltest::loadTestedCommands\fR +.sp +\fBtcltest::loadScript \fI?script?\fR +.sp +\fBtcltest::loadFile \fI?filename?\fR +.sp +\fBtcltest::outputChannel \fI?channelID?\fR +.sp +\fBtcltest::outputFile \fI?filename?\fR +.sp +\fBtcltest::errorChannel \fI?channelID?\fR +.sp +\fBtcltest::errorFile \fI?filename?\fR +.sp +\fBtcltest::makeFile \fIcontents name ?directory?\fR +.sp +\fBtcltest::removeFile \fIname ?directory?\fR +.sp +\fBtcltest::makeDirectory \fIname ?directory?\fR +.sp +\fBtcltest::removeDirectory \fIname ?directory?\fR +.sp +\fBtcltest::viewFile \fIname ?directory?\fR +.sp +\fBtcltest::normalizeMsg \fImsg\fR +.sp +\fBtcltest::normalizePath \fIpathVar\fR +.sp +\fBtcltest::bytestring \fIstring\fR +.sp +\fBtcltest::saveState\fR +.sp +\fBtcltest::restoreState\fR +.sp +\fBtcltest::threadReap\fR .BE .SH DESCRIPTION .PP @@ -55,81 +104,262 @@ The Tcl test suite consists of multiple .test files, each of which contains multiple test cases. Each test case consists of a call to the test command, which specifies the name of test, a short description, any constraints that apply to the test case, the script -to be run, and expected results. See the sections \fI"Tests,"\fR -\fI"Test Constraints,"\fR and \fI"Test Files and How to Run Them"\fR -for more details. +to be run, and expected results. See the \fI"Tests"\fR section for more +details. .PP It is also possible to add to this test harness to create your own customized test harness implementation. For more defails, see the section \fI"How to Customize the Test Harness"\fR. -.PP -This approach to testing was designed and initially implemented by -Mary Ann May-Pumphrey of Sun Microsystems in the early 1990's. Many -thanks to her for donating her work back to the public Tcl release. .SH COMMANDS .TP -\fB::tcltest::test\fP \fIname desc ?constraints? script expectedAnswer\fR -The \fB::tcltest::test\fR command runs\fIscript\fR and compares -its result to \fIexpectedAnswer\fR. It prints an error message if the two do -not match. If \fB::tcltest::verbose\fR contains "p" or "s", it also prints -out a message if the test passed or was skipped. The test will be -skipped if it doesn't match the \fB::tcltest::match\fR variable, if it -matches an element in \fB::tcltest::skip\fR, or if one of the elements -of \fIconstraint\fR turns out not to be true. The -\fB::tcltest::test\fR command has no defined return values. See the -\fI"Writing a new test"\fR section for more details on this command. -.TP -\fB::tcltest::cleanupTests\fP \fI?runningMultipleTests?\fR +\fBtcltest::test\fP \fIname desc ?option value? ?option value? ...\fR +.TP +\fBtcltest::test\fP \fIname desc {?option value? ?option value? ...}\fR +The \fBtcltest::test\fR command runs the value supplied for attribute +\fIscript\fR and compares its result to possible results. +It prints an error message if actual results and expected results do +not match. The \fBtcltest::test\fR command returns 0 if it completes +successfully. Any other return value indicates that an error has +occurred in the tcltest package. See the \fI"Tests"\fR section for +more details on this command. +.TP +\fBtcltest::cleanupTests\fP \fI?runningMultipleTests?\fR This command should be called at the end of a test file. It prints statistics about the tests run and removes files that were created by -\fB::tcltest::makeDirectory\fR and \fB::tcltest::makeFile\fR. Names +\fBtcltest::makeDirectory\fR and \fBtcltest::makeFile\fR. Names of files and directories created outside of -\fB::tcltest::makeFile\fR and \fB::tcltest::makeDirectory\fR and -never deleted are printed to \fB::tcltest::outputChannel\fR. This command +\fBtcltest::makeFile\fR and \fBtcltest::makeDirectory\fR and +never deleted are printed to \fBtcltest::outputChannel\fR. This command also restores the original shell environment, as described by the ::env -array. \fIcalledFromAll\fR should be specified when -\fB::tcltest::cleanupTests\fR is called from an "all.tcl" file. Tcl files -files are generally used to run multiple tests. For more details on how to -run multiple tests, please see the section \fI"Running test files"\fR. -This proc has no defined return value. -.TP -\fB::tcltest::getMatchingTestFiles\fP -This command is used when you want to run multiple test files. It returns -the list of tests that should be sourced in an 'all.tcl' file. See the -section \fI"Running test files"\fR for more information. -.TP -\fB::tcltest::loadTestedCommands\fP +array. \fIcalledFromAll\fR should be specified if +\fBtcltest::cleanupTests\fR is called explicitly from an "all.tcl" +file. Tcl files files are generally used to run multiple tests. For +more details on how to run multiple tests, please see the section +\fI"Running test files"\fR. This proc has no defined return value. +.TP +\fBtcltest::runAllTests\fP +This command should be used in your 'all.tcl' file. It is used to +loop over test files and directories, determining which test files to +run and then running them. Note that this test calls +tcltest::cleanupTests; if using this proc in your 'all.tcl' file, you +should not call tcltest::cleanupTests explicitly in that file. See the +sample 'all.tcl' file in the \fI"Examples"\fR section. +.TP +\fBtcltest::interpreter\fR \fI?executableName?\fR +Sets or returns the name of the executable used to invoke the test +suite. This is the interpreter used in runAllTests to run test files +if singleProcess is set to false. The default value for interpreter +is the name of the interpreter in which the tests were started. +.TP +\fBtcltest::singleProcess\fR \fI?boolean?\fR +Sets or returns a boolean indicating whether test files should be sourced +into the current interpreter by runAllTests or run in their own +processes. If \fIboolean\fR is true (1), tests are sourced into the +current interpreter. If \fIboolean\fR is false (0), tests are run in +the interpreter specified in tcltest::interpreter. The default value +for tcltest::singleProcess is false. +.TP +\fBtcltest::debug\fR \fI?level?\fR +Sets or returns the current debug level. The debug level determines +how much tcltest package debugging information is printed to stdout. +The default debug level is 0. Levels are defined as: +.RS +.IP 0 +Do not display any debug information. +.IP 1 +Display information regarding whether a test is skipped because it +doesn't match any of the tests that were specified using -match or +tcltest::match (userSpecifiedNonMatch) or matches any of the tests +specified by -skip or tcltest::skip (userSpecifiedSkip). +.IP 2 +Display the flag array parsed by the command line processor, the +contents of the ::env array, and all user-defined variables that exist +in the current namespace as they are used. +.IP 3 +Display information regarding what individual procs in the test +harness are doing. +.RE +.TP +\fBtcltest::verbose\fR \fI?levelList?\fR +Sets or returns the current verbosity level. The default verbosity +level is "body". See the "Test output" section for a more detailed +explanation of this option. Levels are defined as: +.RS +.IP body +Display the body of failed tests +.IP pass +Print output when a test passes +.IP skip +Print output when a test is skipped +.IP start +Print output whenever a test starts +.IP error +Print errorInfo and errorCode, if they exist, when a test return code +does not match its expected return code +.RE +.TP +\fBtcltest::preserveCore\fR \fI?level?\fR +Sets or returns the current core preservation level. This level +determines how stringent checks for core files are. The default core +preservation level is 0. Levels are defined as: +.RS +.IP 0 +No checking - do not check for core files at the end of each test +command, but do check for them whenever tcltest::cleanupTests is +called from tcltest::runAllTests. +.IP 1 +Check for core files at the end of each test command and whenever +tcltest::cleanupTests is called from tcltest::runAllTests. +.IP 2 +Check for core files at the end of all test commands and whenever +tcltest::cleanupTests is called from all.tcl. Save any core files +produced in tcltest::temporaryDirectory. +.RE +.TP +\fBtcltest::testConstraint \fIconstraint ?value?\fR +Sets or returns the value associated with the named \fIconstraint\fR. +See the section \fI"Test constraints"\fR for more information. +.TP +\fBtcltest::limitConstraints \fI?constraintList?\fR +Sets or returns a boolean indicating whether testing is being limited +to constraints listed in \fIconstraintList\fR. +If limitConstraints is not false, only those tests with constraints matching +values in \fIconstraintList\fR will be run. +.TP +\fBtcltest::workingDirectory\fR \fI?directoryName?\fR +Sets or returns the directory in which the test suite is being run. +The default value for workingDirectory is the directory in which the +test suite was launched. +.TP +\fBtcltest::temporaryDirectory\fR \fI?directoryName?\fR +Sets or returns the output directory for temporary files created by +tcltest::makeFile and tcltest::makeDirectory. This defaults to the +directory returned by \fItcltest::workingDirectory\fR. +.TP +\fBtcltest::testsDirectory\fR \fI?directoryName?\fR +Sets or returns the directory where the tests reside. This defaults +to the directory returned by \fItcltest::workingDirectory\fR +if the script cannot determine where the \fItests\fR directory is +located. This variable should be explicitly set if tests are being run +from an all.tcl file. +.TP +\fBtcltest::match\fR \fI?globPatternList?\fR +Sets or returns the glob pattern list that determines which tests +should be run. Only tests which match one of the glob patterns in +\fIglobPatternList\fR are run by the test harness. The default value +for \fIglobPatternList\fR is '*'. +.TP +\fBtcltest::matchFiles\fR \fI?globPatternList?\fR +Sets or returns the glob pattern list that determines which test files +should be run. Only test files which match one of the glob patterns in +\fIglobPatternList\fR are run by the test harness. The default value +for \fIglobPatternList\fR is '*.test'. +.TP +\fBtcltest::matchDirectories\fR \fI?globPatternList?\fR +Sets or returns the glob pattern list that determines which test +subdirectories of the current test directory should be run. Only test +subdirectories which match one of the glob patterns in +\fIglobPatternList\fR are run by the test harness. The default value +for \fIglobPatternList\fR is '*'. +.TP +\fBtcltest::skip\fR \fI?globPatternList?\fR +Sets or returns the glob pattern list that determines which tests (of +those matched by tcltest::match) should be skipped. The default value +for \fIglobPatternList\fR is {}. +.TP +\fBtcltest::skipFiles\fR \fI?globPatternList?\fR +Sets or returns the glob pattern list that determines which test files +(of those matched by tcltest::matchFiles) should be skipped. The +default value for \fIglobPatternList\fR is {}. +.TP +\fBtcltest::skipDirectories\fR \fI?globPatternList?\fR +Sets or returns the glob pattern list that determines which test +subdirectories (of those matched by tcltest::matchDirectories) should +be skipped. The default value for \fIglobPatternList\fR is {}. +.TP +\fBtcltest::loadTestedCommands\fP This command uses the script specified via the \fI-load\fR or -\fI-loadfile\fR to load the commands checked by the test suite. -Allowed to be empty, as the tested commands could have been compiled -into the interpreter running the test suite. +\fI-loadfile\fR options or the tcltest::loadScript or +tcltest::loadFile procs to load the commands checked by the test suite. +It is allowed to be empty, as the tested commands could have been +compiled into the interpreter running the test suite. +.TP +\fBtcltest::loadScript\fR \fI?script?\fR +Sets or returns the script executed by \fBloadTestedCommands\fR. +.TP +\fBtcltest::loadFile\fR \fI?filename?\fR +Sets ore returns the file name associated with the script executed +\fBloadTestedCommands\fR. If setting \fIfilename\fR, this proc will +open the file and call \fItcltest::loadScript\fR with the content. +.TP +\fBtcltest::outputChannel\fR \fI?channelID?\fR +Sets or returns the output file ID. This defaults to stdout. +Any test that prints test related output should send +that output to \fItcltest::outputChannel\fR rather than letting +that output default to stdout. +.TP +\fBtcltest::outputFile\fR \fI?filename?\fR +Sets or returns the file name corresponding to the output file. This +defaults to stdout. This proc calls +outputChannel to set the output file channel. +Any test that prints test related output should send +that output to \fItcltest::outputChannel\fR rather than letting +that output default to stdout. +.TP +\fBtcltest::errorChannel\fR \fI?channelID?\fR +Sets or returns the error file ID. This defaults to stderr. +Any test that prints error messages should send +that output to \fItcltest::errorChannel\fR rather than printing +directly to stderr. +.TP +\fBtcltest::errorFile\fR \fI?filename?\fR +Sets or returns the file name corresponding to the error file. This +defaults to stderr. This proc calls +errorChannel to set the error file channel. +Any test that prints test related error output should send +that output to \fItcltest::errorChannel\fR or +\fItcltest::outputChannel\fR rather than letting +that output default to stdout. .TP -\fB::tcltest::makeFile\fP \fIcontents name\fR +\fBtcltest::makeFile\fP \fIcontents name ?directory?\fR Create a file that will be automatically be removed by -\fB::tcltest::cleanupTests\fR at the end of a test file. -This proc has no defined return value. +\fBtcltest::cleanupTests\fR at the end of a test file. This file is +created relative to \fIdirectory\fR. If left unspecified, +\fIdirectory\fR defaults to tcltest::temporaryDirectory. +Returns the full path of the file created. .TP -\fB::tcltest::removeFile\fP \fIname\fR +\fBtcltest::removeFile\fP \fIname ?directory?\fR Force the file referenced by \fIname\fR to be removed. This file name -should be relative to \fI::tcltest::temporaryDirectory\fR. This proc has no -defined return values. +should be relative to \fIdirectory\fR. If left unspecified, +\fIdirectory\fR defaults to tcltest::temporaryDirectory. This proc +has no defined return values. .TP -\fB::tcltest::makeDirectory\fP \fIname\fR +\fBtcltest::makeDirectory\fP \fIname ?directory?\fR Create a directory named \fIname\fR that will automatically be removed -by \fB::tcltest::cleanupTests\fR at the end of a test file. This proc -has no defined return value. -.TP -\fB::tcltest::removeDirectory\fP \fIname\fR -Force the directory referenced by \fIname\fR to be removed. This proc +by \fBtcltest::cleanupTests\fR at the end of a test file. This +directory is created relative to tcltest::temporaryDirectory. +Returns the full path of the directory created. +.TP +\fBtcltest::removeDirectory\fP \fIname\fR +Force the directory referenced by \fIname\fR to be removed. This +directory should be relative to \fIdirectory\fR. If left unspecified, +\fIdirectory\fR defaults to tcltest::temporaryDirectory. This proc has no defined return value. .TP -\fB::tcltest::viewFile\fP \fIfile\fR -Returns the contents of \fIfile\fR. +\fBtcltest::viewFile\fP \fIfile ?directory?\fR +Returns the contents of \fIfile\fR. This file name +should be relative to \fIdirectory\fR. If left unspecified, +\fIdirectory\fR defaults to tcltest::temporaryDirectory. .TP -\fB::tcltest::normalizeMsg\fP \fImsg\fR +\fBtcltest::normalizeMsg\fP \fImsg\fR Remove extra newlines from \fImsg\fR. .TP -\fB::tcltest::bytestring\fP \fIstring\fR +\fBtcltest::normalizePath\fP \fIpathVar\fR +Resolves symlinks in a path, thus creating a path without internal +redirection. It is assumed that \fIpathVar\fR is absolute. +\fIpathVar\fR is modified in place. +.TP +\fBtcltest::bytestring\fP \fIstring\fR Construct a string that consists of the requested sequence of bytes, as opposed to a string of properly formed UTF-8 characters using the value supplied in \fIstring\fR. This allows the tester to create @@ -137,158 +367,178 @@ denormalized or improperly formed strings to pass to C procedures that are supposed to accept strings with embedded NULL types and confirm that a string result has a certain pattern of bytes. .TP -\fB::tcltest::saveState\fP -\fB::tcltest::restoreState\fP -Save and restore the procedure and global variable names. -A test file might contain calls to \fB::tcltest::saveState\fR and +\fBtcltest::saveState\fP +Save procedure and global variable names. +A test file might contain calls to \fBtcltest::saveState\fR and +\fB::tcltest:restoreState\fR if it creates or deletes global variables +or procs. +.TP +\fBtcltest::restoreState\fP +Restore procedure and global variable names. +A test file might contain calls to \fBtcltest::saveState\fR and \fB::tcltest:restoreState\fR if it creates or deletes global variables or procs. .TP -\fB::tcltest::threadReap\fP -\fB::tcltest::threadReap\fR only works if \fItestthread\fR is +\fBtcltest::threadReap\fP +\fBtcltest::threadReap\fR only works if \fItestthread\fR is defined, generally by compiling tcltest. If \fItestthread\fR is -defined, \fB::tcltest::threadReap\fR kills all threads except for the +defined, \fBtcltest::threadReap\fR kills all threads except for the main thread. It gets the ID of the main thread by calling \fItestthread names\fR during initialization. This value is stored in -\fI::tcltest::mainThread\fR. \fB::tcltest::threadReap\fR returns the +\fItcltest::mainThread\fR. \fBtcltest::threadReap\fR returns the number of existing threads at completion. +.TP +\fBtcltest::mainThread\fR +Sets or returns the main thread ID. This defaults to 1. This is the +only thread that is not killed by tcltest::threadReap and is set +according to the return value of \fItestthread names\fR at +initialization. .SH TESTS The \fBtest\fR procedure runs a test script and prints an error message if the script's result does not match the expected result. -The following is the spec for the \fBtest\fR command: +Two syntaxes are provided for specifying the attributes of the tests. +The first uses a separate argument for each of the attributes and +values. The second form places all of the attributes and values +together into a single argument; the argument must have proper list +structure, with teh elements of the list being the attributes and +values. The second form makes it easy to construct multi-line +scripts, since the braces around the whole list make it unnecessary to +include a backslash at the end of each line. In the second form, no +command or variable substitutions are performed on the attribute +names. This makes the behavior of the second form different from the +first form in some cases. +.PP +The first form for the \fBtest\fR command: +.DS +test \fIname\fR \fIdescription\fR + ?-constraints \fIkeywordList|expression\fR + ?-setup \fIsetupScript\fR? + ?-body \fItestScript\fR? + ?-cleanup \fIcleanupScript\fR? + ?-result \fIexpectedAnswer\fR? + ?-output \fIexpectedOutput\fR? + ?-errorOutput \fIexpectedError\fR? + ?-returnCodes \fIcodeList\fR? + ?-match \fIexact|glob|regexp\fR? +.DE +.PP +The second form for the \fBtest\fR command: .DS -test ??