From fb0a8df3a872475d55afadea90cfa60033f81266 Mon Sep 17 00:00:00 2001 From: dgp Date: Tue, 2 Jul 2002 13:28:51 +0000 Subject: * library/tcltest/tcltest.tcl: Simplified logic of [GetMatchingFiles] and [GetMatchingDirectories], removing special case processing. * doc/tcltest.n: More documentation updates. Reference sections are complete. Only examples need adding. --- ChangeLog | 8 + doc/tcltest.n | 598 ++++++++++++++------------------------------ library/tcltest/tcltest.tcl | 126 +++++----- 3 files changed, 262 insertions(+), 470 deletions(-) diff --git a/ChangeLog b/ChangeLog index 0af89ff..105fb39 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,11 @@ +2002-07-02 Don Porter + + * library/tcltest/tcltest.tcl: Simplified logic of [GetMatchingFiles] + and [GetMatchingDirectories], removing special case processing. + + * doc/tcltest.n: More documentation updates. Reference sections + are complete. Only examples need adding. + 2002-07-02 Vince Darley * tests/fCmd.test: diff --git a/doc/tcltest.n b/doc/tcltest.n index bffa05b..4982d7c 100644 --- a/doc/tcltest.n +++ b/doc/tcltest.n @@ -8,7 +8,7 @@ '\" See the file "license.terms" for information on usage and redistribution '\" of this file, and for a DISCLAIMER OF ALL WARRANTIES. '\" -'\" RCS: @(#) $Id: tcltest.n,v 1.24 2002/07/01 22:33:20 dgp Exp $ +'\" RCS: @(#) $Id: tcltest.n,v 1.25 2002/07/02 13:28:51 dgp Exp $ '\" .so man.macros .TH "tcltest" n 2.1 tcltest "Tcl Bundled Packages" @@ -510,16 +510,19 @@ easily capture output with the [\fBconfigure -outfile\fR] and [\fBconfigure -errfile\fR] options, and so that the \fB-output\fR and \fB-errorOutput\fR attributes work properly. .SH "TEST CONSTRAINTS" +.PP Constraints are used to determine whether or not a test should be skipped. Each constraint has a name, which may be any string, and a boolean value. Each [\fBtest\fR] has a \fB-constraints\fR value which is a -list of constraint names. The test will run only if all constraints -in the list are true-valued. Thus, the \fB-constraints\fR option -of [\fBtest\fR] is a convenient, symbolic way to define any conditions -required for the test to be possible or meaningful. For example, -a [\fBtest\fR] with \fB-constraints unix\fR will only be run -if the constraint \fBunix\fR is true, which indicates the test suite -is being run on a Unix platform. +list of constraint names. There are two modes of constraint control. +Most frequently, the default mode is used, indicated by a setting +of [\fBconfigure -limitconstraints\fR] to false. The test will run +only if all constraints in the list are true-valued. Thus, +the \fB-constraints\fR option of [\fBtest\fR] is a convenient, symbolic +way to define any conditions required for the test to be possible or +meaningful. For example, a [\fBtest\fR] with \fB-constraints unix\fR +will only be run if the constraint \fBunix\fR is true, which indicates +the test suite is being run on a Unix platform. .PP Each [\fBtest\fR] should include whatever \fB-constraints\fR are required to constrain it to run only where appropriate. Several @@ -639,35 +642,102 @@ to the "e" format of floating-point numbers. \fIstdio\fR test can only be run if the current app can be spawned via a pipe .PP - -Add description of -constraints and -limitconstraints here. - +The alternative constraint mode is enabled by setting the +value of [\fBconfigure -limitconstraints\fR] to true. When +that configuration is set, all existing constraints other than +those in the constraint list returned by [\fBconfigure -constraints\fR] +are set to false. When the value of [\fBconfigure -constraints\fR] +is set, all those constraints are set to true. The effect is that +when both configuration options \fB-constraints\fR and +\fB-limitconstraints\fR are in use, only those tests including +only constraints from the [\fBconfigure -constraints\fR] list +are run; all others are skipped. For example, one might set +up a configuration with +.CS +configure -constraints knownBug -limitconstraints true -verbose pass +.CE +to run exactly those tests that exercise known bugs, and discover +whether any of them pass, indicating the bug had been fixed. .SH "RUNNING ALL TESTS" - +.PP +The single command [\fBrunAllTests\fR] is evaluated to run an entire +test suite, spanning many files and directories. The configuration +options of \fBtcltest\fR control the precise operations. The +[\fBrunAllTests\fR] command begins by printing a summary of its +configuration to [\fBoutputChannel\fR]. +.PP +Test files to be evaluated are sought in the directory +[\fBconfigure -testdir\fR]. The list of files in that directory +that match any of the patterns in [\fBconfigure -file\fR] and +match none of the patterns in [\fBconfigure -notfile] is generated +and sorted. Then each file will be evaluated in turn. If +[\fBconfigure -singleproc\fR] is true, then each file will +be [\fBsource\fR]d in the caller's context. If if is false, +then a copy of [\fBinterpreter\fR] will be [\fBexec\fR]d to +evaluate each file. The multi-process operation is useful +when testing can cause errors so severe that a process +terminates. Although such an error may terminate a child +process evaluating one file, the master process can continue +with the rest of the test suite. In multi-process operation, +the configuration of \fBtcltest\fR in the master process is +passed to the child processes as command line arguments, +with the exception of [\fBconfigure -outfile\fR]. The +master process collects all output from the child process +and collates their results into one master report. Any +reports of individual test failures, or messages requested +by a [\fBconfigure -verbose\fR] setting are passed directly +on to [\fBoutputChannel\fR] by the master process. +.PP +After evaluating all selected test files, a summary of the +results is printed to [\fBoutputChannel\fR]. The summary +includes the total number of [\fBtest\fR]s evaluated, broken +down into those skipped, those passed, and those failed. +Is also notes the number of files evaluated, and the names +of any files with failing tests, or errors. A list of +the constraints that caused tests to be skipped, and the +number of tests skipped for each is also printed. Also, +messages are printed if it appears that evaluation of +a test file has caused any temporary files to be left +behind in [\fBconfigure -tmpdir\fR]. +.PP +Having completed and summarized all selected test files, +[\fBrunAllTests\fR] then recursively acts on subdirectories +of [\fBconfigure -testdir\fR]. All subdirectories that +match any of the patterns in [\fBconfigure -relateddir\fR] +and do not match any of the patterns in +[\fBconfigure -asidefromdir\fR] are examined. If +a file named \fBall.tcl\fR is found in such a directory, +it will be [\fBsource\fR]d in the caller's context. +Whether or not an examined directory contains an +\fBall.tcl\fR file, its subdirectories are also scanned +against the [\fBconfigure -relateddir\fR] and +[\fBconfigure -asidefromdir\fR] patterns. In this way, +many directories in a directory tree can have all their +test files evaluated by a single [\fBrunAllTests\fR] +command. .SH "CONFIGURABLE OPTIONS" - -.TP -\fBsingleProcess\fR \fI?value?\fR -Sets or returns a boolean indicating whether test files should be sourced -into the current interpreter by runAllTests or run in their own -processes. If \fIvalue\fR is true (1), tests are sourced into the -current interpreter. If \fIvalue\fR is false (0), tests are run in -the interpreter specified in tcltest::interpreter. The default value -for tcltest::singleProcess is false. - -.TP -\fBdebug\fR \fI?level?\fR -Sets or returns the current debug level. The debug level determines -how much tcltest package debugging information is printed to stdout. -The default debug level is 0. Levels are defined as: +The [\fBconfigure\fR] command is used to set and query the configurable +options of \fBtcltest\fR. The valid options are: +.TP +\fB-singleproc \fIboolean\fR +Controls whether or not [\fBrunAllTests\fR] spawns a child process for +each test file. No spawning when \fIboolean\fR is true. Default +value is false. +.TP +\fB-debug \fIlevel\fR +Sets the debug level to \fIlevel\fR, an integer value indicating how +much debugging information should be printed to stdout. Note that +debug messages always go to stdout, independent of the value of +[\fBconfigure -outfile\fR]. Default value is 0. Levels are defined as: .RS .IP 0 Do not display any debug information. .IP 1 Display information regarding whether a test is skipped because it -doesn't match any of the tests that were specified using -match or -tcltest::match (userSpecifiedNonMatch) or matches any of the tests -specified by -skip or tcltest::skip (userSpecifiedSkip). +doesn't match any of the tests that were specified using by +[\fBconfigure -match\fR] (userSpecifiedNonMatch) or matches any of +the tests specified by [\fBconfigure -skip\fR] (userSpecifiedSkip). Also +print warnings about possible lack of cleanup or balance in test files. .IP 2 Display the flag array parsed by the command line processor, the contents of the ::env array, and all user-defined variables that exist @@ -676,319 +746,115 @@ in the current namespace as they are used. Display information regarding what individual procs in the test harness are doing. .RE - .TP -\fBverbose\fR \fI?levelList?\fR -Sets or returns the current verbosity level. The default verbosity -level is "body". See the "Test output" section for a more detailed -explanation of this option. Levels are defined as: +\fB-verbose \fIlevel\fR +Sets the type of output verbosity desired to \fIlevel\fR, +a list of zero or more of the elements \fBbody\fR, \fBpass\fR, +\fBskip\fR, \fBstart\fR, and \fBerror\fR. Default value is \fBbody\fR. +Levels are defined as: .RS -.IP body +.IP body (b) Display the body of failed tests -.IP pass +.IP pass (p) Print output when a test passes -.IP skip +.IP skip (s) Print output when a test is skipped -.IP start +.IP start (t) Print output whenever a test starts -.IP error +.IP error (e) Print errorInfo and errorCode, if they exist, when a test return code does not match its expected return code .RE - -.TP -\fBtcltest::preserveCore\fR \fI?level?\fR -Sets or returns the current core preservation level. This level -determines how stringent checks for core files are. The default core -preservation level is 0. Levels are defined as: +The single letter abbreviations noted above are also recognized +so that [\fBconfigure -verbose pt\fR] is the same as +[\fBconfigure -verbose {pass start}\fR]. +.TP +\fB-preservecore \fIlevel\fR +Sets the core preservation level to \fIlevel\fR. This level +determines how stringent checks for core files are. Default +value is 0. Levels are defined as: .RS .IP 0 No checking - do not check for core files at the end of each test -command, but do check for them whenever tcltest::cleanupTests is -called from tcltest::runAllTests. +command, but do check for them in [\fBrunAllTests\fR] after all +test files have been evaluated. .IP 1 -Check for core files at the end of each test command and whenever -tcltest::cleanupTests is called from tcltest::runAllTests. +Also check for core files at the end of each [\fBtest\fR] command. .IP 2 -Check for core files at the end of all test commands and whenever -tcltest::cleanupTests is called from all.tcl. Save any core files -produced in tcltest::temporaryDirectory. +Check for core files at all times described above, and save a +copy of each core file produced in [\fBconfigure -tmpdir\fR]. .RE - -.TP -\fBtcltest::limitConstraints \fI?value?\fR -Sets or returns a boolean indicating whether testing is being limited -to the list of constraints specified by the \fB-constraints\fR -command line option. If \fIvalue\fR is true, only those tests -with constraints present in the list specified in the \fB-constraints\fR -command line option. - -.TP -\fBtcltest::temporaryDirectory\fR \fI?directoryName?\fR -Sets or returns the output directory for temporary files created by -tcltest::makeFile and tcltest::makeDirectory. This defaults to the -directory returned by \fItcltest::workingDirectory\fR. - -.TP -\fBtcltest::testsDirectory\fR \fI?directoryName?\fR -Sets or returns the directory where the tests reside. This defaults -to the directory returned by \fItcltest::workingDirectory\fR -if the script cannot determine where the \fItests\fR directory is -located. This variable should be explicitly set if tests are being run -from an all.tcl file. - -.TP -\fBtcltest::match\fR \fI?globPatternList?\fR -Sets or returns the glob pattern list that determines which tests -should be run. Only tests which match one of the glob patterns in -\fIglobPatternList\fR are run by the test harness. The default value -for \fIglobPatternList\fR is '*'. - -.TP -\fBtcltest::matchFiles\fR \fI?globPatternList?\fR -Sets or returns the glob pattern list that determines which test files -should be run. Only test files which match one of the glob patterns in -\fIglobPatternList\fR are run by the test harness. The default value -for \fIglobPatternList\fR is '*.test'. - -.TP -\fBtcltest::matchDirectories\fR \fI?globPatternList?\fR -Sets or returns the glob pattern list that determines which test -subdirectories of the current test directory should be run. Only test -subdirectories which match one of the glob patterns in -\fIglobPatternList\fR are run by the test harness. The default value -for \fIglobPatternList\fR is '*'. - -.TP -\fBtcltest::skip\fR \fI?globPatternList?\fR -Sets or returns the glob pattern list that determines which tests (of -those matched by tcltest::match) should be skipped. The default value -for \fIglobPatternList\fR is {}. - -.TP -\fBtcltest::skipFiles\fR \fI?globPatternList?\fR -Sets or returns the glob pattern list that determines which test files -(of those matched by tcltest::matchFiles) should be skipped. The -default value for \fIglobPatternList\fR is {}. - -.TP -\fBtcltest::skipDirectories\fR \fI?globPatternList?\fR -Sets or returns the glob pattern list that determines which test -subdirectories (of those matched by tcltest::matchDirectories) should -be skipped. The default value for \fIglobPatternList\fR is {}. - -.TP -\fBtcltest::loadScript\fR \fI?script?\fR -Sets or returns the script executed by \fBloadTestedCommands\fR. - -.TP -\fBtcltest::loadFile\fR \fI?filename?\fR -Sets ore returns the file name associated with the script executed -\fBloadTestedCommands\fR. If setting \fIfilename\fR, this proc will -open the file and call \fItcltest::loadScript\fR with the content. - -.TP -\fBtcltest::outputFile\fR \fI?filename?\fR -Sets or returns the file name corresponding to the output file. This -defaults to stdout. This proc calls -outputChannel to set the output file channel. -Any test that prints test related output should send -that output to \fItcltest::outputChannel\fR rather than letting -that output default to stdout. - .TP -\fBtcltest::errorFile\fR \fI?filename?\fR -Sets or returns the file name corresponding to the error file. This -defaults to stderr. This proc calls -errorChannel to set the error file channel. -Any test that prints test related error output should send -that output to \fItcltest::errorChannel\fR or -\fItcltest::outputChannel\fR rather than letting -that output default to stdout. +\fB-limitconstraints \fIboolean\fR +Sets the mode by which [\fBtest\fR] honors constraints as described +in TESTS above. Default value is false. +.TP +\fB-constraints \fIlist\fR +Sets all the constraints in \fIlist\fR to true. Also used in +combination with \fB-limitconstraints true\fR to control an +alternative constraint mode as described in TESTS above. Default +value is an empty list. +.TP +\fB-tmpdir \fIdirectory\fR +Sets the temporary directory to be used by [\fBmakeFile\fR], +[\fBmakeDirectory\fR], [\fBviewFile\fR], [\fBremoveFile\fR], +and [\fBremoveDirectory\fR] as the default directory where +temporary files and directories created by test files should +be created. Default value is [\fBworkingDirectory\fR]. +.TP +\fB-testdir \fIdirectory\fR +Sets the directory searched by [\fBrunAllTests\fR] for test files +and subdirectories. Default value is [\fBworkingDirectory\fR]. +.TP +\fB-file \fIpatternList\fR +Sets the list of patterns used by [\fBrunAllTests\fR] to determine +what test files to evaluate. Default value is \fB*.test\fR. +.TP +\fB-notfile \fBpatternList\fR +Sets the list of patterns used by [\fBrunAllTests\fR] to determine +what test files to skip. Default value is \fBl.*.test\fR, so +that any SCCS lock files are skipped. +.TP +\fB-relateddir \fBpatternList\fR +Sets the list of patterns used by [\fBrunAllTests\fR] to determine +what subdirectories to search for an \fBall.tcl\fR file. Default +value is \fB*\fR. +.TP +\fB-asidefromdir \fBpatternList\fR +Sets the list of patterns used by [\fBrunAllTests\fR] to determine +what subdirectories to skip when searching for an \fBall.tcl\fR file. +Default value is an empty list. +.TP +\fB-match \fIpatternList\fR +Set the list of patterns used by [\fBtest\fR] to determine whether +a test should be run. Default value is \fB*\fR. +.TP +\fB-skip \fIpatternList\fR +Set the list of patterns used by [\fBtest\fR] to determine whether +a test should be skipped. Default value is an empty list. +.TP +\fB-load \fIscript\fR +Sets a script to be evaluated by [\fBloadTestedCommands\fR]. +Default value is an empty script. +.TP +\fB-loadfile \fIfilename\fR +Sets the filename from which to read a script to be evaluated +by [\fBloadTestedCommands\fR]. This is an alternative to +\fB-load\fR. They cannot be used together. +.TP +\fB-outfile \fIfilename\fR +Sets the file to which all output produced by tcltest should be +written. A file named \fIfilename\fR will be [\fBopen\fR]ed for writing, +and the resulting channel will be set as the value of [\fBoutputChannel\fR]. +.TP +\fB-errfile \fIfilename\fR +Sets the file to which all error output produced by tcltest +should be written. A file named \fIfilename\fR will be [\fBopen\fR]ed +for writing, and the resulting channel will be set as the value +of [\fBerrorChannel\fR]. .SH "CREATING TEST SUITES WITH TCLTEST" - -.SH "RUNNING TEST FILES" -Use the following command to run a test file that uses package -tcltest: -.CS - ?