From 1f5265171e2ddeade8350713feba432d2d862d2f Mon Sep 17 00:00:00 2001 From: jenn Date: Mon, 26 Jul 1999 22:50:47 +0000 Subject: * tests/tcltest.test: * library/tcltest1.0/tcltest.tcl: * doc/tcltest.n: Cleaned up code in ::tcltest::PrintError, revised documentation, and added tests for the tcltest package. --- ChangeLog | 7 + doc/tcltest.n | 356 ++++++++++++++++++++++++----------------- library/tcltest/tcltest.tcl | 179 ++++++++++++--------- library/tcltest1.0/tcltest.tcl | 179 ++++++++++++--------- 4 files changed, 424 insertions(+), 297 deletions(-) diff --git a/ChangeLog b/ChangeLog index 0045e20..b1a53db 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,10 @@ +1999-07-26 Jennifer Hom + + * tests/tcltest.test: + * library/tcltest1.0/tcltest.tcl: + * doc/tcltest.n: Cleaned up code in ::tcltest::PrintError, revised + documentation, and added tests for the tcltest package. + 1999-07-23 * tests/info.test: diff --git a/doc/tcltest.n b/doc/tcltest.n index 951595b..557fac3 100644 --- a/doc/tcltest.n +++ b/doc/tcltest.n @@ -6,10 +6,10 @@ '\" See the file "license.terms" for information on usage and redistribution '\" of this file, and for a DISCLAIMER OF ALL WARRANTIES. '\" -'\" RCS: @(#) $Id: tcltest.n,v 1.3 1999/07/09 00:01:01 jenn Exp $ +'\" RCS: @(#) $Id: tcltest.n,v 1.4 1999/07/26 22:50:53 jenn Exp $ '\" .so man.macros -.TH "Tcltest" n 8.1 Tcl "Tcl Built-In Commands" +.TH "Tcltest" n 8.2 Tcl "Tcl Built-In Commands" .BS '\" Note: do not modify the .SH NAME line immediately below! .SH NAME @@ -35,14 +35,8 @@ Tcltest \- Test harness support code and utilities .sp \fB::tcltest::normalizeMsg \fImsg\fR .sp -\fB::tcltest::grep \fIexpression list\fR -.sp \fB::tcltest::bytestring \fIstring\fR .sp -\fB::tcltest::set_iso8850_1_locale -.sp -\fB::tcltest::restore_locale -.sp \fB::tcltest::saveState .sp \fB::tcltest::restoreState @@ -59,8 +53,8 @@ The Tcl test suite consists of multiple .test files, each of which contains multiple test cases. Each test case consists of a call to the test command, which specifies the name of test, a short description, any constraints that apply to the test case, the script -to be run, and expected results. See the sections \fI"Tests"\fR and -\fI"Test Constraints"\fR and \fI"Test Files and How to Run Them"\fR +to be run, and expected results. See the sections \fI"Tests,"\fR +\fI"Test Constraints,"\fR and \fI"Test Files and How to Run Them"\fR for more details. .PP It is also possible to add to this test harness to create your own @@ -73,23 +67,29 @@ thanks to her for donating her work back to the public Tcl release. .SH COMMANDS .TP \fB::tcltest::test\fP \fIname desc ?constraints? script expectedAnswer\fR -The \fB::tcltest::test\fR command is used to run a test script defined -within a test file. It prints an error message if the test fails. If -\fB::tcltest::verbose\fR has been set (either by using \fB-verbose\fR -or by manually setting the value of the variable) it can also print +The \fB::tcltest::test\fR command is used \fIscript\fR and compare the +its result to \fIexpectedAnswer\fR. It prints an error message if the two do +not match. If \fB::tcltest::verbose\fR contains "p" or "s", it also prints out a message if the test passed or was skipped. The test will be skipped if it doesn't match the \fB::tcltest::match\fR variable, if it matches an element in \fB::tcltest::skip\fR, or if one of the elements -of \fIconstraint\fR turns out not to be true. See the \fI"Writing a new -test"\fR section for more details on this command. +of \fIconstraint\fR turns out not to be true. The +\fB::tcltest::test\fR command has no defined return values. See the +\fI"Writing a new test"\fR section for more details on this command. .TP \fB::tcltest::cleanupTests\fP \fI?calledFromAll?\fR This command should be called at the end of a test file. It prints statistics about the tests run and removes files that were created by \fB::tcltest::makeDirectory\fR and \fB::tcltest::makeFile\fR. Names -of files created without the \fB::tcltest::makeFile\fR command are -printed. This command also restores the original shell -environment. The default value for \fIcalledFromAll\fR is false. +of files and directories created outside of +\fB::tcltest::makeFile\fR and \fB::tcltest::makeDirectory\fR and +never deleted are printed to \fB::tcltest::outputChannel\fR. This command +also restores the original shell environment, as described by the ::env +array. \fIcalledFromAll\fR should be specified when +\fB::tcltest::cleanupTests\fR is called from an "all.tcl" file. Tcl files +files are generally used to run multiple tests. For more details on how to +run multiple tests, please see the section \fI"Running test files"\fR. +This proc has no defined return value. .TP \fB::tcltest::getMatchingTestFiles\fP This command is used when you want to run multiple test files. It returns @@ -98,56 +98,52 @@ section \fI"Running test files"\fR for more information. .TP \fB::tcltest::makeFile\fP \fIcontents name\fR Create a file that will be automatically be removed by -\fB::tcltest::cleanupTests\fR at the end of a test run. +\fB::tcltest::cleanupTests\fR at the end of a test file. +This proc has no defined return value. .TP \fB::tcltest::removeFile\fP \fIname\fR -Force a file to be removed +Force the file referenced by \fIname\fR to be removed. This file name +should be relative to \fI::tcltest::temporaryDirectory\fR. This proc has no +defined return values. .TP \fB::tcltest::makeDirectory\fP \fIname\fR -Create a directory that will automatically be removed by -\fB::tcltest::cleanupTests\fR at the end of a test run. +Create a directory named \fIname\fR that will automatically be removed +by \fB::tcltest::cleanupTests\fR at the end of a test file. This proc +has no defined return value. .TP \fB::tcltest::removeDirectory\fP \fIname\fR -Force a directory to be removed. +Force the directory referenced by \fIname\fR to be removed. This proc +has no defined return value. .TP -\fB::tcltest::viewFile\fP \fIname\fR -Returns the contents of a file. +\fB::tcltest::viewFile\fP \fIfile\fR +Returns the contents of \fIfile\fR. .TP \fB::tcltest::normalizeMsg\fP \fImsg\fR -Remove extra newlines from a string. -.TP -\fB::tcltest::grep\fP \fIexpression list\fR -Evaluate a given expression against each element of a list and return all -elements for which \fIexpression\fR evaluates to true. Use of the -keyword \fICURRENT_ELEMENT\fR within \fIexpression\fR will flag the -proc to use the value of the current element within the expression. -Use of the \fICURRENT_ELEMENT\fR keyword is optional. If it is left -out, it is assumed to be the final argument to the expression provided. -Examples of usage: -.DS -set subList [grep {CURRENT_ELEMENT == 1} $listOfNumbers] -set subList [grep {regexp {abc} CURRENT_ELEMENT} $listOfStrings] -grep {regexp a} $someList -.DE +Remove extra newlines from \fImsg\fR. .TP \fB::tcltest::bytestring\fP \fIstring\fR Construct a string that consists of the requested sequence of bytes, -as opposed to a string of properly formed UTF-8 characters. -.TP -\fB::tcltest::set_iso8859_1_locale\fP -Set the locale to iso8859_1 -.TP -\fB::tcltest::restore_locale\fP -Restore the locale to its original setting +as opposed to a string of properly formed UTF-8 characters using the +value supplied in \fIstring\fR. This allows the tester to create +denormalized or improperly formed strings to pass to C procedures that +are supposed to accept strings with embedded NULL types and confirm +that a string result has a certain pattern of bytes. .TP \fB::tcltest::saveState\fP -Save the procedure and global variable names -.TP \fB::tcltest::restoreState\fP -Restore the procedure and global variable names +Save and restore the procedure and global variable names. +A test file might contain calls to \fB::tcltest::saveState\fR and +\fB::tcltest:restoreState\fR if it creates or deletes global variables +or procs. .TP \fB::tcltest::threadReap\fP -Kill all threads except for the main thread +\fB::tcltest::threadReap\fR only works if \fItestthread\fR is +defined, generally by compiling tcltest. If \fItestthread\fR is +defined, \fB::tcltest::threadReap\fR kills all threads except for the +main thread. It gets the ID of the main thread by calling +\fItestthread names\fR during initialization. This value is stored in +\fI::tcltest::mainThread\fR. \fB::tcltest::threadReap\fR returns the +number of existing threads at completion. .SH TESTS The \fBtest\fR procedure runs a test script and prints an error message if the script's result does not match the expected result. @@ -165,12 +161,20 @@ target should be the name of the feature being tested. Related tests should share a major number. .PP The argument is a short textual description of the test, -to help humans understand what it does. +to help humans understand what it tests. The name of a Tcl or C +function being tested should be included for regression tests. If the +test case exists to reproduce a bug, include the bug ID in the +description. .PP -The optional argument is list of one or more keywords, -each of which must be the name of an element in the array +The optional argument can be list of one or more +keywords or an expression. If the argument consists of +keywords, each of these keywords must be the name of an element in the array \fI::tcltest::testConstraints\fR. If any of these elements is false or does -not exist, the test is skipped. Add appropriate constraints (e.g., +not exist, the test is skipped. If the argument +consists of an expression, that expression is evaluated. If the +expression evaluates to true, then the test is run. +.PP +Add appropriate constraints (e.g., unixOnly) to any tests that should not always be run. For example, a test that should only be run on Unix should look like the following: .PP @@ -180,6 +184,19 @@ test getAttribute-1.1 {testing file permissions} {unixOnly} { } {00644} .DE .PP +An example of a test that contains an expression: +.PP +.DS +test unixNotfy-1.1 {Tcl_DeleteFileHandler} {unixOnly && !testthread} { + catch {vwait x} + set f [open foo w] + fileevent $f writable {set x 1} + vwait x + close $f + list [catch {vwait x} msg] $msg +} {1 {can't wait for variable "x": would wait forever}} +.DE +.PP See the "Test Constraints" section for a list of built-in constraints and information on how to add your own constraints. .PP @@ -200,47 +217,58 @@ The following variables are also defined in the \fBtcltest\fR namespace and can be used by tests: .TP \fB::tcltest::outputChannel\fR -output file ID - defaults to stdout and can be specified using -outfile +output file ID - defaults to stdout and can be specified using +-outfile on the command line. +Any test that prints test related output should send +that output to \fI::tcltest::outputChannel\fR rather than letting +that output default to stdout. .TP \fB::tcltest::errorChannel\fR error file ID - defaults to stderr and can be specified using -errfile +on the command line. +Any test that prints error messages should send +that output to \fI::tcltest::errorChannel\fR rather than printing +directly to stderr. .TP \fB::tcltest::mainThread\fR -main thread ID - defaults to 1 +main thread ID - defaults to 1. This is the only thread that is not +killed by ::tcltest::threadReap and is set according to the return +value of \fItestthread names\fR at initialization. .TP \fB::tcltest::originalEnv\fR -values of environment variables at the beginning of the test run (::env) +copy of the global "env" array at the beginning of the test run. This +array is used to restore the "env" array to its original state when +\fI::tcltest::cleanupTests\fR is called. .TP \fB::tcltest::workingDirectory\fR -the current working directory ([pwd]) +the directory in which the test suite was launched. .TP \fB::tcltest::temporaryDirectory\fR -the output directory - defaults to the current working directory and can be -specified using -tmpdir +the output directory - defaults to \fI::tcltest::workingDirectory\fR and can be +specified using -tmpdir on the command line. .TP \fB::tcltest::testsDirectory\fR -where the tests reside -.TP -\fB::tcltest::isoLocale\fR -used for internationalization support - default language is French; default -value is fr_FR.ISO_8859-1 for FreeBSD, fr_FR.iso88591 for HP-UX, fr for -Linux and IRIX, iso_8859_1 for other UNIX systems, and French for Windows. +where the tests reside - defaults to \fI::tcltest::workingDirectory\fR +if the script cannot determine where the \fItests\fR directory is +located. This variable should be explicitly set if tests are being +run from an all.tcl file. .TP \fB::tcltest::tcltest\fR -the name of the tcltest executable ([info nameofexecutable]) +the name of the executable used to invoke the test suite. .SH "TEST CONSTRAINTS" Constraints are used to determine whether a test should be skipped. Each constraint is stored as an index in the array -::tcltest::testConstraints. For example, the unixOnly constraint is +\fI::tcltest::testConstraints\fR. For example, the unixOnly constraint is defined as the following: -.PP .DS set ::tcltest::testConstraints(unixOnly) \\ [string equal $tcl_platform(platform) "unix"] .DE -.PP If a test is constrained by "unixOnly", then it will only be run if -the value of ::tcltest::testConstraints(unixOnly) is true. +the value of ::tcltest::testConstraints(unixOnly) is true. Several +constraints are defined in the \fBtcltest\fR package. To add file- or +test-specific constraints, you can set the desired index of the +::tcltest::testsConstraints array in your own test file. .PP The following is a list of constraints defined in the \fBtcltest\fR package: .TP @@ -348,10 +376,11 @@ test can only be run if the current app can be spawned via a pipe Use the following command to run a test file that uses package tcltest: .DS - ?