README -- Tcl test suite design document. RCS: @(#) $Id: README,v 1.11.2.1 2003/04/01 21:13:07 dgp Exp $ Contents: --------- 1. Introduction 2. Running tests 3. Adding tests 4. Incompatibilities with prior Tcl versions 1. Introduction: ---------------- This directory contains a set of validation tests for the Tcl commands and C Library procedures for Tcl. Each of the files whose name ends in ".test" is intended to fully exercise the functions in the C source file that corresponds to the file prefix. The C functions and/or Tcl commands tested by a given file are listed in the first line of the file. 2. Running tests: ----------------- We recommend that you use the "test" target of Tcl's Makefile to run the test suite. From the directory in which you build Tcl, simply type "make test". This will create a special executable named tcltest in which the testing scripts will be evaluated. To create the tcltest executable without running the test suite, simple type "make tcltest". All the configuration options of the tcltest package are available during a "make test" by defining the TESTFLAGS environment variable. For example,if you wish to run only those tests in the file append.test, you can type: make test TESTFLAGS="-file append.test" For interactive testing, the Tcl Makefile provides the "runtest" target. Type "make runtest" in your build directory, and the tcltest executable will be created, if necessary, then it will run interactively. At the command prompt, you may type any Tcl commands. If you type "source ../tests/all.tcl", the test suite will run. You may use the tcltest::configure command to configure the test suite run as an alternative to command line options via TESTFLAGS. You might also wish to use the tcltest::testConstraint command to select the constraints that govern which tests are run. See the documentation for the tcltest package for details. 3. Adding tests: ---------------- Please see the tcltest man page for more information regarding how to write and run tests. Please note that the all.tcl file will source your new test file if the filename matches the tests/*.test pattern (as it should). The names of test files that contain regression (or glass-box) tests should correspond to the Tcl or C code file that they are testing. For example, the test file for the C file "tclCmdAH.c" is "cmdAH.test". Test files that contain black-box tests may not correspond to any Tcl or C code file so they should match the pattern "*_bb.test". Be sure your new test file can be run from any working directory. Be sure no temporary files are left behind by your test file. Use [tcltest::makeFile], [tcltest::removeFile], and [tcltest::cleanupTests] properly to be sure of this. Be sure your tests can run cross-platform in both a build environment as well as an installation environment. If your test file contains tests that should not be run in one or more of those cases, please use the constraints mechanism to skip those tests. 4. Incompatibilities of package tcltest 2.1 with testing machinery of very old versions of Tcl: ------------------------------------------------ 1) Global variables such as VERBOSE, TESTS, and testConfig of the old machinery correspond to the [configure -verbose], [configure -match], and [testConstraint] commands of tcltest 2.1, respectively. 2) VERBOSE values were longer numeric. [configure -verbose] values are lists of keywords. 3) When you run "make test", the working dir for the test suite is now the one from which you called "make test", rather than the "tests" directory. This change allows for both unix and windows test suites to be run simultaneously without interference with each other or with existing files. All tests must now run independently of their working directory. 4) The "all" file is now called "all.tcl" 5) The "defs" and "defs.tcl" files no longer exist. 6) Instead of creating a doAllTests file in the tests directory, to run all nonPortable tests, just use the "-constraints nonPortable" command line flag. If you are running interactively, you can run [tcltest::testConstraint nonPortable 1] (after loading the tcltest package). 7) Direct evaluation of the *.test files by the "source" command is no longer recommended. Instead, "source all.tcl" and use the "-file" and "-notfile" options of tcltest::configure to control which *.test files are evaluated.