README -- Tk test suite design document.

RCS: @(#) $Id: README,v 1.1.4.2 1999/03/11 18:50:35 hershey Exp $


Introduction:
-------------

This directory contains a set of validation tests for the Tk
commands.  Each of the files whose name ends in ".test" is
intended to fully exercise one or a few Tk commands.  The
commands tested by a given file are listed in the first line
of the file.

You can run the tests in three ways:

    (a) type "make test" in ../unix; this will run all of the tests.

    (b) type "tktest <testFile> ?<option> <value>?

    (c) start up tktest in this directory, then "source" the test
        file (for example, type "source parse.test").  To run all
	of the tests, type "source all.tcl".

In all cases, no output will be generated if all goes well, except for
a listing of the tests..  If there are errors then additional messages
will appear in the format described below.  Note that some tests will
be skipped if you run as superuser.

This approach to testing was designed and initially implemented
by Mary Ann May-Pumphrey of Sun Microsystems.  Many thanks to
her for donating her work back to the public Tcl release.

The rest of this file provides additional information on the
features of the testing environment.


Definitions file:
-----------------

The file "defs.tcl" defines the "test" namespace which contains a
collection of procedures and variables used to run the tests.  It is
read in automatically by each of the .test files if needed, but once
it has been read once it will not be read again by the .test files.
Currently, the following procedures are exported from the "test"
namespace and automatically imported:

    cleanupTests dotests saveState restoreState normalizeMsg
    makeFile removeFile makeDirectory removeDirectory viewFile
    safeFetch bytestring set_iso8859_1_locale restore_locale
    setTmpDir setupbg dobg bgReady cleanupbg fixfocus

Please refer to the defs.tcl file for these procedures' specs.

To keep tests from polluting the current working directory with
unwanted files, you can specify a temporary directory, which will
become the current working directory for the tests, by specifying
-tmpdir on the command line or by calling the ::test::setTmpDir
procedure (after sourcing the defs.tcl file).  The default working dir
is the directory from which tktest was called.  Please note that when
you run the test suite by calling "make test", the working dir is
<tk8.1>/tests.


Test output:
------------

Foreach test file, the number of tests passed, skipped, and failed is
printed to stdout.  Aside from this statistical information, output
can be controlled on a per-test basis by the ::test::verbose variable.

::test::verbose can be set to any substring or permutation of "bps".
The default value of ::test::verbose is "b".  If 'b' is present, then
the entire test is printed for each failed test, otherwise only the
test's name, desired output, and actual output, are printed for each
failed test.  If 'p' is present, then a line is printed for each
passed test, otherwise no line is printed for passed tests.  If 's' is
present, then a line (containing the consraints taht cause the test to
be skipped) is printed for each skipped test, otherwise no line is
printed for skipped tests.

You can set ::test::verbose either interactively (after the defs.tcl
file has been sourced) or by the command line argument -verbose, for
example:

      tktest select.test -verbose "psb"


Selecting files to be sourced by all.tcl:
-----------------------------------------

You can specify the files you want all.tcl to source on the command
line with the -file options.  For example, if you call the
following:

     tktest all.tcl -file canv*.test

all files in <tk8.1>/tests that match the pattern canv*.test will be
sourced by the all.tcl file.  Another useful example is if a
particular test hangs, say "grid.test", and you just want to run the
remaining tests, then you can call the following:

     tktest all.tcl -file [h-z]*.test

Note that the argument to -file will be substituted relative to the
directory containing this file.


Selecting tests for execution within a file:
--------------------------------------------

Normally, all the tests in a file are run whenever the file is
sourced.  Each test will be skipped if it doesn't match (using glob
sytle matching) any element in the ::test::matchingTests variable, if
it matches (using glob sytle matching) an element in
::test::skippingTests, or if one of the elements of "constraints"
turns out not to be true.

You can set ::test::matchingTests and/or ::test::skippingTests either
interactively (after the defs.tcl file has been sourced), or by the
command line arguments -match and -skip, for example:

       tktest select.test -match "*2.* *4.*" -skip "*2.33*"

The three constraints: notIfCompiled, knownBug, and nonPortable can be
overridden either interactively (after the defs.tcl file has been
sourced) by setting the ::test::testConfig(<constraint>) variable, or
by using the -constraints command line option with the name of the
constraint in the argument.  The following example shows how to run
tests that are constrained by the knownBug and nonPortable
restricions:

	tktest all.tcl -constraints "knownBug nonPortable"

See the defs.tcl file for information about each of these constraints.
Other constraints can be added at any time.  See the "Writing a new
test" section below for more details about using built-in constraints
and adding new ones.

Adding a New Test File:
-----------------------

If the file matches the tests/*.test pattern (as it should), then it
will automatically be run by the all.tcl file.  Make sure your test
file can be run from any working dir.  Running the following should
work the same from any cwd:

	tktest <Tk8.1>/tests/all.tcl

Make sure no temporary files are left behind by your test file.  Your
test file should call "::test::cleanupTests" before returning.  The
::test::cleanupTests procedure prints statistics about the number of
tests that passed, skipped, and failed, and removes all files the were
created using the ::test::makeFile and ::test::makeDirectory
procedures.

Be sure your tests can run cross-platform in both the build
environment as well as the installation environment.  If your test
file contains tests that should not be run in or more of those cases,
please use the constraints mechanism described in the next section to
skip those tests.


Writing a new test:
-------------------

The following is the spec for the "test" command:

	test <name> <description> ?<constraint>? <script> <answer>

The <name> field should be:

	<target>-<majorNum>.<minorNum>

For white-box (regression) tests, the target should be the name of the
c function or Tk procedure being tested.  For black-box tests, the
target should be the name of the feature being tested.  Related tests
should share a major number.

If your test requires that a file be created on the fly, please use
the ::test::makeFile procedure.  If your test requires that a small
file (<50 lines) be checked in, please consider creating the file on
the fly using the ::test::makeFile procedure.  Files created by the
::test::makeFile procedure will automatically be removed by the
::test::cleanupTests call at the end of each test file.

Add appropriate constraints (e.g., unixOnly) to any tests that should
not always be run.  For example, a test that should only be run on
Unix should look like the following:

    test getAttribute-1.1 {testing file permissions} {unixOnly} {
        lindex [file attributes foo.tcl] 5
    } {00644}

See the defs.tcl file for a list of built-in flags.  You can add any
constraints that you need.  The following is how the defs.tcl file
adds the "unixOnly" constraint:

    set ::test::testConfig(unixOnly) [expr {$tcl_platform(platform) == "unix"}]


Saving keystrokes:
------------------

A convenience procedure named "::test::dotests" is included in file
"defs.tcl".  It takes two arguments--the name of the test file (such
as "parse.test"), and a pattern selecting the tests you want to
execute.  It sets ::test::matching to the second argument, calls
"source" on the file specified in the first argument, and restores
::test::matching to its pre-call value at the end.


Incompatibilities with prior Tk versions:
------------------------------------------

1) Global variables such as VERBOSE, TESTS, and testConfig are now
   renamed to use the new "test" namespace.

   old name   new name
   --------   --------
   VERBOSE    ::test::verbose
   TESTS      ::test::matchingTests
   testConfig ::test::testConfig

   The introduction of the "test" namespace is a precursor to using a
   "test" package.  This next step will be part of a future Tk
   version.

2) VERBOSE values are no longer numeric.  Please see the section above
   on "Test output" for the new usage of the ::test::verbose variable.

3) When you run "make test", the working dir for the test suite is now
   the one from which you called "make test", rather than the
   <tk8.1>/tests directory.  This change allows for both unix and
   windows test suites to be run simultaneously without interference.
   All tests must now run independently of their working directory.
   You can also control the working directory from the tktest command
   line with the -tmpdir option.