Merged revisions 72570,72582-72583,73027,73049,73071,73151,73247 via svnmerge from

svn+ssh://pythondev@svn.python.org/python/trunk

........
  r72570 | michael.foord | 2009-05-11 12:59:43 -0500 (Mon, 11 May 2009) | 7 lines

  Adds a verbosity keyword argument to unittest.main plus a minor fix allowing you to specify test modules / classes
  from the command line.

  Closes issue 5995.

  Michael Foord
........
  r72582 | michael.foord | 2009-05-12 05:46:23 -0500 (Tue, 12 May 2009) | 1 line

  Fix to restore command line behaviour for test modules using unittest.main(). Regression caused by issue 5995. Michael
........
  r72583 | michael.foord | 2009-05-12 05:49:13 -0500 (Tue, 12 May 2009) | 1 line

  Better fix for modules using unittest.main(). Fixes regression caused by commit for issue 5995. Michael Foord
........
  r73027 | michael.foord | 2009-05-29 15:33:46 -0500 (Fri, 29 May 2009) | 1 line

  Add test discovery to unittest. Issue 6001.
........
  r73049 | georg.brandl | 2009-05-30 05:45:40 -0500 (Sat, 30 May 2009) | 1 line

  Rewrap a few long lines.
........
  r73071 | georg.brandl | 2009-05-31 09:15:25 -0500 (Sun, 31 May 2009) | 1 line

  Fix markup.
........
  r73151 | michael.foord | 2009-06-02 13:08:27 -0500 (Tue, 02 Jun 2009) | 1 line

  Restore default testRunner argument in unittest.main to None. Issue 6177
........
  r73247 | michael.foord | 2009-06-05 09:14:34 -0500 (Fri, 05 Jun 2009) | 1 line

  Fix unittest discovery tests for Windows. Issue 6199
........

1 files changed, 203 insertions, 34 deletions

diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
index 61883f6..30e73cc 100644
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -78,15 +78,82 @@ need to derive from a specific class.
       Another test-support module with a very different flavor.
 
    `Simple Smalltalk Testing: With Patterns <http://www.XProgramming.com/testfram.htm>`_
-      Kent Beck's original paper on testing frameworks using the pattern shared by
-      :mod:`unittest`.
+      Kent Beck's original paper on testing frameworks using the pattern shared
+      by :mod:`unittest`.
 
    `Nose <http://code.google.com/p/python-nose/>`_ and `py.test <http://pytest.org>`_
-      Third-party unittest frameworks with a lighter-weight syntax
-      for writing tests.  For example, ``assert func(10) == 42``.
+      Third-party unittest frameworks with a lighter-weight syntax for writing
+      tests.  For example, ``assert func(10) == 42``.
 
    `python-mock <http://python-mock.sourceforge.net/>`_ and `minimock <http://blog.ianbicking.org/minimock.html>`_
-      Tools for creating mock test objects (objects simulating external resources).
+      Tools for creating mock test objects (objects simulating external
+      resources).
+
+
+.. _unittest-command-line-interface:
+
+Command Line Interface
+----------------------
+
+The unittest module can be used from the command line to run tests from
+modules, classes or even individual test methods::
+
+   python -m unittest test_module1 test_module2
+   python -m unittest test_module.TestClass
+   python -m unittest test_module.TestClass.test_method
+
+You can pass in a list with any combination of module names, and fully
+qualified class or method names.
+
+You can run tests with more detail (higher verbosity) by passing in the -v flag::
+
+   python-m unittest -v test_module
+
+For a list of all the command line options::
+
+   python -m unittest -h
+
+..  versionchanged:: 2.7
+   In earlier versions it was only possible to run individual test methods and
+   not modules or classes.
+
+The command line can also be used for test discovery, for running all of the
+tests in a project or just a subset.
+
+
+.. _unittest-test-discovery:
+
+Test Discovery
+--------------
+
+.. versionadded:: 2.7
+
+unittest supports simple test discovery. For a project's tests to be
+compatible with test discovery they must all be importable from the top level
+directory of the project; i.e. they must all be in Python packages.
+
+Test discovery is implemented in :meth:`TestLoader.discover`, but can also be
+used from the command line. The basic command line usage is::
+
+   cd project_directory
+   python -m unittest discover
+
+The ``discover`` sub-command has the following options:
+
+   -v, --verbose    Verbose output
+   -s directory     Directory to start discovery ('.' default)
+   -p pattern       Pattern to match test files ('test*.py' default)
+   -t directory     Top level directory of project (default to
+                    start directory)
+
+The -s, -p, & -t options can be passsed in as positional arguments. The
+following two command lines are equivalent::
+
+   python -m unittest -s project_directory -p '*_test.py'
+   python -m unittest project_directory '*_test.py'
+
+Test modules and packages can customize test loading and discovery by through
+the `load_tests protocol`_.
 
 .. _unittest-minimal-example:
 
@@ -175,7 +242,6 @@ The above examples show the most commonly used :mod:`unittest` features which
 are sufficient to meet many everyday testing needs.  The remainder of the
 documentation explores the full feature set from first principles.
 
-
 .. _organizing-tests:
 
 Organizing test code
@@ -206,13 +272,12 @@ The simplest :class:`TestCase` subclass will simply override the
            self.assertEqual(widget.size(), (50, 50), 'incorrect default size')
 
 Note that in order to test something, we use the one of the :meth:`assert\*`
-methods provided by the :class:`TestCase` base class.  If the
-test fails, an exception will be raised, and :mod:`unittest` will identify the
-test case as a :dfn:`failure`.  Any other exceptions will be treated as
-:dfn:`errors`. This helps you identify where the problem is: :dfn:`failures` are
-caused by incorrect results - a 5 where you expected a 6. :dfn:`Errors` are
-caused by incorrect code - e.g., a :exc:`TypeError` caused by an incorrect
-function call.
+methods provided by the :class:`TestCase` base class.  If the test fails, an
+exception will be raised, and :mod:`unittest` will identify the test case as a
+:dfn:`failure`.  Any other exceptions will be treated as :dfn:`errors`. This
+helps you identify where the problem is: :dfn:`failures` are caused by incorrect
+results - a 5 where you expected a 6. :dfn:`Errors` are caused by incorrect
+code - e.g., a :exc:`TypeError` caused by an incorrect function call.
 
 The way to run a test case will be described later.  For now, note that to
 construct an instance of such a test case, we call its constructor without
@@ -412,10 +477,10 @@ may treat :exc:`AssertionError` differently.
 
 .. note::
 
-   Even though :class:`FunctionTestCase` can be used to quickly convert an existing
-   test base over to a :mod:`unittest`\ -based system, this approach is not
-   recommended.  Taking the time to set up proper :class:`TestCase` subclasses will
-   make future test refactorings infinitely easier.
+   Even though :class:`FunctionTestCase` can be used to quickly convert an
+   existing test base over to a :mod:`unittest`\ -based system, this approach is
+   not recommended.  Taking the time to set up proper :class:`TestCase`
+   subclasses will make future test refactorings infinitely easier.
 
 In some cases, the existing tests may have been written using the :mod:`doctest`
 module.  If so, :mod:`doctest` provides a :class:`DocTestSuite` class that can
@@ -444,7 +509,8 @@ Basic skipping looks like this: ::
        def test_nothing(self):
            self.fail("shouldn't happen")
 
-       @unittest.skipIf(mylib.__version__ < (1, 3), "not supported in this library version")
+       @unittest.skipIf(mylib.__version__ < (1, 3),
+                        "not supported in this library version")
        def test_format(self):
            # Tests that work for only a certain version of the library.
            pass
@@ -1009,10 +1075,10 @@ Test cases
 .. class:: FunctionTestCase(testFunc[, setUp[, tearDown[, description]]])
 
    This class implements the portion of the :class:`TestCase` interface which
-   allows the test runner to drive the test, but does not provide the methods which
-   test code can use to check and report errors. This is used to create test cases
-   using legacy test code, allowing it to be integrated into a :mod:`unittest`\
-   -based test framework.
+   allows the test runner to drive the test, but does not provide the methods
+   which test code can use to check and report errors.  This is used to create
+   test cases using legacy test code, allowing it to be integrated into a
+   :mod:`unittest`-based test framework.
 
 
 .. _testsuite-objects:
@@ -1047,8 +1113,8 @@ Grouping tests
       Add all the tests from an iterable of :class:`TestCase` and :class:`TestSuite`
       instances to this test suite.
 
-      This is equivalent to iterating over *tests*, calling :meth:`addTest` for each
-      element.
+      This is equivalent to iterating over *tests*, calling :meth:`addTest` for
+      each element.
 
    :class:`TestSuite` shares the following methods with :class:`TestCase`:
 
@@ -1126,6 +1192,13 @@ Loading and running tests
          directly does not play well with this method.  Doing so, however, can
          be useful when the fixtures are different and defined in subclasses.
 
+      If a module provides a ``load_tests`` function it will be called to
+      load the tests. This allows modules to customize test loading.
+      This is the `load_tests protocol`_.
+
+      .. versionchanged:: 2.7
+         Support for ``load_tests`` added.
+
 
    .. method:: loadTestsFromName(name[, module])
 
@@ -1142,12 +1215,12 @@ Loading and running tests
       For example, if you have a module :mod:`SampleTests` containing a
       :class:`TestCase`\ -derived class :class:`SampleTestCase` with three test
       methods (:meth:`test_one`, :meth:`test_two`, and :meth:`test_three`), the
-      specifier ``'SampleTests.SampleTestCase'`` would cause this method to return a
-      suite which will run all three test methods.  Using the specifier
-      ``'SampleTests.SampleTestCase.test_two'`` would cause it to return a test suite
-      which will run only the :meth:`test_two` test method.  The specifier can refer
-      to modules and packages which have not been imported; they will be imported as a
-      side-effect.
+      specifier ``'SampleTests.SampleTestCase'`` would cause this method to
+      return a suite which will run all three test methods. Using the specifier
+      ``'SampleTests.SampleTestCase.test_two'`` would cause it to return a test
+      suite which will run only the :meth:`test_two` test method. The specifier
+      can refer to modules and packages which have not been imported; they will
+      be imported as a side-effect.
 
       The method optionally resolves *name* relative to the given *module*.
 
@@ -1164,6 +1237,31 @@ Loading and running tests
       Return a sorted sequence of method names found within *testCaseClass*;
       this should be a subclass of :class:`TestCase`.
 
+
+   .. method:: discover(start_dir, pattern='test*.py', top_level_dir=None)
+
+      Find and return all test modules from the specified start directory,
+      recursing into subdirectories to find them. Only test files that match
+      *pattern* will be loaded. (Using shell style pattern matching.)
+
+      All test modules must be importable from the top level of the project. If
+      the start directory is not the top level directory then the top level
+      directory must be specified separately.
+
+      If a test package name (directory with :file:`__init__.py`) matches the
+      pattern then the package will be checked for a ``load_tests``
+      function. If this exists then it will be called with *loader*, *tests*,
+      *pattern*.
+
+      If load_tests exists then discovery does  *not* recurse into the package,
+      ``load_tests`` is responsible for loading all tests in the package.
+
+      The pattern is deliberately not stored as a loader attribute so that
+      packages can continue discovery themselves. *top_level_dir* is stored so
+      ``load_tests`` does not need to pass this argument in to
+      ``loader.discover()``.
+
+
    The following attributes of a :class:`TestLoader` can be configured either by
    subclassing or assignment on an instance:
 
@@ -1319,8 +1417,8 @@ Loading and running tests
 
    .. method:: addFailure(test, err)
 
-      Called when the test case *test* signals a failure. *err* is a tuple of the form
-      returned by :func:`sys.exc_info`:  ``(type, value, traceback)``.
+      Called when the test case *test* signals a failure. *err* is a tuple of
+      the form returned by :func:`sys.exc_info`: ``(type, value, traceback)``.
 
       The default implementation appends a tuple ``(test, formatted_err)`` to
       the instance's :attr:`failures` attribute, where *formatted_err* is a
@@ -1382,7 +1480,7 @@ Loading and running tests
       subclasses to provide a custom ``TestResult``.
 
 
-.. function:: main([module[, defaultTest[, argv[, testRunner[, testLoader[, exit]]]]]])
+.. function:: main([module[, defaultTest[, argv[, testRunner[, testLoader[, exit, [verbosity]]]]]]])
 
    A command-line program that runs a set of tests; this is primarily for making
    test modules conveniently executable.  The simplest use for this function is to
@@ -1391,6 +1489,12 @@ Loading and running tests
       if __name__ == '__main__':
           unittest.main()
 
+   You can run tests with more detailed information by passing in the verbosity
+   argument::
+
+      if __name__ == '__main__':
+          unittest.main(verbosity=2)
+
    The *testRunner* argument can either be a test runner class or an already
    created instance of it. By default ``main`` calls :func:`sys.exit` with
    an exit code indicating success or failure of the tests run.
@@ -1406,4 +1510,69 @@ Loading and running tests
    This stores the result of the tests run as the ``result`` attribute.
 
    .. versionchanged:: 2.7
-      The ``exit`` parameter was added.
+      The ``exit`` and ``verbosity`` parameters were added.
+
+
+load_tests Protocol
+###################
+
+Modules or packages can customize how tests are loaded from them during normal
+test runs or test discovery by implementing a function called ``load_tests``.
+
+If a test module defines ``load_tests`` it will be called by
+:meth:`TestLoader.loadTestsFromModule` with the following arguments::
+
+    load_tests(loader, standard_tests, None)
+
+It should return a :class:`TestSuite`.
+
+*loader* is the instance of :class:`TestLoader` doing the loading.
+*standard_tests* are the tests that would be loaded by default from the
+module. It is common for test modules to only want to add or remove tests
+from the standard set of tests.
+The third argument is used when loading packages as part of test discovery.
+
+A typical ``load_tests`` function that loads tests from a specific set of
+:class:`TestCase` classes may look like::
+
+    test_cases = (TestCase1, TestCase2, TestCase3)
+
+    def load_tests(loader, tests, pattern):
+        suite = TestSuite()
+        for test_class in test_cases:
+            tests = loader.loadTestsFromTestCase(test_class)
+            suite.addTests(tests)
+        return suite
+
+If discovery is started, either from the command line or by calling
+:meth:`TestLoader.discover`, with a pattern that matches a package
+name then the package :file:`__init__.py` will be checked for ``load_tests``.
+
+.. note::
+
+   The default pattern is 'test*.py'. This matches all python files
+   that start with 'test' but *won't* match any test directories.
+
+   A pattern like 'test*' will match test packages as well as
+   modules.
+
+If the package :file:`__init__.py` defines ``load_tests`` then it will be
+called and discovery not continued into the package. ``load_tests``
+is called with the following arguments::
+
+    load_tests(loader, standard_tests, pattern)
+
+This should return a :class:`TestSuite` representing all the tests
+from the package. (``standard_tests`` will only contain tests
+collected from :file:`__init__.py`.)
+
+Because the pattern is passed into ``load_tests`` the package is free to
+continue (and potentially modify) test discovery. A 'do nothing'
+``load_tests`` function for a test package would look like::
+
+    def load_tests(loader, standard_tests, pattern):
+        # top level directory cached on loader instance
+        this_dir = os.path.dirname(__file__)
+        package_tests = loader.discover(start_dir=this_dir, pattern=pattern)
+        standard_tests.addTests(package_tests)
+        return standard_tests