diff options
Diffstat (limited to 'Doc/lib/libdoctest.tex')
| -rw-r--r-- | Doc/lib/libdoctest.tex | 1974 |
1 files changed, 0 insertions, 1974 deletions
diff --git a/Doc/lib/libdoctest.tex b/Doc/lib/libdoctest.tex deleted file mode 100644 index 9143b84..0000000 --- a/Doc/lib/libdoctest.tex +++ /dev/null @@ -1,1974 +0,0 @@ -\section{\module{doctest} --- - Test interactive Python examples} - -\declaremodule{standard}{doctest} -\moduleauthor{Tim Peters}{tim@python.org} -\sectionauthor{Tim Peters}{tim@python.org} -\sectionauthor{Moshe Zadka}{moshez@debian.org} -\sectionauthor{Edward Loper}{edloper@users.sourceforge.net} - -\modulesynopsis{A framework for verifying interactive Python examples.} - -The \refmodule{doctest} module searches for pieces of text that look like -interactive Python sessions, and then executes those sessions to -verify that they work exactly as shown. There are several common ways to -use doctest: - -\begin{itemize} -\item To check that a module's docstrings are up-to-date by verifying - that all interactive examples still work as documented. -\item To perform regression testing by verifying that interactive - examples from a test file or a test object work as expected. -\item To write tutorial documentation for a package, liberally - illustrated with input-output examples. Depending on whether - the examples or the expository text are emphasized, this has - the flavor of "literate testing" or "executable documentation". -\end{itemize} - -Here's a complete but small example module: - -\begin{verbatim} -""" -This is the "example" module. - -The example module supplies one function, factorial(). For example, - ->>> factorial(5) -120 -""" - -def factorial(n): - """Return the factorial of n, an exact integer >= 0. - - If the result is small enough to fit in an int, return an int. - Else return a long. - - >>> [factorial(n) for n in range(6)] - [1, 1, 2, 6, 24, 120] - >>> [factorial(long(n)) for n in range(6)] - [1, 1, 2, 6, 24, 120] - >>> factorial(30) - 265252859812191058636308480000000L - >>> factorial(30L) - 265252859812191058636308480000000L - >>> factorial(-1) - Traceback (most recent call last): - ... - ValueError: n must be >= 0 - - Factorials of floats are OK, but the float must be an exact integer: - >>> factorial(30.1) - Traceback (most recent call last): - ... - ValueError: n must be exact integer - >>> factorial(30.0) - 265252859812191058636308480000000L - - It must also not be ridiculously large: - >>> factorial(1e100) - Traceback (most recent call last): - ... - OverflowError: n too large - """ - -\end{verbatim} -% allow LaTeX to break here. -\begin{verbatim} - - import math - if not n >= 0: - raise ValueError("n must be >= 0") - if math.floor(n) != n: - raise ValueError("n must be exact integer") - if n+1 == n: # catch a value like 1e300 - raise OverflowError("n too large") - result = 1 - factor = 2 - while factor <= n: - result *= factor - factor += 1 - return result - -def _test(): - import doctest - doctest.testmod() - -if __name__ == "__main__": - _test() -\end{verbatim} - -If you run \file{example.py} directly from the command line, -\refmodule{doctest} works its magic: - -\begin{verbatim} -$ python example.py -$ -\end{verbatim} - -There's no output! That's normal, and it means all the examples -worked. Pass \programopt{-v} to the script, and \refmodule{doctest} -prints a detailed log of what it's trying, and prints a summary at the -end: - -\begin{verbatim} -$ python example.py -v -Trying: - factorial(5) -Expecting: - 120 -ok -Trying: - [factorial(n) for n in range(6)] -Expecting: - [1, 1, 2, 6, 24, 120] -ok -Trying: - [factorial(long(n)) for n in range(6)] -Expecting: - [1, 1, 2, 6, 24, 120] -ok -\end{verbatim} - -And so on, eventually ending with: - -\begin{verbatim} -Trying: - factorial(1e100) -Expecting: - Traceback (most recent call last): - ... - OverflowError: n too large -ok -1 items had no tests: - __main__._test -2 items passed all tests: - 1 tests in __main__ - 8 tests in __main__.factorial -9 tests in 3 items. -9 passed and 0 failed. -Test passed. -$ -\end{verbatim} - -That's all you need to know to start making productive use of -\refmodule{doctest}! Jump in. The following sections provide full -details. Note that there are many examples of doctests in -the standard Python test suite and libraries. Especially useful examples -can be found in the standard test file \file{Lib/test/test_doctest.py}. - -\subsection{Simple Usage: Checking Examples in - Docstrings\label{doctest-simple-testmod}} - -The simplest way to start using doctest (but not necessarily the way -you'll continue to do it) is to end each module \module{M} with: - -\begin{verbatim} -def _test(): - import doctest - doctest.testmod() - -if __name__ == "__main__": - _test() -\end{verbatim} - -\refmodule{doctest} then examines docstrings in module \module{M}. - -Running the module as a script causes the examples in the docstrings -to get executed and verified: - -\begin{verbatim} -python M.py -\end{verbatim} - -This won't display anything unless an example fails, in which case the -failing example(s) and the cause(s) of the failure(s) are printed to stdout, -and the final line of output is -\samp{***Test Failed*** \var{N} failures.}, where \var{N} is the -number of examples that failed. - -Run it with the \programopt{-v} switch instead: - -\begin{verbatim} -python M.py -v -\end{verbatim} - -and a detailed report of all examples tried is printed to standard -output, along with assorted summaries at the end. - -You can force verbose mode by passing \code{verbose=True} to -\function{testmod()}, or -prohibit it by passing \code{verbose=False}. In either of those cases, -\code{sys.argv} is not examined by \function{testmod()} (so passing -\programopt{-v} or not has no effect). - -Since Python 2.6, there is also a command line shortcut for running -\function{testmod()}. You can instruct the Python interpreter to run -the doctest module directly from the standard library and pass the module -name(s) on the command line: - -\begin{verbatim} -python -m doctest -v example.py -\end{verbatim} - -This will import \file{example.py} as a standalone module and run -\function{testmod()} on it. Note that this may not work correctly if the -file is part of a package and imports other submodules from that package. - -For more information on \function{testmod()}, see -section~\ref{doctest-basic-api}. - -\subsection{Simple Usage: Checking Examples in a Text - File\label{doctest-simple-testfile}} - -Another simple application of doctest is testing interactive examples -in a text file. This can be done with the \function{testfile()} -function: - -\begin{verbatim} -import doctest -doctest.testfile("example.txt") -\end{verbatim} - -That short script executes and verifies any interactive Python -examples contained in the file \file{example.txt}. The file content -is treated as if it were a single giant docstring; the file doesn't -need to contain a Python program! For example, perhaps \file{example.txt} -contains this: - -\begin{verbatim} -The ``example`` module -====================== - -Using ``factorial`` -------------------- - -This is an example text file in reStructuredText format. First import -``factorial`` from the ``example`` module: - - >>> from example import factorial - -Now use it: - - >>> factorial(6) - 120 -\end{verbatim} - -Running \code{doctest.testfile("example.txt")} then finds the error -in this documentation: - -\begin{verbatim} -File "./example.txt", line 14, in example.txt -Failed example: - factorial(6) -Expected: - 120 -Got: - 720 -\end{verbatim} - -As with \function{testmod()}, \function{testfile()} won't display anything -unless an example fails. If an example does fail, then the failing -example(s) and the cause(s) of the failure(s) are printed to stdout, using -the same format as \function{testmod()}. - -By default, \function{testfile()} looks for files in the calling -module's directory. See section~\ref{doctest-basic-api} for a -description of the optional arguments that can be used to tell it to -look for files in other locations. - -Like \function{testmod()}, \function{testfile()}'s verbosity can be -set with the \programopt{-v} command-line switch or with the optional -keyword argument \var{verbose}. - -Since Python 2.6, there is also a command line shortcut for running -\function{testfile()}. You can instruct the Python interpreter to run -the doctest module directly from the standard library and pass the file -name(s) on the command line: - -\begin{verbatim} -python -m doctest -v example.txt -\end{verbatim} - -Because the file name does not end with \file{.py}, \module{doctest} infers -that it must be run with \function{testfile()}, not \function{testmod()}. - -For more information on \function{testfile()}, see -section~\ref{doctest-basic-api}. - -\subsection{How It Works\label{doctest-how-it-works}} - -This section examines in detail how doctest works: which docstrings it -looks at, how it finds interactive examples, what execution context it -uses, how it handles exceptions, and how option flags can be used to -control its behavior. This is the information that you need to know -to write doctest examples; for information about actually running -doctest on these examples, see the following sections. - -\subsubsection{Which Docstrings Are Examined?\label{doctest-which-docstrings}} - -The module docstring, and all function, class and method docstrings are -searched. Objects imported into the module are not searched. - -In addition, if \code{M.__test__} exists and "is true", it must be a -dict, and each entry maps a (string) name to a function object, class -object, or string. Function and class object docstrings found from -\code{M.__test__} are searched, and strings are treated as if they -were docstrings. In output, a key \code{K} in \code{M.__test__} appears -with name - -\begin{verbatim} -<name of M>.__test__.K -\end{verbatim} - -Any classes found are recursively searched similarly, to test docstrings in -their contained methods and nested classes. - -\versionchanged[A "private name" concept is deprecated and no longer - documented]{2.4} - -\subsubsection{How are Docstring Examples - Recognized?\label{doctest-finding-examples}} - -In most cases a copy-and-paste of an interactive console session works -fine, but doctest isn't trying to do an exact emulation of any specific -Python shell. All hard tab characters are expanded to spaces, using -8-column tab stops. If you don't believe tabs should mean that, too -bad: don't use hard tabs, or write your own \class{DocTestParser} -class. - -\versionchanged[Expanding tabs to spaces is new; previous versions - tried to preserve hard tabs, with confusing results]{2.4} - -\begin{verbatim} ->>> # comments are ignored ->>> x = 12 ->>> x -12 ->>> if x == 13: -... print "yes" -... else: -... print "no" -... print "NO" -... print "NO!!!" -... -no -NO -NO!!! ->>> -\end{verbatim} - -Any expected output must immediately follow the final -\code{'>>>~'} or \code{'...~'} line containing the code, and -the expected output (if any) extends to the next \code{'>>>~'} -or all-whitespace line. - -The fine print: - -\begin{itemize} - -\item Expected output cannot contain an all-whitespace line, since such a - line is taken to signal the end of expected output. If expected - output does contain a blank line, put \code{<BLANKLINE>} in your - doctest example each place a blank line is expected. - \versionchanged[\code{<BLANKLINE>} was added; there was no way to - use expected output containing empty lines in - previous versions]{2.4} - -\item Output to stdout is captured, but not output to stderr (exception - tracebacks are captured via a different means). - -\item If you continue a line via backslashing in an interactive session, - or for any other reason use a backslash, you should use a raw - docstring, which will preserve your backslashes exactly as you type - them: - -\begin{verbatim} ->>> def f(x): -... r'''Backslashes in a raw docstring: m\n''' ->>> print f.__doc__ -Backslashes in a raw docstring: m\n -\end{verbatim} - - Otherwise, the backslash will be interpreted as part of the string. - For example, the "{\textbackslash}" above would be interpreted as a - newline character. Alternatively, you can double each backslash in the - doctest version (and not use a raw string): - -\begin{verbatim} ->>> def f(x): -... '''Backslashes in a raw docstring: m\\n''' ->>> print f.__doc__ -Backslashes in a raw docstring: m\n -\end{verbatim} - -\item The starting column doesn't matter: - -\begin{verbatim} - >>> assert "Easy!" - >>> import math - >>> math.floor(1.9) - 1.0 -\end{verbatim} - -and as many leading whitespace characters are stripped from the -expected output as appeared in the initial \code{'>>>~'} line -that started the example. -\end{itemize} - -\subsubsection{What's the Execution Context?\label{doctest-execution-context}} - -By default, each time \refmodule{doctest} finds a docstring to test, it -uses a \emph{shallow copy} of \module{M}'s globals, so that running tests -doesn't change the module's real globals, and so that one test in -\module{M} can't leave behind crumbs that accidentally allow another test -to work. This means examples can freely use any names defined at top-level -in \module{M}, and names defined earlier in the docstring being run. -Examples cannot see names defined in other docstrings. - -You can force use of your own dict as the execution context by passing -\code{globs=your_dict} to \function{testmod()} or -\function{testfile()} instead. - -\subsubsection{What About Exceptions?\label{doctest-exceptions}} - -No problem, provided that the traceback is the only output produced by -the example: just paste in the traceback.\footnote{Examples containing - both expected output and an exception are not supported. Trying - to guess where one ends and the other begins is too error-prone, - and that also makes for a confusing test.} -Since tracebacks contain details that are likely to change rapidly (for -example, exact file paths and line numbers), this is one case where doctest -works hard to be flexible in what it accepts. - -Simple example: - -\begin{verbatim} ->>> [1, 2, 3].remove(42) -Traceback (most recent call last): - File "<stdin>", line 1, in ? -ValueError: list.remove(x): x not in list -\end{verbatim} - -That doctest succeeds if \exception{ValueError} is raised, with the -\samp{list.remove(x): x not in list} detail as shown. - -The expected output for an exception must start with a traceback -header, which may be either of the following two lines, indented the -same as the first line of the example: - -\begin{verbatim} -Traceback (most recent call last): -Traceback (innermost last): -\end{verbatim} - -The traceback header is followed by an optional traceback stack, whose -contents are ignored by doctest. The traceback stack is typically -omitted, or copied verbatim from an interactive session. - -The traceback stack is followed by the most interesting part: the -line(s) containing the exception type and detail. This is usually the -last line of a traceback, but can extend across multiple lines if the -exception has a multi-line detail: - -\begin{verbatim} ->>> raise ValueError('multi\n line\ndetail') -Traceback (most recent call last): - File "<stdin>", line 1, in ? -ValueError: multi - line -detail -\end{verbatim} - -The last three lines (starting with \exception{ValueError}) are -compared against the exception's type and detail, and the rest are -ignored. - -Best practice is to omit the traceback stack, unless it adds -significant documentation value to the example. So the last example -is probably better as: - -\begin{verbatim} ->>> raise ValueError('multi\n line\ndetail') -Traceback (most recent call last): - ... -ValueError: multi - line -detail -\end{verbatim} - -Note that tracebacks are treated very specially. In particular, in the -rewritten example, the use of \samp{...} is independent of doctest's -\constant{ELLIPSIS} option. The ellipsis in that example could be left -out, or could just as well be three (or three hundred) commas or digits, -or an indented transcript of a Monty Python skit. - -Some details you should read once, but won't need to remember: - -\begin{itemize} - -\item Doctest can't guess whether your expected output came from an - exception traceback or from ordinary printing. So, e.g., an example - that expects \samp{ValueError: 42 is prime} will pass whether - \exception{ValueError} is actually raised or if the example merely - prints that traceback text. In practice, ordinary output rarely begins - with a traceback header line, so this doesn't create real problems. - -\item Each line of the traceback stack (if present) must be indented - further than the first line of the example, \emph{or} start with a - non-alphanumeric character. The first line following the traceback - header indented the same and starting with an alphanumeric is taken - to be the start of the exception detail. Of course this does the - right thing for genuine tracebacks. - -\item When the \constant{IGNORE_EXCEPTION_DETAIL} doctest option is - is specified, everything following the leftmost colon is ignored. - -\item The interactive shell omits the traceback header line for some - \exception{SyntaxError}s. But doctest uses the traceback header - line to distinguish exceptions from non-exceptions. So in the rare - case where you need to test a \exception{SyntaxError} that omits the - traceback header, you will need to manually add the traceback header - line to your test example. - -\item For some \exception{SyntaxError}s, Python displays the character - position of the syntax error, using a \code{\^} marker: - -\begin{verbatim} ->>> 1 1 - File "<stdin>", line 1 - 1 1 - ^ -SyntaxError: invalid syntax -\end{verbatim} - - Since the lines showing the position of the error come before the - exception type and detail, they are not checked by doctest. For - example, the following test would pass, even though it puts the - \code{\^} marker in the wrong location: - -\begin{verbatim} ->>> 1 1 -Traceback (most recent call last): - File "<stdin>", line 1 - 1 1 - ^ -SyntaxError: invalid syntax -\end{verbatim} - -\end{itemize} - -\versionchanged[The ability to handle a multi-line exception detail, - and the \constant{IGNORE_EXCEPTION_DETAIL} doctest option, - were added]{2.4} - -\subsubsection{Option Flags and Directives\label{doctest-options}} - -A number of option flags control various aspects of doctest's -behavior. Symbolic names for the flags are supplied as module constants, -which can be or'ed together and passed to various functions. The names -can also be used in doctest directives (see below). - -The first group of options define test semantics, controlling -aspects of how doctest decides whether actual output matches an -example's expected output: - -\begin{datadesc}{DONT_ACCEPT_TRUE_FOR_1} - By default, if an expected output block contains just \code{1}, - an actual output block containing just \code{1} or just - \code{True} is considered to be a match, and similarly for \code{0} - versus \code{False}. When \constant{DONT_ACCEPT_TRUE_FOR_1} is - specified, neither substitution is allowed. The default behavior - caters to that Python changed the return type of many functions - from integer to boolean; doctests expecting "little integer" - output still work in these cases. This option will probably go - away, but not for several years. -\end{datadesc} - -\begin{datadesc}{DONT_ACCEPT_BLANKLINE} - By default, if an expected output block contains a line - containing only the string \code{<BLANKLINE>}, then that line - will match a blank line in the actual output. Because a - genuinely blank line delimits the expected output, this is - the only way to communicate that a blank line is expected. When - \constant{DONT_ACCEPT_BLANKLINE} is specified, this substitution - is not allowed. -\end{datadesc} - -\begin{datadesc}{NORMALIZE_WHITESPACE} - When specified, all sequences of whitespace (blanks and newlines) are - treated as equal. Any sequence of whitespace within the expected - output will match any sequence of whitespace within the actual output. - By default, whitespace must match exactly. - \constant{NORMALIZE_WHITESPACE} is especially useful when a line - of expected output is very long, and you want to wrap it across - multiple lines in your source. -\end{datadesc} - -\begin{datadesc}{ELLIPSIS} - When specified, an ellipsis marker (\code{...}) in the expected output - can match any substring in the actual output. This includes - substrings that span line boundaries, and empty substrings, so it's - best to keep usage of this simple. Complicated uses can lead to the - same kinds of "oops, it matched too much!" surprises that \regexp{.*} - is prone to in regular expressions. -\end{datadesc} - -\begin{datadesc}{IGNORE_EXCEPTION_DETAIL} - When specified, an example that expects an exception passes if - an exception of the expected type is raised, even if the exception - detail does not match. For example, an example expecting - \samp{ValueError: 42} will pass if the actual exception raised is - \samp{ValueError: 3*14}, but will fail, e.g., if - \exception{TypeError} is raised. - - Note that a similar effect can be obtained using \constant{ELLIPSIS}, - and \constant{IGNORE_EXCEPTION_DETAIL} may go away when Python releases - prior to 2.4 become uninteresting. Until then, - \constant{IGNORE_EXCEPTION_DETAIL} is the only clear way to write a - doctest that doesn't care about the exception detail yet continues - to pass under Python releases prior to 2.4 (doctest directives - appear to be comments to them). For example, - -\begin{verbatim} ->>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL -Traceback (most recent call last): - File "<stdin>", line 1, in ? -TypeError: object doesn't support item assignment -\end{verbatim} - - passes under Python 2.4 and Python 2.3. The detail changed in 2.4, - to say "does not" instead of "doesn't". - -\end{datadesc} - -\begin{datadesc}{SKIP} - - When specified, do not run the example at all. This can be useful - in contexts where doctest examples serve as both documentation and - test cases, and an example should be included for documentation - purposes, but should not be checked. E.g., the example's output - might be random; or the example might depend on resources which - would be unavailable to the test driver. - - The SKIP flag can also be used for temporarily "commenting out" - examples. - -\end{datadesc} - -\begin{datadesc}{COMPARISON_FLAGS} - A bitmask or'ing together all the comparison flags above. -\end{datadesc} - -The second group of options controls how test failures are reported: - -\begin{datadesc}{REPORT_UDIFF} - When specified, failures that involve multi-line expected and - actual outputs are displayed using a unified diff. -\end{datadesc} - -\begin{datadesc}{REPORT_CDIFF} - When specified, failures that involve multi-line expected and - actual outputs will be displayed using a context diff. -\end{datadesc} - -\begin{datadesc}{REPORT_NDIFF} - When specified, differences are computed by \code{difflib.Differ}, - using the same algorithm as the popular \file{ndiff.py} utility. - This is the only method that marks differences within lines as - well as across lines. For example, if a line of expected output - contains digit \code{1} where actual output contains letter \code{l}, - a line is inserted with a caret marking the mismatching column - positions. -\end{datadesc} - -\begin{datadesc}{REPORT_ONLY_FIRST_FAILURE} - When specified, display the first failing example in each doctest, - but suppress output for all remaining examples. This will prevent - doctest from reporting correct examples that break because of - earlier failures; but it might also hide incorrect examples that - fail independently of the first failure. When - \constant{REPORT_ONLY_FIRST_FAILURE} is specified, the remaining - examples are still run, and still count towards the total number of - failures reported; only the output is suppressed. -\end{datadesc} - -\begin{datadesc}{REPORTING_FLAGS} - A bitmask or'ing together all the reporting flags above. -\end{datadesc} - -"Doctest directives" may be used to modify the option flags for -individual examples. Doctest directives are expressed as a special -Python comment following an example's source code: - -\begin{productionlist}[doctest] - \production{directive} - {"\#" "doctest:" \token{directive_options}} - \production{directive_options} - {\token{directive_option} ("," \token{directive_option})*} - \production{directive_option} - {\token{on_or_off} \token{directive_option_name}} - \production{on_or_off} - {"+" | "-"} - \production{directive_option_name} - {"DONT_ACCEPT_BLANKLINE" | "NORMALIZE_WHITESPACE" | ...} -\end{productionlist} - -Whitespace is not allowed between the \code{+} or \code{-} and the -directive option name. The directive option name can be any of the -option flag names explained above. - -An example's doctest directives modify doctest's behavior for that -single example. Use \code{+} to enable the named behavior, or -\code{-} to disable it. - -For example, this test passes: - -\begin{verbatim} ->>> print range(20) #doctest: +NORMALIZE_WHITESPACE -[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, 11, 12, 13, 14, 15, 16, 17, 18, 19] -\end{verbatim} - -Without the directive it would fail, both because the actual output -doesn't have two blanks before the single-digit list elements, and -because the actual output is on a single line. This test also passes, -and also requires a directive to do so: - -\begin{verbatim} ->>> print range(20) # doctest:+ELLIPSIS -[0, 1, ..., 18, 19] -\end{verbatim} - -Multiple directives can be used on a single physical line, separated -by commas: - -\begin{verbatim} ->>> print range(20) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE -[0, 1, ..., 18, 19] -\end{verbatim} - -If multiple directive comments are used for a single example, then -they are combined: - -\begin{verbatim} ->>> print range(20) # doctest: +ELLIPSIS -... # doctest: +NORMALIZE_WHITESPACE -[0, 1, ..., 18, 19] -\end{verbatim} - -As the previous example shows, you can add \samp{...} lines to your -example containing only directives. This can be useful when an -example is too long for a directive to comfortably fit on the same -line: - -\begin{verbatim} ->>> print range(5) + range(10,20) + range(30,40) + range(50,60) -... # doctest: +ELLIPSIS -[0, ..., 4, 10, ..., 19, 30, ..., 39, 50, ..., 59] -\end{verbatim} - -Note that since all options are disabled by default, and directives apply -only to the example they appear in, enabling options (via \code{+} in a -directive) is usually the only meaningful choice. However, option flags -can also be passed to functions that run doctests, establishing different -defaults. In such cases, disabling an option via \code{-} in a directive -can be useful. - -\versionchanged[Constants \constant{DONT_ACCEPT_BLANKLINE}, - \constant{NORMALIZE_WHITESPACE}, \constant{ELLIPSIS}, - \constant{IGNORE_EXCEPTION_DETAIL}, - \constant{REPORT_UDIFF}, \constant{REPORT_CDIFF}, - \constant{REPORT_NDIFF}, \constant{REPORT_ONLY_FIRST_FAILURE}, - \constant{COMPARISON_FLAGS} and \constant{REPORTING_FLAGS} - were added; by default \code{<BLANKLINE>} in expected output - matches an empty line in actual output; and doctest directives - were added]{2.4} -\versionchanged[Constant \constant{SKIP} was added]{2.5} - -There's also a way to register new option flag names, although this -isn't useful unless you intend to extend \refmodule{doctest} internals -via subclassing: - -\begin{funcdesc}{register_optionflag}{name} - Create a new option flag with a given name, and return the new - flag's integer value. \function{register_optionflag()} can be - used when subclassing \class{OutputChecker} or - \class{DocTestRunner} to create new options that are supported by - your subclasses. \function{register_optionflag} should always be - called using the following idiom: - -\begin{verbatim} - MY_FLAG = register_optionflag('MY_FLAG') -\end{verbatim} - - \versionadded{2.4} -\end{funcdesc} - -\subsubsection{Warnings\label{doctest-warnings}} - -\refmodule{doctest} is serious about requiring exact matches in expected -output. If even a single character doesn't match, the test fails. This -will probably surprise you a few times, as you learn exactly what Python -does and doesn't guarantee about output. For example, when printing a -dict, Python doesn't guarantee that the key-value pairs will be printed -in any particular order, so a test like - -% Hey! What happened to Monty Python examples? -% Tim: ask Guido -- it's his example! -\begin{verbatim} ->>> foo() -{"Hermione": "hippogryph", "Harry": "broomstick"} -\end{verbatim} - -is vulnerable! One workaround is to do - -\begin{verbatim} ->>> foo() == {"Hermione": "hippogryph", "Harry": "broomstick"} -True -\end{verbatim} - -instead. Another is to do - -\begin{verbatim} ->>> d = foo().items() ->>> d.sort() ->>> d -[('Harry', 'broomstick'), ('Hermione', 'hippogryph')] -\end{verbatim} - -There are others, but you get the idea. - -Another bad idea is to print things that embed an object address, like - -\begin{verbatim} ->>> id(1.0) # certain to fail some of the time -7948648 ->>> class C: pass ->>> C() # the default repr() for instances embeds an address -<__main__.C instance at 0x00AC18F0> -\end{verbatim} - -The \constant{ELLIPSIS} directive gives a nice approach for the last -example: - -\begin{verbatim} ->>> C() #doctest: +ELLIPSIS -<__main__.C instance at 0x...> -\end{verbatim} - -Floating-point numbers are also subject to small output variations across -platforms, because Python defers to the platform C library for float -formatting, and C libraries vary widely in quality here. - -\begin{verbatim} ->>> 1./7 # risky -0.14285714285714285 ->>> print 1./7 # safer -0.142857142857 ->>> print round(1./7, 6) # much safer -0.142857 -\end{verbatim} - -Numbers of the form \code{I/2.**J} are safe across all platforms, and I -often contrive doctest examples to produce numbers of that form: - -\begin{verbatim} ->>> 3./4 # utterly safe -0.75 -\end{verbatim} - -Simple fractions are also easier for people to understand, and that makes -for better documentation. - -\subsection{Basic API\label{doctest-basic-api}} - -The functions \function{testmod()} and \function{testfile()} provide a -simple interface to doctest that should be sufficient for most basic -uses. For a less formal introduction to these two functions, see -sections \ref{doctest-simple-testmod} and -\ref{doctest-simple-testfile}. - -\begin{funcdesc}{testfile}{filename\optional{, module_relative}\optional{, - name}\optional{, package}\optional{, - globs}\optional{, verbose}\optional{, - report}\optional{, optionflags}\optional{, - extraglobs}\optional{, raise_on_error}\optional{, - parser}\optional{, encoding}} - - All arguments except \var{filename} are optional, and should be - specified in keyword form. - - Test examples in the file named \var{filename}. Return - \samp{(\var{failure_count}, \var{test_count})}. - - Optional argument \var{module_relative} specifies how the filename - should be interpreted: - - \begin{itemize} - \item If \var{module_relative} is \code{True} (the default), then - \var{filename} specifies an OS-independent module-relative - path. By default, this path is relative to the calling - module's directory; but if the \var{package} argument is - specified, then it is relative to that package. To ensure - OS-independence, \var{filename} should use \code{/} characters - to separate path segments, and may not be an absolute path - (i.e., it may not begin with \code{/}). - \item If \var{module_relative} is \code{False}, then \var{filename} - specifies an OS-specific path. The path may be absolute or - relative; relative paths are resolved with respect to the - current working directory. - \end{itemize} - - Optional argument \var{name} gives the name of the test; by default, - or if \code{None}, \code{os.path.basename(\var{filename})} is used. - - Optional argument \var{package} is a Python package or the name of a - Python package whose directory should be used as the base directory - for a module-relative filename. If no package is specified, then - the calling module's directory is used as the base directory for - module-relative filenames. It is an error to specify \var{package} - if \var{module_relative} is \code{False}. - - Optional argument \var{globs} gives a dict to be used as the globals - when executing examples. A new shallow copy of this dict is - created for the doctest, so its examples start with a clean slate. - By default, or if \code{None}, a new empty dict is used. - - Optional argument \var{extraglobs} gives a dict merged into the - globals used to execute examples. This works like - \method{dict.update()}: if \var{globs} and \var{extraglobs} have a - common key, the associated value in \var{extraglobs} appears in the - combined dict. By default, or if \code{None}, no extra globals are - used. This is an advanced feature that allows parameterization of - doctests. For example, a doctest can be written for a base class, using - a generic name for the class, then reused to test any number of - subclasses by passing an \var{extraglobs} dict mapping the generic - name to the subclass to be tested. - - Optional argument \var{verbose} prints lots of stuff if true, and prints - only failures if false; by default, or if \code{None}, it's true - if and only if \code{'-v'} is in \code{sys.argv}. - - Optional argument \var{report} prints a summary at the end when true, - else prints nothing at the end. In verbose mode, the summary is - detailed, else the summary is very brief (in fact, empty if all tests - passed). - - Optional argument \var{optionflags} or's together option flags. See - section~\ref{doctest-options}. - - Optional argument \var{raise_on_error} defaults to false. If true, - an exception is raised upon the first failure or unexpected exception - in an example. This allows failures to be post-mortem debugged. - Default behavior is to continue running examples. - - Optional argument \var{parser} specifies a \class{DocTestParser} (or - subclass) that should be used to extract tests from the files. It - defaults to a normal parser (i.e., \code{\class{DocTestParser}()}). - - Optional argument \var{encoding} specifies an encoding that should - be used to convert the file to unicode. - - \versionadded{2.4} - - \versionchanged[The parameter \var{encoding} was added]{2.5} - -\end{funcdesc} - -\begin{funcdesc}{testmod}{\optional{m}\optional{, name}\optional{, - globs}\optional{, verbose}\optional{, - report}\optional{, - optionflags}\optional{, extraglobs}\optional{, - raise_on_error}\optional{, exclude_empty}} - - All arguments are optional, and all except for \var{m} should be - specified in keyword form. - - Test examples in docstrings in functions and classes reachable - from module \var{m} (or module \module{__main__} if \var{m} is not - supplied or is \code{None}), starting with \code{\var{m}.__doc__}. - - Also test examples reachable from dict \code{\var{m}.__test__}, if it - exists and is not \code{None}. \code{\var{m}.__test__} maps - names (strings) to functions, classes and strings; function and class - docstrings are searched for examples; strings are searched directly, - as if they were docstrings. - - Only docstrings attached to objects belonging to module \var{m} are - searched. - - Return \samp{(\var{failure_count}, \var{test_count})}. - - Optional argument \var{name} gives the name of the module; by default, - or if \code{None}, \code{\var{m}.__name__} is used. - - Optional argument \var{exclude_empty} defaults to false. If true, - objects for which no doctests are found are excluded from consideration. - The default is a backward compatibility hack, so that code still - using \method{doctest.master.summarize()} in conjunction with - \function{testmod()} continues to get output for objects with no tests. - The \var{exclude_empty} argument to the newer \class{DocTestFinder} - constructor defaults to true. - - Optional arguments \var{extraglobs}, \var{verbose}, \var{report}, - \var{optionflags}, \var{raise_on_error}, and \var{globs} are the same as - for function \function{testfile()} above, except that \var{globs} - defaults to \code{\var{m}.__dict__}. - - \versionchanged[The parameter \var{optionflags} was added]{2.3} - - \versionchanged[The parameters \var{extraglobs}, \var{raise_on_error} - and \var{exclude_empty} were added]{2.4} - - \versionchanged[The optional argument \var{isprivate}, deprecated - in 2.4, was removed]{2.5} - -\end{funcdesc} - -There's also a function to run the doctests associated with a single object. -This function is provided for backward compatibility. There are no plans -to deprecate it, but it's rarely useful: - -\begin{funcdesc}{run_docstring_examples}{f, globs\optional{, - verbose}\optional{, name}\optional{, - compileflags}\optional{, optionflags}} - - Test examples associated with object \var{f}; for example, \var{f} may - be a module, function, or class object. - - A shallow copy of dictionary argument \var{globs} is used for the - execution context. - - Optional argument \var{name} is used in failure messages, and defaults - to \code{"NoName"}. - - If optional argument \var{verbose} is true, output is generated even - if there are no failures. By default, output is generated only in case - of an example failure. - - Optional argument \var{compileflags} gives the set of flags that should - be used by the Python compiler when running the examples. By default, or - if \code{None}, flags are deduced corresponding to the set of future - features found in \var{globs}. - - Optional argument \var{optionflags} works as for function - \function{testfile()} above. -\end{funcdesc} - -\subsection{Unittest API\label{doctest-unittest-api}} - -As your collection of doctest'ed modules grows, you'll want a way to run -all their doctests systematically. Prior to Python 2.4, \refmodule{doctest} -had a barely documented \class{Tester} class that supplied a rudimentary -way to combine doctests from multiple modules. \class{Tester} was feeble, -and in practice most serious Python testing frameworks build on the -\refmodule{unittest} module, which supplies many flexible ways to combine -tests from multiple sources. So, in Python 2.4, \refmodule{doctest}'s -\class{Tester} class is deprecated, and \refmodule{doctest} provides two -functions that can be used to create \refmodule{unittest} test suites from -modules and text files containing doctests. These test suites can then be -run using \refmodule{unittest} test runners: - -\begin{verbatim} -import unittest -import doctest -import my_module_with_doctests, and_another - -suite = unittest.TestSuite() -for mod in my_module_with_doctests, and_another: - suite.addTest(doctest.DocTestSuite(mod)) -runner = unittest.TextTestRunner() -runner.run(suite) -\end{verbatim} - -There are two main functions for creating \class{\refmodule{unittest}.TestSuite} -instances from text files and modules with doctests: - -\begin{funcdesc}{DocFileSuite}{\optional{module_relative}\optional{, - package}\optional{, setUp}\optional{, - tearDown}\optional{, globs}\optional{, - optionflags}\optional{, parser}\optional{, - encoding}} - - Convert doctest tests from one or more text files to a - \class{\refmodule{unittest}.TestSuite}. - - The returned \class{\refmodule{unittest}.TestSuite} is to be run by the - unittest framework and runs the interactive examples in each file. If an - example in any file fails, then the synthesized unit test fails, and a - \exception{failureException} exception is raised showing the name of the - file containing the test and a (sometimes approximate) line number. - - Pass one or more paths (as strings) to text files to be examined. - - Options may be provided as keyword arguments: - - Optional argument \var{module_relative} specifies how - the filenames in \var{paths} should be interpreted: - - \begin{itemize} - \item If \var{module_relative} is \code{True} (the default), then - each filename specifies an OS-independent module-relative - path. By default, this path is relative to the calling - module's directory; but if the \var{package} argument is - specified, then it is relative to that package. To ensure - OS-independence, each filename should use \code{/} characters - to separate path segments, and may not be an absolute path - (i.e., it may not begin with \code{/}). - \item If \var{module_relative} is \code{False}, then each filename - specifies an OS-specific path. The path may be absolute or - relative; relative paths are resolved with respect to the - current working directory. - \end{itemize} - - Optional argument \var{package} is a Python package or the name - of a Python package whose directory should be used as the base - directory for module-relative filenames. If no package is - specified, then the calling module's directory is used as the base - directory for module-relative filenames. It is an error to specify - \var{package} if \var{module_relative} is \code{False}. - - Optional argument \var{setUp} specifies a set-up function for - the test suite. This is called before running the tests in each - file. The \var{setUp} function will be passed a \class{DocTest} - object. The setUp function can access the test globals as the - \var{globs} attribute of the test passed. - - Optional argument \var{tearDown} specifies a tear-down function - for the test suite. This is called after running the tests in each - file. The \var{tearDown} function will be passed a \class{DocTest} - object. The setUp function can access the test globals as the - \var{globs} attribute of the test passed. - - Optional argument \var{globs} is a dictionary containing the - initial global variables for the tests. A new copy of this - dictionary is created for each test. By default, \var{globs} is - a new empty dictionary. - - Optional argument \var{optionflags} specifies the default - doctest options for the tests, created by or-ing together - individual option flags. See section~\ref{doctest-options}. - See function \function{set_unittest_reportflags()} below for - a better way to set reporting options. - - Optional argument \var{parser} specifies a \class{DocTestParser} (or - subclass) that should be used to extract tests from the files. It - defaults to a normal parser (i.e., \code{\class{DocTestParser}()}). - - Optional argument \var{encoding} specifies an encoding that should - be used to convert the file to unicode. - - \versionadded{2.4} - - \versionchanged[The global \code{__file__} was added to the - globals provided to doctests loaded from a text file using - \function{DocFileSuite()}]{2.5} - - \versionchanged[The parameter \var{encoding} was added]{2.5} - -\end{funcdesc} - -\begin{funcdesc}{DocTestSuite}{\optional{module}\optional{, - globs}\optional{, extraglobs}\optional{, - test_finder}\optional{, setUp}\optional{, - tearDown}\optional{, checker}} - Convert doctest tests for a module to a - \class{\refmodule{unittest}.TestSuite}. - - The returned \class{\refmodule{unittest}.TestSuite} is to be run by the - unittest framework and runs each doctest in the module. If any of the - doctests fail, then the synthesized unit test fails, and a - \exception{failureException} exception is raised showing the name of the - file containing the test and a (sometimes approximate) line number. - - Optional argument \var{module} provides the module to be tested. It - can be a module object or a (possibly dotted) module name. If not - specified, the module calling this function is used. - - Optional argument \var{globs} is a dictionary containing the - initial global variables for the tests. A new copy of this - dictionary is created for each test. By default, \var{globs} is - a new empty dictionary. - - Optional argument \var{extraglobs} specifies an extra set of - global variables, which is merged into \var{globs}. By default, no - extra globals are used. - - Optional argument \var{test_finder} is the \class{DocTestFinder} - object (or a drop-in replacement) that is used to extract doctests - from the module. - - Optional arguments \var{setUp}, \var{tearDown}, and \var{optionflags} - are the same as for function \function{DocFileSuite()} above. - - \versionadded{2.3} - - \versionchanged[The parameters \var{globs}, \var{extraglobs}, - \var{test_finder}, \var{setUp}, \var{tearDown}, and - \var{optionflags} were added; this function now uses the same search - technique as \function{testmod()}]{2.4} -\end{funcdesc} - -Under the covers, \function{DocTestSuite()} creates a -\class{\refmodule{unittest}.TestSuite} out of \class{doctest.DocTestCase} -instances, and \class{DocTestCase} is a subclass of -\class{\refmodule{unittest}.TestCase}. \class{DocTestCase} isn't documented -here (it's an internal detail), but studying its code can answer questions -about the exact details of \refmodule{unittest} integration. - -Similarly, \function{DocFileSuite()} creates a -\class{\refmodule{unittest}.TestSuite} out of \class{doctest.DocFileCase} -instances, and \class{DocFileCase} is a subclass of \class{DocTestCase}. - -So both ways of creating a \class{\refmodule{unittest}.TestSuite} run -instances of \class{DocTestCase}. This is important for a subtle reason: -when you run \refmodule{doctest} functions yourself, you can control the -\refmodule{doctest} options in use directly, by passing option flags to -\refmodule{doctest} functions. However, if you're writing a -\refmodule{unittest} framework, \refmodule{unittest} ultimately controls -when and how tests get run. The framework author typically wants to -control \refmodule{doctest} reporting options (perhaps, e.g., specified by -command line options), but there's no way to pass options through -\refmodule{unittest} to \refmodule{doctest} test runners. - -For this reason, \refmodule{doctest} also supports a notion of -\refmodule{doctest} reporting flags specific to \refmodule{unittest} -support, via this function: - -\begin{funcdesc}{set_unittest_reportflags}{flags} - Set the \refmodule{doctest} reporting flags to use. - - Argument \var{flags} or's together option flags. See - section~\ref{doctest-options}. Only "reporting flags" can be used. - - This is a module-global setting, and affects all future doctests run by - module \refmodule{unittest}: the \method{runTest()} method of - \class{DocTestCase} looks at the option flags specified for the test case - when the \class{DocTestCase} instance was constructed. If no reporting - flags were specified (which is the typical and expected case), - \refmodule{doctest}'s \refmodule{unittest} reporting flags are or'ed into - the option flags, and the option flags so augmented are passed to the - \class{DocTestRunner} instance created to run the doctest. If any - reporting flags were specified when the \class{DocTestCase} instance was - constructed, \refmodule{doctest}'s \refmodule{unittest} reporting flags - are ignored. - - The value of the \refmodule{unittest} reporting flags in effect before the - function was called is returned by the function. - - \versionadded{2.4} -\end{funcdesc} - - -\subsection{Advanced API\label{doctest-advanced-api}} - -The basic API is a simple wrapper that's intended to make doctest easy -to use. It is fairly flexible, and should meet most users' needs; -however, if you require more fine-grained control over testing, or -wish to extend doctest's capabilities, then you should use the -advanced API. - -The advanced API revolves around two container classes, which are used -to store the interactive examples extracted from doctest cases: - -\begin{itemize} -\item \class{Example}: A single python statement, paired with its - expected output. -\item \class{DocTest}: A collection of \class{Example}s, typically - extracted from a single docstring or text file. -\end{itemize} - -Additional processing classes are defined to find, parse, and run, and -check doctest examples: - -\begin{itemize} -\item \class{DocTestFinder}: Finds all docstrings in a given module, - and uses a \class{DocTestParser} to create a \class{DocTest} - from every docstring that contains interactive examples. -\item \class{DocTestParser}: Creates a \class{DocTest} object from - a string (such as an object's docstring). -\item \class{DocTestRunner}: Executes the examples in a - \class{DocTest}, and uses an \class{OutputChecker} to verify - their output. -\item \class{OutputChecker}: Compares the actual output from a - doctest example with the expected output, and decides whether - they match. -\end{itemize} - -The relationships among these processing classes are summarized in the -following diagram: - -\begin{verbatim} - list of: -+------+ +---------+ -|module| --DocTestFinder-> | DocTest | --DocTestRunner-> results -+------+ | ^ +---------+ | ^ (printed) - | | | Example | | | - v | | ... | v | - DocTestParser | Example | OutputChecker - +---------+ -\end{verbatim} - -\subsubsection{DocTest Objects\label{doctest-DocTest}} -\begin{classdesc}{DocTest}{examples, globs, name, filename, lineno, - docstring} - A collection of doctest examples that should be run in a single - namespace. The constructor arguments are used to initialize the - member variables of the same names. - \versionadded{2.4} -\end{classdesc} - -\class{DocTest} defines the following member variables. They are -initialized by the constructor, and should not be modified directly. - -\begin{memberdesc}{examples} - A list of \class{Example} objects encoding the individual - interactive Python examples that should be run by this test. -\end{memberdesc} - -\begin{memberdesc}{globs} - The namespace (aka globals) that the examples should be run in. - This is a dictionary mapping names to values. Any changes to the - namespace made by the examples (such as binding new variables) - will be reflected in \member{globs} after the test is run. -\end{memberdesc} - -\begin{memberdesc}{name} - A string name identifying the \class{DocTest}. Typically, this is - the name of the object or file that the test was extracted from. -\end{memberdesc} - -\begin{memberdesc}{filename} - The name of the file that this \class{DocTest} was extracted from; - or \code{None} if the filename is unknown, or if the - \class{DocTest} was not extracted from a file. -\end{memberdesc} - -\begin{memberdesc}{lineno} - The line number within \member{filename} where this - \class{DocTest} begins, or \code{None} if the line number is - unavailable. This line number is zero-based with respect to the - beginning of the file. -\end{memberdesc} - -\begin{memberdesc}{docstring} - The string that the test was extracted from, or `None` if the - string is unavailable, or if the test was not extracted from a - string. -\end{memberdesc} - -\subsubsection{Example Objects\label{doctest-Example}} -\begin{classdesc}{Example}{source, want\optional{, - exc_msg}\optional{, lineno}\optional{, - indent}\optional{, options}} - A single interactive example, consisting of a Python statement and - its expected output. The constructor arguments are used to - initialize the member variables of the same names. - \versionadded{2.4} -\end{classdesc} - -\class{Example} defines the following member variables. They are -initialized by the constructor, and should not be modified directly. - -\begin{memberdesc}{source} - A string containing the example's source code. This source code - consists of a single Python statement, and always ends with a - newline; the constructor adds a newline when necessary. -\end{memberdesc} - -\begin{memberdesc}{want} - The expected output from running the example's source code (either - from stdout, or a traceback in case of exception). \member{want} - ends with a newline unless no output is expected, in which case - it's an empty string. The constructor adds a newline when - necessary. -\end{memberdesc} - -\begin{memberdesc}{exc_msg} - The exception message generated by the example, if the example is - expected to generate an exception; or \code{None} if it is not - expected to generate an exception. This exception message is - compared against the return value of - \function{traceback.format_exception_only()}. \member{exc_msg} - ends with a newline unless it's \code{None}. The constructor adds - a newline if needed. -\end{memberdesc} - -\begin{memberdesc}{lineno} - The line number within the string containing this example where - the example begins. This line number is zero-based with respect - to the beginning of the containing string. -\end{memberdesc} - -\begin{memberdesc}{indent} - The example's indentation in the containing string, i.e., the - number of space characters that precede the example's first - prompt. -\end{memberdesc} - -\begin{memberdesc}{options} - A dictionary mapping from option flags to \code{True} or - \code{False}, which is used to override default options for this - example. Any option flags not contained in this dictionary are - left at their default value (as specified by the - \class{DocTestRunner}'s \member{optionflags}). - By default, no options are set. -\end{memberdesc} - -\subsubsection{DocTestFinder objects\label{doctest-DocTestFinder}} -\begin{classdesc}{DocTestFinder}{\optional{verbose}\optional{, - parser}\optional{, recurse}\optional{, - exclude_empty}} - A processing class used to extract the \class{DocTest}s that are - relevant to a given object, from its docstring and the docstrings - of its contained objects. \class{DocTest}s can currently be - extracted from the following object types: modules, functions, - classes, methods, staticmethods, classmethods, and properties. - - The optional argument \var{verbose} can be used to display the - objects searched by the finder. It defaults to \code{False} (no - output). - - The optional argument \var{parser} specifies the - \class{DocTestParser} object (or a drop-in replacement) that is - used to extract doctests from docstrings. - - If the optional argument \var{recurse} is false, then - \method{DocTestFinder.find()} will only examine the given object, - and not any contained objects. - - If the optional argument \var{exclude_empty} is false, then - \method{DocTestFinder.find()} will include tests for objects with - empty docstrings. - - \versionadded{2.4} -\end{classdesc} - -\class{DocTestFinder} defines the following method: - -\begin{methoddesc}{find}{obj\optional{, name}\optional{, - module}\optional{, globs}\optional{, extraglobs}} - Return a list of the \class{DocTest}s that are defined by - \var{obj}'s docstring, or by any of its contained objects' - docstrings. - - The optional argument \var{name} specifies the object's name; this - name will be used to construct names for the returned - \class{DocTest}s. If \var{name} is not specified, then - \code{\var{obj}.__name__} is used. - - The optional parameter \var{module} is the module that contains - the given object. If the module is not specified or is None, then - the test finder will attempt to automatically determine the - correct module. The object's module is used: - - \begin{itemize} - \item As a default namespace, if \var{globs} is not specified. - \item To prevent the DocTestFinder from extracting DocTests - from objects that are imported from other modules. (Contained - objects with modules other than \var{module} are ignored.) - \item To find the name of the file containing the object. - \item To help find the line number of the object within its file. - \end{itemize} - - If \var{module} is \code{False}, no attempt to find the module - will be made. This is obscure, of use mostly in testing doctest - itself: if \var{module} is \code{False}, or is \code{None} but - cannot be found automatically, then all objects are considered to - belong to the (non-existent) module, so all contained objects will - (recursively) be searched for doctests. - - The globals for each \class{DocTest} is formed by combining - \var{globs} and \var{extraglobs} (bindings in \var{extraglobs} - override bindings in \var{globs}). A new shallow copy of the globals - dictionary is created for each \class{DocTest}. If \var{globs} is - not specified, then it defaults to the module's \var{__dict__}, if - specified, or \code{\{\}} otherwise. If \var{extraglobs} is not - specified, then it defaults to \code{\{\}}. -\end{methoddesc} - -\subsubsection{DocTestParser objects\label{doctest-DocTestParser}} -\begin{classdesc}{DocTestParser}{} - A processing class used to extract interactive examples from a - string, and use them to create a \class{DocTest} object. - \versionadded{2.4} -\end{classdesc} - -\class{DocTestParser} defines the following methods: - -\begin{methoddesc}{get_doctest}{string, globs, name, filename, lineno} - Extract all doctest examples from the given string, and collect - them into a \class{DocTest} object. - - \var{globs}, \var{name}, \var{filename}, and \var{lineno} are - attributes for the new \class{DocTest} object. See the - documentation for \class{DocTest} for more information. -\end{methoddesc} - -\begin{methoddesc}{get_examples}{string\optional{, name}} - Extract all doctest examples from the given string, and return - them as a list of \class{Example} objects. Line numbers are - 0-based. The optional argument \var{name} is a name identifying - this string, and is only used for error messages. -\end{methoddesc} - -\begin{methoddesc}{parse}{string\optional{, name}} - Divide the given string into examples and intervening text, and - return them as a list of alternating \class{Example}s and strings. - Line numbers for the \class{Example}s are 0-based. The optional - argument \var{name} is a name identifying this string, and is only - used for error messages. -\end{methoddesc} - -\subsubsection{DocTestRunner objects\label{doctest-DocTestRunner}} -\begin{classdesc}{DocTestRunner}{\optional{checker}\optional{, - verbose}\optional{, optionflags}} - A processing class used to execute and verify the interactive - examples in a \class{DocTest}. - - The comparison between expected outputs and actual outputs is done - by an \class{OutputChecker}. This comparison may be customized - with a number of option flags; see section~\ref{doctest-options} - for more information. If the option flags are insufficient, then - the comparison may also be customized by passing a subclass of - \class{OutputChecker} to the constructor. - - The test runner's display output can be controlled in two ways. - First, an output function can be passed to - \method{TestRunner.run()}; this function will be called with - strings that should be displayed. It defaults to - \code{sys.stdout.write}. If capturing the output is not - sufficient, then the display output can be also customized by - subclassing DocTestRunner, and overriding the methods - \method{report_start}, \method{report_success}, - \method{report_unexpected_exception}, and \method{report_failure}. - - The optional keyword argument \var{checker} specifies the - \class{OutputChecker} object (or drop-in replacement) that should - be used to compare the expected outputs to the actual outputs of - doctest examples. - - The optional keyword argument \var{verbose} controls the - \class{DocTestRunner}'s verbosity. If \var{verbose} is - \code{True}, then information is printed about each example, as it - is run. If \var{verbose} is \code{False}, then only failures are - printed. If \var{verbose} is unspecified, or \code{None}, then - verbose output is used iff the command-line switch \programopt{-v} - is used. - - The optional keyword argument \var{optionflags} can be used to - control how the test runner compares expected output to actual - output, and how it displays failures. For more information, see - section~\ref{doctest-options}. - - \versionadded{2.4} -\end{classdesc} - -\class{DocTestParser} defines the following methods: - -\begin{methoddesc}{report_start}{out, test, example} - Report that the test runner is about to process the given example. - This method is provided to allow subclasses of - \class{DocTestRunner} to customize their output; it should not be - called directly. - - \var{example} is the example about to be processed. \var{test} is - the test containing \var{example}. \var{out} is the output - function that was passed to \method{DocTestRunner.run()}. -\end{methoddesc} - -\begin{methoddesc}{report_success}{out, test, example, got} - Report that the given example ran successfully. This method is - provided to allow subclasses of \class{DocTestRunner} to customize - their output; it should not be called directly. - - \var{example} is the example about to be processed. \var{got} is - the actual output from the example. \var{test} is the test - containing \var{example}. \var{out} is the output function that - was passed to \method{DocTestRunner.run()}. -\end{methoddesc} - -\begin{methoddesc}{report_failure}{out, test, example, got} - Report that the given example failed. This method is provided to - allow subclasses of \class{DocTestRunner} to customize their - output; it should not be called directly. - - \var{example} is the example about to be processed. \var{got} is - the actual output from the example. \var{test} is the test - containing \var{example}. \var{out} is the output function that - was passed to \method{DocTestRunner.run()}. -\end{methoddesc} - -\begin{methoddesc}{report_unexpected_exception}{out, test, example, exc_info} - Report that the given example raised an unexpected exception. - This method is provided to allow subclasses of - \class{DocTestRunner} to customize their output; it should not be - called directly. - - \var{example} is the example about to be processed. - \var{exc_info} is a tuple containing information about the - unexpected exception (as returned by \function{sys.exc_info()}). - \var{test} is the test containing \var{example}. \var{out} is the - output function that was passed to \method{DocTestRunner.run()}. -\end{methoddesc} - -\begin{methoddesc}{run}{test\optional{, compileflags}\optional{, - out}\optional{, clear_globs}} - Run the examples in \var{test} (a \class{DocTest} object), and - display the results using the writer function \var{out}. - - The examples are run in the namespace \code{test.globs}. If - \var{clear_globs} is true (the default), then this namespace will - be cleared after the test runs, to help with garbage collection. - If you would like to examine the namespace after the test - completes, then use \var{clear_globs=False}. - - \var{compileflags} gives the set of flags that should be used by - the Python compiler when running the examples. If not specified, - then it will default to the set of future-import flags that apply - to \var{globs}. - - The output of each example is checked using the - \class{DocTestRunner}'s output checker, and the results are - formatted by the \method{DocTestRunner.report_*} methods. -\end{methoddesc} - -\begin{methoddesc}{summarize}{\optional{verbose}} - Print a summary of all the test cases that have been run by this - DocTestRunner, and return a tuple \samp{(\var{failure_count}, - \var{test_count})}. - - The optional \var{verbose} argument controls how detailed the - summary is. If the verbosity is not specified, then the - \class{DocTestRunner}'s verbosity is used. -\end{methoddesc} - -\subsubsection{OutputChecker objects\label{doctest-OutputChecker}} - -\begin{classdesc}{OutputChecker}{} - A class used to check the whether the actual output from a doctest - example matches the expected output. \class{OutputChecker} - defines two methods: \method{check_output}, which compares a given - pair of outputs, and returns true if they match; and - \method{output_difference}, which returns a string describing the - differences between two outputs. - \versionadded{2.4} -\end{classdesc} - -\class{OutputChecker} defines the following methods: - -\begin{methoddesc}{check_output}{want, got, optionflags} - Return \code{True} iff the actual output from an example - (\var{got}) matches the expected output (\var{want}). These - strings are always considered to match if they are identical; but - depending on what option flags the test runner is using, several - non-exact match types are also possible. See - section~\ref{doctest-options} for more information about option - flags. -\end{methoddesc} - -\begin{methoddesc}{output_difference}{example, got, optionflags} - Return a string describing the differences between the expected - output for a given example (\var{example}) and the actual output - (\var{got}). \var{optionflags} is the set of option flags used to - compare \var{want} and \var{got}. -\end{methoddesc} - -\subsection{Debugging\label{doctest-debugging}} - -Doctest provides several mechanisms for debugging doctest examples: - -\begin{itemize} -\item Several functions convert doctests to executable Python - programs, which can be run under the Python debugger, \refmodule{pdb}. -\item The \class{DebugRunner} class is a subclass of - \class{DocTestRunner} that raises an exception for the first - failing example, containing information about that example. - This information can be used to perform post-mortem debugging on - the example. -\item The \refmodule{unittest} cases generated by \function{DocTestSuite()} - support the \method{debug()} method defined by - \class{\refmodule{unittest}.TestCase}. -\item You can add a call to \function{\refmodule{pdb}.set_trace()} in a - doctest example, and you'll drop into the Python debugger when that - line is executed. Then you can inspect current values of variables, - and so on. For example, suppose \file{a.py} contains just this - module docstring: - -\begin{verbatim} -""" ->>> def f(x): -... g(x*2) ->>> def g(x): -... print x+3 -... import pdb; pdb.set_trace() ->>> f(3) -9 -""" -\end{verbatim} - - Then an interactive Python session may look like this: - -\begin{verbatim} ->>> import a, doctest ->>> doctest.testmod(a) ---Return-- -> <doctest a[1]>(3)g()->None --> import pdb; pdb.set_trace() -(Pdb) list - 1 def g(x): - 2 print x+3 - 3 -> import pdb; pdb.set_trace() -[EOF] -(Pdb) print x -6 -(Pdb) step ---Return-- -> <doctest a[0]>(2)f()->None --> g(x*2) -(Pdb) list - 1 def f(x): - 2 -> g(x*2) -[EOF] -(Pdb) print x -3 -(Pdb) step ---Return-- -> <doctest a[2]>(1)?()->None --> f(3) -(Pdb) cont -(0, 3) ->>> -\end{verbatim} - - \versionchanged[The ability to use \function{\refmodule{pdb}.set_trace()} - usefully inside doctests was added]{2.4} -\end{itemize} - -Functions that convert doctests to Python code, and possibly run -the synthesized code under the debugger: - -\begin{funcdesc}{script_from_examples}{s} - Convert text with examples to a script. - - Argument \var{s} is a string containing doctest examples. The string - is converted to a Python script, where doctest examples in \var{s} - are converted to regular code, and everything else is converted to - Python comments. The generated script is returned as a string. - For example, - - \begin{verbatim} - import doctest - print doctest.script_from_examples(r""" - Set x and y to 1 and 2. - >>> x, y = 1, 2 - - Print their sum: - >>> print x+y - 3 - """) - \end{verbatim} - - displays: - - \begin{verbatim} - # Set x and y to 1 and 2. - x, y = 1, 2 - # - # Print their sum: - print x+y - # Expected: - ## 3 - \end{verbatim} - - This function is used internally by other functions (see below), but - can also be useful when you want to transform an interactive Python - session into a Python script. - - \versionadded{2.4} -\end{funcdesc} - -\begin{funcdesc}{testsource}{module, name} - Convert the doctest for an object to a script. - - Argument \var{module} is a module object, or dotted name of a module, - containing the object whose doctests are of interest. Argument - \var{name} is the name (within the module) of the object with the - doctests of interest. The result is a string, containing the - object's docstring converted to a Python script, as described for - \function{script_from_examples()} above. For example, if module - \file{a.py} contains a top-level function \function{f()}, then - -\begin{verbatim} -import a, doctest -print doctest.testsource(a, "a.f") -\end{verbatim} - - prints a script version of function \function{f()}'s docstring, - with doctests converted to code, and the rest placed in comments. - - \versionadded{2.3} -\end{funcdesc} - -\begin{funcdesc}{debug}{module, name\optional{, pm}} - Debug the doctests for an object. - - The \var{module} and \var{name} arguments are the same as for function - \function{testsource()} above. The synthesized Python script for the - named object's docstring is written to a temporary file, and then that - file is run under the control of the Python debugger, \refmodule{pdb}. - - A shallow copy of \code{\var{module}.__dict__} is used for both local - and global execution context. - - Optional argument \var{pm} controls whether post-mortem debugging is - used. If \var{pm} has a true value, the script file is run directly, and - the debugger gets involved only if the script terminates via raising an - unhandled exception. If it does, then post-mortem debugging is invoked, - via \function{\refmodule{pdb}.post_mortem()}, passing the traceback object - from the unhandled exception. If \var{pm} is not specified, or is false, - the script is run under the debugger from the start, via passing an - appropriate \function{exec()} call to \function{\refmodule{pdb}.run()}. - - \versionadded{2.3} - - \versionchanged[The \var{pm} argument was added]{2.4} -\end{funcdesc} - -\begin{funcdesc}{debug_src}{src\optional{, pm}\optional{, globs}} - Debug the doctests in a string. - - This is like function \function{debug()} above, except that - a string containing doctest examples is specified directly, via - the \var{src} argument. - - Optional argument \var{pm} has the same meaning as in function - \function{debug()} above. - - Optional argument \var{globs} gives a dictionary to use as both - local and global execution context. If not specified, or \code{None}, - an empty dictionary is used. If specified, a shallow copy of the - dictionary is used. - - \versionadded{2.4} -\end{funcdesc} - -The \class{DebugRunner} class, and the special exceptions it may raise, -are of most interest to testing framework authors, and will only be -sketched here. See the source code, and especially \class{DebugRunner}'s -docstring (which is a doctest!) for more details: - -\begin{classdesc}{DebugRunner}{\optional{checker}\optional{, - verbose}\optional{, optionflags}} - - A subclass of \class{DocTestRunner} that raises an exception as - soon as a failure is encountered. If an unexpected exception - occurs, an \exception{UnexpectedException} exception is raised, - containing the test, the example, and the original exception. If - the output doesn't match, then a \exception{DocTestFailure} - exception is raised, containing the test, the example, and the - actual output. - - For information about the constructor parameters and methods, see - the documentation for \class{DocTestRunner} in - section~\ref{doctest-advanced-api}. -\end{classdesc} - -There are two exceptions that may be raised by \class{DebugRunner} -instances: - -\begin{excclassdesc}{DocTestFailure}{test, example, got} - An exception thrown by \class{DocTestRunner} to signal that a - doctest example's actual output did not match its expected output. - The constructor arguments are used to initialize the member - variables of the same names. -\end{excclassdesc} -\exception{DocTestFailure} defines the following member variables: -\begin{memberdesc}{test} - The \class{DocTest} object that was being run when the example failed. -\end{memberdesc} -\begin{memberdesc}{example} - The \class{Example} that failed. -\end{memberdesc} -\begin{memberdesc}{got} - The example's actual output. -\end{memberdesc} - -\begin{excclassdesc}{UnexpectedException}{test, example, exc_info} - An exception thrown by \class{DocTestRunner} to signal that a - doctest example raised an unexpected exception. The constructor - arguments are used to initialize the member variables of the same - names. -\end{excclassdesc} -\exception{UnexpectedException} defines the following member variables: -\begin{memberdesc}{test} - The \class{DocTest} object that was being run when the example failed. -\end{memberdesc} -\begin{memberdesc}{example} - The \class{Example} that failed. -\end{memberdesc} -\begin{memberdesc}{exc_info} - A tuple containing information about the unexpected exception, as - returned by \function{sys.exc_info()}. -\end{memberdesc} - -\subsection{Soapbox\label{doctest-soapbox}} - -As mentioned in the introduction, \refmodule{doctest} has grown to have -three primary uses: - -\begin{enumerate} -\item Checking examples in docstrings. -\item Regression testing. -\item Executable documentation / literate testing. -\end{enumerate} - -These uses have different requirements, and it is important to -distinguish them. In particular, filling your docstrings with obscure -test cases makes for bad documentation. - -When writing a docstring, choose docstring examples with care. -There's an art to this that needs to be learned---it may not be -natural at first. Examples should add genuine value to the -documentation. A good example can often be worth many words. -If done with care, the examples will be invaluable for your users, and -will pay back the time it takes to collect them many times over as the -years go by and things change. I'm still amazed at how often one of -my \refmodule{doctest} examples stops working after a "harmless" -change. - -Doctest also makes an excellent tool for regression testing, especially if -you don't skimp on explanatory text. By interleaving prose and examples, -it becomes much easier to keep track of what's actually being tested, and -why. When a test fails, good prose can make it much easier to figure out -what the problem is, and how it should be fixed. It's true that you could -write extensive comments in code-based testing, but few programmers do. -Many have found that using doctest approaches instead leads to much clearer -tests. Perhaps this is simply because doctest makes writing prose a little -easier than writing code, while writing comments in code is a little -harder. I think it goes deeper than just that: the natural attitude -when writing a doctest-based test is that you want to explain the fine -points of your software, and illustrate them with examples. This in -turn naturally leads to test files that start with the simplest features, -and logically progress to complications and edge cases. A coherent -narrative is the result, instead of a collection of isolated functions -that test isolated bits of functionality seemingly at random. It's -a different attitude, and produces different results, blurring the -distinction between testing and explaining. - -Regression testing is best confined to dedicated objects or files. There -are several options for organizing tests: - -\begin{itemize} -\item Write text files containing test cases as interactive examples, - and test the files using \function{testfile()} or - \function{DocFileSuite()}. This is recommended, although is - easiest to do for new projects, designed from the start to use - doctest. -\item Define functions named \code{_regrtest_\textit{topic}} that - consist of single docstrings, containing test cases for the - named topics. These functions can be included in the same file - as the module, or separated out into a separate test file. -\item Define a \code{__test__} dictionary mapping from regression test - topics to docstrings containing test cases. -\end{itemize} |