diff options
Diffstat (limited to 'Doc/library')
204 files changed, 15136 insertions, 5900 deletions
diff --git a/Doc/library/2to3.rst b/Doc/library/2to3.rst index de31251..b3efeab 100644 --- a/Doc/library/2to3.rst +++ b/Doc/library/2to3.rst @@ -141,7 +141,7 @@ and off individually. They are described here in more detail. .. 2to3fixer:: exec - Converts the :keyword:`exec` statement to the :func:`exec` function. + Converts the ``exec`` statement to the :func:`exec` function. .. 2to3fixer:: execfile @@ -267,6 +267,25 @@ and off individually. They are described here in more detail. Converts octal literals into the new syntax. +.. 2to3fixer:: operator + + Converts calls to various functions in the :mod:`operator` module to other, + but equivalent, function calls. When needed, the appropriate ``import`` + statements are added, e.g. ``import collections``. The following mapping + are made: + + ================================== ========================================== + From To + ================================== ========================================== + ``operator.isCallable(obj)`` ``hasattr(obj, '__call__')`` + ``operator.sequenceIncludes(obj)`` ``operator.contains(obj)`` + ``operator.isSequenceType(obj)`` ``isinstance(obj, collections.Sequence)`` + ``operator.isMappingType(obj)`` ``isinstance(obj, collections.Mapping)`` + ``operator.isNumberType(obj)`` ``isinstance(obj, numbers.Number)`` + ``operator.repeat(obj, n)`` ``operator.mul(obj, n)`` + ``operator.irepeat(obj, n)`` ``operator.imul(obj, n)`` + ================================== ========================================== + .. 2to3fixer:: paren Add extra parenthesis where they are required in list comprehensions. For @@ -274,7 +293,7 @@ and off individually. They are described here in more detail. .. 2to3fixer:: print - Converts the :keyword:`print` statement to the :func:`print` function. + Converts the ``print`` statement to the :func:`print` function. .. 2to3fixer:: raise diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst index 2348217..0acccc5 100644 --- a/Doc/library/__future__.rst +++ b/Doc/library/__future__.rst @@ -4,6 +4,9 @@ .. module:: __future__ :synopsis: Future statement definitions +**Source code:** :source:`Lib/__future__.py` + +-------------- :mod:`__future__` is a real module, and serves three purposes: diff --git a/Doc/library/_dummy_thread.rst b/Doc/library/_dummy_thread.rst index 62e5708..83aec12 100644 --- a/Doc/library/_dummy_thread.rst +++ b/Doc/library/_dummy_thread.rst @@ -4,6 +4,9 @@ .. module:: _dummy_thread :synopsis: Drop-in replacement for the _thread module. +**Source code:** :source:`Lib/_dummy_thread.py` + +-------------- This module provides a duplicate interface to the :mod:`_thread` module. It is meant to be imported when the :mod:`_thread` module is not provided on a diff --git a/Doc/library/_thread.rst b/Doc/library/_thread.rst index cb62407..369e9cd 100644 --- a/Doc/library/_thread.rst +++ b/Doc/library/_thread.rst @@ -28,7 +28,7 @@ implementation. For systems lacking the :mod:`_thread` module, the :mod:`_dummy_thread` module is available. It duplicates this module's interface and can be used as a drop-in replacement. -It defines the following constant and functions: +It defines the following constants and functions: .. exception:: error @@ -103,18 +103,42 @@ It defines the following constant and functions: Availability: Windows, systems with POSIX threads. +.. data:: TIMEOUT_MAX + + The maximum value allowed for the *timeout* parameter of + :meth:`Lock.acquire`. Specifying a timeout greater than this value will + raise an :exc:`OverflowError`. + + .. versionadded:: 3.2 + + Lock objects have the following methods: -.. method:: lock.acquire([waitflag]) +.. method:: lock.acquire(waitflag=1, timeout=-1) - Without the optional argument, this method acquires the lock unconditionally, if + Without any optional argument, this method acquires the lock unconditionally, if necessary waiting until it is released by another thread (only one thread at a - time can acquire a lock --- that's their reason for existence). If the integer - *waitflag* argument is present, the action depends on its value: if it is zero, - the lock is only acquired if it can be acquired immediately without waiting, - while if it is nonzero, the lock is acquired unconditionally as before. The - return value is ``True`` if the lock is acquired successfully, ``False`` if not. + time can acquire a lock --- that's their reason for existence). + + If the integer *waitflag* argument is present, the action depends on its + value: if it is zero, the lock is only acquired if it can be acquired + immediately without waiting, while if it is nonzero, the lock is acquired + unconditionally as above. + + If the floating-point *timeout* argument is present and positive, it + specifies the maximum wait time in seconds before returning. A negative + *timeout* argument specifies an unbounded wait. You cannot specify + a *timeout* if *waitflag* is zero. + + The return value is ``True`` if the lock is acquired successfully, + ``False`` if not. + + .. versionchanged:: 3.2 + The *timeout* parameter is new. + + .. versionchanged:: 3.2 + Lock acquires can now be interrupted by signals on POSIX. .. method:: lock.release() @@ -156,12 +180,10 @@ In addition to these methods, lock objects can also be used via the * It is not possible to interrupt the :meth:`acquire` method on a lock --- the :exc:`KeyboardInterrupt` exception will happen after the lock has been acquired. - .. index:: pair: threads; IRIX - * When the main thread exits, it is system defined whether the other threads - survive. On SGI IRIX using the native thread implementation, they survive. On - most other systems, they are killed without executing :keyword:`try` ... - :keyword:`finally` clauses or executing object destructors. + survive. On most systems, they are killed without executing + :keyword:`try` ... :keyword:`finally` clauses or executing object + destructors. * When the main thread exits, it does not do any of its usual cleanup (except that :keyword:`try` ... :keyword:`finally` clauses are honored), and the diff --git a/Doc/library/abc.rst b/Doc/library/abc.rst index aa1cc78..9fadbd2 100644 --- a/Doc/library/abc.rst +++ b/Doc/library/abc.rst @@ -7,6 +7,10 @@ .. sectionauthor:: Georg Brandl .. much of the content adapted from docstrings +**Source code:** :source:`Lib/abc.py` + +-------------- + This module provides the infrastructure for defining an :term:`abstract base class` (ABCs) in Python, as outlined in :pep:`3119`; see the PEP for why this was added to Python. (See also :pep:`3141` and the :mod:`numbers` module @@ -122,7 +126,7 @@ This module provides the following class: It also provides the following decorators: -.. function:: abstractmethod(function) +.. decorator:: abstractmethod(function) A decorator indicating abstract methods. @@ -157,6 +161,36 @@ It also provides the following decorators: multiple-inheritance. +.. decorator:: abstractclassmethod(function) + + A subclass of the built-in :func:`classmethod`, indicating an abstract + classmethod. Otherwise it is similar to :func:`abstractmethod`. + + Usage:: + + class C(metaclass=ABCMeta): + @abstractclassmethod + def my_abstract_classmethod(cls, ...): + ... + + .. versionadded:: 3.2 + + +.. decorator:: abstractstaticmethod(function) + + A subclass of the built-in :func:`staticmethod`, indicating an abstract + staticmethod. Otherwise it is similar to :func:`abstractmethod`. + + Usage:: + + class C(metaclass=ABCMeta): + @abstractstaticmethod + def my_abstract_staticmethod(...): + ... + + .. versionadded:: 3.2 + + .. function:: abstractproperty(fget=None, fset=None, fdel=None, doc=None) A subclass of the built-in :func:`property`, indicating an abstract property. diff --git a/Doc/library/aifc.rst b/Doc/library/aifc.rst index 304437d..999bad8 100644 --- a/Doc/library/aifc.rst +++ b/Doc/library/aifc.rst @@ -10,6 +10,10 @@ single: AIFF single: AIFF-C +**Source code:** :source:`Lib/aifc.py` + +-------------- + This module provides support for reading and writing AIFF and AIFF-C files. AIFF is Audio Interchange File Format, a format for storing digital audio samples in a file. AIFF-C is a newer version of the format that includes the diff --git a/Doc/library/allos.rst b/Doc/library/allos.rst index f25c3b8..bf91717 100644 --- a/Doc/library/allos.rst +++ b/Doc/library/allos.rst @@ -15,9 +15,12 @@ but they are available on most other systems as well. Here's an overview: os.rst io.rst time.rst + argparse.rst optparse.rst getopt.rst logging.rst + logging.config.rst + logging.handlers.rst getpass.rst curses.rst curses.ascii.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst new file mode 100644 index 0000000..61e1786 --- /dev/null +++ b/Doc/library/argparse.rst @@ -0,0 +1,1816 @@ +:mod:`argparse` --- Parser for command-line options, arguments and sub-commands +=============================================================================== + +.. module:: argparse + :synopsis: Command-line option and argument-parsing library. +.. moduleauthor:: Steven Bethard <steven.bethard@gmail.com> +.. sectionauthor:: Steven Bethard <steven.bethard@gmail.com> + +**Source code:** :source:`Lib/argparse.py` + +.. versionadded:: 3.2 + +-------------- + +The :mod:`argparse` module makes it easy to write user-friendly command-line +interfaces. The program defines what arguments it requires, and :mod:`argparse` +will figure out how to parse those out of :data:`sys.argv`. The :mod:`argparse` +module also automatically generates help and usage messages and issues errors +when users give the program invalid arguments. + + +Example +------- + +The following code is a Python program that takes a list of integers and +produces either the sum or the max:: + + import argparse + + parser = argparse.ArgumentParser(description='Process some integers.') + parser.add_argument('integers', metavar='N', type=int, nargs='+', + help='an integer for the accumulator') + parser.add_argument('--sum', dest='accumulate', action='store_const', + const=sum, default=max, + help='sum the integers (default: find the max)') + + args = parser.parse_args() + print(args.accumulate(args.integers)) + +Assuming the Python code above is saved into a file called ``prog.py``, it can +be run at the command line and provides useful help messages:: + + $ prog.py -h + usage: prog.py [-h] [--sum] N [N ...] + + Process some integers. + + positional arguments: + N an integer for the accumulator + + optional arguments: + -h, --help show this help message and exit + --sum sum the integers (default: find the max) + +When run with the appropriate arguments, it prints either the sum or the max of +the command-line integers:: + + $ prog.py 1 2 3 4 + 4 + + $ prog.py 1 2 3 4 --sum + 10 + +If invalid arguments are passed in, it will issue an error:: + + $ prog.py a b c + usage: prog.py [-h] [--sum] N [N ...] + prog.py: error: argument N: invalid int value: 'a' + +The following sections walk you through this example. + + +Creating a parser +^^^^^^^^^^^^^^^^^ + +The first step in using the :mod:`argparse` is creating an +:class:`ArgumentParser` object:: + + >>> parser = argparse.ArgumentParser(description='Process some integers.') + +The :class:`ArgumentParser` object will hold all the information necessary to +parse the command line into Python data types. + + +Adding arguments +^^^^^^^^^^^^^^^^ + +Filling an :class:`ArgumentParser` with information about program arguments is +done by making calls to the :meth:`~ArgumentParser.add_argument` method. +Generally, these calls tell the :class:`ArgumentParser` how to take the strings +on the command line and turn them into objects. This information is stored and +used when :meth:`~ArgumentParser.parse_args` is called. For example:: + + >>> parser.add_argument('integers', metavar='N', type=int, nargs='+', + ... help='an integer for the accumulator') + >>> parser.add_argument('--sum', dest='accumulate', action='store_const', + ... const=sum, default=max, + ... help='sum the integers (default: find the max)') + +Later, calling :meth:`~ArgumentParser.parse_args` will return an object with +two attributes, ``integers`` and ``accumulate``. The ``integers`` attribute +will be a list of one or more ints, and the ``accumulate`` attribute will be +either the :func:`sum` function, if ``--sum`` was specified at the command line, +or the :func:`max` function if it was not. + + +Parsing arguments +^^^^^^^^^^^^^^^^^ + +:class:`ArgumentParser` parses args through the +:meth:`~ArgumentParser.parse_args` method. This will inspect the command line, +convert each arg to the appropriate type and then invoke the appropriate action. +In most cases, this means a simple namespace object will be built up from +attributes parsed out of the command line:: + + >>> parser.parse_args(['--sum', '7', '-1', '42']) + Namespace(accumulate=<built-in function sum>, integers=[7, -1, 42]) + +In a script, :meth:`~ArgumentParser.parse_args` will typically be called with no +arguments, and the :class:`ArgumentParser` will automatically determine the +command-line args from :data:`sys.argv`. + + +ArgumentParser objects +---------------------- + +.. class:: ArgumentParser([description], [epilog], [prog], [usage], [add_help], \ + [argument_default], [parents], [prefix_chars], \ + [conflict_handler], [formatter_class]) + + Create a new :class:`ArgumentParser` object. Each parameter has its own more + detailed description below, but in short they are: + + * description_ - Text to display before the argument help. + + * epilog_ - Text to display after the argument help. + + * add_help_ - Add a -h/--help option to the parser. (default: ``True``) + + * argument_default_ - Set the global default value for arguments. + (default: ``None``) + + * parents_ - A list of :class:`ArgumentParser` objects whose arguments should + also be included. + + * prefix_chars_ - The set of characters that prefix optional arguments. + (default: '-') + + * fromfile_prefix_chars_ - The set of characters that prefix files from + which additional arguments should be read. (default: ``None``) + + * formatter_class_ - A class for customizing the help output. + + * conflict_handler_ - Usually unnecessary, defines strategy for resolving + conflicting optionals. + + * prog_ - The name of the program (default: + :data:`sys.argv[0]`) + + * usage_ - The string describing the program usage (default: generated) + +The following sections describe how each of these are used. + + +description +^^^^^^^^^^^ + +Most calls to the :class:`ArgumentParser` constructor will use the +``description=`` keyword argument. This argument gives a brief description of +what the program does and how it works. In help messages, the description is +displayed between the command-line usage string and the help messages for the +various arguments:: + + >>> parser = argparse.ArgumentParser(description='A foo that bars') + >>> parser.print_help() + usage: argparse.py [-h] + + A foo that bars + + optional arguments: + -h, --help show this help message and exit + +By default, the description will be line-wrapped so that it fits within the +given space. To change this behavior, see the formatter_class_ argument. + + +epilog +^^^^^^ + +Some programs like to display additional description of the program after the +description of the arguments. Such text can be specified using the ``epilog=`` +argument to :class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser( + ... description='A foo that bars', + ... epilog="And that's how you'd foo a bar") + >>> parser.print_help() + usage: argparse.py [-h] + + A foo that bars + + optional arguments: + -h, --help show this help message and exit + + And that's how you'd foo a bar + +As with the description_ argument, the ``epilog=`` text is by default +line-wrapped, but this behavior can be adjusted with the formatter_class_ +argument to :class:`ArgumentParser`. + + +add_help +^^^^^^^^ + +By default, ArgumentParser objects add an option which simply displays +the parser's help message. For example, consider a file named +``myprogram.py`` containing the following code:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument('--foo', help='foo help') + args = parser.parse_args() + +If ``-h`` or ``--help`` is supplied at the command line, the ArgumentParser +help will be printed:: + + $ python myprogram.py --help + usage: myprogram.py [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo help + +Occasionally, it may be useful to disable the addition of this help option. +This can be achieved by passing ``False`` as the ``add_help=`` argument to +:class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) + >>> parser.add_argument('--foo', help='foo help') + >>> parser.print_help() + usage: PROG [--foo FOO] + + optional arguments: + --foo FOO foo help + +The help option is typically ``-h/--help``. The exception to this is +if the ``prefix_chars=`` is specified and does not include ``'-'``, in +which case ``-h`` and ``--help`` are not valid options. In +this case, the first character in ``prefix_chars`` is used to prefix +the help options:: + + >>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='+/') + >>> parser.print_help() + usage: PROG [+h] + + optional arguments: + +h, ++help show this help message and exit + + +prefix_chars +^^^^^^^^^^^^ + +Most command-line options will use ``'-'`` as the prefix, e.g. ``-f/--foo``. +Parsers that need to support different or additional prefix +characters, e.g. for options +like ``+f`` or ``/foo``, may specify them using the ``prefix_chars=`` argument +to the ArgumentParser constructor:: + + >>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='-+') + >>> parser.add_argument('+f') + >>> parser.add_argument('++bar') + >>> parser.parse_args('+f X ++bar Y'.split()) + Namespace(bar='Y', f='X') + +The ``prefix_chars=`` argument defaults to ``'-'``. Supplying a set of +characters that does not include ``'-'`` will cause ``-f/--foo`` options to be +disallowed. + + +fromfile_prefix_chars +^^^^^^^^^^^^^^^^^^^^^ + +Sometimes, for example when dealing with a particularly long argument lists, it +may make sense to keep the list of arguments in a file rather than typing it out +at the command line. If the ``fromfile_prefix_chars=`` argument is given to the +:class:`ArgumentParser` constructor, then arguments that start with any of the +specified characters will be treated as files, and will be replaced by the +arguments they contain. For example:: + + >>> with open('args.txt', 'w') as fp: + ... fp.write('-f\nbar') + >>> parser = argparse.ArgumentParser(fromfile_prefix_chars='@') + >>> parser.add_argument('-f') + >>> parser.parse_args(['-f', 'foo', '@args.txt']) + Namespace(f='bar') + +Arguments read from a file must by default be one per line (but see also +:meth:`~ArgumentParser.convert_arg_line_to_args`) and are treated as if they +were in the same place as the original file referencing argument on the command +line. So in the example above, the expression ``['-f', 'foo', '@args.txt']`` +is considered equivalent to the expression ``['-f', 'foo', '-f', 'bar']``. + +The ``fromfile_prefix_chars=`` argument defaults to ``None``, meaning that +arguments will never be treated as file references. + + +argument_default +^^^^^^^^^^^^^^^^ + +Generally, argument defaults are specified either by passing a default to +:meth:`~ArgumentParser.add_argument` or by calling the +:meth:`~ArgumentParser.set_defaults` methods with a specific set of name-value +pairs. Sometimes however, it may be useful to specify a single parser-wide +default for arguments. This can be accomplished by passing the +``argument_default=`` keyword argument to :class:`ArgumentParser`. For example, +to globally suppress attribute creation on :meth:`~ArgumentParser.parse_args` +calls, we supply ``argument_default=SUPPRESS``:: + + >>> parser = argparse.ArgumentParser(argument_default=argparse.SUPPRESS) + >>> parser.add_argument('--foo') + >>> parser.add_argument('bar', nargs='?') + >>> parser.parse_args(['--foo', '1', 'BAR']) + Namespace(bar='BAR', foo='1') + >>> parser.parse_args([]) + Namespace() + + +parents +^^^^^^^ + +Sometimes, several parsers share a common set of arguments. Rather than +repeating the definitions of these arguments, a single parser with all the +shared arguments and passed to ``parents=`` argument to :class:`ArgumentParser` +can be used. The ``parents=`` argument takes a list of :class:`ArgumentParser` +objects, collects all the positional and optional actions from them, and adds +these actions to the :class:`ArgumentParser` object being constructed:: + + >>> parent_parser = argparse.ArgumentParser(add_help=False) + >>> parent_parser.add_argument('--parent', type=int) + + >>> foo_parser = argparse.ArgumentParser(parents=[parent_parser]) + >>> foo_parser.add_argument('foo') + >>> foo_parser.parse_args(['--parent', '2', 'XXX']) + Namespace(foo='XXX', parent=2) + + >>> bar_parser = argparse.ArgumentParser(parents=[parent_parser]) + >>> bar_parser.add_argument('--bar') + >>> bar_parser.parse_args(['--bar', 'YYY']) + Namespace(bar='YYY', parent=None) + +Note that most parent parsers will specify ``add_help=False``. Otherwise, the +:class:`ArgumentParser` will see two ``-h/--help`` options (one in the parent +and one in the child) and raise an error. + +.. note:: + You must fully initialize the parsers before passing them via ``parents=``. + If you change the parent parsers after the child parser, those changes will + not be reflected in the child. + + +formatter_class +^^^^^^^^^^^^^^^ + +:class:`ArgumentParser` objects allow the help formatting to be customized by +specifying an alternate formatting class. Currently, there are three such +classes: + +.. class:: RawDescriptionHelpFormatter + RawTextHelpFormatter + ArgumentDefaultsHelpFormatter + +The first two allow more control over how textual descriptions are displayed, +while the last automatically adds information about argument default values. + +By default, :class:`ArgumentParser` objects line-wrap the description_ and +epilog_ texts in command-line help messages:: + + >>> parser = argparse.ArgumentParser( + ... prog='PROG', + ... description='''this description + ... was indented weird + ... but that is okay''', + ... epilog=''' + ... likewise for this epilog whose whitespace will + ... be cleaned up and whose words will be wrapped + ... across a couple lines''') + >>> parser.print_help() + usage: PROG [-h] + + this description was indented weird but that is okay + + optional arguments: + -h, --help show this help message and exit + + likewise for this epilog whose whitespace will be cleaned up and whose words + will be wrapped across a couple lines + +Passing :class:`~argparse.RawDescriptionHelpFormatter` as ``formatter_class=`` +indicates that description_ and epilog_ are already correctly formatted and +should not be line-wrapped:: + + >>> parser = argparse.ArgumentParser( + ... prog='PROG', + ... formatter_class=argparse.RawDescriptionHelpFormatter, + ... description=textwrap.dedent('''\ + ... Please do not mess up this text! + ... -------------------------------- + ... I have indented it + ... exactly the way + ... I want it + ... ''')) + >>> parser.print_help() + usage: PROG [-h] + + Please do not mess up this text! + -------------------------------- + I have indented it + exactly the way + I want it + + optional arguments: + -h, --help show this help message and exit + +:class:`RawTextHelpFormatter` maintains whitespace for all sorts of help text +including argument descriptions. + +The other formatter class available, :class:`ArgumentDefaultsHelpFormatter`, +will add information about the default value of each of the arguments:: + + >>> parser = argparse.ArgumentParser( + ... prog='PROG', + ... formatter_class=argparse.ArgumentDefaultsHelpFormatter) + >>> parser.add_argument('--foo', type=int, default=42, help='FOO!') + >>> parser.add_argument('bar', nargs='*', default=[1, 2, 3], help='BAR!') + >>> parser.print_help() + usage: PROG [-h] [--foo FOO] [bar [bar ...]] + + positional arguments: + bar BAR! (default: [1, 2, 3]) + + optional arguments: + -h, --help show this help message and exit + --foo FOO FOO! (default: 42) + + +conflict_handler +^^^^^^^^^^^^^^^^ + +:class:`ArgumentParser` objects do not allow two actions with the same option +string. By default, :class:`ArgumentParser` objects raises an exception if an +attempt is made to create an argument with an option string that is already in +use:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-f', '--foo', help='old foo help') + >>> parser.add_argument('--foo', help='new foo help') + Traceback (most recent call last): + .. + ArgumentError: argument --foo: conflicting option string(s): --foo + +Sometimes (e.g. when using parents_) it may be useful to simply override any +older arguments with the same option string. To get this behavior, the value +``'resolve'`` can be supplied to the ``conflict_handler=`` argument of +:class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser(prog='PROG', conflict_handler='resolve') + >>> parser.add_argument('-f', '--foo', help='old foo help') + >>> parser.add_argument('--foo', help='new foo help') + >>> parser.print_help() + usage: PROG [-h] [-f FOO] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + -f FOO old foo help + --foo FOO new foo help + +Note that :class:`ArgumentParser` objects only remove an action if all of its +option strings are overridden. So, in the example above, the old ``-f/--foo`` +action is retained as the ``-f`` action, because only the ``--foo`` option +string was overridden. + + +prog +^^^^ + +By default, :class:`ArgumentParser` objects uses ``sys.argv[0]`` to determine +how to display the name of the program in help messages. This default is almost +always desirable because it will make the help messages match how the program was +invoked on the command line. For example, consider a file named +``myprogram.py`` with the following code:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument('--foo', help='foo help') + args = parser.parse_args() + +The help for this program will display ``myprogram.py`` as the program name +(regardless of where the program was invoked from):: + + $ python myprogram.py --help + usage: myprogram.py [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo help + $ cd .. + $ python subdir\myprogram.py --help + usage: myprogram.py [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo help + +To change this default behavior, another value can be supplied using the +``prog=`` argument to :class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser(prog='myprogram') + >>> parser.print_help() + usage: myprogram [-h] + + optional arguments: + -h, --help show this help message and exit + +Note that the program name, whether determined from ``sys.argv[0]`` or from the +``prog=`` argument, is available to help messages using the ``%(prog)s`` format +specifier. + +:: + + >>> parser = argparse.ArgumentParser(prog='myprogram') + >>> parser.add_argument('--foo', help='foo of the %(prog)s program') + >>> parser.print_help() + usage: myprogram [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo of the myprogram program + + +usage +^^^^^ + +By default, :class:`ArgumentParser` calculates the usage message from the +arguments it contains:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--foo', nargs='?', help='foo help') + >>> parser.add_argument('bar', nargs='+', help='bar help') + >>> parser.print_help() + usage: PROG [-h] [--foo [FOO]] bar [bar ...] + + positional arguments: + bar bar help + + optional arguments: + -h, --help show this help message and exit + --foo [FOO] foo help + +The default message can be overridden with the ``usage=`` keyword argument:: + + >>> parser = argparse.ArgumentParser(prog='PROG', usage='%(prog)s [options]') + >>> parser.add_argument('--foo', nargs='?', help='foo help') + >>> parser.add_argument('bar', nargs='+', help='bar help') + >>> parser.print_help() + usage: PROG [options] + + positional arguments: + bar bar help + + optional arguments: + -h, --help show this help message and exit + --foo [FOO] foo help + +The ``%(prog)s`` format specifier is available to fill in the program name in +your usage messages. + + +The add_argument() method +------------------------- + +.. method:: ArgumentParser.add_argument(name or flags..., [action], [nargs], \ + [const], [default], [type], [choices], [required], \ + [help], [metavar], [dest]) + + Define how a single command-line argument should be parsed. Each parameter + has its own more detailed description below, but in short they are: + + * `name or flags`_ - Either a name or a list of option strings, e.g. ``foo`` + or ``-f, --foo``. + + * action_ - The basic type of action to be taken when this argument is + encountered at the command line. + + * nargs_ - The number of command-line arguments that should be consumed. + + * const_ - A constant value required by some action_ and nargs_ selections. + + * default_ - The value produced if the argument is absent from the + command line. + + * type_ - The type to which the command-line argument should be converted. + + * choices_ - A container of the allowable values for the argument. + + * required_ - Whether or not the command-line option may be omitted + (optionals only). + + * help_ - A brief description of what the argument does. + + * metavar_ - A name for the argument in usage messages. + + * dest_ - The name of the attribute to be added to the object returned by + :meth:`parse_args`. + +The following sections describe how each of these are used. + + +name or flags +^^^^^^^^^^^^^ + +The :meth:`~ArgumentParser.add_argument` method must know whether an optional +argument, like ``-f`` or ``--foo``, or a positional argument, like a list of +filenames, is expected. The first arguments passed to +:meth:`~ArgumentParser.add_argument` must therefore be either a series of +flags, or a simple argument name. For example, an optional argument could +be created like:: + + >>> parser.add_argument('-f', '--foo') + +while a positional argument could be created like:: + + >>> parser.add_argument('bar') + +When :meth:`~ArgumentParser.parse_args` is called, optional arguments will be +identified by the ``-`` prefix, and the remaining arguments will be assumed to +be positional:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-f', '--foo') + >>> parser.add_argument('bar') + >>> parser.parse_args(['BAR']) + Namespace(bar='BAR', foo=None) + >>> parser.parse_args(['BAR', '--foo', 'FOO']) + Namespace(bar='BAR', foo='FOO') + >>> parser.parse_args(['--foo', 'FOO']) + usage: PROG [-h] [-f FOO] bar + PROG: error: too few arguments + + +action +^^^^^^ + +:class:`ArgumentParser` objects associate command-line args with actions. These +actions can do just about anything with the command-line args associated with +them, though most actions simply add an attribute to the object returned by +:meth:`~ArgumentParser.parse_args`. The ``action`` keyword argument specifies +how the command-line args should be handled. The supported actions are: + +* ``'store'`` - This just stores the argument's value. This is the default + action. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> parser.parse_args('--foo 1'.split()) + Namespace(foo='1') + +* ``'store_const'`` - This stores the value specified by the const_ keyword + argument. (Note that the const_ keyword argument defaults to the rather + unhelpful ``None``.) The ``'store_const'`` action is most commonly used with + optional arguments that specify some sort of flag. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='store_const', const=42) + >>> parser.parse_args('--foo'.split()) + Namespace(foo=42) + +* ``'store_true'`` and ``'store_false'`` - These store the values ``True`` and + ``False`` respectively. These are special cases of ``'store_const'``. For + example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='store_true') + >>> parser.add_argument('--bar', action='store_false') + >>> parser.parse_args('--foo --bar'.split()) + Namespace(bar=False, foo=True) + +* ``'append'`` - This stores a list, and appends each argument value to the + list. This is useful to allow an option to be specified multiple times. + Example usage:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='append') + >>> parser.parse_args('--foo 1 --foo 2'.split()) + Namespace(foo=['1', '2']) + +* ``'append_const'`` - This stores a list, and appends the value specified by + the const_ keyword argument to the list. (Note that the const_ keyword + argument defaults to ``None``.) The ``'append_const'`` action is typically + useful when multiple arguments need to store constants to the same list. For + example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--str', dest='types', action='append_const', const=str) + >>> parser.add_argument('--int', dest='types', action='append_const', const=int) + >>> parser.parse_args('--str --int'.split()) + Namespace(types=[<type 'str'>, <type 'int'>]) + +* ``'version'`` - This expects a ``version=`` keyword argument in the + :meth:`~ArgumentParser.add_argument` call, and prints version information + and exits when invoked. + + >>> import argparse + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--version', action='version', version='%(prog)s 2.0') + >>> parser.parse_args(['--version']) + PROG 2.0 + +You can also specify an arbitrary action by passing an object that implements +the Action API. The easiest way to do this is to extend +:class:`argparse.Action`, supplying an appropriate ``__call__`` method. The +``__call__`` method should accept four parameters: + +* ``parser`` - The ArgumentParser object which contains this action. + +* ``namespace`` - The namespace object that will be returned by + :meth:`~ArgumentParser.parse_args`. Most actions add an attribute to this + object. + +* ``values`` - The associated command-line args, with any type-conversions + applied. (Type-conversions are specified with the type_ keyword argument to + :meth:`~ArgumentParser.add_argument`. + +* ``option_string`` - The option string that was used to invoke this action. + The ``option_string`` argument is optional, and will be absent if the action + is associated with a positional argument. + +An example of a custom action:: + + >>> class FooAction(argparse.Action): + ... def __call__(self, parser, namespace, values, option_string=None): + ... print('%r %r %r' % (namespace, values, option_string)) + ... setattr(namespace, self.dest, values) + ... + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action=FooAction) + >>> parser.add_argument('bar', action=FooAction) + >>> args = parser.parse_args('1 --foo 2'.split()) + Namespace(bar=None, foo=None) '1' None + Namespace(bar='1', foo=None) '2' '--foo' + >>> args + Namespace(bar='1', foo='2') + + +nargs +^^^^^ + +ArgumentParser objects usually associate a single command-line argument with a +single action to be taken. The ``nargs`` keyword argument associates a +different number of command-line arguments with a single action. The supported +values are: + +* N (an integer). N args from the command line will be gathered together into a + list. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', nargs=2) + >>> parser.add_argument('bar', nargs=1) + >>> parser.parse_args('c --foo a b'.split()) + Namespace(bar=['c'], foo=['a', 'b']) + + Note that ``nargs=1`` produces a list of one item. This is different from + the default, in which the item is produced by itself. + +* ``'?'``. One arg will be consumed from the command line if possible, and + produced as a single item. If no command-line arg is present, the value from + default_ will be produced. Note that for optional arguments, there is an + additional case - the option string is present but not followed by a + command-line arg. In this case the value from const_ will be produced. Some + examples to illustrate this:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', nargs='?', const='c', default='d') + >>> parser.add_argument('bar', nargs='?', default='d') + >>> parser.parse_args('XX --foo YY'.split()) + Namespace(bar='XX', foo='YY') + >>> parser.parse_args('XX --foo'.split()) + Namespace(bar='XX', foo='c') + >>> parser.parse_args(''.split()) + Namespace(bar='d', foo='d') + + One of the more common uses of ``nargs='?'`` is to allow optional input and + output files:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('infile', nargs='?', type=argparse.FileType('r'), + ... default=sys.stdin) + >>> parser.add_argument('outfile', nargs='?', type=argparse.FileType('w'), + ... default=sys.stdout) + >>> parser.parse_args(['input.txt', 'output.txt']) + Namespace(infile=<_io.TextIOWrapper name='input.txt' encoding='UTF-8'>, + outfile=<_io.TextIOWrapper name='output.txt' encoding='UTF-8'>) + >>> parser.parse_args([]) + Namespace(infile=<_io.TextIOWrapper name='<stdin>' encoding='UTF-8'>, + outfile=<_io.TextIOWrapper name='<stdout>' encoding='UTF-8'>) + +* ``'*'``. All command-line args present are gathered into a list. Note that + it generally doesn't make much sense to have more than one positional argument + with ``nargs='*'``, but multiple optional arguments with ``nargs='*'`` is + possible. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', nargs='*') + >>> parser.add_argument('--bar', nargs='*') + >>> parser.add_argument('baz', nargs='*') + >>> parser.parse_args('a b --foo x y --bar 1 2'.split()) + Namespace(bar=['1', '2'], baz=['a', 'b'], foo=['x', 'y']) + +* ``'+'``. Just like ``'*'``, all command-line args present are gathered into a + list. Additionally, an error message will be generated if there wasn't at + least one command-line arg present. For example:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', nargs='+') + >>> parser.parse_args('a b'.split()) + Namespace(foo=['a', 'b']) + >>> parser.parse_args(''.split()) + usage: PROG [-h] foo [foo ...] + PROG: error: too few arguments + +If the ``nargs`` keyword argument is not provided, the number of args consumed +is determined by the action_. Generally this means a single command-line arg +will be consumed and a single item (not a list) will be produced. + + +const +^^^^^ + +The ``const`` argument of :meth:`~ArgumentParser.add_argument` is used to hold +constant values that are not read from the command line but are required for +the various :class:`ArgumentParser` actions. The two most common uses of it are: + +* When :meth:`~ArgumentParser.add_argument` is called with + ``action='store_const'`` or ``action='append_const'``. These actions add the + ``const`` value to one of the attributes of the object returned by :meth:`~ArgumentParser.parse_args`. See the action_ description for examples. + +* When :meth:`~ArgumentParser.add_argument` is called with option strings + (like ``-f`` or ``--foo``) and ``nargs='?'``. This creates an optional + argument that can be followed by zero or one command-line args. + When parsing the command line, if the option string is encountered with no + command-line arg following it, the value of ``const`` will be assumed instead. + See the nargs_ description for examples. + +The ``const`` keyword argument defaults to ``None``. + + +default +^^^^^^^ + +All optional arguments and some positional arguments may be omitted at the +command line. The ``default`` keyword argument of +:meth:`~ArgumentParser.add_argument`, whose value defaults to ``None``, +specifies what value should be used if the command-line arg is not present. +For optional arguments, the ``default`` value is used when the option string +was not present at the command line:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default=42) + >>> parser.parse_args('--foo 2'.split()) + Namespace(foo='2') + >>> parser.parse_args(''.split()) + Namespace(foo=42) + +For positional arguments with nargs_ ``='?'`` or ``'*'``, the ``default`` value +is used when no command-line arg was present:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('foo', nargs='?', default=42) + >>> parser.parse_args('a'.split()) + Namespace(foo='a') + >>> parser.parse_args(''.split()) + Namespace(foo=42) + + +Providing ``default=argparse.SUPPRESS`` causes no attribute to be added if the +command-line argument was not present.:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default=argparse.SUPPRESS) + >>> parser.parse_args([]) + Namespace() + >>> parser.parse_args(['--foo', '1']) + Namespace(foo='1') + + +type +^^^^ + +By default, :class:`ArgumentParser` objects read command-line args in as simple +strings. However, quite often the command-line string should instead be +interpreted as another type, like a :class:`float` or :class:`int`. The +``type`` keyword argument of :meth:`~ArgumentParser.add_argument` allows any +necessary type-checking and type-conversions to be performed. Common built-in +types and functions can be used directly as the value of the ``type`` argument:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('foo', type=int) + >>> parser.add_argument('bar', type=open) + >>> parser.parse_args('2 temp.txt'.split()) + Namespace(bar=<_io.TextIOWrapper name='temp.txt' encoding='UTF-8'>, foo=2) + +To ease the use of various types of files, the argparse module provides the +factory FileType which takes the ``mode=`` and ``bufsize=`` arguments of the +:func:`open` function. For example, ``FileType('w')`` can be used to create a +writable file:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('bar', type=argparse.FileType('w')) + >>> parser.parse_args(['out.txt']) + Namespace(bar=<_io.TextIOWrapper name='out.txt' encoding='UTF-8'>) + +``type=`` can take any callable that takes a single string argument and returns +the type-converted value:: + + >>> def perfect_square(string): + ... value = int(string) + ... sqrt = math.sqrt(value) + ... if sqrt != int(sqrt): + ... msg = "%r is not a perfect square" % string + ... raise argparse.ArgumentTypeError(msg) + ... return value + ... + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', type=perfect_square) + >>> parser.parse_args('9'.split()) + Namespace(foo=9) + >>> parser.parse_args('7'.split()) + usage: PROG [-h] foo + PROG: error: argument foo: '7' is not a perfect square + +The choices_ keyword argument may be more convenient for type checkers that +simply check against a range of values:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', type=int, choices=range(5, 10)) + >>> parser.parse_args('7'.split()) + Namespace(foo=7) + >>> parser.parse_args('11'.split()) + usage: PROG [-h] {5,6,7,8,9} + PROG: error: argument foo: invalid choice: 11 (choose from 5, 6, 7, 8, 9) + +See the choices_ section for more details. + + +choices +^^^^^^^ + +Some command-line args should be selected from a restricted set of values. +These can be handled by passing a container object as the ``choices`` keyword +argument to :meth:`~ArgumentParser.add_argument`. When the command line is +parsed, arg values will be checked, and an error message will be displayed if +the arg was not one of the acceptable values:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', choices='abc') + >>> parser.parse_args('c'.split()) + Namespace(foo='c') + >>> parser.parse_args('X'.split()) + usage: PROG [-h] {a,b,c} + PROG: error: argument foo: invalid choice: 'X' (choose from 'a', 'b', 'c') + +Note that inclusion in the ``choices`` container is checked after any type_ +conversions have been performed, so the type of the objects in the ``choices`` +container should match the type_ specified:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', type=complex, choices=[1, 1j]) + >>> parser.parse_args('1j'.split()) + Namespace(foo=1j) + >>> parser.parse_args('-- -4'.split()) + usage: PROG [-h] {1,1j} + PROG: error: argument foo: invalid choice: (-4+0j) (choose from 1, 1j) + +Any object that supports the ``in`` operator can be passed as the ``choices`` +value, so :class:`dict` objects, :class:`set` objects, custom containers, +etc. are all supported. + + +required +^^^^^^^^ + +In general, the :mod:`argparse` module assumes that flags like ``-f`` and ``--bar`` +indicate *optional* arguments, which can always be omitted at the command line. +To make an option *required*, ``True`` can be specified for the ``required=`` +keyword argument to :meth:`~ArgumentParser.add_argument`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', required=True) + >>> parser.parse_args(['--foo', 'BAR']) + Namespace(foo='BAR') + >>> parser.parse_args([]) + usage: argparse.py [-h] [--foo FOO] + argparse.py: error: option --foo is required + +As the example shows, if an option is marked as ``required``, +:meth:`~ArgumentParser.parse_args` will report an error if that option is not +present at the command line. + +.. note:: + + Required options are generally considered bad form because users expect + *options* to be *optional*, and thus they should be avoided when possible. + + +help +^^^^ + +The ``help`` value is a string containing a brief description of the argument. +When a user requests help (usually by using ``-h`` or ``--help`` at the +command line), these ``help`` descriptions will be displayed with each +argument:: + + >>> parser = argparse.ArgumentParser(prog='frobble') + >>> parser.add_argument('--foo', action='store_true', + ... help='foo the bars before frobbling') + >>> parser.add_argument('bar', nargs='+', + ... help='one of the bars to be frobbled') + >>> parser.parse_args('-h'.split()) + usage: frobble [-h] [--foo] bar [bar ...] + + positional arguments: + bar one of the bars to be frobbled + + optional arguments: + -h, --help show this help message and exit + --foo foo the bars before frobbling + +The ``help`` strings can include various format specifiers to avoid repetition +of things like the program name or the argument default_. The available +specifiers include the program name, ``%(prog)s`` and most keyword arguments to +:meth:`~ArgumentParser.add_argument`, e.g. ``%(default)s``, ``%(type)s``, etc.:: + + >>> parser = argparse.ArgumentParser(prog='frobble') + >>> parser.add_argument('bar', nargs='?', type=int, default=42, + ... help='the bar to %(prog)s (default: %(default)s)') + >>> parser.print_help() + usage: frobble [-h] [bar] + + positional arguments: + bar the bar to frobble (default: 42) + + optional arguments: + -h, --help show this help message and exit + + +metavar +^^^^^^^ + +When :class:`ArgumentParser` generates help messages, it need some way to refer +to each expected argument. By default, ArgumentParser objects use the dest_ +value as the "name" of each object. By default, for positional argument +actions, the dest_ value is used directly, and for optional argument actions, +the dest_ value is uppercased. So, a single positional argument with +``dest='bar'`` will that argument will be referred to as ``bar``. A single +optional argument ``--foo`` that should be followed by a single command-line arg +will be referred to as ``FOO``. An example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> parser.add_argument('bar') + >>> parser.parse_args('X --foo Y'.split()) + Namespace(bar='X', foo='Y') + >>> parser.print_help() + usage: [-h] [--foo FOO] bar + + positional arguments: + bar + + optional arguments: + -h, --help show this help message and exit + --foo FOO + +An alternative name can be specified with ``metavar``:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', metavar='YYY') + >>> parser.add_argument('bar', metavar='XXX') + >>> parser.parse_args('X --foo Y'.split()) + Namespace(bar='X', foo='Y') + >>> parser.print_help() + usage: [-h] [--foo YYY] XXX + + positional arguments: + XXX + + optional arguments: + -h, --help show this help message and exit + --foo YYY + +Note that ``metavar`` only changes the *displayed* name - the name of the +attribute on the :meth:`~ArgumentParser.parse_args` object is still determined +by the dest_ value. + +Different values of ``nargs`` may cause the metavar to be used multiple times. +Providing a tuple to ``metavar`` specifies a different display for each of the +arguments:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x', nargs=2) + >>> parser.add_argument('--foo', nargs=2, metavar=('bar', 'baz')) + >>> parser.print_help() + usage: PROG [-h] [-x X X] [--foo bar baz] + + optional arguments: + -h, --help show this help message and exit + -x X X + --foo bar baz + + +dest +^^^^ + +Most :class:`ArgumentParser` actions add some value as an attribute of the +object returned by :meth:`~ArgumentParser.parse_args`. The name of this +attribute is determined by the ``dest`` keyword argument of +:meth:`~ArgumentParser.add_argument`. For positional argument actions, +``dest`` is normally supplied as the first argument to +:meth:`~ArgumentParser.add_argument`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('bar') + >>> parser.parse_args('XXX'.split()) + Namespace(bar='XXX') + +For optional argument actions, the value of ``dest`` is normally inferred from +the option strings. :class:`ArgumentParser` generates the value of ``dest`` by +taking the first long option string and stripping away the initial ``'--'`` +string. If no long option strings were supplied, ``dest`` will be derived from +the first short option string by stripping the initial ``'-'`` character. Any +internal ``'-'`` characters will be converted to ``'_'`` characters to make sure +the string is a valid attribute name. The examples below illustrate this +behavior:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('-f', '--foo-bar', '--foo') + >>> parser.add_argument('-x', '-y') + >>> parser.parse_args('-f 1 -x 2'.split()) + Namespace(foo_bar='1', x='2') + >>> parser.parse_args('--foo 1 -y 2'.split()) + Namespace(foo_bar='1', x='2') + +``dest`` allows a custom attribute name to be provided:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', dest='bar') + >>> parser.parse_args('--foo XXX'.split()) + Namespace(bar='XXX') + + +The parse_args() method +----------------------- + +.. method:: ArgumentParser.parse_args(args=None, namespace=None) + + Convert argument strings to objects and assign them as attributes of the + namespace. Return the populated namespace. + + Previous calls to :meth:`add_argument` determine exactly what objects are + created and how they are assigned. See the documentation for + :meth:`add_argument` for details. + + By default, the arg strings are taken from :data:`sys.argv`, and a new empty + :class:`Namespace` object is created for the attributes. + + +Option value syntax +^^^^^^^^^^^^^^^^^^^ + +The :meth:`~ArgumentParser.parse_args` method supports several ways of +specifying the value of an option (if it takes one). In the simplest case, the +option and its value are passed as two separate arguments:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x') + >>> parser.add_argument('--foo') + >>> parser.parse_args('-x X'.split()) + Namespace(foo=None, x='X') + >>> parser.parse_args('--foo FOO'.split()) + Namespace(foo='FOO', x=None) + +For long options (options with names longer than a single character), the option +and value can also be passed as a single command-line argument, using ``=`` to +separate them:: + + >>> parser.parse_args('--foo=FOO'.split()) + Namespace(foo='FOO', x=None) + +For short options (options only one character long), the option and its value +can be concatenated:: + + >>> parser.parse_args('-xX'.split()) + Namespace(foo=None, x='X') + +Several short options can be joined together, using only a single ``-`` prefix, +as long as only the last option (or none of them) requires a value:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x', action='store_true') + >>> parser.add_argument('-y', action='store_true') + >>> parser.add_argument('-z') + >>> parser.parse_args('-xyzZ'.split()) + Namespace(x=True, y=True, z='Z') + + +Invalid arguments +^^^^^^^^^^^^^^^^^ + +While parsing the command line, :meth:`~ArgumentParser.parse_args` checks for a +variety of errors, including ambiguous options, invalid types, invalid options, +wrong number of positional arguments, etc. When it encounters such an error, +it exits and prints the error along with a usage message:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--foo', type=int) + >>> parser.add_argument('bar', nargs='?') + + >>> # invalid type + >>> parser.parse_args(['--foo', 'spam']) + usage: PROG [-h] [--foo FOO] [bar] + PROG: error: argument --foo: invalid int value: 'spam' + + >>> # invalid option + >>> parser.parse_args(['--bar']) + usage: PROG [-h] [--foo FOO] [bar] + PROG: error: no such option: --bar + + >>> # wrong number of arguments + >>> parser.parse_args(['spam', 'badger']) + usage: PROG [-h] [--foo FOO] [bar] + PROG: error: extra arguments found: badger + + +Arguments containing ``"-"`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The :meth:`~ArgumentParser.parse_args` method attempts to give errors whenever +the user has clearly made a mistake, but some situations are inherently +ambiguous. For example, the command-line arg ``'-1'`` could either be an +attempt to specify an option or an attempt to provide a positional argument. +The :meth:`~ArgumentParser.parse_args` method is cautious here: positional +arguments may only begin with ``'-'`` if they look like negative numbers and +there are no options in the parser that look like negative numbers:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x') + >>> parser.add_argument('foo', nargs='?') + + >>> # no negative number options, so -1 is a positional argument + >>> parser.parse_args(['-x', '-1']) + Namespace(foo=None, x='-1') + + >>> # no negative number options, so -1 and -5 are positional arguments + >>> parser.parse_args(['-x', '-1', '-5']) + Namespace(foo='-5', x='-1') + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-1', dest='one') + >>> parser.add_argument('foo', nargs='?') + + >>> # negative number options present, so -1 is an option + >>> parser.parse_args(['-1', 'X']) + Namespace(foo=None, one='X') + + >>> # negative number options present, so -2 is an option + >>> parser.parse_args(['-2']) + usage: PROG [-h] [-1 ONE] [foo] + PROG: error: no such option: -2 + + >>> # negative number options present, so both -1s are options + >>> parser.parse_args(['-1', '-1']) + usage: PROG [-h] [-1 ONE] [foo] + PROG: error: argument -1: expected one argument + +If you have positional arguments that must begin with ``'-'`` and don't look +like negative numbers, you can insert the pseudo-argument ``'--'`` which tells +:meth:`~ArgumentParser.parse_args` that everything after that is a positional +argument:: + + >>> parser.parse_args(['--', '-f']) + Namespace(foo='-f', one=None) + + +Argument abbreviations +^^^^^^^^^^^^^^^^^^^^^^ + +The :meth:`~ArgumentParser.parse_args` method allows long options to be +abbreviated if the abbreviation is unambiguous:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-bacon') + >>> parser.add_argument('-badger') + >>> parser.parse_args('-bac MMM'.split()) + Namespace(bacon='MMM', badger=None) + >>> parser.parse_args('-bad WOOD'.split()) + Namespace(bacon=None, badger='WOOD') + >>> parser.parse_args('-ba BA'.split()) + usage: PROG [-h] [-bacon BACON] [-badger BADGER] + PROG: error: ambiguous option: -ba could match -badger, -bacon + +An error is produced for arguments that could produce more than one options. + + +Beyond ``sys.argv`` +^^^^^^^^^^^^^^^^^^^ + +Sometimes it may be useful to have an ArgumentParser parse args other than those +of :data:`sys.argv`. This can be accomplished by passing a list of strings to +:meth:`~ArgumentParser.parse_args`. This is useful for testing at the +interactive prompt:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument( + ... 'integers', metavar='int', type=int, choices=range(10), + ... nargs='+', help='an integer in the range 0..9') + >>> parser.add_argument( + ... '--sum', dest='accumulate', action='store_const', const=sum, + ... default=max, help='sum the integers (default: find the max)') + >>> parser.parse_args(['1', '2', '3', '4']) + Namespace(accumulate=<built-in function max>, integers=[1, 2, 3, 4]) + >>> parser.parse_args('1 2 3 4 --sum'.split()) + Namespace(accumulate=<built-in function sum>, integers=[1, 2, 3, 4]) + + +The Namespace object +^^^^^^^^^^^^^^^^^^^^ + +By default, :meth:`~ArgumentParser.parse_args` will return a new object of type +:class:`Namespace` where the necessary attributes have been set. This class is +deliberately simple, just an :class:`object` subclass with a readable string +representation. If you prefer to have dict-like view of the attributes, you +can use the standard Python idiom via :func:`vars`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> args = parser.parse_args(['--foo', 'BAR']) + >>> vars(args) + {'foo': 'BAR'} + +It may also be useful to have an :class:`ArgumentParser` assign attributes to an +already existing object, rather than a new :class:`Namespace` object. This can +be achieved by specifying the ``namespace=`` keyword argument:: + + >>> class C: + ... pass + ... + >>> c = C() + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> parser.parse_args(args=['--foo', 'BAR'], namespace=c) + >>> c.foo + 'BAR' + + +Other utilities +--------------- + +Sub-commands +^^^^^^^^^^^^ + +.. method:: ArgumentParser.add_subparsers() + + Many programs split up their functionality into a number of sub-commands, + for example, the ``svn`` program can invoke sub-commands like ``svn + checkout``, ``svn update``, and ``svn commit``. Splitting up functionality + this way can be a particularly good idea when a program performs several + different functions which require different kinds of command-line arguments. + :class:`ArgumentParser` supports the creation of such sub-commands with the + :meth:`add_subparsers` method. The :meth:`add_subparsers` method is normally + called with no arguments and returns an special action object. This object + has a single method, :meth:`~ArgumentParser.add_parser`, which takes a + command name and any :class:`ArgumentParser` constructor arguments, and + returns an :class:`ArgumentParser` object that can be modified as usual. + + Some example usage:: + + >>> # create the top-level parser + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--foo', action='store_true', help='foo help') + >>> subparsers = parser.add_subparsers(help='sub-command help') + >>> + >>> # create the parser for the "a" command + >>> parser_a = subparsers.add_parser('a', help='a help') + >>> parser_a.add_argument('bar', type=int, help='bar help') + >>> + >>> # create the parser for the "b" command + >>> parser_b = subparsers.add_parser('b', help='b help') + >>> parser_b.add_argument('--baz', choices='XYZ', help='baz help') + >>> + >>> # parse some arg lists + >>> parser.parse_args(['a', '12']) + Namespace(bar=12, foo=False) + >>> parser.parse_args(['--foo', 'b', '--baz', 'Z']) + Namespace(baz='Z', foo=True) + + Note that the object returned by :meth:`parse_args` will only contain + attributes for the main parser and the subparser that was selected by the + command line (and not any other subparsers). So in the example above, when + the ``"a"`` command is specified, only the ``foo`` and ``bar`` attributes are + present, and when the ``"b"`` command is specified, only the ``foo`` and + ``baz`` attributes are present. + + Similarly, when a help message is requested from a subparser, only the help + for that particular parser will be printed. The help message will not + include parent parser or sibling parser messages. (A help message for each + subparser command, however, can be given by supplying the ``help=`` argument + to :meth:`add_parser` as above.) + + :: + + >>> parser.parse_args(['--help']) + usage: PROG [-h] [--foo] {a,b} ... + + positional arguments: + {a,b} sub-command help + a a help + b b help + + optional arguments: + -h, --help show this help message and exit + --foo foo help + + >>> parser.parse_args(['a', '--help']) + usage: PROG a [-h] bar + + positional arguments: + bar bar help + + optional arguments: + -h, --help show this help message and exit + + >>> parser.parse_args(['b', '--help']) + usage: PROG b [-h] [--baz {X,Y,Z}] + + optional arguments: + -h, --help show this help message and exit + --baz {X,Y,Z} baz help + + The :meth:`add_subparsers` method also supports ``title`` and ``description`` + keyword arguments. When either is present, the subparser's commands will + appear in their own group in the help output. For example:: + + >>> parser = argparse.ArgumentParser() + >>> subparsers = parser.add_subparsers(title='subcommands', + ... description='valid subcommands', + ... help='additional help') + >>> subparsers.add_parser('foo') + >>> subparsers.add_parser('bar') + >>> parser.parse_args(['-h']) + usage: [-h] {foo,bar} ... + + optional arguments: + -h, --help show this help message and exit + + subcommands: + valid subcommands + + {foo,bar} additional help + + Furthermore, ``add_parser`` supports an additional ``aliases`` argument, + which allows multiple strings to refer to the same subparser. This example, + like ``svn``, aliases ``co`` as a shorthand for ``checkout``:: + + >>> parser = argparse.ArgumentParser() + >>> subparsers = parser.add_subparsers() + >>> checkout = subparsers.add_parser('checkout', aliases=['co']) + >>> checkout.add_argument('foo') + >>> parser.parse_args(['co', 'bar']) + Namespace(foo='bar') + + One particularly effective way of handling sub-commands is to combine the use + of the :meth:`add_subparsers` method with calls to :meth:`set_defaults` so + that each subparser knows which Python function it should execute. For + example:: + + >>> # sub-command functions + >>> def foo(args): + ... print(args.x * args.y) + ... + >>> def bar(args): + ... print('((%s))' % args.z) + ... + >>> # create the top-level parser + >>> parser = argparse.ArgumentParser() + >>> subparsers = parser.add_subparsers() + >>> + >>> # create the parser for the "foo" command + >>> parser_foo = subparsers.add_parser('foo') + >>> parser_foo.add_argument('-x', type=int, default=1) + >>> parser_foo.add_argument('y', type=float) + >>> parser_foo.set_defaults(func=foo) + >>> + >>> # create the parser for the "bar" command + >>> parser_bar = subparsers.add_parser('bar') + >>> parser_bar.add_argument('z') + >>> parser_bar.set_defaults(func=bar) + >>> + >>> # parse the args and call whatever function was selected + >>> args = parser.parse_args('foo 1 -x 2'.split()) + >>> args.func(args) + 2.0 + >>> + >>> # parse the args and call whatever function was selected + >>> args = parser.parse_args('bar XYZYX'.split()) + >>> args.func(args) + ((XYZYX)) + + This way, you can let :meth:`parse_args` do the job of calling the + appropriate function after argument parsing is complete. Associating + functions with actions like this is typically the easiest way to handle the + different actions for each of your subparsers. However, if it is necessary + to check the name of the subparser that was invoked, the ``dest`` keyword + argument to the :meth:`add_subparsers` call will work:: + + >>> parser = argparse.ArgumentParser() + >>> subparsers = parser.add_subparsers(dest='subparser_name') + >>> subparser1 = subparsers.add_parser('1') + >>> subparser1.add_argument('-x') + >>> subparser2 = subparsers.add_parser('2') + >>> subparser2.add_argument('y') + >>> parser.parse_args(['2', 'frobble']) + Namespace(subparser_name='2', y='frobble') + + +FileType objects +^^^^^^^^^^^^^^^^ + +.. class:: FileType(mode='r', bufsize=None) + + The :class:`FileType` factory creates objects that can be passed to the type + argument of :meth:`ArgumentParser.add_argument`. Arguments that have + :class:`FileType` objects as their type will open command-line args as files + with the requested modes and buffer sizes: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--output', type=argparse.FileType('wb', 0)) + >>> parser.parse_args(['--output', 'out']) + Namespace(output=<_io.BufferedWriter name='out'>) + + FileType objects understand the pseudo-argument ``'-'`` and automatically + convert this into ``sys.stdin`` for readable :class:`FileType` objects and + ``sys.stdout`` for writable :class:`FileType` objects: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('infile', type=argparse.FileType('r')) + >>> parser.parse_args(['-']) + Namespace(infile=<_io.TextIOWrapper name='<stdin>' encoding='UTF-8'>) + + +Argument groups +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.add_argument_group(title=None, description=None) + + By default, :class:`ArgumentParser` groups command-line arguments into + "positional arguments" and "optional arguments" when displaying help + messages. When there is a better conceptual grouping of arguments than this + default one, appropriate groups can be created using the + :meth:`add_argument_group` method:: + + >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) + >>> group = parser.add_argument_group('group') + >>> group.add_argument('--foo', help='foo help') + >>> group.add_argument('bar', help='bar help') + >>> parser.print_help() + usage: PROG [--foo FOO] bar + + group: + bar bar help + --foo FOO foo help + + The :meth:`add_argument_group` method returns an argument group object which + has an :meth:`~ArgumentParser.add_argument` method just like a regular + :class:`ArgumentParser`. When an argument is added to the group, the parser + treats it just like a normal argument, but displays the argument in a + separate group for help messages. The :meth:`add_argument_group` method + accepts *title* and *description* arguments which can be used to + customize this display:: + + >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) + >>> group1 = parser.add_argument_group('group1', 'group1 description') + >>> group1.add_argument('foo', help='foo help') + >>> group2 = parser.add_argument_group('group2', 'group2 description') + >>> group2.add_argument('--bar', help='bar help') + >>> parser.print_help() + usage: PROG [--bar BAR] foo + + group1: + group1 description + + foo foo help + + group2: + group2 description + + --bar BAR bar help + + Note that any arguments not your user defined groups will end up back in the + usual "positional arguments" and "optional arguments" sections. + + +Mutual exclusion +^^^^^^^^^^^^^^^^ + +.. method:: add_mutually_exclusive_group(required=False) + + Create a mutually exclusive group. :mod:`argparse` will make sure that only + one of the arguments in the mutually exclusive group was present on the + command line:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> group = parser.add_mutually_exclusive_group() + >>> group.add_argument('--foo', action='store_true') + >>> group.add_argument('--bar', action='store_false') + >>> parser.parse_args(['--foo']) + Namespace(bar=True, foo=True) + >>> parser.parse_args(['--bar']) + Namespace(bar=False, foo=False) + >>> parser.parse_args(['--foo', '--bar']) + usage: PROG [-h] [--foo | --bar] + PROG: error: argument --bar: not allowed with argument --foo + + The :meth:`add_mutually_exclusive_group` method also accepts a *required* + argument, to indicate that at least one of the mutually exclusive arguments + is required:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> group = parser.add_mutually_exclusive_group(required=True) + >>> group.add_argument('--foo', action='store_true') + >>> group.add_argument('--bar', action='store_false') + >>> parser.parse_args([]) + usage: PROG [-h] (--foo | --bar) + PROG: error: one of the arguments --foo --bar is required + + Note that currently mutually exclusive argument groups do not support the + *title* and *description* arguments of + :meth:`~ArgumentParser.add_argument_group`. + + +Parser defaults +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.set_defaults(**kwargs) + + Most of the time, the attributes of the object returned by :meth:`parse_args` + will be fully determined by inspecting the command-line args and the argument + actions. :meth:`set_defaults` allows some additional + attributes that are determined without any inspection of the command line to + be added:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('foo', type=int) + >>> parser.set_defaults(bar=42, baz='badger') + >>> parser.parse_args(['736']) + Namespace(bar=42, baz='badger', foo=736) + + Note that parser-level defaults always override argument-level defaults:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default='bar') + >>> parser.set_defaults(foo='spam') + >>> parser.parse_args([]) + Namespace(foo='spam') + + Parser-level defaults can be particularly useful when working with multiple + parsers. See the :meth:`~ArgumentParser.add_subparsers` method for an + example of this type. + +.. method:: ArgumentParser.get_default(dest) + + Get the default value for a namespace attribute, as set by either + :meth:`~ArgumentParser.add_argument` or by + :meth:`~ArgumentParser.set_defaults`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default='badger') + >>> parser.get_default('foo') + 'badger' + + +Printing help +^^^^^^^^^^^^^ + +In most typical applications, :meth:`~ArgumentParser.parse_args` will take +care of formatting and printing any usage or error messages. However, several +formatting methods are available: + +.. method:: ArgumentParser.print_usage(file=None) + + Print a brief description of how the :class:`ArgumentParser` should be + invoked on the command line. If *file* is ``None``, :data:`sys.stdout` is + assumed. + +.. method:: ArgumentParser.print_help(file=None) + + Print a help message, including the program usage and information about the + arguments registered with the :class:`ArgumentParser`. If *file* is + ``None``, :data:`sys.stdout` is assumed. + +There are also variants of these methods that simply return a string instead of +printing it: + +.. method:: ArgumentParser.format_usage() + + Return a string containing a brief description of how the + :class:`ArgumentParser` should be invoked on the command line. + +.. method:: ArgumentParser.format_help() + + Return a string containing a help message, including the program usage and + information about the arguments registered with the :class:`ArgumentParser`. + + +Partial parsing +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.parse_known_args(args=None, namespace=None) + +Sometimes a script may only parse a few of the command-line arguments, passing +the remaining arguments on to another script or program. In these cases, the +:meth:`~ArgumentParser.parse_known_args` method can be useful. It works much like +:meth:`~ArgumentParser.parse_args` except that it does not produce an error when +extra arguments are present. Instead, it returns a two item tuple containing +the populated namespace and the list of remaining argument strings. + +:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='store_true') + >>> parser.add_argument('bar') + >>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam']) + (Namespace(bar='BAR', foo=True), ['--badger', 'spam']) + + +Customizing file parsing +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.convert_arg_line_to_args(arg_line) + + Arguments that are read from a file (see the *fromfile_prefix_chars* + keyword argument to the :class:`ArgumentParser` constructor) are read one + argument per line. :meth:`convert_arg_line_to_args` can be overriden for + fancier reading. + + This method takes a single argument *arg_line* which is a string read from + the argument file. It returns a list of arguments parsed from this string. + The method is called once per line read from the argument file, in order. + + A useful override of this method is one that treats each space-separated word + as an argument:: + + def convert_arg_line_to_args(self, arg_line): + for arg in arg_line.split(): + if not arg.strip(): + continue + yield arg + + +Exiting methods +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.exit(status=0, message=None) + + This method terminates the program, exiting with the specified *status* + and, if given, it prints a *message* before that. + +.. method:: ArgumentParser.error(message) + + This method prints a usage message including the *message* to the + standard output and terminates the program with a status code of 2. + +.. _upgrading-optparse-code: + +Upgrading optparse code +----------------------- + +Originally, the :mod:`argparse` module had attempted to maintain compatibility +with :mod:`optparse`. However, :mod:`optparse` was difficult to extend +transparently, particularly with the changes required to support the new +``nargs=`` specifiers and better usage messages. When most everything in +:mod:`optparse` had either been copy-pasted over or monkey-patched, it no +longer seemed practical to try to maintain the backwards compatibility. + +A partial upgrade path from :mod:`optparse` to :mod:`argparse`: + +* Replace all :meth:`optparse.OptionParser.add_option` calls with + :meth:`ArgumentParser.add_argument` calls. + +* Replace ``options, args = parser.parse_args()`` with ``args = + parser.parse_args()`` and add additional :meth:`ArgumentParser.add_argument` + calls for the positional arguments. + +* Replace callback actions and the ``callback_*`` keyword arguments with + ``type`` or ``action`` arguments. + +* Replace string names for ``type`` keyword arguments with the corresponding + type objects (e.g. int, float, complex, etc). + +* Replace :class:`optparse.Values` with :class:`Namespace` and + :exc:`optparse.OptionError` and :exc:`optparse.OptionValueError` with + :exc:`ArgumentError`. + +* Replace strings with implicit arguments such as ``%default`` or ``%prog`` with + the standard Python syntax to use dictionaries to format strings, that is, + ``%(default)s`` and ``%(prog)s``. + +* Replace the OptionParser constructor ``version`` argument with a call to + ``parser.add_argument('--version', action='version', version='<the version>')`` diff --git a/Doc/library/array.rst b/Doc/library/array.rst index 3ffc64d..d563cce 100644 --- a/Doc/library/array.rst +++ b/Doc/library/array.rst @@ -60,7 +60,7 @@ The module defines the following type: appropriate type. If given a list or string, the initializer is passed to the new array's - :meth:`fromlist`, :meth:`fromstring`, or :meth:`fromunicode` method (see below) + :meth:`fromlist`, :meth:`frombytes`, or :meth:`fromunicode` method (see below) to add initial items to the array. Otherwise, the iterable initializer is passed to the :meth:`extend` method. @@ -99,7 +99,7 @@ The following data items and methods are also supported: memory buffer in bytes can be computed as ``array.buffer_info()[1] * array.itemsize``. This is occasionally useful when working with low-level (and inherently unsafe) I/O interfaces that require memory addresses, such as certain - :cfunc:`ioctl` operations. The returned numbers are valid as long as the array + :c:func:`ioctl` operations. The returned numbers are valid as long as the array exists and no length-changing operations are applied to it. .. note:: @@ -132,6 +132,15 @@ The following data items and methods are also supported: must be the right type to be appended to the array. +.. method:: array.frombytes(s) + + Appends items from the string, interpreting the string as an array of machine + values (as if it had been read from a file using the :meth:`fromfile` method). + + .. versionadded:: 3.2 + :meth:`fromstring` is renamed to :meth:`frombytes` for clarity. + + .. method:: array.fromfile(f, n) Read *n* items (as machine values) from the :term:`file object` *f* and append @@ -147,17 +156,16 @@ The following data items and methods are also supported: a.append(x)`` except that if there is a type error, the array is unchanged. -.. method:: array.fromstring(s) +.. method:: array.fromstring() - Appends items from the string, interpreting the string as an array of machine - values (as if it had been read from a file using the :meth:`fromfile` method). + Deprecated alias for :meth:`frombytes`. .. method:: array.fromunicode(s) Extends this array with data from the given unicode string. The array must be a type ``'u'`` array; otherwise a :exc:`ValueError` is raised. Use - ``array.fromstring(unicodestring.encode(enc))`` to append Unicode data to an + ``array.frombytes(unicodestring.encode(enc))`` to append Unicode data to an array of some other type. @@ -190,6 +198,16 @@ The following data items and methods are also supported: Reverse the order of the items in the array. +.. method:: array.tobytes() + + Convert the array to an array of machine values and return the bytes + representation (the same sequence of bytes that would be written to a file by + the :meth:`tofile` method.) + + .. versionadded:: 3.2 + :meth:`tostring` is renamed to :meth:`tobytes` for clarity. + + .. method:: array.tofile(f) Write all items (as machine values) to the :term:`file object` *f*. @@ -202,15 +220,13 @@ The following data items and methods are also supported: .. method:: array.tostring() - Convert the array to an array of machine values and return the string - representation (the same sequence of bytes that would be written to a file by - the :meth:`tofile` method.) + Deprecated alias for :meth:`tobytes`. .. method:: array.tounicode() Convert the array to a unicode string. The array must be a type ``'u'`` array; - otherwise a :exc:`ValueError` is raised. Use ``array.tostring().decode(enc)`` to + otherwise a :exc:`ValueError` is raised. Use ``array.tobytes().decode(enc)`` to obtain a unicode string from an array of some other type. diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index 08ee714..e2c0b6d 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -7,6 +7,9 @@ .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> .. sectionauthor:: Georg Brandl <georg@python.org> +**Source code:** :source:`Lib/ast.py` + +-------------- The :mod:`ast` module helps Python applications to process trees of the Python abstract syntax grammar. The abstract syntax itself might change with each @@ -117,12 +120,15 @@ and classes for traversing abstract syntax trees: Safely evaluate an expression node or a string containing a Python expression. The string or node provided may only consist of the following - Python literal structures: strings, numbers, tuples, lists, dicts, booleans, - and ``None``. + Python literal structures: strings, bytes, numbers, tuples, lists, dicts, + sets, booleans, and ``None``. This can be used for safely evaluating strings containing Python expressions from untrusted sources without the need to parse the values oneself. + .. versionchanged:: 3.2 + Now allows bytes and set literals. + .. function:: get_docstring(node, clean=True) diff --git a/Doc/library/asynchat.rst b/Doc/library/asynchat.rst index 3b8eb12..75b3cda 100644 --- a/Doc/library/asynchat.rst +++ b/Doc/library/asynchat.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Sam Rushing <rushing@nightmare.com> .. sectionauthor:: Steve Holden <sholden@holdenweb.com> +**Source code:** :source:`Lib/asynchat.py` + +-------------- This module builds on the :mod:`asyncore` infrastructure, simplifying asynchronous clients and servers and making it easier to handle protocols @@ -31,7 +34,7 @@ connection requests. Like :class:`asyncore.dispatcher`, :class:`async_chat` defines a set of events that are generated by an analysis of socket conditions after a - :cfunc:`select` call. Once the polling loop has been started the + :c:func:`select` call. Once the polling loop has been started the :class:`async_chat` object's methods are called by the event-processing framework with no action on the part of the programmer. diff --git a/Doc/library/asyncore.rst b/Doc/library/asyncore.rst index b6fe2bb..5f95d41 100644 --- a/Doc/library/asyncore.rst +++ b/Doc/library/asyncore.rst @@ -9,6 +9,9 @@ .. sectionauthor:: Steve Holden <sholden@holdenweb.com> .. heavily adapted from original documentation by Sam Rushing +**Source code:** :source:`Lib/asyncore.py` + +-------------- This module provides the basic infrastructure for writing asynchronous socket service clients and servers. @@ -22,7 +25,7 @@ bound. If your program is processor bound, then pre-emptive scheduled threads are probably what you really need. Network servers are rarely processor bound, however. -If your operating system supports the :cfunc:`select` system call in its I/O +If your operating system supports the :c:func:`select` system call in its I/O library (and nearly all do), then you can use it to juggle multiple communication channels at once; doing other work while your I/O is taking place in the "background." Although this strategy can seem strange and @@ -86,14 +89,14 @@ any that have been added to the map during asynchronous service) is closed. | ``handle_close()`` | Implied by a read event with no data | | | available | +----------------------+----------------------------------------+ - | ``handle_accept()`` | Implied by a read event on a listening | + | ``handle_accepted()``| Implied by a read event on a listening | | | socket | +----------------------+----------------------------------------+ During asynchronous processing, each mapped channel's :meth:`readable` and :meth:`writable` methods are used to determine whether the channel's socket - should be added to the list of channels :cfunc:`select`\ ed or - :cfunc:`poll`\ ed for read and write events. + should be added to the list of channels :c:func:`select`\ ed or + :c:func:`poll`\ ed for read and write events. Thus, the set of channel events is larger than the basic socket events. The full set of methods that can be overridden in your subclass follows: @@ -144,7 +147,21 @@ any that have been added to the map during asynchronous service) is closed. Called on listening channels (passive openers) when a connection can be established with a new remote endpoint that has issued a :meth:`connect` - call for the local endpoint. + call for the local endpoint. Deprecated in version 3.2; use + :meth:`handle_accepted` instead. + + .. deprecated:: 3.2 + + + .. method:: handle_accepted(sock, addr) + + Called on listening channels (passive openers) when a connection has been + established with a new remote endpoint that has issued a :meth:`connect` + call for the local endpoint. *conn* is a *new* socket object usable to + send and receive data on the connection, and *address* is the address + bound to the socket on the other end of the connection. + + .. versionadded:: 3.2 .. method:: readable() @@ -201,7 +218,8 @@ any that have been added to the map during asynchronous service) is closed. .. method:: bind(address) Bind the socket to *address*. The socket must not already be bound. (The - format of *address* depends on the address family --- see above.) To mark + format of *address* depends on the address family --- refer to the + :mod:`socket` documentation for more information.) To mark the socket as re-usable (setting the :const:`SO_REUSEADDR` option), call the :class:`dispatcher` object's :meth:`set_reuse_addr` method. @@ -225,6 +243,7 @@ any that have been added to the map during asynchronous service) is closed. flushed). Sockets are automatically closed when they are garbage-collected. + .. class:: dispatcher_with_send() A :class:`dispatcher` subclass which adds simple buffered output capability, @@ -234,9 +253,9 @@ any that have been added to the map during asynchronous service) is closed. .. class:: file_dispatcher() A file_dispatcher takes a file descriptor or :term:`file object` along - with an optional map argument and wraps it for use with the :cfunc:`poll` - or :cfunc:`loop` functions. If provided a file object or anything with a - :cfunc:`fileno` method, that method will be called and passed to the + with an optional map argument and wraps it for use with the :c:func:`poll` + or :c:func:`loop` functions. If provided a file object or anything with a + :c:func:`fileno` method, that method will be called and passed to the :class:`file_wrapper` constructor. Availability: UNIX. .. class:: file_wrapper() @@ -283,15 +302,15 @@ implement its socket handling:: self.buffer = self.buffer[sent:] - client = HTTPClient('www.python.org', '/') - asyncore.loop() + client = HTTPClient('www.python.org', '/') + asyncore.loop() .. _asyncore-example-2: asyncore Example basic echo server ---------------------------------- -Here is abasic echo server that uses the :class:`dispatcher` class to accept +Here is a basic echo server that uses the :class:`dispatcher` class to accept connections and dispatches the incoming connections to a handler:: import asyncore @@ -301,7 +320,8 @@ connections and dispatches the incoming connections to a handler:: def handle_read(self): data = self.recv(8192) - self.send(data) + if data: + self.send(data) class EchoServer(asyncore.dispatcher): @@ -312,14 +332,9 @@ connections and dispatches the incoming connections to a handler:: self.bind((host, port)) self.listen(5) - def handle_accept(self): - pair = self.accept() - if pair is None: - return - else: - sock, addr = pair - print('Incoming connection from %s' % repr(addr)) - handler = EchoHandler(sock) + def handle_accepted(self, sock, addr): + print('Incoming connection from %s' % repr(addr)) + handler = EchoHandler(sock) server = EchoServer('localhost', 8080) asyncore.loop() diff --git a/Doc/library/atexit.rst b/Doc/library/atexit.rst index 104c730..cc1051b 100644 --- a/Doc/library/atexit.rst +++ b/Doc/library/atexit.rst @@ -98,3 +98,4 @@ Usage as a :term:`decorator`:: This obviously only works with functions that don't take arguments. + diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index c10a74a..2401ae7 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -37,7 +37,7 @@ The modern interface provides: The encoded byte string is returned. -.. function:: b64decode(s, altchars=None) +.. function:: b64decode(s, altchars=None, validate=False) Decode a Base64 encoded byte string. @@ -45,9 +45,13 @@ The modern interface provides: at least length 2 (additional characters are ignored) which specifies the alternative alphabet used instead of the ``+`` and ``/`` characters. - The decoded byte string is returned. A :exc:`TypeError` is raised if *s* were - incorrectly padded or if there are non-alphabet characters present in the - string. + The decoded string is returned. A `binascii.Error` is raised if *s* is + incorrectly padded. + + If *validate* is ``False`` (the default), non-base64-alphabet characters are + discarded prior to the padding check. If *validate* is ``True``, + non-base64-alphabet characters in the input result in a + :exc:`binascii.Error`. .. function:: standard_b64encode(s) diff --git a/Doc/library/bdb.rst b/Doc/library/bdb.rst index 4795ec9..0737602 100644 --- a/Doc/library/bdb.rst +++ b/Doc/library/bdb.rst @@ -4,6 +4,10 @@ .. module:: bdb :synopsis: Debugger framework. +**Source code:** :source:`Lib/bdb.py` + +-------------- + The :mod:`bdb` module handles basic debugger functions, like setting breakpoints or managing execution via the debugger. @@ -50,9 +54,10 @@ The :mod:`bdb` module also defines two classes: Mark the breakpoint as disabled. - .. method:: bpprint(out=None) + .. method:: bpformat() - Print all the information about the breakpoint: + Return a string with all the information about the breakpoint, nicely + formatted: * The breakpoint number. * If it is temporary or not. @@ -61,6 +66,13 @@ The :mod:`bdb` module also defines two classes: * If it must be ignored the next N times. * The breakpoint hit count. + .. versionadded:: 3.2 + + .. method:: bpprint(out=None) + + Print the output of :meth:`bpformat` to the file *out*, or if it is + ``None``, to standard output. + .. class:: Bdb(skip=None) @@ -267,6 +279,15 @@ The :mod:`bdb` module also defines two classes: Delete all existing breakpoints. + .. method:: get_bpbynumber(arg) + + Return a breakpoint specified by the given number. If *arg* is a string, + it will be converted to a number. If *arg* is a non-numeric string, if + the given breakpoint never existed or has been deleted, a + :exc:`ValueError` is raised. + + .. versionadded:: 3.2 + .. method:: get_break(filename, lineno) Check if there is a breakpoint for *lineno* of *filename*. diff --git a/Doc/library/binascii.rst b/Doc/library/binascii.rst index 2f7851a..2aa3702 100644 --- a/Doc/library/binascii.rst +++ b/Doc/library/binascii.rst @@ -18,6 +18,11 @@ use these functions directly but use wrapper modules like :mod:`uu`, low-level functions written in C for greater speed that are used by the higher-level modules. +.. note:: + + Encoding and decoding functions do not accept Unicode strings. Only bytestring + and bytearray objects can be processed. + The :mod:`binascii` module defines the following functions: @@ -54,6 +59,9 @@ The :mod:`binascii` module defines the following functions: data. More than one line may be passed at a time. If the optional argument *header* is present and true, underscores will be decoded as spaces. + .. versionchanged:: 3.2 + Accept only bytestring or bytearray objects as input. + .. function:: b2a_qp(data, quotetabs=False, istext=True, header=False) @@ -83,6 +91,9 @@ The :mod:`binascii` module defines the following functions: decompressed data, unless data input data ends in an orphaned repeat indicator, in which case the :exc:`Incomplete` exception is raised. + .. versionchanged:: 3.2 + Accept only bytestring or bytearray objects as input. + .. function:: rlecode_hqx(data) @@ -139,6 +150,9 @@ The :mod:`binascii` module defines the following functions: of hexadecimal digits (which can be upper or lower case), otherwise a :exc:`TypeError` is raised. + .. versionchanged:: 3.2 + Accept only bytestring or bytearray objects as input. + .. exception:: Error @@ -164,4 +178,3 @@ The :mod:`binascii` module defines the following functions: Module :mod:`quopri` Support for quoted-printable encoding used in MIME email messages. - diff --git a/Doc/library/bisect.rst b/Doc/library/bisect.rst index eb23159..13b0147 100644 --- a/Doc/library/bisect.rst +++ b/Doc/library/bisect.rst @@ -7,6 +7,10 @@ .. sectionauthor:: Raymond Hettinger <python at rcn.com> .. example based on the PyModules FAQ entry by Aaron Watters <arw@pythonpros.com> +**Source code:** :source:`Lib/bisect.py` + +-------------- + This module provides support for maintaining a list in sorted order without having to sort the list after each insertion. For long lists of items with expensive comparison operations, this can be an improvement over the more common diff --git a/Doc/library/builtins.rst b/Doc/library/builtins.rst index f0d2a60..c495728 100644 --- a/Doc/library/builtins.rst +++ b/Doc/library/builtins.rst @@ -7,7 +7,7 @@ This module provides direct access to all 'built-in' identifiers of Python; for example, ``builtins.open`` is the full name for the built-in function -:func:`open`. See chapter :ref:`builtin`. +:func:`open`. This module is not normally accessed explicitly by most applications, but can be useful in modules that provide objects with the same name as a built-in value, diff --git a/Doc/library/calendar.rst b/Doc/library/calendar.rst index c8dac49..f495271 100644 --- a/Doc/library/calendar.rst +++ b/Doc/library/calendar.rst @@ -6,6 +6,9 @@ of the Unix cal program. .. sectionauthor:: Drew Csillag <drew_csillag@geocities.com> +**Source code:** :source:`Lib/calendar.py` + +-------------- This module allows you to output calendars like the Unix :program:`cal` program, and provides additional useful functions related to the calendar. By default, @@ -16,7 +19,7 @@ are given as integers. For related functionality, see also the :mod:`datetime` and :mod:`time` modules. Most of these functions and classes rely on the :mod:`datetime` module which -uses an idealized calendar, the current Gregorian calendar indefinitely extended +uses an idealized calendar, the current Gregorian calendar extended in both directions. This matches the definition of the "proleptic Gregorian" calendar in Dershowitz and Reingold's book "Calendrical Calculations", where it's the base calendar for all computations. @@ -309,4 +312,3 @@ The :mod:`calendar` module exports the following data attributes: Module :mod:`time` Low-level time related functions. - diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst index 734031d..1e2498d 100644 --- a/Doc/library/cgi.rst +++ b/Doc/library/cgi.rst @@ -13,6 +13,10 @@ single: URL single: Common Gateway Interface +**Source code:** :source:`Lib/cgi.py` + +-------------- + Support module for Common Gateway Interface (CGI) scripts. This module defines a number of utilities for use by CGI scripts written in @@ -324,15 +328,13 @@ algorithms implemented in this module in other circumstances. Convert the characters ``'&'``, ``'<'`` and ``'>'`` in string *s* to HTML-safe sequences. Use this if you need to display text that might contain such characters in HTML. If the optional flag *quote* is true, the quotation mark - character (``'"'``) is also translated; this helps for inclusion in an HTML - attribute value, as in ``<A HREF="...">``. If the value to be quoted might - include single- or double-quote characters, or both, consider using the - :func:`~xml.sax.saxutils.quoteattr` function in the :mod:`xml.sax.saxutils` - module instead. + character (``"``) is also translated; this helps for inclusion in an HTML + attribute value delimited by double quotes, as in ``<a href="...">``. Note + that single quotes are never translated. - If the value to be quoted might include single- or double-quote characters, - or both, consider using the :func:`quoteattr` function in the - :mod:`xml.sax.saxutils` module instead. + .. deprecated:: 3.2 + This function is unsafe because *quote* is false by default, and therefore + deprecated. Use :func:`html.escape` instead. .. _cgi-security: @@ -510,8 +512,8 @@ Common problems and solutions .. rubric:: Footnotes -.. [#] Note that some recent versions of the HTML specification do state what order the - field values should be supplied in, but knowing whether a request was - received from a conforming browser, or even from a browser at all, is tedious - and error-prone. +.. [#] Note that some recent versions of the HTML specification do state what + order the field values should be supplied in, but knowing whether a request + was received from a conforming browser, or even from a browser at all, is + tedious and error-prone. diff --git a/Doc/library/cmath.rst b/Doc/library/cmath.rst index 14b909b..d7778df 100644 --- a/Doc/library/cmath.rst +++ b/Doc/library/cmath.rst @@ -187,15 +187,24 @@ Hyperbolic functions Classification functions ------------------------ +.. function:: isfinite(x) + + Return ``True`` if both the real and imaginary parts of *x* are finite, and + ``False`` otherwise. + + .. versionadded:: 3.2 + + .. function:: isinf(x) - Return *True* if the real or the imaginary part of x is positive - or negative infinity. + Return ``True`` if either the real or the imaginary part of *x* is an + infinity, and ``False`` otherwise. .. function:: isnan(x) - Return *True* if the real or imaginary part of x is not a number (NaN). + Return ``True`` if either the real or the imaginary part of *x* is a NaN, + and ``False`` otherwise. Constants diff --git a/Doc/library/cmd.rst b/Doc/library/cmd.rst index d789270..464764d 100644 --- a/Doc/library/cmd.rst +++ b/Doc/library/cmd.rst @@ -5,13 +5,15 @@ :synopsis: Build line-oriented command interpreters. .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> +**Source code:** :source:`Lib/cmd.py` + +-------------- The :class:`Cmd` class provides a simple framework for writing line-oriented command interpreters. These are often useful for test harnesses, administrative tools, and prototypes that will later be wrapped in a more sophisticated interface. - .. class:: Cmd(completekey='tab', stdin=None, stdout=None) A :class:`Cmd` instance or subclass instance is a line-oriented interpreter @@ -203,3 +205,161 @@ Instances of :class:`Cmd` subclasses have some public instance variables: :mod:`readline`, on systems that support it, the interpreter will automatically support :program:`Emacs`\ -like line editing and command-history keystrokes.) +Cmd Example +----------- + +.. sectionauthor:: Raymond Hettinger <python at rcn dot com> + +The :mod:`cmd` module is mainly useful for building custom shells that let a +user work with a program interactively. + +This section presents a simple example of how to build a shell around a few of +the commands in the :mod:`turtle` module. + +Basic turtle commands such as :meth:`~turtle.forward` are added to a +:class:`Cmd` subclass with method named :meth:`do_forward`. The argument is +converted to a number and dispatched to the turtle module. The docstring is +used in the help utility provided by the shell. + +The example also includes a basic record and playback facility implemented with +the :meth:`~Cmd.precmd` method which is responsible for converting the input to +lowercase and writing the commands to a file. The :meth:`do_playback` method +reads the file and adds the recorded commands to the :attr:`cmdqueue` for +immediate playback:: + + import cmd, sys + from turtle import * + + class TurtleShell(cmd.Cmd): + intro = 'Welcome to the turtle shell. Type help or ? to list commands.\n' + prompt = '(turtle) ' + file = None + + # ----- basic turtle commands ----- + def do_forward(self, arg): + 'Move the turtle forward by the specified distance: FORWARD 10' + forward(*parse(arg)) + def do_right(self, arg): + 'Turn turtle right by given number of degrees: RIGHT 20' + right(*parse(arg)) + def do_left(self, arg): + 'Turn turtle left by given number of degrees: LEFT 90' + right(*parse(arg)) + def do_goto(self, arg): + 'Move turtle to an absolute position with changing orientation. GOTO 100 200' + goto(*parse(arg)) + def do_home(self, arg): + 'Return turtle to the home postion: HOME' + home() + def do_circle(self, arg): + 'Draw circle with given radius an options extent and steps: CIRCLE 50' + circle(*parse(arg)) + def do_position(self, arg): + 'Print the current turle position: POSITION' + print('Current position is %d %d\n' % position()) + def do_heading(self, arg): + 'Print the current turle heading in degrees: HEADING' + print('Current heading is %d\n' % (heading(),)) + def do_color(self, arg): + 'Set the color: COLOR BLUE' + color(arg.lower()) + def do_undo(self, arg): + 'Undo (repeatedly) the last turtle action(s): UNDO' + def do_reset(self, arg): + 'Clear the screen and return turtle to center: RESET' + reset() + def do_bye(self, arg): + 'Stop recording, close the turtle window, and exit: BYE' + print('Thank you for using Turtle') + self.close() + bye() + sys.exit(0) + + # ----- record and playback ----- + def do_record(self, arg): + 'Save future commands to filename: RECORD rose.cmd' + self.file = open(arg, 'w') + def do_playback(self, arg): + 'Playback commands from a file: PLAYBACK rose.cmd' + self.close() + cmds = open(arg).read().splitlines() + self.cmdqueue.extend(cmds) + def precmd(self, line): + line = line.lower() + if self.file and 'playback' not in line: + print(line, file=self.file) + return line + def close(self): + if self.file: + self.file.close() + self.file = None + + def parse(arg): + 'Convert a series of zero or more numbers to an argument tuple' + return tuple(map(int, arg.split())) + + if __name__ == '__main__': + TurtleShell().cmdloop() + + +Here is a sample session with the turtle shell showing the help functions, using +blank lines to repeat commands, and the simple record and playback facility:: + + Welcome to the turtle shell. Type help or ? to list commands. + + (turtle) ? + + Documented commands (type help <topic>): + ======================================== + bye color goto home playback record right + circle forward heading left position reset undo + + (turtle) help forward + Move the turtle forward by the specified distance: FORWARD 10 + (turtle) record spiral.cmd + (turtle) position + Current position is 0 0 + + (turtle) heading + Current heading is 0 + + (turtle) reset + (turtle) circle 20 + (turtle) right 30 + (turtle) circle 40 + (turtle) right 30 + (turtle) circle 60 + (turtle) right 30 + (turtle) circle 80 + (turtle) right 30 + (turtle) circle 100 + (turtle) right 30 + (turtle) circle 120 + (turtle) right 30 + (turtle) circle 120 + (turtle) heading + Current heading is 180 + + (turtle) forward 100 + (turtle) + (turtle) right 90 + (turtle) forward 100 + (turtle) + (turtle) right 90 + (turtle) forward 400 + (turtle) right 90 + (turtle) forward 500 + (turtle) right 90 + (turtle) forward 400 + (turtle) right 90 + (turtle) forward 300 + (turtle) playback spiral.cmd + Current position is 0 0 + + Current heading is 0 + + Current heading is 180 + + (turtle) bye + Thank you for using Turtle + diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index d07dafa..922bcf4 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -787,9 +787,9 @@ Encodings and Unicode --------------------- Strings are stored internally as sequences of codepoints (to be precise -as :ctype:`Py_UNICODE` arrays). Depending on the way Python is compiled (either +as :c:type:`Py_UNICODE` arrays). Depending on the way Python is compiled (either via ``--without-wide-unicode`` or ``--with-wide-unicode``, with the -former being the default) :ctype:`Py_UNICODE` is either a 16-bit or 32-bit data +former being the default) :c:type:`Py_UNICODE` is either a 16-bit or 32-bit data type. Once a string object is used outside of CPU and memory, CPU endianness and how these arrays are stored as bytes become an issue. Transforming a string object into a sequence of bytes is called encoding and recreating the @@ -936,6 +936,8 @@ particular, the following variants typically exist: | cp500 | EBCDIC-CP-BE, EBCDIC-CP-CH, | Western Europe | | | IBM500 | | +-----------------+--------------------------------+--------------------------------+ +| cp720 | | Arabic | ++-----------------+--------------------------------+--------------------------------+ | cp737 | | Greek | +-----------------+--------------------------------+--------------------------------+ | cp775 | IBM775 | Baltic languages | @@ -951,6 +953,8 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | cp857 | 857, IBM857 | Turkish | +-----------------+--------------------------------+--------------------------------+ +| cp858 | 858, IBM858 | Western Europe | ++-----------------+--------------------------------+--------------------------------+ | cp860 | 860, IBM860 | Portuguese | +-----------------+--------------------------------+--------------------------------+ | cp861 | 861, CP-IS, IBM861 | Icelandic | @@ -1086,7 +1090,7 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | mac_latin2 | maclatin2, maccentraleurope | Central and Eastern Europe | +-----------------+--------------------------------+--------------------------------+ -| mac_roman | macroman | Western Europe | +| mac_roman | macroman, macintosh | Western Europe | +-----------------+--------------------------------+--------------------------------+ | mac_turkish | macturkish | Turkish | +-----------------+--------------------------------+--------------------------------+ @@ -1110,9 +1114,9 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | utf_16 | U16, utf16 | all languages | +-----------------+--------------------------------+--------------------------------+ -| utf_16_be | UTF-16BE | all languages (BMP only) | +| utf_16_be | UTF-16BE | all languages | +-----------------+--------------------------------+--------------------------------+ -| utf_16_le | UTF-16LE | all languages (BMP only) | +| utf_16_le | UTF-16LE | all languages | +-----------------+--------------------------------+--------------------------------+ | utf_7 | U7, unicode-1-1-utf-7 | all languages | +-----------------+--------------------------------+--------------------------------+ @@ -1161,6 +1165,44 @@ particular, the following variants typically exist: | | | operand | +--------------------+---------+---------------------------+ +The following codecs provide bytes-to-bytes mappings. + ++--------------------+---------------------------+---------------------------+ +| Codec | Aliases | Purpose | ++====================+===========================+===========================+ +| base64_codec | base64, base-64 | Convert operand to MIME | +| | | base64 | ++--------------------+---------------------------+---------------------------+ +| bz2_codec | bz2 | Compress the operand | +| | | using bz2 | ++--------------------+---------------------------+---------------------------+ +| hex_codec | hex | Convert operand to | +| | | hexadecimal | +| | | representation, with two | +| | | digits per byte | ++--------------------+---------------------------+---------------------------+ +| quopri_codec | quopri, quoted-printable, | Convert operand to MIME | +| | quotedprintable | quoted printable | ++--------------------+---------------------------+---------------------------+ +| uu_codec | uu | Convert the operand using | +| | | uuencode | ++--------------------+---------------------------+---------------------------+ +| zlib_codec | zip, zlib | Compress the operand | +| | | using gzip | ++--------------------+---------------------------+---------------------------+ + +The following codecs provide string-to-string mappings. + ++--------------------+---------------------------+---------------------------+ +| Codec | Aliases | Purpose | ++====================+===========================+===========================+ +| rot_13 | rot13 | Returns the Caesar-cypher | +| | | encryption of the operand | ++--------------------+---------------------------+---------------------------+ + +.. versionadded:: 3.2 + bytes-to-bytes and string-to-string codecs. + :mod:`encodings.idna` --- Internationalized Domain Names in Applications ------------------------------------------------------------------------ @@ -1227,6 +1269,23 @@ functions can be used directly if desired. Convert a label to Unicode, as specified in :rfc:`3490`. +:mod:`encodings.mbcs` --- Windows ANSI codepage +----------------------------------------------- + +.. module:: encodings.mbcs + :synopsis: Windows ANSI codepage + +Encode operand according to the ANSI codepage (CP_ACP). This codec only +supports ``'strict'`` and ``'replace'`` error handlers to encode, and +``'strict'`` and ``'ignore'`` error handlers to decode. + +Availability: Windows only. + +.. versionchanged:: 3.2 + Before 3.2, the *errors* argument was ignored; ``'replace'`` was always used + to encode, and ``'ignore'`` to decode. + + :mod:`encodings.utf_8_sig` --- UTF-8 codec with BOM signature ------------------------------------------------------------- diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 68a79f1..d29bc17 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -13,6 +13,10 @@ import itertools __name__ = '<doctest>' +**Source code:** :source:`Lib/collections.py` and :source:`Lib/_abcoll.py` + +-------------- + This module implements specialized container datatypes providing alternatives to Python's general purpose built-in containers, :class:`dict`, :class:`list`, :class:`set`, and :class:`tuple`. @@ -29,7 +33,7 @@ Python's general purpose built-in containers, :class:`dict`, :class:`list`, ===================== ==================================================================== In addition to the concrete container classes, the collections module provides -ABCs (abstract base classes) that can be used to test whether a class provides a +:ref:`abstract-base-classes` that can be used to test whether a class provides a particular interface, for example, whether it is hashable or a mapping. @@ -85,7 +89,7 @@ For example:: .. versionadded:: 3.1 - Counter objects support two methods beyond those available for all + Counter objects support three methods beyond those available for all dictionaries: .. method:: elements() @@ -108,6 +112,19 @@ For example:: >>> Counter('abracadabra').most_common(3) [('a', 5), ('r', 2), ('b', 2)] + .. method:: subtract([iterable-or-mapping]) + + Elements are subtracted from an *iterable* or from another *mapping* + (or counter). Like :meth:`dict.update` but subtracts counts instead + of replacing them. Both inputs and outputs may be zero or negative. + + >>> c = Counter(a=4, b=2, c=0, d=-2) + >>> d = Counter(a=1, b=2, c=3, d=4) + >>> c.subtract(d) + Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6}) + + .. versionadded:: 3.2 + The usual dictionary methods are available for :class:`Counter` objects except for two which work differently for counters. @@ -188,14 +205,14 @@ counts, but the output will exclude results with counts of zero or less. * `Bag class <http://www.gnu.org/software/smalltalk/manual-base/html_node/Bag.html>`_ in Smalltalk. - * Wikipedia entry for `Multisets <http://en.wikipedia.org/wiki/Multiset>`_\. + * Wikipedia entry for `Multisets <http://en.wikipedia.org/wiki/Multiset>`_. * `C++ multisets <http://www.demo2s.com/Tutorial/Cpp/0380__set-multiset/Catalog0380__set-multiset.htm>`_ tutorial with examples. * For mathematical operations on multisets and their use cases, see *Knuth, Donald. The Art of Computer Programming Volume II, - Section 4.6.3, Exercise 19*\. + Section 4.6.3, Exercise 19*. * To enumerate all distinct multisets of a given size over a given set of elements, see :func:`itertools.combinations_with_replacement`. @@ -248,6 +265,13 @@ counts, but the output will exclude results with counts of zero or less. Remove all elements from the deque leaving it with length 0. + .. method:: count(x) + + Count the number of deque elements equal to *x*. + + .. versionadded:: 3.2 + + .. method:: extend(iterable) Extend the right side of the deque by appending elements from the iterable @@ -279,6 +303,13 @@ counts, but the output will exclude results with counts of zero or less. :exc:`ValueError`. + .. method:: reverse() + + Reverse the elements of the deque in-place and then return ``None``. + + .. versionadded:: 3.2 + + .. method:: rotate(n) Rotate the deque *n* steps to the right. If *n* is negative, rotate to @@ -555,11 +586,15 @@ they add the ability to access fields by name instead of position index. .. versionchanged:: 3.1 Added support for *rename*. -Example: .. doctest:: :options: +NORMALIZE_WHITESPACE + >>> # Basic example + >>> Point = namedtuple('Point', ['x', 'y']) + >>> p = Point(x=10, y=11) + + >>> # Example using the verbose option to print the class definition >>> Point = namedtuple('Point', 'x y', verbose=True) class Point(tuple): 'Point(x, y)' @@ -569,6 +604,7 @@ Example: _fields = ('x', 'y') <BLANKLINE> def __new__(_cls, x, y): + 'Create a new instance of Point(x, y)' return _tuple.__new__(_cls, (x, y)) <BLANKLINE> @classmethod @@ -580,7 +616,8 @@ Example: return result <BLANKLINE> def __repr__(self): - return 'Point(x=%r, y=%r)' % self + 'Return a nicely formatted representation string' + return self.__class__.__name__ + '(x=%r, y=%r)' % self <BLANKLINE> def _asdict(self): 'Return a new OrderedDict which maps field names to their values' @@ -594,10 +631,11 @@ Example: return result <BLANKLINE> def __getnewargs__(self): + 'Return self as a plain tuple. Used by copy and pickle.' return tuple(self) <BLANKLINE> - x = _property(_itemgetter(0)) - y = _property(_itemgetter(1)) + x = _property(_itemgetter(0), doc='Alias for field number 0') + y = _property(_itemgetter(1), doc='Alias for field number 1') >>> p = Point(11, y=22) # instantiate with positional or keyword arguments >>> p[0] + p[1] # indexable like the plain tuple (11, 22) @@ -698,15 +736,15 @@ functionality with a subclass. Here is how to add a calculated field and a fixed-width print format: >>> class Point(namedtuple('Point', 'x y')): - ... __slots__ = () - ... @property - ... def hypot(self): - ... return (self.x ** 2 + self.y ** 2) ** 0.5 - ... def __str__(self): - ... return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot) + __slots__ = () + @property + def hypot(self): + return (self.x ** 2 + self.y ** 2) ** 0.5 + def __str__(self): + return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot) >>> for p in Point(3, 4), Point(14, 5/7): - ... print(p) + print(p) Point: x= 3.000 y= 4.000 hypot= 5.000 Point: x=14.000 y= 0.714 hypot=14.018 @@ -733,12 +771,19 @@ and more efficient to use a simple class declaration: >>> Status.open, Status.pending, Status.closed (0, 1, 2) >>> class Status: - ... open, pending, closed = range(3) + open, pending, closed = range(3) .. seealso:: - `Named tuple recipe <http://code.activestate.com/recipes/500261/>`_ - adapted for Python 2.4. + * `Named tuple recipe <http://code.activestate.com/recipes/500261/>`_ + adapted for Python 2.4. + + * `Recipe for named tuple abstract base class with a metaclass mix-in + <http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/>`_ + by Jan Kaliszewski. Besides providing an :term:`abstract base class` for + named tuples, it also supports an alternate :term:`metaclass`-based + constructor that is convenient for use cases where named tuples are being + subclassed. :class:`OrderedDict` objects @@ -764,6 +809,23 @@ the items are returned in the order their keys were first added. (key, value) pair. The pairs are returned in LIFO order if *last* is true or FIFO order if false. + .. method:: move_to_end(key, last=True) + + Move an existing *key* to either end of an ordered dictionary. The item + is moved to the right end if *last* is true (the default) or to the + beginning if *last* is false. Raises :exc:`KeyError` if the *key* does + not exist:: + + >>> d = OrderedDict.fromkeys('abcde') + >>> d.move_to_end('b') + >>> ''.join(d.keys) + 'acdeb' + >>> d.move_to_end('b', last=False) + >>> ''.join(d.keys) + 'bacde' + + .. versionadded:: 3.2 + In addition to the usual mapping methods, ordered dictionaries also support reverse iteration using :func:`reversed`. @@ -864,6 +926,7 @@ attribute. class. + :class:`UserList` objects ------------------------- @@ -923,6 +986,7 @@ attribute. subclass) or an arbitrary sequence which can be converted into a string using the built-in :func:`str` function. +.. _abstract-base-classes: ABCs - abstract base classes ---------------------------- @@ -939,7 +1003,7 @@ ABC Inherits from Abstract Methods Mixin :class:`Sized` ``__len__`` :class:`Callable` ``__call__`` -:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``. ``__iter__``, ``__reversed__``, +:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``, ``__iter__``, ``__reversed__``, :class:`Iterable`, ``index``, and ``count`` :class:`Container` @@ -1074,6 +1138,9 @@ Notes on using :class:`Set` and :class:`MutableSet` as a mixin: .. seealso:: + * Latest version of the :source:`Python source code for the collections + abstract base classes <Lib/_abcoll.py>` + * `OrderedSet recipe <http://code.activestate.com/recipes/576694/>`_ for an example built on :class:`MutableSet`. diff --git a/Doc/library/colorsys.rst b/Doc/library/colorsys.rst index 2cbc704..dbab706 100644 --- a/Doc/library/colorsys.rst +++ b/Doc/library/colorsys.rst @@ -5,6 +5,9 @@ :synopsis: Conversion functions between RGB and other color systems. .. sectionauthor:: David Ascher <da@python.net> +**Source code:** :source:`Lib/colorsys.py` + +-------------- The :mod:`colorsys` module defines bidirectional conversions of color values between colors expressed in the RGB (Red Green Blue) color space used in diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 8a93f9b..cb7a09c 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -52,11 +52,30 @@ compile Python sources. regex is used to search the full path to each file considered for compilation, and if the regex produces a match, the file is skipped. +.. cmdoption:: -i list + + Read the file ``list`` and add each line that it contains to the list of + files and directories to compile. If ``list`` is ``-``, read lines from + ``stdin``. + +.. cmdoption:: -b + + Write the byte-code files to their legacy locations and names, which may + overwrite byte-code files created by another version of Python. The default + is to write files to their :pep:`3147` locations and names, which allows + byte-code files from multiple versions of Python to coexist. + +.. versionchanged:: 3.2 + Added the ``-i``, ``-b`` and ``-h`` options. + +There is no command-line option to control the optimization level used by the +:func:`compile` function, because the Python interpreter itself already +provides the option: :program:`python -O -m compileall`. Public functions ---------------- -.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False) +.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) Recursively descend the directory tree named by *dir*, compiling all :file:`.py` files along the way. @@ -80,7 +99,49 @@ Public functions If *quiet* is true, nothing is printed to the standard output unless errors occur. -.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False) + If *legacy* is true, byte-code files are written to their legacy locations + and names, which may overwrite byte-code files created by another version of + Python. The default is to write files to their :pep:`3147` locations and + names, which allows byte-code files from multiple versions of Python to + coexist. + + *optimize* specifies the optimization level for the compiler. It is passed to + the built-in :func:`compile` function. + + .. versionchanged:: 3.2 + Added the *legacy* and *optimize* parameter. + + +.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) + + Compile the file with path *fullname*. + + If *ddir* is given, it is prepended to the path to the file being compiled + for use in compilation time tracebacks, and is also compiled in to the + byte-code file, where it will be used in tracebacks and other messages in + cases where the source file does not exist at the time the byte-code file is + executed. + + If *rx* is given, its search method is passed the full path name to the + file being compiled, and if it returns a true value, the file is not + compiled and ``True`` is returned. + + If *quiet* is true, nothing is printed to the standard output unless errors + occur. + + If *legacy* is true, byte-code files are written to their legacy locations + and names, which may overwrite byte-code files created by another version of + Python. The default is to write files to their :pep:`3147` locations and + names, which allows byte-code files from multiple versions of Python to + coexist. + + *optimize* specifies the optimization level for the compiler. It is passed to + the built-in :func:`compile` function. + + .. versionadded:: 3.2 + + +.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False, optimize=-1) Byte-compile all the :file:`.py` files found along ``sys.path``. If *skip_curdir* is true (the default), the current directory is not included @@ -88,6 +149,10 @@ Public functions function. Note that unlike the other compile functions, ``maxlevels`` defaults to ``0``. + .. versionchanged:: 3.2 + Added the *legacy* and *optimize* parameter. + + To force a recompile of all the :file:`.py` files in the :file:`Lib/` subdirectory and all its subdirectories:: diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst new file mode 100644 index 0000000..e5d13f3 --- /dev/null +++ b/Doc/library/concurrent.futures.rst @@ -0,0 +1,371 @@ +:mod:`concurrent.futures` --- Launching parallel tasks +====================================================== + +.. module:: concurrent.futures + :synopsis: Execute computations concurrently using threads or processes. + +**Source code:** :source:`Lib/concurrent/futures/thread.py` +and :source:`Lib/concurrent/futures/process.py` + +.. versionadded:: 3.2 + +-------------- + +The :mod:`concurrent.futures` module provides a high-level interface for +asynchronously executing callables. + +The asynchronous execution can be be performed with threads, using +:class:`ThreadPoolExecutor`, or separate processes, using +:class:`ProcessPoolExecutor`. Both implement the same interface, which is +defined by the abstract :class:`Executor` class. + + +Executor Objects +---------------- + +.. class:: Executor + + An abstract class that provides methods to execute calls asynchronously. It + should not be used directly, but through its concrete subclasses. + + .. method:: submit(fn, *args, **kwargs) + + Schedules the callable, *fn*, to be executed as ``fn(*args **kwargs)`` + and returns a :class:`Future` object representing the execution of the + callable. :: + + with ThreadPoolExecutor(max_workers=1) as executor: + future = executor.submit(pow, 323, 1235) + print(future.result()) + + .. method:: map(func, *iterables, timeout=None) + + Equivalent to ``map(func, *iterables)`` except *func* is executed + asynchronously and several calls to *func* may be made concurrently. The + returned iterator raises a :exc:`TimeoutError` if :meth:`__next__()` is + called and the result isn't available after *timeout* seconds from the + original call to :meth:`Executor.map`. *timeout* can be an int or a + float. If *timeout* is not specified or ``None``, there is no limit to + the wait time. If a call raises an exception, then that exception will + be raised when its value is retrieved from the iterator. + + .. method:: shutdown(wait=True) + + Signal the executor that it should free any resources that it is using + when the currently pending futures are done executing. Calls to + :meth:`Executor.submit` and :meth:`Executor.map` made after shutdown will + raise :exc:`RuntimeError`. + + If *wait* is ``True`` then this method will not return until all the + pending futures are done executing and the resources associated with the + executor have been freed. If *wait* is ``False`` then this method will + return immediately and the resources associated with the executor will be + freed when all pending futures are done executing. Regardless of the + value of *wait*, the entire Python program will not exit until all + pending futures are done executing. + + You can avoid having to call this method explicitly if you use the + :keyword:`with` statement, which will shutdown the :class:`Executor` + (waiting as if :meth:`Executor.shutdown` were called with *wait* set to + ``True``):: + + import shutil + with ThreadPoolExecutor(max_workers=4) as e: + e.submit(shutil.copy, 'src1.txt', 'dest1.txt') + e.submit(shutil.copy, 'src2.txt', 'dest2.txt') + e.submit(shutil.copy, 'src3.txt', 'dest3.txt') + e.submit(shutil.copy, 'src3.txt', 'dest4.txt') + + +ThreadPoolExecutor +------------------ + +:class:`ThreadPoolExecutor` is a :class:`Executor` subclass that uses a pool of +threads to execute calls asynchronously. + +Deadlocks can occur when the callable associated with a :class:`Future` waits on +the results of another :class:`Future`. For example:: + + import time + def wait_on_b(): + time.sleep(5) + print(b.result()) # b will never complete because it is waiting on a. + return 5 + + def wait_on_a(): + time.sleep(5) + print(a.result()) # a will never complete because it is waiting on b. + return 6 + + + executor = ThreadPoolExecutor(max_workers=2) + a = executor.submit(wait_on_b) + b = executor.submit(wait_on_a) + +And:: + + def wait_on_future(): + f = executor.submit(pow, 5, 2) + # This will never complete because there is only one worker thread and + # it is executing this function. + print(f.result()) + + executor = ThreadPoolExecutor(max_workers=1) + executor.submit(wait_on_future) + + +.. class:: ThreadPoolExecutor(max_workers) + + An :class:`Executor` subclass that uses a pool of at most *max_workers* + threads to execute calls asynchronously. + + +.. _threadpoolexecutor-example: + +ThreadPoolExecutor Example +~~~~~~~~~~~~~~~~~~~~~~~~~~ +:: + + import concurrent.futures + import urllib.request + + URLS = ['http://www.foxnews.com/', + 'http://www.cnn.com/', + 'http://europe.wsj.com/', + 'http://www.bbc.co.uk/', + 'http://some-made-up-domain.com/'] + + def load_url(url, timeout): + return urllib.request.urlopen(url, timeout=timeout).read() + + with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: + future_to_url = dict((executor.submit(load_url, url, 60), url) + for url in URLS) + + for future in concurrent.futures.as_completed(future_to_url): + url = future_to_url[future] + if future.exception() is not None: + print('%r generated an exception: %s' % (url, + future.exception())) + else: + print('%r page is %d bytes' % (url, len(future.result()))) + + +ProcessPoolExecutor +------------------- + +The :class:`ProcessPoolExecutor` class is an :class:`Executor` subclass that +uses a pool of processes to execute calls asynchronously. +:class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which +allows it to side-step the :term:`Global Interpreter Lock` but also means that +only picklable objects can be executed and returned. + +Calling :class:`Executor` or :class:`Future` methods from a callable submitted +to a :class:`ProcessPoolExecutor` will result in deadlock. + +.. class:: ProcessPoolExecutor(max_workers=None) + + An :class:`Executor` subclass that executes calls asynchronously using a pool + of at most *max_workers* processes. If *max_workers* is ``None`` or not + given, it will default to the number of processors on the machine. + + +.. _processpoolexecutor-example: + +ProcessPoolExecutor Example +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +:: + + import concurrent.futures + import math + + PRIMES = [ + 112272535095293, + 112582705942171, + 112272535095293, + 115280095190773, + 115797848077099, + 1099726899285419] + + def is_prime(n): + if n % 2 == 0: + return False + + sqrt_n = int(math.floor(math.sqrt(n))) + for i in range(3, sqrt_n + 1, 2): + if n % i == 0: + return False + return True + + def main(): + with concurrent.futures.ProcessPoolExecutor() as executor: + for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)): + print('%d is prime: %s' % (number, prime)) + + if __name__ == '__main__': + main() + + +Future Objects +-------------- + +The :class:`Future` class encapsulates the asynchronous execution of a callable. +:class:`Future` instances are created by :meth:`Executor.submit`. + +.. class:: Future + + Encapsulates the asynchronous execution of a callable. :class:`Future` + instances are created by :meth:`Executor.submit` and should not be created + directly except for testing. + + .. method:: cancel() + + Attempt to cancel the call. If the call is currently being executed and + cannot be cancelled then the method will return ``False``, otherwise the + call will be cancelled and the method will return ``True``. + + .. method:: cancelled() + + Return ``True`` if the call was successfully cancelled. + + .. method:: running() + + Return ``True`` if the call is currently being executed and cannot be + cancelled. + + .. method:: done() + + Return ``True`` if the call was successfully cancelled or finished + running. + + .. method:: result(timeout=None) + + Return the value returned by the call. If the call hasn't yet completed + then this method will wait up to *timeout* seconds. If the call hasn't + completed in *timeout* seconds, then a :exc:`TimeoutError` will be + raised. *timeout* can be an int or float. If *timeout* is not specified + or ``None``, there is no limit to the wait time. + + If the future is cancelled before completing then :exc:`CancelledError` + will be raised. + + If the call raised, this method will raise the same exception. + + .. method:: exception(timeout=None) + + Return the exception raised by the call. If the call hasn't yet + completed then this method will wait up to *timeout* seconds. If the + call hasn't completed in *timeout* seconds, then a :exc:`TimeoutError` + will be raised. *timeout* can be an int or float. If *timeout* is not + specified or ``None``, there is no limit to the wait time. + + If the future is cancelled before completing then :exc:`CancelledError` + will be raised. + + If the call completed without raising, ``None`` is returned. + + .. method:: add_done_callback(fn) + + Attaches the callable *fn* to the future. *fn* will be called, with the + future as its only argument, when the future is cancelled or finishes + running. + + Added callables are called in the order that they were added and are + always called in a thread belonging to the process that added them. If + the callable raises a :exc:`Exception` subclass, it will be logged and + ignored. If the callable raises a :exc:`BaseException` subclass, the + behavior is undefined. + + If the future has already completed or been cancelled, *fn* will be + called immediately. + + The following :class:`Future` methods are meant for use in unit tests and + :class:`Executor` implementations. + + .. method:: set_running_or_notify_cancel() + + This method should only be called by :class:`Executor` implementations + before executing the work associated with the :class:`Future` and by unit + tests. + + If the method returns ``False`` then the :class:`Future` was cancelled, + i.e. :meth:`Future.cancel` was called and returned `True`. Any threads + waiting on the :class:`Future` completing (i.e. through + :func:`as_completed` or :func:`wait`) will be woken up. + + If the method returns ``True`` then the :class:`Future` was not cancelled + and has been put in the running state, i.e. calls to + :meth:`Future.running` will return `True`. + + This method can only be called once and cannot be called after + :meth:`Future.set_result` or :meth:`Future.set_exception` have been + called. + + .. method:: set_result(result) + + Sets the result of the work associated with the :class:`Future` to + *result*. + + This method should only be used by :class:`Executor` implementations and + unit tests. + + .. method:: set_exception(exception) + + Sets the result of the work associated with the :class:`Future` to the + :class:`Exception` *exception*. + + This method should only be used by :class:`Executor` implementations and + unit tests. + + +Module Functions +---------------- + +.. function:: wait(fs, timeout=None, return_when=ALL_COMPLETED) + + Wait for the :class:`Future` instances (possibly created by different + :class:`Executor` instances) given by *fs* to complete. Returns a named + 2-tuple of sets. The first set, named ``done``, contains the futures that + completed (finished or were cancelled) before the wait completed. The second + set, named ``not_done``, contains uncompleted futures. + + *timeout* can be used to control the maximum number of seconds to wait before + returning. *timeout* can be an int or float. If *timeout* is not specified + or ``None``, there is no limit to the wait time. + + *return_when* indicates when this function should return. It must be one of + the following constants: + + +-----------------------------+----------------------------------------+ + | Constant | Description | + +=============================+========================================+ + | :const:`FIRST_COMPLETED` | The function will return when any | + | | future finishes or is cancelled. | + +-----------------------------+----------------------------------------+ + | :const:`FIRST_EXCEPTION` | The function will return when any | + | | future finishes by raising an | + | | exception. If no future raises an | + | | exception then it is equivalent to | + | | :const:`ALL_COMPLETED`. | + +-----------------------------+----------------------------------------+ + | :const:`ALL_COMPLETED` | The function will return when all | + | | futures finish or are cancelled. | + +-----------------------------+----------------------------------------+ + +.. function:: as_completed(fs, timeout=None) + + Returns an iterator over the :class:`Future` instances (possibly created by + different :class:`Executor` instances) given by *fs* that yields futures as + they complete (finished or were cancelled). Any futures that completed + before :func:`as_completed` is called will be yielded first. The returned + iterator raises a :exc:`TimeoutError` if :meth:`__next__` is called and the + result isn't available after *timeout* seconds from the original call to + :func:`as_completed`. *timeout* can be an int or float. If *timeout* is not + specified or ``None``, there is no limit to the wait time. + + +.. seealso:: + + :pep:`3148` -- futures - execute computations asynchronously + The proposal which described this feature for inclusion in the Python + standard library. diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst index 2cef232..1a88bbd 100644 --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -7,7 +7,9 @@ .. moduleauthor:: Ken Manheimer <klm@zope.com> .. moduleauthor:: Barry Warsaw <bwarsaw@python.org> .. moduleauthor:: Eric S. Raymond <esr@thyrsus.com> +.. moduleauthor:: Łukasz Langa <lukasz@langa.pl> .. sectionauthor:: Christopher G. Petrilli <petrilli@amber.org> +.. sectionauthor:: Łukasz Langa <lukasz@langa.pl> .. index:: pair: .ini; file @@ -15,11 +17,10 @@ single: ini file single: Windows ini file -This module defines the class :class:`ConfigParser`. The :class:`ConfigParser` -class implements a basic configuration file parser language which provides a -structure similar to what you would find on Microsoft Windows INI files. You -can use this to write Python programs which can be customized by end users -easily. +This module provides the :class:`ConfigParser` class which implements a basic +configuration language which provides a structure similar to what's found in +Microsoft Windows INI files. You can use this to write Python programs which +can be customized by end users easily. .. note:: @@ -36,444 +37,1234 @@ easily. The json module implements a subset of JavaScript syntax which can also be used for this purpose. -The configuration file consists of sections, led by a ``[section]`` header and -followed by ``name: value`` entries, with continuations in the style of -:rfc:`822` (see section 3.1.1, "LONG HEADER FIELDS"); ``name=value`` is also -accepted. Note that leading whitespace is removed from values. The optional -values can contain format strings which refer to other values in the same -section, or values in a special ``DEFAULT`` section. Additional defaults can be -provided on initialization and retrieval. Lines beginning with ``'#'`` or -``';'`` are ignored and may be used to provide comments. -Configuration files may include comments, prefixed by specific characters (``#`` -and ``;``). Comments may appear on their own in an otherwise empty line, or may -be entered in lines holding values or section names. In the latter case, they -need to be preceded by a whitespace character to be recognized as a comment. -(For backwards compatibility, only ``;`` starts an inline comment, while ``#`` -does not.) +Quick Start +----------- + +Let's take a very basic configuration file that looks like this: + +.. code-block:: ini + + [DEFAULT] + ServerAliveInterval = 45 + Compression = yes + CompressionLevel = 9 + ForwardX11 = yes + + [bitbucket.org] + User = hg + + [topsecret.server.com] + Port = 50022 + ForwardX11 = no + +The structure of INI files is described `in the following section +<#supported-ini-file-structure>`_. Essentially, the file +consists of sections, each of which contains keys with values. +:mod:`configparser` classes can read and write such files. Let's start by +creating the above configuration file programatically. + +.. doctest:: + + >>> import configparser + >>> config = configparser.ConfigParser() + >>> config['DEFAULT'] = {'ServerAliveInterval': '45', + ... 'Compression': 'yes', + ... 'CompressionLevel': '9'} + >>> config['bitbucket.org'] = {} + >>> config['bitbucket.org']['User'] = 'hg' + >>> config['topsecret.server.com'] = {} + >>> topsecret = config['topsecret.server.com'] + >>> topsecret['Port'] = '50022' # mutates the parser + >>> topsecret['ForwardX11'] = 'no' # same here + >>> config['DEFAULT']['ForwardX11'] = 'yes' + >>> with open('example.ini', 'w') as configfile: + ... config.write(configfile) + ... + +As you can see, we can treat a config parser much like a dictionary. +There are differences, `outlined later <#mapping-protocol-access>`_, but +the behavior is very close to what you would expect from a dictionary. + +Now that we have created and saved a configuration file, let's read it +back and explore the data it holds. + +.. doctest:: + + >>> import configparser + >>> config = configparser.ConfigParser() + >>> config.sections() + [] + >>> config.read('example.ini') + ['example.ini'] + >>> config.sections() + ['bitbucket.org', 'topsecret.server.com'] + >>> 'bitbucket.org' in config + True + >>> 'bytebong.com' in config + False + >>> config['bitbucket.org']['User'] + 'hg' + >>> config['DEFAULT']['Compression'] + 'yes' + >>> topsecret = config['topsecret.server.com'] + >>> topsecret['ForwardX11'] + 'no' + >>> topsecret['Port'] + '50022' + >>> for key in config['bitbucket.org']: print(key) + ... + user + compressionlevel + serveraliveinterval + compression + forwardx11 + >>> config['bitbucket.org']['ForwardX11'] + 'yes' + +As we can see above, the API is pretty straightforward. The only bit of magic +involves the ``DEFAULT`` section which provides default values for all other +sections [1]_. Note also that keys in sections are +case-insensitive and stored in lowercase [1]_. + + +Supported Datatypes +------------------- + +Config parsers do not guess datatypes of values in configuration files, always +storing them internally as strings. This means that if you need other +datatypes, you should convert on your own: + +.. doctest:: + + >>> int(topsecret['Port']) + 50022 + >>> float(topsecret['CompressionLevel']) + 9.0 + +Extracting Boolean values is not that simple, though. Passing the value +to ``bool()`` would do no good since ``bool('False')`` is still +``True``. This is why config parsers also provide :meth:`getboolean`. +This method is case-insensitive and recognizes Boolean values from +``'yes'``/``'no'``, ``'on'``/``'off'`` and ``'1'``/``'0'`` [1]_. +For example: + +.. doctest:: + + >>> topsecret.getboolean('ForwardX11') + False + >>> config['bitbucket.org'].getboolean('ForwardX11') + True + >>> config.getboolean('bitbucket.org', 'Compression') + True + +Apart from :meth:`getboolean`, config parsers also provide equivalent +:meth:`getint` and :meth:`getfloat` methods, but these are far less +useful since conversion using :func:`int` and :func:`float` is +sufficient for these types. + + +Fallback Values +--------------- + +As with a dictionary, you can use a section's :meth:`get` method to +provide fallback values: + +.. doctest:: + + >>> topsecret.get('Port') + '50022' + >>> topsecret.get('CompressionLevel') + '9' + >>> topsecret.get('Cipher') + >>> topsecret.get('Cipher', '3des-cbc') + '3des-cbc' + +Please note that default values have precedence over fallback values. +For instance, in our example the ``'CompressionLevel'`` key was +specified only in the ``'DEFAULT'`` section. If we try to get it from +the section ``'topsecret.server.com'``, we will always get the default, +even if we specify a fallback: + +.. doctest:: + + >>> topsecret.get('CompressionLevel', '3') + '9' + +One more thing to be aware of is that the parser-level :meth:`get` method +provides a custom, more complex interface, maintained for backwards +compatibility. When using this method, a fallback value can be provided via +the ``fallback`` keyword-only argument: + +.. doctest:: + + >>> config.get('bitbucket.org', 'monster', + ... fallback='No such things as monsters') + 'No such things as monsters' + +The same ``fallback`` argument can be used with the :meth:`getint`, +:meth:`getfloat` and :meth:`getboolean` methods, for example: + +.. doctest:: + + >>> 'BatchMode' in topsecret + False + >>> topsecret.getboolean('BatchMode', fallback=True) + True + >>> config['DEFAULT']['BatchMode'] = 'no' + >>> topsecret.getboolean('BatchMode', fallback=True) + False + + +Supported INI File Structure +---------------------------- + +A configuration file consists of sections, each led by a ``[section]`` header, +followed by key/value entries separated by a specific string (``=`` or ``:`` by +default [1]_). By default, section names are case sensitive but keys are not +[1]_. Leading and trailing whitespace is removed from keys and values. +Values can be omitted, in which case the key/value delimiter may also be left +out. Values can also span multiple lines, as long as they are indented deeper +than the first line of the value. Depending on the parser's mode, blank lines +may be treated as parts of multiline values or ignored. + +Configuration files may include comments, prefixed by specific +characters (``#`` and ``;`` by default [1]_). Comments may appear on +their own on an otherwise empty line, possibly indented. [1]_ + +For example: + +.. code-block:: ini + + [Simple Values] + key=value + spaces in keys=allowed + spaces in values=allowed as well + spaces around the delimiter = obviously + you can also use : to delimit keys from values + + [All Values Are Strings] + values like this: 1000000 + or this: 3.14159265359 + are they treated as numbers? : no + integers, floats and booleans are held as: strings + can use the API to get converted values directly: true + + [Multiline Values] + chorus: I'm a lumberjack, and I'm okay + I sleep all night and I work all day + + [No Values] + key_without_value + empty string value here = -On top of the core functionality, :class:`SafeConfigParser` supports -interpolation. This means values can contain format strings which refer to -other values in the same section, or values in a special ``DEFAULT`` section. -Additional defaults can be provided on initialization. + [You can use comments] + # like this + ; or this -For example:: + # By default only in an empty line. + # Inline comments can be harmful because they prevent users + # from using the delimiting characters as parts of values. + # That being said, this can be customized. + + [Sections Can Be Indented] + can_values_be_as_well = True + does_that_mean_anything_special = False + purpose = formatting for readability + multiline_values = are + handled just fine as + long as they are indented + deeper than the first line + of a value + # Did I mention we can indent comments, too? + + +Interpolation of values +----------------------- - [My Section] - foodir: %(dir)s/whatever - dir=frob - long: this value continues - in the next line +On top of the core functionality, :class:`ConfigParser` supports +interpolation. This means values can be preprocessed before returning them +from ``get()`` calls. -would resolve the ``%(dir)s`` to the value of ``dir`` (``frob`` in this case). -All reference expansions are done on demand. +.. class:: BasicInterpolation() -Default values can be specified by passing them into the :class:`ConfigParser` -constructor as a dictionary. Additional defaults may be passed into the -:meth:`get` method which will override all others. + The default implementation used by :class:`ConfigParser`. It enables + values to contain format strings which refer to other values in the same + section, or values in the special default section [1]_. Additional default + values can be provided on initialization. -Sections are normally stored in a built-in dictionary. An alternative dictionary -type can be passed to the :class:`ConfigParser` constructor. For example, if a -dictionary type is passed that sorts its keys, the sections will be sorted on -write-back, as will be the keys within each section. + For example: + .. code-block:: ini -.. class:: RawConfigParser(defaults=None, dict_type=collections.OrderedDict) + [Paths] + home_dir: /Users + my_dir: %(home_dir)s/lumberjack + my_pictures: %(my_dir)s/Pictures - The basic configuration object. When *defaults* is given, it is initialized - into the dictionary of intrinsic defaults. When *dict_type* is given, it will - be used to create the dictionary objects for the list of sections, for the - options within a section, and for the default values. This class does not - support the magical interpolation behavior. - .. versionchanged:: 3.1 - The default *dict_type* is :class:`collections.OrderedDict`. + In the example above, :class:`ConfigParser` with *interpolation* set to + ``BasicInterpolation()`` would resolve ``%(home_dir)s`` to the value of + ``home_dir`` (``/Users`` in this case). ``%(my_dir)s`` in effect would + resolve to ``/Users/lumberjack``. All interpolations are done on demand so + keys used in the chain of references do not have to be specified in any + specific order in the configuration file. + With ``interpolation`` set to ``None``, the parser would simply return + ``%(my_dir)s/Pictures`` as the value of ``my_pictures`` and + ``%(home_dir)s/lumberjack`` as the value of ``my_dir``. -.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict) +.. class:: ExtendedInterpolation() - Derived class of :class:`RawConfigParser` that implements the magical - interpolation feature and adds optional arguments to the :meth:`get` and - :meth:`items` methods. The values in *defaults* must be appropriate for the - ``%()s`` string interpolation. Note that *__name__* is an intrinsic default; - its value is the section name, and will override any value provided in - *defaults*. + An alternative handler for interpolation which implements a more advanced + syntax, used for instance in ``zc.buildout``. Extended interpolation is + using ``${section:option}`` to denote a value from a foreign section. + Interpolation can span multiple levels. For convenience, if the ``section:`` + part is omitted, interpolation defaults to the current section (and possibly + the default values from the special section). - All option names used in interpolation will be passed through the - :meth:`optionxform` method just like any other option name reference. For - example, using the default implementation of :meth:`optionxform` (which converts - option names to lower case), the values ``foo %(bar)s`` and ``foo %(BAR)s`` are - equivalent. + For example, the configuration specified above with basic interpolation, + would look like this with extended interpolation: - .. versionchanged:: 3.1 - The default *dict_type* is :class:`collections.OrderedDict`. + .. code-block:: ini + + [Paths] + home_dir: /Users + my_dir: ${home_dir}/lumberjack + my_pictures: ${my_dir}/Pictures + + Values from other sections can be fetched as well: + + .. code-block:: ini + + [Common] + home_dir: /Users + library_dir: /Library + system_dir: /System + macports_dir: /opt/local + [Frameworks] + Python: 3.2 + path: ${Common:system_dir}/Library/Frameworks/ -.. class:: SafeConfigParser(defaults=None, dict_type=collections.OrderedDict) + [Arthur] + nickname: Two Sheds + last_name: Jackson + my_dir: ${Common:home_dir}/twosheds + my_pictures: ${my_dir}/Pictures + python_dir: ${Frameworks:path}/Python/Versions/${Frameworks:Python} - Derived class of :class:`ConfigParser` that implements a more-sane variant of - the magical interpolation feature. This implementation is more predictable as - well. New applications should prefer this version if they don't need to be - compatible with older versions of Python. +Mapping Protocol Access +----------------------- + +.. versionadded:: 3.2 + +Mapping protocol access is a generic name for functionality that enables using +custom objects as if they were dictionaries. In case of :mod:`configparser`, +the mapping interface implementation is using the +``parser['section']['option']`` notation. + +``parser['section']`` in particular returns a proxy for the section's data in +the parser. This means that the values are not copied but they are taken from +the original parser on demand. What's even more important is that when values +are changed on a section proxy, they are actually mutated in the original +parser. + +:mod:`configparser` objects behave as close to actual dictionaries as possible. +The mapping interface is complete and adheres to the ``MutableMapping`` ABC. +However, there are a few differences that should be taken into account: + +* By default, all keys in sections are accessible in a case-insensitive manner + [1]_. E.g. ``for option in parser["section"]`` yields only ``optionxform``'ed + option key names. This means lowercased keys by default. At the same time, + for a section that holds the key ``'a'``, both expressions return ``True``:: + + "a" in parser["section"] + "A" in parser["section"] + +* All sections include ``DEFAULTSECT`` values as well which means that + ``.clear()`` on a section may not leave the section visibly empty. This is + because default values cannot be deleted from the section (because technically + they are not there). If they are overriden in the section, deleting causes + the default value to be visible again. Trying to delete a default value + causes a ``KeyError``. + +* Trying to delete the ``DEFAULTSECT`` raises ``ValueError``. + +* ``parser.get(section, option, **kwargs)`` - the second argument is **not** + a fallback value. Note however that the section-level ``get()`` methods are + compatible both with the mapping protocol and the classic configparser API. + +* ``parser.items()`` is compatible with the mapping protocol (returns a list of + *section_name*, *section_proxy* pairs including the DEFAULTSECT). However, + this method can also be invoked with arguments: ``parser.items(section, raw, + vars)``. The latter call returns a list of *option*, *value* pairs for + a specified ``section``, with all interpolations expanded (unless + ``raw=True`` is provided). + +The mapping protocol is implemented on top of the existing legacy API so that +subclasses overriding the original interface still should have mappings working +as expected. + + +Customizing Parser Behaviour +---------------------------- + +There are nearly as many INI format variants as there are applications using it. +:mod:`configparser` goes a long way to provide support for the largest sensible +set of INI styles available. The default functionality is mainly dictated by +historical background and it's very likely that you will want to customize some +of the features. + +The most common way to change the way a specific config parser works is to use +the :meth:`__init__` options: + +* *defaults*, default value: ``None`` + + This option accepts a dictionary of key-value pairs which will be initially + put in the ``DEFAULT`` section. This makes for an elegant way to support + concise configuration files that don't specify values which are the same as + the documented default. + + Hint: if you want to specify default values for a specific section, use + :meth:`read_dict` before you read the actual file. + +* *dict_type*, default value: :class:`collections.OrderedDict` + + This option has a major impact on how the mapping protocol will behave and how + the written configuration files look. With the default ordered + dictionary, every section is stored in the order they were added to the + parser. Same goes for options within sections. + + An alternative dictionary type can be used for example to sort sections and + options on write-back. You can also use a regular dictionary for performance + reasons. + + Please note: there are ways to add a set of key-value pairs in a single + operation. When you use a regular dictionary in those operations, the order + of the keys may be random. For example: + + .. doctest:: + + >>> parser = configparser.ConfigParser() + >>> parser.read_dict({'section1': {'key1': 'value1', + ... 'key2': 'value2', + ... 'key3': 'value3'}, + ... 'section2': {'keyA': 'valueA', + ... 'keyB': 'valueB', + ... 'keyC': 'valueC'}, + ... 'section3': {'foo': 'x', + ... 'bar': 'y', + ... 'baz': 'z'} + ... }) + >>> parser.sections() + ['section3', 'section2', 'section1'] + >>> [option for option in parser['section3']] + ['baz', 'foo', 'bar'] + + In these operations you need to use an ordered dictionary as well: + + .. doctest:: + + >>> from collections import OrderedDict + >>> parser = configparser.ConfigParser() + >>> parser.read_dict( + ... OrderedDict(( + ... ('s1', + ... OrderedDict(( + ... ('1', '2'), + ... ('3', '4'), + ... ('5', '6'), + ... )) + ... ), + ... ('s2', + ... OrderedDict(( + ... ('a', 'b'), + ... ('c', 'd'), + ... ('e', 'f'), + ... )) + ... ), + ... )) + ... ) + >>> parser.sections() + ['s1', 's2'] + >>> [option for option in parser['s1']] + ['1', '3', '5'] + >>> [option for option in parser['s2'].values()] + ['b', 'd', 'f'] + +* *allow_no_value*, default value: ``False`` + + Some configuration files are known to include settings without values, but + which otherwise conform to the syntax supported by :mod:`configparser`. The + *allow_no_value* parameter to the constructor can be used to + indicate that such values should be accepted: + + .. doctest:: + + >>> import configparser + + >>> sample_config = """ + ... [mysqld] + ... user = mysql + ... pid-file = /var/run/mysqld/mysqld.pid + ... skip-external-locking + ... old_passwords = 1 + ... skip-bdb + ... # we don't need ACID today + ... skip-innodb + ... """ + >>> config = configparser.ConfigParser(allow_no_value=True) + >>> config.read_string(sample_config) + + >>> # Settings with values are treated as before: + >>> config["mysqld"]["user"] + 'mysql' + + >>> # Settings without values provide None: + >>> config["mysqld"]["skip-bdb"] + + >>> # Settings which aren't specified still raise an error: + >>> config["mysqld"]["does-not-exist"] + Traceback (most recent call last): + ... + KeyError: 'does-not-exist' + +* *delimiters*, default value: ``('=', ':')`` + + Delimiters are substrings that delimit keys from values within a section. The + first occurence of a delimiting substring on a line is considered a delimiter. + This means values (but not keys) can contain the delimiters. + + See also the *space_around_delimiters* argument to + :meth:`ConfigParser.write`. + +* *comment_prefixes*, default value: ``('#', ';')`` + +* *inline_comment_prefixes*, default value: ``None`` + + Comment prefixes are strings that indicate the start of a valid comment within + a config file. *comment_prefixes* are used only on otherwise empty lines + (optionally indented) whereas *inline_comment_prefixes* can be used after + every valid value (e.g. section names, options and empty lines as well). By + default inline comments are disabled and ``'#'`` and ``';'`` are used as + prefixes for whole line comments. + + .. versionchanged:: 3.2 + In previous versions of :mod:`configparser` behaviour matched + ``comment_prefixes=('#',';')`` and ``inline_comment_prefixes=(';',)``. + + Please note that config parsers don't support escaping of comment prefixes so + using *inline_comment_prefixes* may prevent users from specifying option + values with characters used as comment prefixes. When in doubt, avoid setting + *inline_comment_prefixes*. In any circumstances, the only way of storing + comment prefix characters at the beginning of a line in multiline values is to + interpolate the prefix, for example:: + + >>> from configparser import ConfigParser, ExtendedInterpolation + >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) + >>> # the default BasicInterpolation could be used as well + >>> parser.read_string(""" + ... [DEFAULT] + ... hash = # + ... + ... [hashes] + ... shebang = + ... ${hash}!/usr/bin/env python + ... ${hash} -*- coding: utf-8 -*- + ... + ... extensions = + ... enabled_extension + ... another_extension + ... #disabled_by_comment + ... yet_another_extension + ... + ... interpolation not necessary = if # is not at line start + ... even in multiline values = line #1 + ... line #2 + ... line #3 + ... """) + >>> print(parser['hashes']['shebang']) + + #!/usr/bin/env python + # -*- coding: utf-8 -*- + >>> print(parser['hashes']['extensions']) + + enabled_extension + another_extension + yet_another_extension + >>> print(parser['hashes']['interpolation not necessary']) + if # is not at line start + >>> print(parser['hashes']['even in multiline values']) + line #1 + line #2 + line #3 + +* *strict*, default value: ``True`` + + When set to ``True``, the parser will not allow for any section or option + duplicates while reading from a single source (using :meth:`read_file`, + :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict + parsers in new applications. + + .. versionchanged:: 3.2 + In previous versions of :mod:`configparser` behaviour matched + ``strict=False``. + +* *empty_lines_in_values*, default value: ``True`` + + In config parsers, values can span multiple lines as long as they are + indented more than the key that holds them. By default parsers also let + empty lines to be parts of values. At the same time, keys can be arbitrarily + indented themselves to improve readability. In consequence, when + configuration files get big and complex, it is easy for the user to lose + track of the file structure. Take for instance: + + .. code-block:: ini + + [Section] + key = multiline + value with a gotcha + + this = is still a part of the multiline value of 'key' + + This can be especially problematic for the user to see if she's using a + proportional font to edit the file. That is why when your application does + not need values with empty lines, you should consider disallowing them. This + will make empty lines split keys every time. In the example above, it would + produce two keys, ``key`` and ``this``. + +* *default_section*, default value: ``configparser.DEFAULTSECT`` (that is: + ``"DEFAULT"``) + + The convention of allowing a special section of default values for other + sections or interpolation purposes is a powerful concept of this library, + letting users create complex declarative configurations. This section is + normally called ``"DEFAULT"`` but this can be customized to point to any + other valid section name. Some typical values include: ``"general"`` or + ``"common"``. The name provided is used for recognizing default sections when + reading from any source and is used when writing configuration back to + a file. Its current value can be retrieved using the + ``parser_instance.default_section`` attribute and may be modified at runtime + (i.e. to convert files from one format to another). + +* *interpolation*, default value: ``configparser.BasicInterpolation`` + + Interpolation behaviour may be customized by providing a custom handler + through the *interpolation* argument. ``None`` can be used to turn off + interpolation completely, ``ExtendedInterpolation()`` provides a more + advanced variant inspired by ``zc.buildout``. More on the subject in the + `dedicated documentation section <#interpolation-of-values>`_. + :class:`RawConfigParser` has a default value of ``None``. + + +More advanced customization may be achieved by overriding default values of +these parser attributes. The defaults are defined on the classes, so they +may be overriden by subclasses or by attribute assignment. + +.. attribute:: BOOLEAN_STATES + + By default when using :meth:`getboolean`, config parsers consider the + following values ``True``: ``'1'``, ``'yes'``, ``'true'``, ``'on'`` and the + following values ``False``: ``'0'``, ``'no'``, ``'false'``, ``'off'``. You + can override this by specifying a custom dictionary of strings and their + Boolean outcomes. For example: + + .. doctest:: + + >>> custom = configparser.ConfigParser() + >>> custom['section1'] = {'funky': 'nope'} + >>> custom['section1'].getboolean('funky') + Traceback (most recent call last): + ... + ValueError: Not a boolean: nope + >>> custom.BOOLEAN_STATES = {'sure': True, 'nope': False} + >>> custom['section1'].getboolean('funky') + False + + Other typical Boolean pairs include ``accept``/``reject`` or + ``enabled``/``disabled``. + +.. method:: optionxform(option) + + This method transforms option names on every read, get, or set + operation. The default converts the name to lowercase. This also + means that when a configuration file gets written, all keys will be + lowercase. Override this method if that's unsuitable. + For example: + + .. doctest:: + + >>> config = """ + ... [Section1] + ... Key = Value + ... + ... [Section2] + ... AnotherKey = Value + ... """ + >>> typical = configparser.ConfigParser() + >>> typical.read_string(config) + >>> list(typical['Section1'].keys()) + ['key'] + >>> list(typical['Section2'].keys()) + ['anotherkey'] + >>> custom = configparser.RawConfigParser() + >>> custom.optionxform = lambda option: option + >>> custom.read_string(config) + >>> list(custom['Section1'].keys()) + ['Key'] + >>> list(custom['Section2'].keys()) + ['AnotherKey'] + +.. attribute:: SECTCRE + + A compiled regular expression used to parse section headers. The default + matches ``[section]`` to the name ``"section"``. Whitespace is considered part + of the section name, thus ``[ larch ]`` will be read as a section of name + ``" larch "``. Override this attribute if that's unsuitable. For example: + + .. doctest:: + + >>> config = """ + ... [Section 1] + ... option = value + ... + ... [ Section 2 ] + ... another = val + ... """ + >>> typical = ConfigParser() + >>> typical.read_string(config) + >>> typical.sections() + ['Section 1', ' Section 2 '] + >>> custom = ConfigParser() + >>> custom.SECTCRE = re.compile(r"\[ *(?P<header>[^]]+?) *\]") + >>> custom.read_string(config) + >>> custom.sections() + ['Section 1', 'Section 2'] + + .. note:: + + While ConfigParser objects also use an ``OPTCRE`` attribute for recognizing + option lines, it's not recommended to override it because that would + interfere with constructor options *allow_no_value* and *delimiters*. + + +Legacy API Examples +------------------- + +Mainly because of backwards compatibility concerns, :mod:`configparser` +provides also a legacy API with explicit ``get``/``set`` methods. While there +are valid use cases for the methods outlined below, mapping protocol access is +preferred for new projects. The legacy API is at times more advanced, +low-level and downright counterintuitive. + +An example of writing to a configuration file:: + + import configparser - .. XXX Need to explain what's safer/more predictable about it. + config = configparser.RawConfigParser() + + # Please note that using RawConfigParser's set functions, you can assign + # non-string values to keys internally, but will receive an error when + # attempting to write to a file or when you get it in non-raw mode. Setting + # values using the mapping protocol or ConfigParser's set() does not allow + # such assignments to take place. + config.add_section('Section1') + config.set('Section1', 'int', '15') + config.set('Section1', 'bool', 'true') + config.set('Section1', 'float', '3.1415') + config.set('Section1', 'baz', 'fun') + config.set('Section1', 'bar', 'Python') + config.set('Section1', 'foo', '%(bar)s is %(baz)s!') + + # Writing our configuration file to 'example.cfg' + with open('example.cfg', 'w') as configfile: + config.write(configfile) + +An example of reading the configuration file again:: + + import configparser + + config = configparser.RawConfigParser() + config.read('example.cfg') + + # getfloat() raises an exception if the value is not a float + # getint() and getboolean() also do this for their respective types + float = config.getfloat('Section1', 'float') + int = config.getint('Section1', 'int') + print(float + int) + + # Notice that the next output does not interpolate '%(bar)s' or '%(baz)s'. + # This is because we are using a RawConfigParser(). + if config.getboolean('Section1', 'bool'): + print(config.get('Section1', 'foo')) + +To get interpolation, use :class:`ConfigParser`:: + + import configparser + + cfg = configparser.ConfigParser() + cfg.read('example.cfg') + + # Set the optional `raw` argument of get() to True if you wish to disable + # interpolation in a single get operation. + print(cfg.get('Section1', 'foo', raw=False)) # -> "Python is fun!" + print(cfg.get('Section1', 'foo', raw=True)) # -> "%(bar)s is %(baz)s!" + + # The optional `vars` argument is a dict with members that will take + # precedence in interpolation. + print(cfg.get('Section1', 'foo', vars={'bar': 'Documentation', + 'baz': 'evil'})) + + # The optional `fallback` argument can be used to provide a fallback value + print(cfg.get('Section1', 'foo')) + # -> "Python is fun!" + + print(cfg.get('Section1', 'foo', fallback='Monty is not.')) + # -> "Python is fun!" + + print(cfg.get('Section1', 'monster', fallback='No such things as monsters.')) + # -> "No such things as monsters." + + # A bare print(cfg.get('Section1', 'monster')) would raise NoOptionError + # but we can also use: + + print(cfg.get('Section1', 'monster', fallback=None)) + # -> None + +Default values are available in both types of ConfigParsers. They are used in +interpolation if an option used is not defined elsewhere. :: + + import configparser + + # New instance with 'bar' and 'baz' defaulting to 'Life' and 'hard' each + config = configparser.ConfigParser({'bar': 'Life', 'baz': 'hard'}) + config.read('example.cfg') + + print(config.get('Section1', 'foo')) # -> "Python is fun!" + config.remove_option('Section1', 'bar') + config.remove_option('Section1', 'baz') + print(config.get('Section1', 'foo')) # -> "Life is hard!" + + +.. _configparser-objects: + +ConfigParser Objects +-------------------- + +.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation()) + + The main configuration parser. When *defaults* is given, it is initialized + into the dictionary of intrinsic defaults. When *dict_type* is given, it + will be used to create the dictionary objects for the list of sections, for + the options within a section, and for the default values. + + When *delimiters* is given, it is used as the set of substrings that + divide keys from values. When *comment_prefixes* is given, it will be used + as the set of substrings that prefix comments in otherwise empty lines. + Comments can be indented. When *inline_comment_prefixes* is given, it will be + used as the set of substrings that prefix comments in non-empty lines. + + line and inline comments. For backwards compatibility, the default value for + *comment_prefixes* is a special value that indicates that ``;`` and ``#`` can + start whole line comments while only ``;`` can start inline comments. + + When *strict* is ``True`` (the default), the parser won't allow for + any section or option duplicates while reading from a single source (file, + string or dictionary), raising :exc:`DuplicateSectionError` or + :exc:`DuplicateOptionError`. When *empty_lines_in_values* is ``False`` + (default: ``True``), each empty line marks the end of an option. Otherwise, + internal empty lines of a multiline option are kept as part of the value. + When *allow_no_value* is ``True`` (default: ``False``), options without + values are accepted; the value held for these is ``None`` and they are + serialized without the trailing delimiter. + + When *default_section* is given, it specifies the name for the special + section holding default values for other sections and interpolation purposes + (normally named ``"DEFAULT"``). This value can be retrieved and changed on + runtime using the ``default_section`` instance attribute. + + Interpolation behaviour may be customized by providing a custom handler + through the *interpolation* argument. ``None`` can be used to turn off + interpolation completely, ``ExtendedInterpolation()`` provides a more + advanced variant inspired by ``zc.buildout``. More on the subject in the + `dedicated documentation section <#interpolation-of-values>`_. + + All option names used in interpolation will be passed through the + :meth:`optionxform` method just like any other option name reference. For + example, using the default implementation of :meth:`optionxform` (which + converts option names to lower case), the values ``foo %(bar)s`` and ``foo + %(BAR)s`` are equivalent. .. versionchanged:: 3.1 The default *dict_type* is :class:`collections.OrderedDict`. + .. versionchanged:: 3.2 + *allow_no_value*, *delimiters*, *comment_prefixes*, *strict*, + *empty_lines_in_values*, *default_section* and *interpolation* were + added. -.. exception:: Error - Base class for all other configparser exceptions. + .. method:: defaults() + Return a dictionary containing the instance-wide defaults. -.. exception:: NoSectionError - Exception raised when a specified section is not found. + .. method:: sections() + Return a list of the sections available; the *default section* is not + included in the list. -.. exception:: DuplicateSectionError - Exception raised if :meth:`add_section` is called with the name of a section - that is already present. + .. method:: add_section(section) + Add a section named *section* to the instance. If a section by the given + name already exists, :exc:`DuplicateSectionError` is raised. If the + *default section* name is passed, :exc:`ValueError` is raised. The name + of the section must be a string; if not, :exc:`TypeError` is raised. -.. exception:: NoOptionError + .. versionchanged:: 3.2 + Non-string section names raise :exc:`TypeError`. - Exception raised when a specified option is not found in the specified section. + .. method:: has_section(section) -.. exception:: InterpolationError + Indicates whether the named *section* is present in the configuration. + The *default section* is not acknowledged. - Base class for exceptions raised when problems occur performing string - interpolation. + .. method:: options(section) -.. exception:: InterpolationDepthError + Return a list of options available in the specified *section*. - Exception raised when string interpolation cannot be completed because the - number of iterations exceeds :const:`MAX_INTERPOLATION_DEPTH`. Subclass of - :exc:`InterpolationError`. + .. method:: has_option(section, option) -.. exception:: InterpolationMissingOptionError + If the given *section* exists, and contains the given *option*, return + :const:`True`; otherwise return :const:`False`. If the specified + *section* is :const:`None` or an empty string, DEFAULT is assumed. - Exception raised when an option referenced from a value does not exist. Subclass - of :exc:`InterpolationError`. + .. method:: read(filenames, encoding=None) -.. exception:: InterpolationSyntaxError + Attempt to read and parse a list of filenames, returning a list of + filenames which were successfully parsed. If *filenames* is a string, it + is treated as a single filename. If a file named in *filenames* cannot + be opened, that file will be ignored. This is designed so that you can + specify a list of potential configuration file locations (for example, + the current directory, the user's home directory, and some system-wide + directory), and all existing configuration files in the list will be + read. If none of the named files exist, the :class:`ConfigParser` + instance will contain an empty dataset. An application which requires + initial values to be loaded from a file should load the required file or + files using :meth:`read_file` before calling :meth:`read` for any + optional files:: - Exception raised when the source text into which substitutions are made does not - conform to the required syntax. Subclass of :exc:`InterpolationError`. + import configparser, os + config = configparser.ConfigParser() + config.read_file(open('defaults.cfg')) + config.read(['site.cfg', os.path.expanduser('~/.myapp.cfg')], + encoding='cp1250') -.. exception:: MissingSectionHeaderError + .. versionadded:: 3.2 + The *encoding* parameter. Previously, all files were read using the + default encoding for :func:`open`. - Exception raised when attempting to parse a file which has no section headers. + .. method:: read_file(f, source=None) -.. exception:: ParsingError + Read and parse configuration data from *f* which must be an iterable + yielding Unicode strings (for example files opened in text mode). - Exception raised when errors occur attempting to parse a file. + Optional argument *source* specifies the name of the file being read. If + not given and *f* has a :attr:`name` attribute, that is used for + *source*; the default is ``'<???>'``. + .. versionadded:: 3.2 + Replaces :meth:`readfp`. -.. data:: MAX_INTERPOLATION_DEPTH + .. method:: read_string(string, source='<string>') - The maximum depth for recursive interpolation for :meth:`get` when the *raw* - parameter is false. This is relevant only for the :class:`ConfigParser` class. + Parse configuration data from a string. + Optional argument *source* specifies a context-specific name of the + string passed. If not given, ``'<string>'`` is used. This should + commonly be a filesystem path or a URL. -.. seealso:: + .. versionadded:: 3.2 - Module :mod:`shlex` - Support for a creating Unix shell-like mini-languages which can be used as an - alternate format for application configuration files. + .. method:: read_dict(dictionary, source='<dict>') -.. _rawconfigparser-objects: + Load configuration from any object that provides a dict-like ``items()`` + method. Keys are section names, values are dictionaries with keys and + values that should be present in the section. If the used dictionary + type preserves order, sections and their keys will be added in order. + Values are automatically converted to strings. -RawConfigParser Objects ------------------------ + Optional argument *source* specifies a context-specific name of the + dictionary passed. If not given, ``<dict>`` is used. -:class:`RawConfigParser` instances have the following methods: + This method can be used to copy state between parsers. + .. versionadded:: 3.2 -.. method:: RawConfigParser.defaults() - Return a dictionary containing the instance-wide defaults. + .. method:: get(section, option, raw=False, [vars, fallback]) + Get an *option* value for the named *section*. If *vars* is provided, it + must be a dictionary. The *option* is looked up in *vars* (if provided), + *section*, and in *DEFAULTSECT* in that order. If the key is not found + and *fallback* is provided, it is used as a fallback value. ``None`` can + be provided as a *fallback* value. -.. method:: RawConfigParser.sections() + All the ``'%'`` interpolations are expanded in the return values, unless + the *raw* argument is true. Values for interpolation keys are looked up + in the same manner as the option. - Return a list of the sections available; ``DEFAULT`` is not included in the - list. + .. versionchanged:: 3.2 + Arguments *raw*, *vars* and *fallback* are keyword only to protect + users from trying to use the third argument as the *fallback* fallback + (especially when using the mapping protocol). -.. method:: RawConfigParser.add_section(section) + .. method:: getint(section, option, raw=False, [vars, fallback]) - Add a section named *section* to the instance. If a section by the given name - already exists, :exc:`DuplicateSectionError` is raised. If the name - ``DEFAULT`` (or any of it's case-insensitive variants) is passed, - :exc:`ValueError` is raised. + A convenience method which coerces the *option* in the specified *section* + to an integer. See :meth:`get` for explanation of *raw*, *vars* and + *fallback*. -.. method:: RawConfigParser.has_section(section) - Indicates whether the named section is present in the configuration. The - ``DEFAULT`` section is not acknowledged. + .. method:: getfloat(section, option, raw=False, [vars, fallback]) + A convenience method which coerces the *option* in the specified *section* + to a floating point number. See :meth:`get` for explanation of *raw*, + *vars* and *fallback*. -.. method:: RawConfigParser.options(section) - Returns a list of options available in the specified *section*. + .. method:: getboolean(section, option, raw=False, [vars, fallback]) + A convenience method which coerces the *option* in the specified *section* + to a Boolean value. Note that the accepted values for the option are + ``'1'``, ``'yes'``, ``'true'``, and ``'on'``, which cause this method to + return ``True``, and ``'0'``, ``'no'``, ``'false'``, and ``'off'``, which + cause it to return ``False``. These string values are checked in a + case-insensitive manner. Any other value will cause it to raise + :exc:`ValueError`. See :meth:`get` for explanation of *raw*, *vars* and + *fallback*. -.. method:: RawConfigParser.has_option(section, option) - If the given section exists, and contains the given option, return - :const:`True`; otherwise return :const:`False`. + .. method:: items([section], raw=False, vars=None) + When *section* is not given, return a list of *section_name*, + *section_proxy* pairs, including DEFAULTSECT. -.. method:: RawConfigParser.read(filenames) + Otherwise, return a list of *name*, *value* pairs for the options in the + given *section*. Optional arguments have the same meaning as for the + :meth:`get` method. - Attempt to read and parse a list of filenames, returning a list of filenames - which were successfully parsed. If *filenames* is a string, - it is treated as a single filename. If a file named in *filenames* cannot be - opened, that file will be ignored. This is designed so that you can specify a - list of potential configuration file locations (for example, the current - directory, the user's home directory, and some system-wide directory), and all - existing configuration files in the list will be read. If none of the named - files exist, the :class:`ConfigParser` instance will contain an empty dataset. - An application which requires initial values to be loaded from a file should - load the required file or files using :meth:`readfp` before calling :meth:`read` - for any optional files:: - import configparser, os + .. method:: set(section, option, value) - config = configparser.ConfigParser() - config.readfp(open('defaults.cfg')) - config.read(['site.cfg', os.path.expanduser('~/.myapp.cfg')]) + If the given section exists, set the given option to the specified value; + otherwise raise :exc:`NoSectionError`. *option* and *value* must be + strings; if not, :exc:`TypeError` is raised. -.. method:: RawConfigParser.readfp(fp, filename=None) + .. method:: write(fileobject, space_around_delimiters=True) - Read and parse configuration data from the file or file-like object in *fp* - (only the :meth:`readline` method is used). The file-like object must - operate in text mode, i.e. return strings from :meth:`readline`. + Write a representation of the configuration to the specified :term:`file + object`, which must be opened in text mode (accepting strings). This + representation can be parsed by a future :meth:`read` call. If + *space_around_delimiters* is true, delimiters between + keys and values are surrounded by spaces. - If *filename* is omitted and *fp* has a :attr:`name` attribute, that is used - for *filename*; the default is ``<???>``. + .. method:: remove_option(section, option) -.. method:: RawConfigParser.get(section, option) + Remove the specified *option* from the specified *section*. If the + section does not exist, raise :exc:`NoSectionError`. If the option + existed to be removed, return :const:`True`; otherwise return + :const:`False`. - Get an *option* value for the named *section*. + .. method:: remove_section(section) -.. method:: RawConfigParser.getint(section, option) + Remove the specified *section* from the configuration. If the section in + fact existed, return ``True``. Otherwise return ``False``. - A convenience method which coerces the *option* in the specified *section* to an - integer. + .. method:: optionxform(option) -.. method:: RawConfigParser.getfloat(section, option) + Transforms the option name *option* as found in an input file or as passed + in by client code to the form that should be used in the internal + structures. The default implementation returns a lower-case version of + *option*; subclasses may override this or client code can set an attribute + of this name on instances to affect this behavior. - A convenience method which coerces the *option* in the specified *section* to a - floating point number. + You don't need to subclass the parser to use this method, you can also + set it on an instance, to a function that takes a string argument and + returns a string. Setting it to ``str``, for example, would make option + names case sensitive:: + cfgparser = ConfigParser() + cfgparser.optionxform = str -.. method:: RawConfigParser.getboolean(section, option) + Note that when reading configuration files, whitespace around the option + names is stripped before :meth:`optionxform` is called. - A convenience method which coerces the *option* in the specified *section* to a - Boolean value. Note that the accepted values for the option are ``"1"``, - ``"yes"``, ``"true"``, and ``"on"``, which cause this method to return ``True``, - and ``"0"``, ``"no"``, ``"false"``, and ``"off"``, which cause it to return - ``False``. These string values are checked in a case-insensitive manner. Any - other value will cause it to raise :exc:`ValueError`. + .. method:: readfp(fp, filename=None) -.. method:: RawConfigParser.items(section) + .. deprecated:: 3.2 + Use :meth:`read_file` instead. - Return a list of ``(name, value)`` pairs for each option in the given *section*. + .. versionchanged:: 3.2 + :meth:`readfp` now iterates on *f* instead of calling ``f.readline()``. + For existing code calling :meth:`readfp` with arguments which don't + support iteration, the following generator may be used as a wrapper + around the file-like object:: -.. method:: RawConfigParser.set(section, option, value) + def readline_generator(f): + line = f.readline() + while line: + yield line + line = f.readline() - If the given section exists, set the given option to the specified value; - otherwise raise :exc:`NoSectionError`. While it is possible to use - :class:`RawConfigParser` (or :class:`ConfigParser` with *raw* parameters set to - true) for *internal* storage of non-string values, full functionality (including - interpolation and output to files) can only be achieved using string values. + Instead of ``parser.readfp(f)`` use + ``parser.read_file(readline_generator(f))``. -.. method:: RawConfigParser.write(fileobject) +.. data:: MAX_INTERPOLATION_DEPTH - Write a representation of the configuration to the specified :term:`file object`, - which must be opened in text mode (accepting strings). This representation - can be parsed by a future :meth:`read` call. + The maximum depth for recursive interpolation for :meth:`get` when the *raw* + parameter is false. This is relevant only when the default *interpolation* + is used. -.. method:: RawConfigParser.remove_option(section, option) +.. _rawconfigparser-objects: - Remove the specified *option* from the specified *section*. If the section does - not exist, raise :exc:`NoSectionError`. If the option existed to be removed, - return :const:`True`; otherwise return :const:`False`. +RawConfigParser Objects +----------------------- +.. class:: RawConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configaparser.DEFAULTSECT, interpolation=None) -.. method:: RawConfigParser.remove_section(section) + Legacy variant of the :class:`ConfigParser` with interpolation disabled + by default and unsafe ``add_section`` and ``set`` methods. - Remove the specified *section* from the configuration. If the section in fact - existed, return ``True``. Otherwise return ``False``. + .. note:: + Consider using :class:`ConfigParser` instead which checks types of + the values to be stored internally. If you don't want interpolation, you + can use ``ConfigParser(interpolation=None)``. -.. method:: RawConfigParser.optionxform(option) + .. method:: add_section(section) - Transforms the option name *option* as found in an input file or as passed in - by client code to the form that should be used in the internal structures. - The default implementation returns a lower-case version of *option*; - subclasses may override this or client code can set an attribute of this name - on instances to affect this behavior. + Add a section named *section* to the instance. If a section by the given + name already exists, :exc:`DuplicateSectionError` is raised. If the + *default section* name is passed, :exc:`ValueError` is raised. - You don't necessarily need to subclass a ConfigParser to use this method, you - can also re-set it on an instance, to a function that takes a string - argument. Setting it to ``str``, for example, would make option names case - sensitive:: + Type of *section* is not checked which lets users create non-string named + sections. This behaviour is unsupported and may cause internal errors. - cfgparser = ConfigParser() - ... - cfgparser.optionxform = str - Note that when reading configuration files, whitespace around the - option names are stripped before :meth:`optionxform` is called. + .. method:: set(section, option, value) + If the given section exists, set the given option to the specified value; + otherwise raise :exc:`NoSectionError`. While it is possible to use + :class:`RawConfigParser` (or :class:`ConfigParser` with *raw* parameters + set to true) for *internal* storage of non-string values, full + functionality (including interpolation and output to files) can only be + achieved using string values. -.. _configparser-objects: + This method lets users assign non-string values to keys internally. This + behaviour is unsupported and will cause errors when attempting to write + to a file or get it in non-raw mode. **Use the mapping protocol API** + which does not allow such assignments to take place. -ConfigParser Objects --------------------- -The :class:`ConfigParser` class extends some methods of the -:class:`RawConfigParser` interface, adding some optional arguments. +Exceptions +---------- +.. exception:: Error -.. method:: ConfigParser.get(section, option, raw=False, vars=None) + Base class for all other :mod:`configparser` exceptions. - Get an *option* value for the named *section*. If *vars* is provided, it - must be a dictionary. The *option* is looked up in *vars* (if provided), - *section*, and in *defaults* in that order. - All the ``'%'`` interpolations are expanded in the return values, unless the - *raw* argument is true. Values for interpolation keys are looked up in the - same manner as the option. +.. exception:: NoSectionError + Exception raised when a specified section is not found. -.. method:: ConfigParser.items(section, raw=False, vars=None) - Return a list of ``(name, value)`` pairs for each option in the given *section*. - Optional arguments have the same meaning as for the :meth:`get` method. +.. exception:: DuplicateSectionError + Exception raised if :meth:`add_section` is called with the name of a section + that is already present or in strict parsers when a section if found more + than once in a single input file, string or dictionary. -.. _safeconfigparser-objects: + .. versionadded:: 3.2 + Optional ``source`` and ``lineno`` attributes and arguments to + :meth:`__init__` were added. -SafeConfigParser Objects ------------------------- -The :class:`SafeConfigParser` class implements the same extended interface as -:class:`ConfigParser`, with the following addition: +.. exception:: DuplicateOptionError + Exception raised by strict parsers if a single option appears twice during + reading from a single file, string or dictionary. This catches misspellings + and case sensitivity-related errors, e.g. a dictionary may have two keys + representing the same case-insensitive configuration key. -.. method:: SafeConfigParser.set(section, option, value) - If the given section exists, set the given option to the specified value; - otherwise raise :exc:`NoSectionError`. *value* must be a string; if it is - not, :exc:`TypeError` is raised. +.. exception:: NoOptionError + Exception raised when a specified option is not found in the specified + section. -Examples --------- -An example of writing to a configuration file:: +.. exception:: InterpolationError - import configparser + Base class for exceptions raised when problems occur performing string + interpolation. - config = configparser.RawConfigParser() - # When adding sections or items, add them in the reverse order of - # how you want them to be displayed in the actual file. - # In addition, please note that using RawConfigParser's and the raw - # mode of ConfigParser's respective set functions, you can assign - # non-string values to keys internally, but will receive an error - # when attempting to write to a file or when you get it in non-raw - # mode. SafeConfigParser does not allow such assignments to take place. - config.add_section('Section1') - config.set('Section1', 'int', '15') - config.set('Section1', 'bool', 'true') - config.set('Section1', 'float', '3.1415') - config.set('Section1', 'baz', 'fun') - config.set('Section1', 'bar', 'Python') - config.set('Section1', 'foo', '%(bar)s is %(baz)s!') +.. exception:: InterpolationDepthError - # Writing our configuration file to 'example.cfg' - with open('example.cfg', 'w') as configfile: - config.write(configfile) + Exception raised when string interpolation cannot be completed because the + number of iterations exceeds :const:`MAX_INTERPOLATION_DEPTH`. Subclass of + :exc:`InterpolationError`. -An example of reading the configuration file again:: - import configparser +.. exception:: InterpolationMissingOptionError - config = configparser.RawConfigParser() - config.read('example.cfg') + Exception raised when an option referenced from a value does not exist. + Subclass of :exc:`InterpolationError`. - # getfloat() raises an exception if the value is not a float - # getint() and getboolean() also do this for their respective types - float = config.getfloat('Section1', 'float') - int = config.getint('Section1', 'int') - print(float + int) - # Notice that the next output does not interpolate '%(bar)s' or '%(baz)s'. - # This is because we are using a RawConfigParser(). - if config.getboolean('Section1', 'bool'): - print(config.get('Section1', 'foo')) +.. exception:: InterpolationSyntaxError -To get interpolation, you will need to use a :class:`ConfigParser` or -:class:`SafeConfigParser`:: + Exception raised when the source text into which substitutions are made does + not conform to the required syntax. Subclass of :exc:`InterpolationError`. - import configparser - config = configparser.ConfigParser() - config.read('example.cfg') +.. exception:: MissingSectionHeaderError - # Set the third, optional argument of get to 1 if you wish to use raw mode. - print(config.get('Section1', 'foo', 0)) # -> "Python is fun!" - print(config.get('Section1', 'foo', 1)) # -> "%(bar)s is %(baz)s!" + Exception raised when attempting to parse a file which has no section + headers. - # The optional fourth argument is a dict with members that will take - # precedence in interpolation. - print(config.get('Section1', 'foo', 0, {'bar': 'Documentation', - 'baz': 'evil'})) -Defaults are available in all three types of ConfigParsers. They are used in -interpolation if an option used is not defined elsewhere. :: +.. exception:: ParsingError - import configparser + Exception raised when errors occur attempting to parse a file. - # New instance with 'bar' and 'baz' defaulting to 'Life' and 'hard' each - config = configparser.SafeConfigParser({'bar': 'Life', 'baz': 'hard'}) - config.read('example.cfg') + .. versionchanged:: 3.2 + The ``filename`` attribute and :meth:`__init__` argument were renamed to + ``source`` for consistency. - print(config.get('Section1', 'foo')) # -> "Python is fun!" - config.remove_option('Section1', 'bar') - config.remove_option('Section1', 'baz') - print(config.get('Section1', 'foo')) # -> "Life is hard!" -The function ``opt_move`` below can be used to move options between sections:: - - def opt_move(config, section1, section2, option): - try: - config.set(section2, option, config.get(section1, option, 1)) - except configparser.NoSectionError: - # Create non-existent section - config.add_section(section2) - opt_move(config, section1, section2, option) - else: - config.remove_option(section1, option) +.. rubric:: Footnotes + +.. [1] Config parsers allow for heavy customization. If you are interested in + changing the behaviour outlined by the footnote reference, consult the + `Customizing Parser Behaviour`_ section. diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index ca37f0f..a35ea56 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -4,6 +4,9 @@ .. module:: contextlib :synopsis: Utilities for with-statement contexts. +**Source code:** :source:`Lib/contextlib.py` + +-------------- This module provides utilities for common tasks involving the :keyword:`with` statement. For more information see also :ref:`typecontextmanager` and @@ -12,7 +15,7 @@ statement. For more information see also :ref:`typecontextmanager` and Functions provided: -.. function:: contextmanager(func) +.. decorator:: contextmanager This function is a :term:`decorator` that can be used to define a factory function for :keyword:`with` statement context managers, without needing to @@ -51,54 +54,11 @@ Functions provided: the exception has been handled, and execution will resume with the statement immediately following the :keyword:`with` statement. + contextmanager uses :class:`ContextDecorator` so the context managers it + creates can be used as decorators as well as in :keyword:`with` statements. -.. function:: nested(mgr1[, mgr2[, ...]]) - - Combine multiple context managers into a single nested context manager. - - This function has been deprecated in favour of the multiple manager form - of the :keyword:`with` statement. - - The one advantage of this function over the multiple manager form of the - :keyword:`with` statement is that argument unpacking allows it to be - used with a variable number of context managers as follows:: - - from contextlib import nested - - with nested(*managers): - do_something() - - Note that if the :meth:`__exit__` method of one of the nested context managers - indicates an exception should be suppressed, no exception information will be - passed to any remaining outer context managers. Similarly, if the - :meth:`__exit__` method of one of the nested managers raises an exception, any - previous exception state will be lost; the new exception will be passed to the - :meth:`__exit__` methods of any remaining outer context managers. In general, - :meth:`__exit__` methods should avoid raising exceptions, and in particular they - should not re-raise a passed-in exception. - - This function has two major quirks that have led to it being deprecated. Firstly, - as the context managers are all constructed before the function is invoked, the - :meth:`__new__` and :meth:`__init__` methods of the inner context managers are - not actually covered by the scope of the outer context managers. That means, for - example, that using :func:`nested` to open two files is a programming error as the - first file will not be closed promptly if an exception is thrown when opening - the second file. - - Secondly, if the :meth:`__enter__` method of one of the inner context managers - raises an exception that is caught and suppressed by the :meth:`__exit__` method - of one of the outer context managers, this construct will raise - :exc:`RuntimeError` rather than skipping the body of the :keyword:`with` - statement. - - Developers that need to support nesting of a variable number of context managers - can either use the :mod:`warnings` module to suppress the DeprecationWarning - raised by this function or else use this function as a model for an application - specific implementation. - - .. deprecated:: 3.1 - The with-statement now supports this functionality directly (without the - confusing error prone quirks). + .. versionchanged:: 3.2 + Use of :class:`ContextDecorator`. .. function:: closing(thing) @@ -128,6 +88,76 @@ Functions provided: ``page.close()`` will be called when the :keyword:`with` block is exited. +.. class:: ContextDecorator() + + A base class that enables a context manager to also be used as a decorator. + + Context managers inheriting from ``ContextDecorator`` have to implement + ``__enter__`` and ``__exit__`` as normal. ``__exit__`` retains its optional + exception handling even when used as a decorator. + + ``ContextDecorator`` is used by :func:`contextmanager`, so you get this + functionality automatically. + + Example of ``ContextDecorator``:: + + from contextlib import ContextDecorator + + class mycontext(ContextDecorator): + def __enter__(self): + print('Starting') + return self + + def __exit__(self, *exc): + print('Finishing') + return False + + >>> @mycontext() + ... def function(): + ... print('The bit in the middle') + ... + >>> function() + Starting + The bit in the middle + Finishing + + >>> with mycontext(): + ... print('The bit in the middle') + ... + Starting + The bit in the middle + Finishing + + This change is just syntactic sugar for any construct of the following form:: + + def f(): + with cm(): + # Do stuff + + ``ContextDecorator`` lets you instead write:: + + @cm() + def f(): + # Do stuff + + It makes it clear that the ``cm`` applies to the whole function, rather than + just a piece of it (and saving an indentation level is nice, too). + + Existing context managers that already have a base class can be extended by + using ``ContextDecorator`` as a mixin class:: + + from contextlib import ContextDecorator + + class mycontext(ContextBaseClass, ContextDecorator): + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + .. versionadded:: 3.2 + + .. seealso:: :pep:`0343` - The "with" statement diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index 92ba6da..edbe726 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -188,6 +188,15 @@ The :mod:`csv` module defines the following classes: TAB-delimited file. It is registered with the dialect name ``'excel-tab'``. +.. class:: unix_dialect() + + The :class:`unix_dialect` class defines the usual properties of a CSV file + generated on UNIX systems, i.e. using ``'\n'`` as line terminator and quoting + all fields. It is registered with the dialect name ``'unix'``. + + .. versionadded:: 3.2 + + .. class:: Sniffer() The :class:`Sniffer` class is used to deduce the format of a CSV file. @@ -393,6 +402,16 @@ Writer objects have the following public attribute: A read-only description of the dialect in use by the writer. +DictWriter objects have the following public method: + + +.. method:: DictWriter.writeheader() + + Write a row with the field names (as specified in the constructor). + + .. versionadded:: 3.2 + + .. _csv-examples: Examples diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index d1d025b..33d1f7e 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -38,7 +38,7 @@ You load libraries by accessing them as attributes of these objects. *cdll* loads libraries which export functions using the standard ``cdecl`` calling convention, while *windll* libraries call functions using the ``stdcall`` calling convention. *oledll* also uses the ``stdcall`` calling convention, and -assumes the functions return a Windows :ctype:`HRESULT` error code. The error +assumes the functions return a Windows :c:type:`HRESULT` error code. The error code is used to automatically raise a :class:`WindowsError` exception when the function call fails. @@ -198,9 +198,9 @@ should be careful anyway. ``None``, integers, bytes objects and (unicode) strings are the only native Python objects that can directly be used as parameters in these function calls. ``None`` is passed as a C ``NULL`` pointer, bytes objects and strings are passed -as pointer to the memory block that contains their data (:ctype:`char *` or -:ctype:`wchar_t *`). Python integers are passed as the platforms default C -:ctype:`int` type, their value is masked to fit into the C type. +as pointer to the memory block that contains their data (:c:type:`char *` or +:c:type:`wchar_t *`). Python integers are passed as the platforms default C +:c:type:`int` type, their value is masked to fit into the C type. Before we move on calling functions with other parameter types, we have to learn more about :mod:`ctypes` data types. @@ -213,48 +213,48 @@ Fundamental data types :mod:`ctypes` defines a number of primitive C compatible data types : -+----------------------+----------------------------------------+----------------------------+ -| ctypes type | C type | Python type | -+======================+========================================+============================+ -| :class:`c_bool` | :ctype:`_Bool` | bool (1) | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_char` | :ctype:`char` | 1-character bytes object | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_wchar` | :ctype:`wchar_t` | 1-character string | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_byte` | :ctype:`char` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_ubyte` | :ctype:`unsigned char` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_short` | :ctype:`short` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_ushort` | :ctype:`unsigned short` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_int` | :ctype:`int` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_uint` | :ctype:`unsigned int` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_long` | :ctype:`long` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_ulong` | :ctype:`unsigned long` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_longlong` | :ctype:`__int64` or :ctype:`long long` | int | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_ulonglong` | :ctype:`unsigned __int64` or | int | -| | :ctype:`unsigned long long` | | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_float` | :ctype:`float` | float | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_double` | :ctype:`double` | float | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_longdouble`| :ctype:`long double` | float | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_char_p` | :ctype:`char *` (NUL terminated) | bytes object or ``None`` | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_wchar_p` | :ctype:`wchar_t *` (NUL terminated) | string or ``None`` | -+----------------------+----------------------------------------+----------------------------+ -| :class:`c_void_p` | :ctype:`void *` | int or ``None`` | -+----------------------+----------------------------------------+----------------------------+ ++----------------------+------------------------------------------+----------------------------+ +| ctypes type | C type | Python type | ++======================+==========================================+============================+ +| :class:`c_bool` | :c:type:`_Bool` | bool (1) | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_char` | :c:type:`char` | 1-character bytes object | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_wchar` | :c:type:`wchar_t` | 1-character string | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_byte` | :c:type:`char` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_ubyte` | :c:type:`unsigned char` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_short` | :c:type:`short` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_ushort` | :c:type:`unsigned short` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_int` | :c:type:`int` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_uint` | :c:type:`unsigned int` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_long` | :c:type:`long` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_ulong` | :c:type:`unsigned long` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_longlong` | :c:type:`__int64` or :c:type:`long long` | int | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_ulonglong` | :c:type:`unsigned __int64` or | int | +| | :c:type:`unsigned long long` | | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_float` | :c:type:`float` | float | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_double` | :c:type:`double` | float | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_longdouble`| :c:type:`long double` | float | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_char_p` | :c:type:`char *` (NUL terminated) | bytes object or ``None`` | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_wchar_p` | :c:type:`wchar_t *` (NUL terminated) | string or ``None`` | ++----------------------+------------------------------------------+----------------------------+ +| :class:`c_void_p` | :c:type:`void *` | int or ``None`` | ++----------------------+------------------------------------------+----------------------------+ (1) The constructor accepts any object with a truth value. @@ -325,7 +325,7 @@ property:: The :func:`create_string_buffer` function replaces the :func:`c_buffer` function (which is still available as an alias), as well as the :func:`c_string` function from earlier ctypes releases. To create a mutable memory block containing -unicode characters of the C type :ctype:`wchar_t` use the +unicode characters of the C type :c:type:`wchar_t` use the :func:`create_unicode_buffer` function. @@ -436,7 +436,7 @@ integer, string, bytes, a :mod:`ctypes` instance, or an object with an Return types ^^^^^^^^^^^^ -By default functions are assumed to return the C :ctype:`int` type. Other +By default functions are assumed to return the C :c:type:`int` type. Other return types can be specified by setting the :attr:`restype` attribute of the function object. @@ -935,8 +935,8 @@ argument, and the callback functions expected argument types as the remaining arguments. I will present an example here which uses the standard C library's -:cfunc:`qsort` function, this is used to sort items with the help of a callback -function. :cfunc:`qsort` will be used to sort an array of integers:: +:c:func:`qsort` function, this is used to sort items with the help of a callback +function. :c:func:`qsort` will be used to sort an array of integers:: >>> IntArray5 = c_int * 5 >>> ia = IntArray5(5, 1, 7, 33, 99) @@ -1077,7 +1077,7 @@ Accessing values exported from dlls ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Some shared libraries not only export functions, they also export variables. An -example in the Python library itself is the :cdata:`Py_OptimizeFlag`, an integer +example in the Python library itself is the :c:data:`Py_OptimizeFlag`, an integer set to 0, 1, or 2, depending on the :option:`-O` or :option:`-OO` flag given on startup. @@ -1095,11 +1095,11 @@ have printed ``c_long(1)``, or ``c_long(2)`` if :option:`-OO` would have been specified. An extended example which also demonstrates the use of pointers accesses the -:cdata:`PyImport_FrozenModules` pointer exported by Python. +:c:data:`PyImport_FrozenModules` pointer exported by Python. Quoting the docs for that value: - This pointer is initialized to point to an array of :ctype:`struct _frozen` + This pointer is initialized to point to an array of :c:type:`struct _frozen` records, terminated by one whose members are all *NULL* or zero. When a frozen module is imported, it is searched in this table. Third-party code could play tricks with this to provide a dynamically created collection of frozen modules. @@ -1116,7 +1116,7 @@ size, we show only how this table can be read with :mod:`ctypes`:: ... >>> -We have defined the :ctype:`struct _frozen` data type, so we can get the pointer +We have defined the :c:type:`struct _frozen` data type, so we can get the pointer to the table:: >>> FrozenTable = POINTER(struct_frozen) @@ -1335,7 +1335,7 @@ way is to instantiate one of the following classes: Instances of this class represent loaded shared libraries. Functions in these libraries use the standard C calling convention, and are assumed to return - :ctype:`int`. + :c:type:`int`. .. class:: OleDLL(name, mode=DEFAULT_MODE, handle=None, use_errno=False, use_last_error=False) @@ -1352,7 +1352,7 @@ way is to instantiate one of the following classes: Windows only: Instances of this class represent loaded shared libraries, functions in these libraries use the ``stdcall`` calling convention, and are - assumed to return :ctype:`int` by default. + assumed to return :c:type:`int` by default. On Windows CE only the standard calling convention is used, for convenience the :class:`WinDLL` and :class:`OleDLL` use the standard calling convention on this @@ -1493,7 +1493,7 @@ object is available: An instance of :class:`PyDLL` that exposes Python C API functions as attributes. Note that all these functions are assumed to return C - :ctype:`int`, which is of course not always the truth, so you have to assign + :c:type:`int`, which is of course not always the truth, so you have to assign the correct :attr:`restype` attribute to use these functions. @@ -1522,10 +1522,10 @@ They are instances of a private class: .. attribute:: restype Assign a ctypes type to specify the result type of the foreign function. - Use ``None`` for :ctype:`void`, a function not returning anything. + Use ``None`` for :c:type:`void`, a function not returning anything. It is possible to assign a callable Python object that is not a ctypes - type, in this case the function is assumed to return a C :ctype:`int`, and + type, in this case the function is assumed to return a C :c:type:`int`, and the callable will be called with this integer, allowing to do further processing or error checking. Using this is deprecated, for more flexible post processing or error checking use a ctypes data type as @@ -1935,22 +1935,6 @@ Utility functions but it is possible to enlarge the buffer. -.. function:: set_conversion_mode(encoding, errors) - - This function sets the rules that ctypes objects use when converting between - bytes objects and (unicode) strings. *encoding* must be a string specifying an - encoding, like ``'utf-8'`` or ``'mbcs'``, *errors* must be a string specifying - the error handling on encoding/decoding errors. Examples of possible values are - ``'strict'``, ``'replace'``, or ``'ignore'``. - - :func:`set_conversion_mode` returns a 2-tuple containing the previous - conversion rules. On windows, the initial conversion rules are ``('mbcs', - 'ignore')``, on other systems ``('ascii', 'strict')``. - - You can set the *encoding* to ``'undefined'`` to completely disable automatic - conversions. - - .. function:: set_errno(value) Set the current value of the ctypes-private copy of the system :data:`errno` @@ -2120,21 +2104,21 @@ These are the fundamental ctypes data types: .. class:: c_byte - Represents the C :ctype:`signed char` datatype, and interprets the value as + Represents the C :c:type:`signed char` datatype, and interprets the value as small integer. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_char - Represents the C :ctype:`char` datatype, and interprets the value as a single + Represents the C :c:type:`char` datatype, and interprets the value as a single character. The constructor accepts an optional string initializer, the length of the string must be exactly one character. .. class:: c_char_p - Represents the C :ctype:`char *` datatype when it points to a zero-terminated + Represents the C :c:type:`char *` datatype when it points to a zero-terminated string. For a general character pointer that may also point to binary data, ``POINTER(c_char)`` must be used. The constructor accepts an integer address, or a bytes object. @@ -2142,173 +2126,180 @@ These are the fundamental ctypes data types: .. class:: c_double - Represents the C :ctype:`double` datatype. The constructor accepts an + Represents the C :c:type:`double` datatype. The constructor accepts an optional float initializer. .. class:: c_longdouble - Represents the C :ctype:`long double` datatype. The constructor accepts an + Represents the C :c:type:`long double` datatype. The constructor accepts an optional float initializer. On platforms where ``sizeof(long double) == sizeof(double)`` it is an alias to :class:`c_double`. .. class:: c_float - Represents the C :ctype:`float` datatype. The constructor accepts an + Represents the C :c:type:`float` datatype. The constructor accepts an optional float initializer. .. class:: c_int - Represents the C :ctype:`signed int` datatype. The constructor accepts an + Represents the C :c:type:`signed int` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. On platforms where ``sizeof(int) == sizeof(long)`` it is an alias to :class:`c_long`. .. class:: c_int8 - Represents the C 8-bit :ctype:`signed int` datatype. Usually an alias for + Represents the C 8-bit :c:type:`signed int` datatype. Usually an alias for :class:`c_byte`. .. class:: c_int16 - Represents the C 16-bit :ctype:`signed int` datatype. Usually an alias for + Represents the C 16-bit :c:type:`signed int` datatype. Usually an alias for :class:`c_short`. .. class:: c_int32 - Represents the C 32-bit :ctype:`signed int` datatype. Usually an alias for + Represents the C 32-bit :c:type:`signed int` datatype. Usually an alias for :class:`c_int`. .. class:: c_int64 - Represents the C 64-bit :ctype:`signed int` datatype. Usually an alias for + Represents the C 64-bit :c:type:`signed int` datatype. Usually an alias for :class:`c_longlong`. .. class:: c_long - Represents the C :ctype:`signed long` datatype. The constructor accepts an + Represents the C :c:type:`signed long` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_longlong - Represents the C :ctype:`signed long long` datatype. The constructor accepts + Represents the C :c:type:`signed long long` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_short - Represents the C :ctype:`signed short` datatype. The constructor accepts an + Represents the C :c:type:`signed short` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_size_t - Represents the C :ctype:`size_t` datatype. + Represents the C :c:type:`size_t` datatype. + + +.. class:: c_ssize_t + + Represents the C :c:type:`ssize_t` datatype. + + .. versionadded:: 3.2 .. class:: c_ubyte - Represents the C :ctype:`unsigned char` datatype, it interprets the value as + Represents the C :c:type:`unsigned char` datatype, it interprets the value as small integer. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_uint - Represents the C :ctype:`unsigned int` datatype. The constructor accepts an + Represents the C :c:type:`unsigned int` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. On platforms where ``sizeof(int) == sizeof(long)`` it is an alias for :class:`c_ulong`. .. class:: c_uint8 - Represents the C 8-bit :ctype:`unsigned int` datatype. Usually an alias for + Represents the C 8-bit :c:type:`unsigned int` datatype. Usually an alias for :class:`c_ubyte`. .. class:: c_uint16 - Represents the C 16-bit :ctype:`unsigned int` datatype. Usually an alias for + Represents the C 16-bit :c:type:`unsigned int` datatype. Usually an alias for :class:`c_ushort`. .. class:: c_uint32 - Represents the C 32-bit :ctype:`unsigned int` datatype. Usually an alias for + Represents the C 32-bit :c:type:`unsigned int` datatype. Usually an alias for :class:`c_uint`. .. class:: c_uint64 - Represents the C 64-bit :ctype:`unsigned int` datatype. Usually an alias for + Represents the C 64-bit :c:type:`unsigned int` datatype. Usually an alias for :class:`c_ulonglong`. .. class:: c_ulong - Represents the C :ctype:`unsigned long` datatype. The constructor accepts an + Represents the C :c:type:`unsigned long` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_ulonglong - Represents the C :ctype:`unsigned long long` datatype. The constructor + Represents the C :c:type:`unsigned long long` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_ushort - Represents the C :ctype:`unsigned short` datatype. The constructor accepts + Represents the C :c:type:`unsigned short` datatype. The constructor accepts an optional integer initializer; no overflow checking is done. .. class:: c_void_p - Represents the C :ctype:`void *` type. The value is represented as integer. + Represents the C :c:type:`void *` type. The value is represented as integer. The constructor accepts an optional integer initializer. .. class:: c_wchar - Represents the C :ctype:`wchar_t` datatype, and interprets the value as a + Represents the C :c:type:`wchar_t` datatype, and interprets the value as a single character unicode string. The constructor accepts an optional string initializer, the length of the string must be exactly one character. .. class:: c_wchar_p - Represents the C :ctype:`wchar_t *` datatype, which must be a pointer to a + Represents the C :c:type:`wchar_t *` datatype, which must be a pointer to a zero-terminated wide character string. The constructor accepts an integer address, or a string. .. class:: c_bool - Represent the C :ctype:`bool` datatype (more accurately, :ctype:`_Bool` from + Represent the C :c:type:`bool` datatype (more accurately, :c:type:`_Bool` from C99). Its value can be True or False, and the constructor accepts any object that has a truth value. .. class:: HRESULT - Windows only: Represents a :ctype:`HRESULT` value, which contains success or + Windows only: Represents a :c:type:`HRESULT` value, which contains success or error information for a function or method call. .. class:: py_object - Represents the C :ctype:`PyObject *` datatype. Calling this without an - argument creates a ``NULL`` :ctype:`PyObject *` pointer. + Represents the C :c:type:`PyObject *` datatype. Calling this without an + argument creates a ``NULL`` :c:type:`PyObject *` pointer. The :mod:`ctypes.wintypes` module provides quite some other Windows specific -data types, for example :ctype:`HWND`, :ctype:`WPARAM`, or :ctype:`DWORD`. Some -useful structures like :ctype:`MSG` or :ctype:`RECT` are also defined. +data types, for example :c:type:`HWND`, :c:type:`WPARAM`, or :c:type:`DWORD`. Some +useful structures like :c:type:`MSG` or :c:type:`RECT` are also defined. .. _ctypes-structured-data-types: diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index 6d2baa0..1ca22c8 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -49,7 +49,7 @@ Linux and the BSD variants of Unix. Tutorial material on using curses with Python, by Andrew Kuchling and Eric Raymond. - The :file:`Demo/curses/` directory in the Python source distribution contains + The :file:`Tools/demo/` directory in the Python source distribution contains some example programs using the curses bindings provided by this module. diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 26d9946..de9ad44 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -28,11 +28,14 @@ For applications requiring more, :class:`datetime` and :class:`time` objects have an optional time zone information member, :attr:`tzinfo`, that can contain an instance of a subclass of the abstract :class:`tzinfo` class. These :class:`tzinfo` objects capture information about the offset from UTC time, the -time zone name, and whether Daylight Saving Time is in effect. Note that no -concrete :class:`tzinfo` classes are supplied by the :mod:`datetime` module. -Supporting timezones at whatever level of detail is required is up to the -application. The rules for time adjustment across the world are more political -than rational, and there is no standard suitable for every application. +time zone name, and whether Daylight Saving Time is in effect. Note that only +one concrete :class:`tzinfo` class, the :class:`timezone` class, is supplied by the +:mod:`datetime` module. The :class:`timezone` class can reprsent simple +timezones with fixed offset from UTC such as UTC itself or North American EST and +EDT timezones. Supporting timezones at whatever level of detail is +required is up to the application. The rules for time adjustment across the +world are more political than rational, change frequently, and there is no +standard suitable for every application aside from UTC. The :mod:`datetime` module exports the following constants: @@ -99,6 +102,14 @@ Available Types time adjustment (for example, to account for time zone and/or daylight saving time). +.. class:: timezone + + A class that implements the :class:`tzinfo` abstract base class as a + fixed offset from the UTC. + + .. versionadded:: 3.2 + + Objects of these types are immutable. Objects of the :class:`date` type are always naive. @@ -116,6 +127,7 @@ Subclass relationships:: object timedelta tzinfo + timezone time date datetime @@ -220,8 +232,28 @@ Supported operations: | | In general, *t1* \* i == *t1* \* (i-1) + *t1* | | | is true. (1) | +--------------------------------+-----------------------------------------------+ -| ``t1 = t2 // i`` | The floor is computed and the remainder (if | -| | any) is thrown away. (3) | +| ``t1 = t2 * f or t1 = f * t2`` | Delta multiplied by a float. The result is | +| | rounded to the nearest multiple of | +| | timedelta.resolution using round-half-to-even.| ++--------------------------------+-----------------------------------------------+ +| ``f = t2 / t3`` | Division (3) of *t2* by *t3*. Returns a | +| | :class:`float` object. | ++--------------------------------+-----------------------------------------------+ +| ``t1 = t2 / f or t1 = t2 / i`` | Delta divided by a float or an int. The result| +| | is rounded to the nearest multiple of | +| | timedelta.resolution using round-half-to-even.| ++--------------------------------+-----------------------------------------------+ +| ``t1 = t2 // i`` or | The floor is computed and the remainder (if | +| ``t1 = t2 // t3`` | any) is thrown away. In the second case, an | +| | integer is returned. (3) | ++--------------------------------+-----------------------------------------------+ +| ``t1 = t2 % t3`` | The remainder is computed as a | +| | :class:`timedelta` object. (3) | ++--------------------------------+-----------------------------------------------+ +| ``q, r = divmod(t1, t2)`` | Computes the quotient and the remainder: | +| | ``q = t1 // t2`` (3) and ``r = t1 % t2``. | +| | q is an integer and r is a :class:`timedelta` | +| | object. | +--------------------------------+-----------------------------------------------+ | ``+t1`` | Returns a :class:`timedelta` object with the | | | same value. (2) | @@ -270,6 +302,13 @@ In addition to the operations listed above :class:`timedelta` objects support certain additions and subtractions with :class:`date` and :class:`datetime` objects (see below). +.. versionchanged:: 3.2 + Floor division and true division of a :class:`timedelta` object by another + :class:`timedelta` object are now supported, as are remainder operations and + the :func:`divmod` function. True division and multiplication of a + :class:`timedelta` object by a :class:`float` object are now supported. + + Comparisons of :class:`timedelta` objects are supported with the :class:`timedelta` object representing the smaller duration considered to be the smaller timedelta. In order to stop mixed-type comparisons from falling back to @@ -282,12 +321,27 @@ comparison is ``==`` or ``!=``. The latter cases return :const:`False` or efficient pickling, and in Boolean contexts, a :class:`timedelta` object is considered to be true if and only if it isn't equal to ``timedelta(0)``. +Instance methods: + +.. method:: timedelta.total_seconds() + + Return the total number of seconds contained in the duration. Equivalent to + ``td / timedelta(seconds=1)``. + + Note that for very large time intervals (greater than 270 years on + most platforms) this method will lose microsecond accuracy. + + .. versionadded:: 3.2 + + Example usage: >>> from datetime import timedelta >>> year = timedelta(days=365) >>> another_year = timedelta(weeks=40, days=84, hours=23, ... minutes=50, seconds=600) # adds up to 365 days + >>> year.total_seconds() + 31536000.0 >>> year == another_year True >>> ten_years = 10 * year @@ -342,7 +396,7 @@ Other constructors, all class methods: Return the local date corresponding to the POSIX timestamp, such as is returned by :func:`time.time`. This may raise :exc:`ValueError`, if the timestamp is out - of the range of values supported by the platform C :cfunc:`localtime` function. + of the range of values supported by the platform C :c:func:`localtime` function. It's common for this to be restricted to years from 1970 through 2038. Note that on non-POSIX systems that include leap seconds in their notion of a timestamp, leap seconds are ignored by :meth:`fromtimestamp`. @@ -516,7 +570,7 @@ Instance methods: Return a string representing the date, for example ``date(2002, 12, 4).ctime() == 'Wed Dec 4 00:00:00 2002'``. ``d.ctime()`` is equivalent to ``time.ctime(time.mktime(d.timetuple()))`` on platforms where the native C - :cfunc:`ctime` function (which :func:`time.ctime` invokes, but which + :c:func:`ctime` function (which :func:`time.ctime` invokes, but which :meth:`date.ctime` does not invoke) conforms to the C standard. @@ -623,7 +677,7 @@ Other constructors, all class methods: or not specified, this is like :meth:`today`, but, if possible, supplies more precision than can be gotten from going through a :func:`time.time` timestamp (for example, this may be possible on platforms supplying the C - :cfunc:`gettimeofday` function). + :c:func:`gettimeofday` function). Else *tz* must be an instance of a class :class:`tzinfo` subclass, and the current date and time are converted to *tz*'s time zone. In this case the @@ -635,8 +689,8 @@ Other constructors, all class methods: Return the current UTC date and time, with :attr:`tzinfo` ``None``. This is like :meth:`now`, but returns the current UTC date and time, as a naive - :class:`datetime` object. See also :meth:`now`. - + :class:`datetime` object. An aware current UTC datetime can be obtained by + calling ``datetime.now(timezone.utc)``. See also :meth:`now`. .. classmethod:: datetime.fromtimestamp(timestamp, tz=None) @@ -651,8 +705,8 @@ Other constructors, all class methods: ``tz.fromutc(datetime.utcfromtimestamp(timestamp).replace(tzinfo=tz))``. :meth:`fromtimestamp` may raise :exc:`ValueError`, if the timestamp is out of - the range of values supported by the platform C :cfunc:`localtime` or - :cfunc:`gmtime` functions. It's common for this to be restricted to years in + the range of values supported by the platform C :c:func:`localtime` or + :c:func:`gmtime` functions. It's common for this to be restricted to years in 1970 through 2038. Note that on non-POSIX systems that include leap seconds in their notion of a timestamp, leap seconds are ignored by :meth:`fromtimestamp`, and then it's possible to have two timestamps differing by a second that yield @@ -663,7 +717,7 @@ Other constructors, all class methods: Return the UTC :class:`datetime` corresponding to the POSIX timestamp, with :attr:`tzinfo` ``None``. This may raise :exc:`ValueError`, if the timestamp is - out of the range of values supported by the platform C :cfunc:`gmtime` function. + out of the range of values supported by the platform C :c:func:`gmtime` function. It's common for this to be restricted to years in 1970 through 2038. See also :meth:`fromtimestamp`. @@ -927,7 +981,7 @@ Instance methods: of the result is set according to the :meth:`dst` method: :attr:`tzinfo` is ``None`` or :meth:`dst` returns ``None``, :attr:`tm_isdst` is set to ``-1``; else if :meth:`dst` returns a non-zero value, :attr:`tm_isdst` is set to ``1``; - else ``tm_isdst`` is set to ``0``. + else :attr:`tm_isdst` is set to ``0``. .. method:: datetime.utctimetuple() @@ -937,10 +991,10 @@ Instance methods: ``d.dst()`` returns. DST is never in effect for a UTC time. If *d* is aware, *d* is normalized to UTC time, by subtracting - ``d.utcoffset()``, and a :class:`time.struct_time` for the normalized time is - returned. :attr:`tm_isdst` is forced to 0. Note that the result's - :attr:`tm_year` member may be :const:`MINYEAR`\ -1 or :const:`MAXYEAR`\ +1, if - *d*.year was ``MINYEAR`` or ``MAXYEAR`` and UTC adjustment spills over a year + ``d.utcoffset()``, and a :class:`time.struct_time` for the + normalized time is returned. :attr:`tm_isdst` is forced to 0. Note + that an :exc:`OverflowError` may be raised if *d*.year was + ``MINYEAR`` or ``MAXYEAR`` and UTC adjustment spills over a year boundary. @@ -1002,7 +1056,7 @@ Instance methods: Return a string representing the date and time, for example ``datetime(2002, 12, 4, 20, 30, 40).ctime() == 'Wed Dec 4 20:30:40 2002'``. ``d.ctime()`` is equivalent to ``time.ctime(time.mktime(d.timetuple()))`` on platforms where the - native C :cfunc:`ctime` function (which :func:`time.ctime` invokes, but which + native C :c:func:`ctime` function (which :func:`time.ctime` invokes, but which :meth:`datetime.ctime` does not invoke) conforms to the C standard. @@ -1293,8 +1347,10 @@ Example: :class:`tzinfo` is an abstract base class, meaning that this class should not be instantiated directly. You need to derive a concrete subclass, and (at least) supply implementations of the standard :class:`tzinfo` methods needed by the -:class:`datetime` methods you use. The :mod:`datetime` module does not supply -any concrete subclasses of :class:`tzinfo`. +:class:`datetime` methods you use. The :mod:`datetime` module supplies +a simple concrete subclass of :class:`tzinfo` :class:`timezone` which can reprsent +timezones with fixed offset from UTC such as UTC itself or North American EST and +EDT. An instance of (a concrete subclass of) :class:`tzinfo` can be passed to the constructors for :class:`datetime` and :class:`time` objects. The latter objects @@ -1312,7 +1368,7 @@ methods. Exactly which methods are needed depends on the uses made of aware :mod:`datetime` objects. If in doubt, simply implement all of them. -.. method:: tzinfo.utcoffset(self, dt) +.. method:: tzinfo.utcoffset(dt) Return offset of local time from UTC, in minutes east of UTC. If local time is west of UTC, this should be negative. Note that this is intended to be the @@ -1334,7 +1390,7 @@ methods. Exactly which methods are needed depends on the uses made of aware :exc:`NotImplementedError`. -.. method:: tzinfo.dst(self, dt) +.. method:: tzinfo.dst(dt) Return the daylight saving time (DST) adjustment, in minutes east of UTC, or ``None`` if DST information isn't known. Return ``timedelta(0)`` if DST is not @@ -1382,7 +1438,7 @@ methods. Exactly which methods are needed depends on the uses made of aware The default implementation of :meth:`dst` raises :exc:`NotImplementedError`. -.. method:: tzinfo.tzname(self, dt) +.. method:: tzinfo.tzname(dt) Return the time zone name corresponding to the :class:`datetime` object *dt*, as a string. Nothing about string names is defined by the :mod:`datetime` module, @@ -1418,7 +1474,7 @@ time, and not need worry about objects in other timezones. There is one more :class:`tzinfo` method that a subclass may wish to override: -.. method:: tzinfo.fromutc(self, dt) +.. method:: tzinfo.fromutc(dt) This is called from the default :class:`datetime.astimezone()` implementation. When called from that, ``dt.tzinfo`` is *self*, and *dt*'s date and time members @@ -1495,9 +1551,65 @@ arranged, as in the example, by expressing DST switch times in the time zone's standard local time. Applications that can't bear such ambiguities should avoid using hybrid -:class:`tzinfo` subclasses; there are no ambiguities when using UTC, or any -other fixed-offset :class:`tzinfo` subclass (such as a class representing only -EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). +:class:`tzinfo` subclasses; there are no ambiguities when using :class:`timezone`, +or any other fixed-offset :class:`tzinfo` subclass (such as a class representing +only EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). + + +.. _datetime-timezone: + +:class:`timezone` Objects +-------------------------- + +A :class:`timezone` object represents a timezone that is defined by a +fixed offset from UTC. Note that objects of this class cannot be used +to represent timezone information in the locations where different +offsets are used in different days of the year or where historical +changes have been made to civil time. + + +.. class:: timezone(offset[, name]) + + The *offset* argument must be specified as a :class:`timedelta` + object representing the difference between the local time and UTC. It must + be strictly between ``-timedelta(hours=24)`` and + ``timedelta(hours=24)`` and represent a whole number of minutes, + otherwise :exc:`ValueError` is raised. + + The *name* argument is optional. If specified it must be a string that + is used as the value returned by the ``tzname(dt)`` method. Otherwise, + ``tzname(dt)`` returns a string 'UTCsHH:MM', where s is the sign of + *offset*, HH and MM are two digits of ``offset.hours`` and + ``offset.minutes`` respectively. + +.. method:: timezone.utcoffset(dt) + + Return the fixed value specified when the :class:`timezone` instance is + constructed. The *dt* argument is ignored. The return value is a + :class:`timedelta` instance equal to the difference between the + local time and UTC. + +.. method:: timezone.tzname(dt) + + Return the fixed value specified when the :class:`timezone` instance is + constructed or a string 'UTCsHH:MM', where s is the sign of + *offset*, HH and MM are two digits of ``offset.hours`` and + ``offset.minutes`` respectively. + +.. method:: timezone.dst(dt) + + Always returns ``None``. + +.. method:: timezone.fromutc(dt) + + Return ``dt + offset``. The *dt* argument must be an aware + :class:`datetime` instance, with ``tzinfo`` set to ``self``. + +Class attributes: + +.. attribute:: timezone.utc + + The UTC timezone, ``timezone(timedelta(0))``. .. _strftime-strptime-behavior: @@ -1549,9 +1661,6 @@ version) requires, and these work on all platforms with a standard C implementation. Note that the 1999 version of the C standard added additional format codes. -The exact range of years for which :meth:`strftime` works also varies across -platforms. Regardless of platform, years before 1900 cannot be used. - +-----------+--------------------------------+-------+ | Directive | Meaning | Notes | +===========+================================+=======+ @@ -1594,7 +1703,7 @@ platforms. Regardless of platform, years before 1900 cannot be used. | | AM or PM. | | +-----------+--------------------------------+-------+ | ``%S`` | Second as a decimal number | \(3) | -| | [00,61]. | | +| | [00,59]. | | +-----------+--------------------------------+-------+ | ``%U`` | Week number of the year | \(4) | | | (Sunday as the first day of | | @@ -1624,10 +1733,11 @@ platforms. Regardless of platform, years before 1900 cannot be used. | ``%y`` | Year without century as a | | | | decimal number [00,99]. | | +-----------+--------------------------------+-------+ -| ``%Y`` | Year with century as a decimal | | -| | number. | | +| ``%Y`` | Year with century as a decimal | \(5) | +| | number [0001,9999] (strptime), | | +| | [1000,9999] (strftime). | | +-----------+--------------------------------+-------+ -| ``%z`` | UTC offset in the form +HHMM | \(5) | +| ``%z`` | UTC offset in the form +HHMM | \(6) | | | or -HHMM (empty string if the | | | | the object is naive). | | +-----------+--------------------------------+-------+ @@ -1651,17 +1761,30 @@ Notes: the output hour field if the ``%I`` directive is used to parse the hour. (3) - The range really is ``0`` to ``61``; according to the Posix standard this - accounts for leap seconds and the (very rare) double leap seconds. - The :mod:`time` module may produce and does accept leap seconds since - it is based on the Posix standard, but the :mod:`datetime` module - does not accept leap seconds in :meth:`strptime` input nor will it - produce them in :func:`strftime` output. + Unlike :mod:`time` module, :mod:`datetime` module does not support + leap seconds. (4) When used with the :meth:`strptime` method, ``%U`` and ``%W`` are only used in calculations when the day of the week and the year are specified. (5) + For technical reasons, :meth:`strftime` method does not support + dates before year 1000: ``t.strftime(format)`` will raise a + :exc:`ValueError` when ``t.year < 1000`` even if ``format`` does + not contain ``%Y`` directive. The :meth:`strptime` method can + parse years in the full [1, 9999] range, but years < 1000 must be + zero-filled to 4-digit width. + + .. versionchanged:: 3.2 + In previous versions, :meth:`strftime` method was restricted to + years >= 1900. + +(6) For example, if :meth:`utcoffset` returns ``timedelta(hours=-3, minutes=-30)``, ``%z`` is replaced with the string ``'-0330'``. + +.. versionchanged:: 3.2 + When the ``%z`` directive is provided to the :meth:`strptime` method, an + aware :class:`datetime` object will be produced. The ``tzinfo`` of the + result will be set to a :class:`timezone` instance. diff --git a/Doc/library/dbm.rst b/Doc/library/dbm.rst index cbefc1a..e3d50b9 100644 --- a/Doc/library/dbm.rst +++ b/Doc/library/dbm.rst @@ -61,10 +61,15 @@ the Oracle Berkeley DB. modified by the prevailing umask). -The object returned by :func:`.open` supports most of the same functionality as +The object returned by :func:`.open` supports the same basic functionality as dictionaries; keys and their corresponding values can be stored, retrieved, and deleted, and the :keyword:`in` operator and the :meth:`keys` method are -available. Key and values are always stored as bytes. This means that when +available, as well as :meth:`get` and :meth:`setdefault`. + +.. versionchanged:: 3.2 + :meth:`get` and :meth:`setdefault` are now available in all database modules. + +Key and values are always stored as bytes. This means that when strings are used they are implicitly converted to the default encoding before being stored. diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index 758dcce..9d5b32f 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -123,15 +123,14 @@ precision, rounding, or enabled traps:: >>> from decimal import * >>> getcontext() Context(prec=28, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999, - capitals=1, flags=[], traps=[Overflow, DivisionByZero, + capitals=1, clamp=0, flags=[], traps=[Overflow, DivisionByZero, InvalidOperation]) >>> getcontext().prec = 7 # Set a new precision -Decimal instances can be constructed from integers, strings, or tuples. To -create a Decimal from a :class:`float`, first convert it to a string. This -serves as an explicit reminder of the details of the conversion (including -representation error). Decimal numbers include special values such as +Decimal instances can be constructed from integers, strings, floats, or tuples. +Construction from an integer or a float performs an exact conversion of the +value of that integer or float. Decimal numbers include special values such as :const:`NaN` which stands for "Not a number", positive and negative :const:`Infinity`, and :const:`-0`. @@ -140,10 +139,12 @@ representation error). Decimal numbers include special values such as Decimal('10') >>> Decimal('3.14') Decimal('3.14') + >>> Decimal(3.14) + Decimal('3.140000000000000124344978758017532527446746826171875') >>> Decimal((0, (3, 1, 4), -2)) Decimal('3.14') >>> Decimal(str(2.0 ** 0.5)) - Decimal('1.41421356237') + Decimal('1.4142135623730951') >>> Decimal(2) ** Decimal('0.5') Decimal('1.414213562373095048801688724') >>> Decimal('NaN') @@ -244,7 +245,7 @@ enabled: >>> ExtendedContext Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999, - capitals=1, flags=[], traps=[]) + capitals=1, clamp=0, flags=[], traps=[]) >>> setcontext(ExtendedContext) >>> Decimal(1) / Decimal(7) Decimal('0.142857143') @@ -269,7 +270,7 @@ using the :meth:`clear_flags` method. :: Decimal('3.14159292') >>> getcontext() Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999, - capitals=1, flags=[Inexact, Rounded], traps=[]) + capitals=1, clamp=0, flags=[Inexact, Rounded], traps=[]) The *flags* entry shows that the rational approximation to :const:`Pi` was rounded (digits beyond the context precision were thrown away) and that the @@ -309,7 +310,7 @@ Decimal objects Construct a new :class:`Decimal` object based from *value*. - *value* can be an integer, string, tuple, or another :class:`Decimal` + *value* can be an integer, string, tuple, :class:`float`, or another :class:`Decimal` object. If no *value* is given, returns ``Decimal('0')``. If *value* is a string, it should conform to the decimal numeric string syntax after leading and trailing whitespace characters are removed:: @@ -335,6 +336,12 @@ Decimal objects digits, and an integer exponent. For example, ``Decimal((0, (1, 4, 1, 4), -3))`` returns ``Decimal('1.414')``. + If *value* is a :class:`float`, the binary floating point value is losslessly + converted to its exact decimal equivalent. This conversion can often require + 53 or more digits of precision. For example, ``Decimal(float('1.1'))`` + converts to + ``Decimal('1.100000000000000088817841970012523233890533447265625')``. + The *context* precision does not affect how many digits are stored. That is determined exclusively by the number of digits in *value*. For example, ``Decimal('3.00000')`` records all five zeros even if the context precision is @@ -347,6 +354,10 @@ Decimal objects Once constructed, :class:`Decimal` objects are immutable. + .. versionchanged:: 3.2 + The argument to the constructor is now permitted to be a :class:`float` + instance. + Decimal floating point objects share many properties with the other built-in numeric types such as :class:`float` and :class:`int`. All of the usual math operations and special methods apply. Likewise, decimal objects can be @@ -354,6 +365,18 @@ Decimal objects compared, sorted, and coerced to another type (such as :class:`float` or :class:`int`). + Decimal objects cannot generally be combined with floats or + instances of :class:`fractions.Fraction` in arithmetic operations: + an attempt to add a :class:`Decimal` to a :class:`float`, for + example, will raise a :exc:`TypeError`. However, it is possible to + use Python's comparison operators to compare a :class:`Decimal` + instance ``x`` with another number ``y``. This avoids confusing results + when doing equality comparisons between numbers of different types. + + .. versionchanged:: 3.2 + Mixed-type comparisons between :class:`Decimal` instances and other + numeric types are now fully supported. + In addition to the standard numeric properties, decimal floating point objects also have a number of specialized methods: @@ -468,6 +491,9 @@ Decimal objects `0x1.999999999999ap-4`. That equivalent value in decimal is `0.1000000000000000055511151231257827021181583404541015625`. + .. note:: From Python 3.2 onwards, a :class:`Decimal` instance + can also be constructed directly from a :class:`float`. + .. doctest:: >>> Decimal.from_float(0.1) @@ -861,7 +887,7 @@ In addition to the three supplied contexts, new contexts can be created with the :class:`Context` constructor. -.. class:: Context(prec=None, rounding=None, traps=None, flags=None, Emin=None, Emax=None, capitals=1) +.. class:: Context(prec=None, rounding=None, traps=None, flags=None, Emin=None, Emax=None, capitals=None, clamp=None) Creates a new context. If a field is not specified or is :const:`None`, the default values are copied from the :const:`DefaultContext`. If the *flags* @@ -892,13 +918,33 @@ In addition to the three supplied contexts, new contexts can be created with the :const:`1`, exponents are printed with a capital :const:`E`; otherwise, a lowercase :const:`e` is used: :const:`Decimal('6.02e+23')`. + The *clamp* field is either :const:`0` (the default) or :const:`1`. + If set to :const:`1`, the exponent ``e`` of a :class:`Decimal` + instance representable in this context is strictly limited to the + range ``Emin - prec + 1 <= e <= Emax - prec + 1``. If *clamp* is + :const:`0` then a weaker condition holds: the adjusted exponent of + the :class:`Decimal` instance is at most ``Emax``. When *clamp* is + :const:`1`, a large normal number will, where possible, have its + exponent reduced and a corresponding number of zeros added to its + coefficient, in order to fit the exponent constraints; this + preserves the value of the number but loses information about + significant trailing zeros. For example:: + + >>> Context(prec=6, Emax=999, clamp=1).create_decimal('1.23e999') + Decimal('1.23000E+999') + + A *clamp* value of :const:`1` allows compatibility with the + fixed-width decimal interchange formats specified in IEEE 754. The :class:`Context` class defines several general purpose methods as well as a large number of methods for doing arithmetic directly in a given context. In addition, for each of the :class:`Decimal` methods described above (with the exception of the :meth:`adjusted` and :meth:`as_tuple` methods) there is - a corresponding :class:`Context` method. For example, ``C.exp(x)`` is - equivalent to ``x.exp(context=C)``. + a corresponding :class:`Context` method. For example, for a :class:`Context` + instance ``C`` and :class:`Decimal` instance ``x``, ``C.exp(x)`` is + equivalent to ``x.exp(context=C)``. Each :class:`Context` method accepts a + Python integer (an instance of :class:`int`) anywhere that a + Decimal instance is accepted. .. method:: clear_flags() @@ -963,7 +1009,6 @@ In addition to the three supplied contexts, new contexts can be created with the value for subnormal results. When underflow occurs, the exponent is set to :const:`Etiny`. - .. method:: Etop() Returns a value equal to ``Emax - prec + 1``. @@ -1612,7 +1657,8 @@ to work with the :class:`Decimal` class:: build(trailneg) for i in range(places): build(next() if digits else '0') - build(dp) + if places: + build(dp) if not digits: build('0') i = 0 @@ -1672,6 +1718,9 @@ to work with the :class:`Decimal` class:: def cos(x): """Return the cosine of x as measured in radians. + The Taylor series approximation works best for a small value of x. + For larger values, first compute x = x % (2 * pi). + >>> print(cos(Decimal('0.5'))) 0.8775825618903727161162815826 >>> print(cos(0.5)) @@ -1695,6 +1744,9 @@ to work with the :class:`Decimal` class:: def sin(x): """Return the sine of x as measured in radians. + The Taylor series approximation works best for a small value of x. + For larger values, first compute x = x % (2 * pi). + >>> print(sin(Decimal('0.5'))) 0.4794255386042030002732879352 >>> print(sin(0.5)) @@ -1821,37 +1873,15 @@ value unchanged: Q. Is there a way to convert a regular float to a :class:`Decimal`? -A. Yes, all binary floating point numbers can be exactly expressed as a -Decimal. An exact conversion may take more precision than intuition would -suggest, so we trap :const:`Inexact` to signal a need for more precision: - -.. testcode:: - - def float_to_decimal(f): - "Convert a floating point number to a Decimal with no loss of information" - n, d = f.as_integer_ratio() - with localcontext() as ctx: - ctx.traps[Inexact] = True - while True: - try: - return Decimal(n) / Decimal(d) - except Inexact: - ctx.prec += 1 +A. Yes, any binary floating point number can be exactly expressed as a +Decimal though an exact conversion may take more precision than intuition would +suggest: .. doctest:: - >>> float_to_decimal(math.pi) + >>> Decimal(math.pi) Decimal('3.141592653589793115997963468544185161590576171875') -Q. Why isn't the :func:`float_to_decimal` routine included in the module? - -A. There is some question about whether it is advisable to mix binary and -decimal floating point. Also, its use requires some care to avoid the -representation issues associated with binary floating point: - - >>> float_to_decimal(1.1) - Decimal('1.100000000000000088817841970012523233890533447265625') - Q. Within a complex calculation, how can I make sure that I haven't gotten a spurious result because of insufficient precision or rounding anomalies. diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index 6dea8c1..bdc37b3 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -17,6 +17,7 @@ can be used for example, for comparing files, and can produce difference information in various formats, including HTML and context and unified diffs. For comparing directories and files, see also, the :mod:`filecmp` module. + .. class:: SequenceMatcher This is a flexible class for comparing pairs of sequences of any type, so long @@ -35,11 +36,17 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. complicated way on how many elements the sequences have in common; best case time is linear. - **Heuristic:** To speed-up matching, items whose duplicates appear more than 1% of - the time in sequences of at least 200 items are treated as junk. This has the - unfortunate side-effect of giving bad results for sequences constructed from - a small set of items. An option to turn off the heuristic will be added to - Python 3.2. + **Automatic junk heuristic:** :class:`SequenceMatcher` supports a heuristic that + automatically treats certain sequence items as junk. The heuristic counts how many + times each individual item appears in the sequence. If an item's duplicates (after + the first one) account for more than 1% of the sequence and the sequence is at least + 200 items long, this item is marked as "popular" and is treated as junk for + the purpose of sequence matching. This heuristic can be turned off by setting + the ``autojunk`` argument to ``False`` when creating the :class:`SequenceMatcher`. + + .. versionadded:: 3.2 + The *autojunk* parameter. + .. class:: Differ @@ -145,8 +152,8 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The context diff format normally has a header for filenames and modification times. Any or all of these may be specified using strings for *fromfile*, - *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally - expressed in the format returned by :func:`time.ctime`. If not specified, the + *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally + expressed in the ISO 8601 format. If not specified, the strings default to blanks. >>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n'] @@ -277,8 +284,8 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The context diff format normally has a header for filenames and modification times. Any or all of these may be specified using strings for *fromfile*, - *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally - expressed in the format returned by :func:`time.ctime`. If not specified, the + *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally + expressed in the ISO 8601 format. If not specified, the strings default to blanks. @@ -329,7 +336,7 @@ SequenceMatcher Objects The :class:`SequenceMatcher` class has this constructor: -.. class:: SequenceMatcher(isjunk=None, a='', b='') +.. class:: SequenceMatcher(isjunk=None, a='', b='', autojunk=True) Optional argument *isjunk* must be ``None`` (the default) or a one-argument function that takes a sequence element and returns true if and only if the @@ -345,6 +352,22 @@ The :class:`SequenceMatcher` class has this constructor: The optional arguments *a* and *b* are sequences to be compared; both default to empty strings. The elements of both sequences must be :term:`hashable`. + The optional argument *autojunk* can be used to disable the automatic junk + heuristic. + + .. versionadded:: 3.2 + The *autojunk* parameter. + + SequenceMatcher objects get three data attributes: *bjunk* is the + set of elements of *b* for which *isjunk* is True; *bpopular* is the set of + non-junk elements considered popular by the heuristic (if it is not + disabled); *b2j* is a dict mapping the remaining elements of *b* to a list + of positions where they occur. All three are reset whenever *b* is reset + with :meth:`set_seqs` or :meth:`set_seq2`. + + .. versionadded:: 3.2 + The *bjunk* and *bpopular* attributes. + :class:`SequenceMatcher` objects have the following methods: .. method:: set_seqs(a, b) @@ -460,13 +483,15 @@ The :class:`SequenceMatcher` class has this constructor: >>> b = "abycdf" >>> s = SequenceMatcher(None, a, b) >>> for tag, i1, i2, j1, j2 in s.get_opcodes(): - ... print(("%7s a[%d:%d] (%s) b[%d:%d] (%s)" % - ... (tag, i1, i2, a[i1:i2], j1, j2, b[j1:j2]))) - delete a[0:1] (q) b[0:0] () - equal a[1:3] (ab) b[0:2] (ab) - replace a[3:4] (x) b[2:3] (y) - equal a[4:6] (cd) b[3:5] (cd) - insert a[6:6] () b[5:6] (f) + print('{:7} a[{}:{}] --> b[{}:{}] {!r:>8} --> {!r}'.format( + tag, i1, i2, j1, j2, a[i1:i2], b[j1:j2])) + + + delete a[0:1] --> b[0:0] 'q' --> '' + equal a[1:3] --> b[0:2] 'ab' --> 'ab' + replace a[3:4] --> b[2:3] 'x' --> 'y' + equal a[4:6] --> b[3:5] 'cd' --> 'cd' + insert a[6:6] --> b[5:6] '' --> 'f' .. method:: get_grouped_opcodes(n=3) @@ -524,7 +549,7 @@ different results due to differing levels of approximation, although SequenceMatcher Examples ------------------------ -This example compares two strings, considering blanks to be "junk:" +This example compares two strings, considering blanks to be "junk": >>> s = SequenceMatcher(lambda x: x == " ", ... "private Thread currentThread;", diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 4aae06b..79cc583 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -4,6 +4,9 @@ .. module:: dis :synopsis: Disassembler for Python bytecode. +**Source code:** :source:`Lib/dis.py` + +-------------- The :mod:`dis` module supports the analysis of CPython :term:`bytecode` by disassembling it. The CPython bytecode which this module takes as an @@ -12,7 +15,7 @@ and the interpreter. .. impl-detail:: - Bytecode is an implementation detail of the CPython interpreter! No + Bytecode is an implementation detail of the CPython interpreter. No guarantees are made that bytecode will not be added, removed, or changed between versions of Python. Use of this module should not be considered to work across Python VMs or Python releases. @@ -36,6 +39,28 @@ the following command can be used to get the disassembly of :func:`myfunc`:: The :mod:`dis` module defines the following functions and constants: +.. function:: code_info(x) + + Return a formatted multi-line string with detailed code object information + for the supplied function, method, source code string or code object. + + Note that the exact contents of code info strings are highly implementation + dependent and they may change arbitrarily across Python VMs or Python + releases. + + .. versionadded:: 3.2 + + +.. function:: show_code(x) + + Print detailed code object information for the supplied function, method, + source code string or code object to stdout. + + This is a convenient shorthand for ``print(code_info(x))``, intended for + interactive exploration at the interpreter prompt. + + .. versionadded:: 3.2 + .. function:: dis(x=None) Disassemble the *x* object. *x* can denote either a module, a class, a @@ -172,15 +197,15 @@ The Python compiler currently generates the following bytecode instructions. three. -.. opcode:: ROT_FOUR +.. opcode:: DUP_TOP - Lifts second, third and forth stack item one position up, moves top down to - position four. + Duplicates the reference on top of the stack. -.. opcode:: DUP_TOP +.. opcode:: DUP_TOP_TWO - Duplicates the reference on top of the stack. + Duplicates the two references on top of the stack, leaving them in the + same order. **Unary operations** @@ -445,6 +470,18 @@ the stack so that it is available for further iterations of the loop. by ``CALL_FUNCTION`` to construct a class. +.. opcode:: SETUP_WITH (delta) + + This opcode performs several operations before a with block starts. First, + it loads :meth:`~object.__exit__` from the context manager and pushes it onto + the stack for later use by :opcode:`WITH_CLEANUP`. Then, + :meth:`~object.__enter__` is called, and a finally block pointing to *delta* + is pushed. Finally, the result of calling the enter method is pushed onto + the stack. The next opcode will either ignore it (:opcode:`POP_TOP`), or + store it in (a) variable(s) (:opcode:`STORE_FAST`, :opcode:`STORE_NAME`, or + :opcode:`UNPACK_SEQUENCE`). + + .. opcode:: WITH_CLEANUP Cleans up the stack when a :keyword:`with` statement block exits. TOS is @@ -507,12 +544,6 @@ the more significant byte last. are put onto the stack right-to-left. -.. opcode:: DUP_TOPX (count) - - Duplicate *count* items, keeping them in the same order. Due to implementation - limits, *count* should be between 1 and 5 inclusive. - - .. opcode:: STORE_ATTR (namei) Implements ``TOS.name = TOS1``, where *namei* is the index of name in @@ -695,6 +726,12 @@ the more significant byte last. storage. +.. opcode:: DELETE_DEREF (i) + + Empties the cell contained in slot *i* of the cell and free variable storage. + Used by the :keyword:`del` statement. + + .. opcode:: RAISE_VARARGS (argc) Raises an exception. *argc* indicates the number of parameters to the raise diff --git a/Doc/library/doctest.rst b/Doc/library/doctest.rst index 5f40432..29fbd64 100644 --- a/Doc/library/doctest.rst +++ b/Doc/library/doctest.rst @@ -452,8 +452,9 @@ Some details you should read once, but won't need to remember: with an alphanumeric is taken to be the start of the exception detail. Of course this does the right thing for genuine tracebacks. -* When the :const:`IGNORE_EXCEPTION_DETAIL` doctest option is is specified, - everything following the leftmost colon is ignored. +* When the :const:`IGNORE_EXCEPTION_DETAIL` doctest option is specified, + everything following the leftmost colon and any module information in the + exception name is ignored. * The interactive shell omits the traceback header line for some :exc:`SyntaxError`\ s. But doctest uses the traceback header line to @@ -543,20 +544,38 @@ doctest decides whether actual output matches an example's expected output: exception raised is ``ValueError: 3*14``, but will fail, e.g., if :exc:`TypeError` is raised. - Note that a similar effect can be obtained using :const:`ELLIPSIS`, and - :const:`IGNORE_EXCEPTION_DETAIL` may go away when Python releases prior to 2.4 - become uninteresting. Until then, :const:`IGNORE_EXCEPTION_DETAIL` is the only - clear way to write a doctest that doesn't care about the exception detail yet - continues to pass under Python releases prior to 2.4 (doctest directives appear - to be comments to them). For example, :: + It will also ignore the module name used in Python 3 doctest reports. Hence + both these variations will work regardless of whether the test is run under + Python 2.7 or Python 3.2 (or later versions): + + >>> raise CustomError('message') #doctest: +IGNORE_EXCEPTION_DETAIL + Traceback (most recent call last): + CustomError: message + + >>> raise CustomError('message') #doctest: +IGNORE_EXCEPTION_DETAIL + Traceback (most recent call last): + my_module.CustomError: message + + Note that :const:`ELLIPSIS` can also be used to ignore the + details of the exception message, but such a test may still fail based + on whether or not the module details are printed as part of the + exception name. Using :const:`IGNORE_EXCEPTION_DETAIL` and the details + from Python 2.3 is also the only clear way to write a doctest that doesn't + care about the exception detail yet continues to pass under Python 2.3 or + earlier (those releases do not support doctest directives and ignore them + as irrelevant comments). For example, :: >>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: object doesn't support item assignment - passes under Python 2.4 and Python 2.3. The detail changed in 2.4, to say "does - not" instead of "doesn't". + passes under Python 2.3 and later Python versions, even though the detail + changed in Python 2.4 to say "does not" instead of "doesn't". + + .. versionchanged:: 3.2 + :const:`IGNORE_EXCEPTION_DETAIL` now also ignores any information relating + to the module containing the exception under test. .. data:: SKIP @@ -671,7 +690,6 @@ usually the only meaningful choice. However, option flags can also be passed to functions that run doctests, establishing different defaults. In such cases, disabling an option via ``-`` in a directive can be useful. - There's also a way to register new option flag names, although this isn't useful unless you intend to extend :mod:`doctest` internals via subclassing: @@ -895,18 +913,16 @@ Unittest API As your collection of doctest'ed modules grows, you'll want a way to run all their doctests systematically. :mod:`doctest` provides two functions that can be used to create :mod:`unittest` test suites from modules and text files -containing doctests. These test suites can then be run using :mod:`unittest` -test runners:: +containing doctests. To integrate with :mod:`unittest` test discovery, include +a :func:`load_tests` function in your test module:: import unittest import doctest - import my_module_with_doctests, and_another + import my_module_with_doctests - suite = unittest.TestSuite() - for mod in my_module_with_doctests, and_another: - suite.addTest(doctest.DocTestSuite(mod)) - runner = unittest.TextTestRunner() - runner.run(suite) + def load_tests(loader, tests, ignore): + tests.addTests(doctest.DocTestSuite(my_module_with_doctests)) + return tests There are two main functions for creating :class:`unittest.TestSuite` instances from text files and modules with doctests: diff --git a/Doc/library/dummy_threading.rst b/Doc/library/dummy_threading.rst index 0658df2..b578324 100644 --- a/Doc/library/dummy_threading.rst +++ b/Doc/library/dummy_threading.rst @@ -4,6 +4,9 @@ .. module:: dummy_threading :synopsis: Drop-in replacement for the threading module. +**Source code:** :source:`Lib/dummy_threading.py` + +-------------- This module provides a duplicate interface to the :mod:`threading` module. It is meant to be imported when the :mod:`_thread` module is not provided on a diff --git a/Doc/library/email-examples.rst b/Doc/library/email-examples.rst index c1b16da..32cecf3 100644 --- a/Doc/library/email-examples.rst +++ b/Doc/library/email-examples.rst @@ -11,6 +11,12 @@ First, let's see how to create and send a simple text message: .. literalinclude:: ../includes/email-simple.py +And parsing RFC822 headers can easily be done by the parse(filename) or +parsestr(message_as_string) methods of the Parser() class: + +.. literalinclude:: ../includes/email-headers.py + + Here's an example of how to send a MIME message containing a bunch of family pictures that may be residing in a directory: diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst index 930905a..85b32fe 100644 --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -22,6 +22,12 @@ the Generator on a :class:`~email.message.Message` constructed by program may result in changes to the :class:`~email.message.Message` object as defaults are filled in. +:class:`bytes` output can be generated using the :class:`BytesGenerator` class. +If the message object structure contains non-ASCII bytes, this generator's +:meth:`~BytesGenerator.flatten` method will emit the original bytes. Parsing a +binary message and then flattening it with :class:`BytesGenerator` should be +idempotent for standards compliant messages. + Here are the public methods of the :class:`Generator` class, imported from the :mod:`email.generator` module: @@ -50,7 +56,7 @@ Here are the public methods of the :class:`Generator` class, imported from the The other public :class:`Generator` methods are: - .. method:: flatten(msg, unixfrom=False) + .. method:: flatten(msg, unixfrom=False, linesep='\\n') Print the textual representation of the message object structure rooted at *msg* to the output file specified when the :class:`Generator` instance @@ -65,6 +71,21 @@ Here are the public methods of the :class:`Generator` class, imported from the Note that for subparts, no envelope header is ever printed. + Optional *linesep* specifies the line separator character used to + terminate lines in the output. It defaults to ``\n`` because that is + the most useful value for Python application code (other library packages + expect ``\n`` separated lines). ``linesep=\r\n`` can be used to + generate output with RFC-compliant line separators. + + Messages parsed with a Bytes parser that have a + :mailheader:`Content-Transfer-Encoding` of 8bit will be converted to a + use a 7bit Content-Transfer-Encoding. Non-ASCII bytes in the headers + will be :rfc:`2047` encoded with a charset of `unknown-8bit`. + + .. versionchanged:: 3.2 + Added support for re-encoding 8bit message bodies, and the *linesep* + argument. + .. method:: clone(fp) Return an independent clone of this :class:`Generator` instance with the @@ -76,11 +97,81 @@ Here are the public methods of the :class:`Generator` class, imported from the :class:`Generator`'s constructor. This provides just enough file-like API for :class:`Generator` instances to be used in the :func:`print` function. -As a convenience, see the methods :meth:`Message.as_string` and -``str(aMessage)``, a.k.a. :meth:`Message.__str__`, which simplify the generation -of a formatted string representation of a message object. For more detail, see +As a convenience, see the :class:`~email.message.Message` methods +:meth:`~email.message.Message.as_string` and ``str(aMessage)``, a.k.a. +:meth:`~email.message.Message.__str__`, which simplify the generation of a +formatted string representation of a message object. For more detail, see :mod:`email.message`. +.. class:: BytesGenerator(outfp, mangle_from_=True, maxheaderlen=78) + + The constructor for the :class:`BytesGenerator` class takes a binary + :term:`file-like object` called *outfp* for an argument. *outfp* must + support a :meth:`write` method that accepts binary data. + + Optional *mangle_from_* is a flag that, when ``True``, puts a ``>`` + character in front of any line in the body that starts exactly as ``From``, + i.e. ``From`` followed by a space at the beginning of the line. This is the + only guaranteed portable way to avoid having such lines be mistaken for a + Unix mailbox format envelope header separator (see `WHY THE CONTENT-LENGTH + FORMAT IS BAD <http://www.jwz.org/doc/content-length.html>`_ for details). + *mangle_from_* defaults to ``True``, but you might want to set this to + ``False`` if you are not writing Unix mailbox format files. + + Optional *maxheaderlen* specifies the longest length for a non-continued + header. When a header line is longer than *maxheaderlen* (in characters, + with tabs expanded to 8 spaces), the header will be split as defined in the + :class:`~email.header.Header` class. Set to zero to disable header + wrapping. The default is 78, as recommended (but not required) by + :rfc:`2822`. + + The other public :class:`BytesGenerator` methods are: + + + .. method:: flatten(msg, unixfrom=False, linesep='\n') + + Print the textual representation of the message object structure rooted + at *msg* to the output file specified when the :class:`BytesGenerator` + instance was created. Subparts are visited depth-first and the resulting + text will be properly MIME encoded. If the input that created the *msg* + contained bytes with the high bit set and those bytes have not been + modified, they will be copied faithfully to the output, even if doing so + is not strictly RFC compliant. (To produce strictly RFC compliant + output, use the :class:`Generator` class.) + + Messages parsed with a Bytes parser that have a + :mailheader:`Content-Transfer-Encoding` of 8bit will be reconstructed + as 8bit if they have not been modified. + + Optional *unixfrom* is a flag that forces the printing of the envelope + header delimiter before the first :rfc:`2822` header of the root message + object. If the root object has no envelope header, a standard one is + crafted. By default, this is set to ``False`` to inhibit the printing of + the envelope delimiter. + + Note that for subparts, no envelope header is ever printed. + + Optional *linesep* specifies the line separator character used to + terminate lines in the output. It defaults to ``\n`` because that is + the most useful value for Python application code (other library packages + expect ``\n`` separated lines). ``linesep=\r\n`` can be used to + generate output with RFC-compliant line separators. + + .. method:: clone(fp) + + Return an independent clone of this :class:`BytesGenerator` instance with + the exact same options. + + .. method:: write(s) + + Write the string *s* to the underlying file object. *s* is encoded using + the ``ASCII`` codec and written to the *write* method of the *outfp* + *outfp* passed to the :class:`BytesGenerator`'s constructor. This + provides just enough file-like API for :class:`BytesGenerator` instances + to be used in the :func:`print` function. + + .. versionadded:: 3.2 + The :mod:`email.generator` module also provides a derived class, called :class:`DecodedGenerator` which is like the :class:`Generator` base class, except that non-\ :mimetype:`text` parts are substituted with a format string diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst index 2202637..c385cf3 100644 --- a/Doc/library/email.header.rst +++ b/Doc/library/email.header.rst @@ -104,25 +104,48 @@ Here is the :class:`Header` class description: Optional *errors* is passed as the errors argument to the decode call if *s* is a byte string. - .. method:: encode(splitchars=';, \\t', maxlinelen=None) + + .. method:: encode(splitchars=';, \\t', maxlinelen=None, linesep='\\n') Encode a message header into an RFC-compliant format, possibly wrapping long lines and encapsulating non-ASCII parts in base64 or quoted-printable - encodings. Optional *splitchars* is a string containing characters to - split long ASCII lines on, in rough support of :rfc:`2822`'s *highest - level syntactic breaks*. This doesn't affect :rfc:`2047` encoded lines. + encodings. + + Optional *splitchars* is a string containing characters which should be + given extra weight by the splitting algorithm during normal header + wrapping. This is in very rough support of :RFC:`2822`\'s 'higher level + syntactic breaks': split points preceded by a splitchar are preferred + during line splitting, with the characters preferred in the order in + which they appear in the string. Space and tab may be included in the + string to indicate whether preference should be given to one over the + other as a split point when other split chars do not appear in the line + being split. Splitchars does not affect :RFC:`2047` encoded lines. *maxlinelen*, if given, overrides the instance's value for the maximum line length. + *linesep* specifies the characters used to separate the lines of the + folded header. It defaults to the most useful value for Python + application code (``\n``), but ``\r\n`` can be specified in order + to produce headers with RFC-compliant line separators. + + .. versionchanged:: 3.2 + Added the *linesep* argument. + The :class:`Header` class also provides a number of methods to support standard operators and built-in functions. .. method:: __str__() - A helper for :class:`str`'s :func:`encode` method. Returns the header as - a Unicode string. + Returns an approximation of the :class:`Header` as a string, using an + unlimited line length. All pieces are converted to unicode using the + specified encoding and joined together appropriately. Any pieces with a + charset of `unknown-8bit` are decoded as `ASCII` using the `replace` + error handler. + + .. versionchanged:: 3.2 + Added handling for the `unknown-8bit` charset. .. method:: __eq__(other) diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst index 4b23f6a..1e6a485 100644 --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -112,9 +112,18 @@ Here are the methods of the :class:`Message` class: be decoded if this header's value is ``quoted-printable`` or ``base64``. If some other encoding is used, or :mailheader:`Content-Transfer-Encoding` header is missing, or if the payload has bogus base64 data, the payload is - returned as-is (undecoded). If the message is a multipart and the - *decode* flag is ``True``, then ``None`` is returned. The default for - *decode* is ``False``. + returned as-is (undecoded). In all cases the returned value is binary + data. If the message is a multipart and the *decode* flag is ``True``, + then ``None`` is returned. + + When *decode* is ``False`` (the default) the body is returned as a string + without decoding the :mailheader:`Content-Transfer-Encoding`. However, + for a :mailheader:`Content-Transfer-Encoding` of 8bit, an attempt is made + to decode the original bytes using the ``charset`` specified by the + :mailheader:`Content-Type` header, using the ``replace`` error handler. + If no ``charset`` is specified, or if the ``charset`` given is not + recognized by the email package, the body is decoded using the default + ASCII charset. .. method:: set_payload(payload, charset=None) @@ -168,6 +177,11 @@ Here are the methods of the :class:`Message` class: Note that in all cases, any envelope header present in the message is not included in the mapping interface. + In a model generated from bytes, any header values that (in contravention of + the RFCs) contain non-ASCII bytes will, when retrieved through this + interface, be represented as :class:`~email.header.Header` objects with + a charset of `unknown-8bit`. + .. method:: __len__() @@ -263,10 +277,10 @@ Here are the methods of the :class:`Message` class: it can be specified as a three tuple in the format ``(CHARSET, LANGUAGE, VALUE)``, where ``CHARSET`` is a string naming the charset to be used to encode the value, ``LANGUAGE`` can usually be set - to ``None`` or the empty string (see :RFC:`2231` for other possibilities), + to ``None`` or the empty string (see :rfc:`2231` for other possibilities), and ``VALUE`` is the string value containing non-ASCII code points. If a three tuple is not passed and the value contains non-ASCII characters, - it is automatically encoded in :RFC`2231` format using a ``CHARSET`` + it is automatically encoded in :rfc:`2231` format using a ``CHARSET`` of ``utf-8`` and a ``LANGUAGE`` of ``None``. Here's an example:: diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst index 32f4ff1..77a0b69 100644 --- a/Doc/library/email.parser.rst +++ b/Doc/library/email.parser.rst @@ -80,6 +80,14 @@ Here is the API for the :class:`FeedParser`: if you feed more data to a closed :class:`FeedParser`. +.. class:: BytesFeedParser(_factory=email.message.Message) + + Works exactly like :class:`FeedParser` except that the input to the + :meth:`~FeedParser.feed` method must be bytes and not string. + + .. versionadded:: 3.2 + + Parser class API ^^^^^^^^^^^^^^^^ @@ -131,7 +139,7 @@ class. Similar to the :meth:`parse` method, except it takes a string object instead of a file-like object. Calling this method on a string is exactly - equivalent to wrapping *text* in a :class:`StringIO` instance first and + equivalent to wrapping *text* in a :class:`~io.StringIO` instance first and calling :meth:`parse`. Optional *headersonly* is a flag specifying whether to stop parsing after @@ -139,25 +147,78 @@ class. the entire contents of the file. +.. class:: BytesParser(_class=email.message.Message, strict=None) + + This class is exactly parallel to :class:`Parser`, but handles bytes input. + The *_class* and *strict* arguments are interpreted in the same way as for + the :class:`Parser` constructor. *strict* is supported only to make porting + code easier; it is deprecated. + + .. method:: parse(fp, headeronly=False) + + Read all the data from the binary file-like object *fp*, parse the + resulting bytes, and return the message object. *fp* must support + both the :meth:`readline` and the :meth:`read` methods on file-like + objects. + + The bytes contained in *fp* must be formatted as a block of :rfc:`2822` + style headers and header continuation lines, optionally preceded by a + envelope header. The header block is terminated either by the end of the + data or by a blank line. Following the header block is the body of the + message (which may contain MIME-encoded subparts, including subparts + with a :mailheader:`Content-Transfer-Encoding` of ``8bit``. + + Optional *headersonly* is a flag specifying whether to stop parsing after + reading the headers or not. The default is ``False``, meaning it parses + the entire contents of the file. + + .. method:: parsebytes(bytes, headersonly=False) + + Similar to the :meth:`parse` method, except it takes a byte string object + instead of a file-like object. Calling this method on a byte string is + exactly equivalent to wrapping *text* in a :class:`~io.BytesIO` instance + first and calling :meth:`parse`. + + Optional *headersonly* is as with the :meth:`parse` method. + + .. versionadded:: 3.2 + + Since creating a message object structure from a string or a file object is such -a common task, two functions are provided as a convenience. They are available +a common task, four functions are provided as a convenience. They are available in the top-level :mod:`email` package namespace. .. currentmodule:: email -.. function:: message_from_string(s[, _class][, strict]) +.. function:: message_from_string(s, _class=email.message.Message, strict=None) Return a message object structure from a string. This is exactly equivalent to ``Parser().parsestr(s)``. Optional *_class* and *strict* are interpreted as with the :class:`Parser` class constructor. +.. function:: message_from_bytes(s, _class=email.message.Message, strict=None) + + Return a message object structure from a byte string. This is exactly + equivalent to ``BytesParser().parsebytes(s)``. Optional *_class* and + *strict* are interpreted as with the :class:`Parser` class constructor. + + .. versionadded:: 3.2 -.. function:: message_from_file(fp[, _class][, strict]) +.. function:: message_from_file(fp, _class=email.message.Message, strict=None) Return a message object structure tree from an open :term:`file object`. This is exactly equivalent to ``Parser().parse(fp)``. Optional *_class* and *strict* are interpreted as with the :class:`Parser` class constructor. +.. function:: message_from_binary_file(fp, _class=email.message.Message, strict=None) + + Return a message object structure tree from an open binary :term:`file + object`. This is exactly equivalent to ``BytesParser().parse(fp)``. + Optional *_class* and *strict* are interpreted as with the :class:`Parser` + class constructor. + + .. versionadded:: 3.2 + Here's an example of how you might use this at an interactive Python prompt:: >>> import email diff --git a/Doc/library/email.rst b/Doc/library/email.rst index d3f1908..4530b95 100644 --- a/Doc/library/email.rst +++ b/Doc/library/email.rst @@ -6,7 +6,7 @@ email messages, including MIME documents. .. moduleauthor:: Barry A. Warsaw <barry@python.org> .. sectionauthor:: Barry A. Warsaw <barry@python.org> -.. Copyright (C) 2001-2007 Python Software Foundation +.. Copyright (C) 2001-2010 Python Software Foundation The :mod:`email` package is a library for managing email messages, including @@ -92,6 +92,44 @@ table also describes the Python compatibility of each version of the package. +---------------+------------------------------+-----------------------+ | :const:`4.0` | Python 2.5 | Python 2.3 to 2.5 | +---------------+------------------------------+-----------------------+ +| :const:`5.0` | Python 3.0 and Python 3.1 | Python 3.0 to 3.2 | ++---------------+------------------------------+-----------------------+ +| :const:`5.1` | Python 3.2 | Python 3.0 to 3.2 | ++---------------+------------------------------+-----------------------+ + +Here are the major differences between :mod:`email` version 5.1 and +version 5.0: + +* It is once again possible to parse messages containing non-ASCII bytes, + and to reproduce such messages if the data containing the non-ASCII + bytes is not modified. + +* New functions :func:`message_from_bytes` and :func:`message_from_binary_file`, + and new classes :class:`~email.parser.BytesFeedParser` and + :class:`~email.parser.BytesParser` allow binary message data to be parsed + into model objects. + +* Given bytes input to the model, :meth:`~email.message.Message.get_payload` + will by default decode a message body that has a + :mailheader:`Content-Transfer-Encoding` of ``8bit`` using the charset + specified in the MIME headers and return the resulting string. + +* Given bytes input to the model, :class:`~email.generator.Generator` will + convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of + 8bit to instead have a 7bit Content-Transfer-Encoding. + +* New class :class:`~email.generator.BytesGenerator` produces bytes + as output, preserving any unchanged non-ASCII data that was + present in the input used to build the model, including message bodies + with a :mailheader:`Content-Transfer-Encoding` of 8bit. + +Here are the major differences between :mod:`email` version 5.0 and version 4: + +* All operations are on unicode strings. Text inputs must be strings, + text outputs are strings. Outputs are limited to the ASCII character + set and so can be encoded to ASCII for transmission. Inputs are also + limited to ASCII; this is an acknowledged limitation of email 5.0 and + means it can only be used to parse email that is 7bit clean. Here are the major differences between :mod:`email` version 4 and version 3: diff --git a/Doc/library/email.util.rst b/Doc/library/email.util.rst index a1ce301..f7b777a 100644 --- a/Doc/library/email.util.rst +++ b/Doc/library/email.util.rst @@ -105,11 +105,17 @@ There are several useful utilities provided in the :mod:`email.utils` module: ``False``. The default is ``False``. -.. function:: make_msgid(idstring=None) +.. function:: make_msgid(idstring=None, domain=None) Returns a string suitable for an :rfc:`2822`\ -compliant :mailheader:`Message-ID` header. Optional *idstring* if given, is a string - used to strengthen the uniqueness of the message id. + used to strengthen the uniqueness of the message id. Optional *domain* if + given provides the portion of the msgid after the '@'. The default is the + local hostname. It is not normally necessary to override this default, but + may be useful certain cases, such as a constructing distributed system that + uses a consistent domain name across multiple hosts. + + .. versionchanged:: 3.2 domain keyword added .. function:: decode_rfc2231(s) diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 4159287..528febd 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -197,7 +197,7 @@ The following exceptions are the exceptions that are usually raised. Raised when an operation runs out of memory but the situation may still be rescued (by deleting some objects). The associated value is a string indicating what kind of (internal) operation ran out of memory. Note that because of the - underlying memory management architecture (C's :cfunc:`malloc` function), the + underlying memory management architecture (C's :c:func:`malloc` function), the interpreter may not always be able to completely recover from this situation; it nevertheless raises an exception so that a stack traceback can be printed, in case a run-away program was the cause. @@ -224,8 +224,8 @@ The following exceptions are the exceptions that are usually raised. This exception is derived from :exc:`EnvironmentError`. It is raised when a function returns a system-related error (not for illegal argument types or other incidental errors). The :attr:`errno` attribute is a numeric error - code from :cdata:`errno`, and the :attr:`strerror` attribute is the - corresponding string, as would be printed by the C function :cfunc:`perror`. + code from :c:data:`errno`, and the :attr:`strerror` attribute is the + corresponding string, as would be printed by the C function :c:func:`perror`. See the module :mod:`errno`, which contains names for the error codes defined by the underlying operating system. @@ -261,8 +261,8 @@ The following exceptions are the exceptions that are usually raised. .. exception:: StopIteration - Raised by builtin :func:`next` and an :term:`iterator`\'s :meth:`__next__` - method to signal that there are no further values. + Raised by built-in function :func:`next` and an :term:`iterator`\'s + :meth:`__next__` method to signal that there are no further values. .. exception:: SyntaxError @@ -307,7 +307,7 @@ The following exceptions are the exceptions that are usually raised. This exception is raised by the :func:`sys.exit` function. When it is not handled, the Python interpreter exits; no stack traceback is printed. If the associated value is an integer, it specifies the system exit status (passed - to C's :cfunc:`exit` function); if it is ``None``, the exit status is zero; + to C's :c:func:`exit` function); if it is ``None``, the exit status is zero; if it has another type (such as a string), the object's value is printed and the exit status is one. @@ -380,9 +380,9 @@ The following exceptions are the exceptions that are usually raised. .. exception:: WindowsError Raised when a Windows-specific error occurs or when the error number does not - correspond to an :cdata:`errno` value. The :attr:`winerror` and + correspond to an :c:data:`errno` value. The :attr:`winerror` and :attr:`strerror` values are created from the return values of the - :cfunc:`GetLastError` and :cfunc:`FormatMessage` functions from the Windows + :c:func:`GetLastError` and :c:func:`FormatMessage` functions from the Windows Platform API. The :attr:`errno` value maps the :attr:`winerror` value to corresponding ``errno.h`` values. This is a subclass of :exc:`OSError`. @@ -442,10 +442,20 @@ module for more information. Base class for warnings related to Unicode. + .. exception:: BytesWarning Base class for warnings related to :class:`bytes` and :class:`buffer`. + +.. exception:: ResourceWarning + + Base class for warnings related to resource usage. + + .. versionadded:: 3.2 + + + Exception hierarchy ------------------- diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index dd76d65..6192400 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -12,7 +12,7 @@ pair: UNIX; I/O control This module performs file control and I/O control on file descriptors. It is an -interface to the :cfunc:`fcntl` and :cfunc:`ioctl` Unix routines. +interface to the :c:func:`fcntl` and :c:func:`ioctl` Unix routines. All functions in this module take a file descriptor *fd* as their first argument. This can be an integer file descriptor, such as returned by @@ -30,17 +30,17 @@ The module defines the following functions: :mod:`fcntl` module. The argument *arg* is optional, and defaults to the integer value ``0``. When present, it can either be an integer value, or a string. With the argument missing or an integer value, the return value of this function - is the integer return value of the C :cfunc:`fcntl` call. When the argument is + is the integer return value of the C :c:func:`fcntl` call. When the argument is a string it represents a binary structure, e.g. created by :func:`struct.pack`. The binary data is copied to a buffer whose address is passed to the C - :cfunc:`fcntl` call. The return value after a successful call is the contents + :c:func:`fcntl` call. The return value after a successful call is the contents of the buffer, converted to a string object. The length of the returned string will be the same as the length of the *arg* argument. This is limited to 1024 bytes. If the information returned in the buffer by the operating system is larger than 1024 bytes, this is most likely to result in a segmentation violation or a more subtle data corruption. - If the :cfunc:`fcntl` fails, an :exc:`IOError` is raised. + If the :c:func:`fcntl` fails, an :exc:`IOError` is raised. .. function:: ioctl(fd, op[, arg[, mutate_flag]]) @@ -91,7 +91,7 @@ The module defines the following functions: Perform the lock operation *op* on file descriptor *fd* (file objects providing a :meth:`fileno` method are accepted as well). See the Unix manual :manpage:`flock(2)` for details. (On some systems, this function is emulated - using :cfunc:`fcntl`.) + using :c:func:`fcntl`.) .. function:: lockf(fd, operation, [length, [start, [whence]]]) diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst index f57dcce..e0ffff7 100644 --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -5,6 +5,9 @@ :synopsis: Compare files efficiently. .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/filecmp.py` + +-------------- The :mod:`filecmp` module defines functions to compare files and directories, with various optional time/correctness trade-offs. For comparing files, diff --git a/Doc/library/fileformats.rst b/Doc/library/fileformats.rst index 980d4f5..e9c2e1f 100644 --- a/Doc/library/fileformats.rst +++ b/Doc/library/fileformats.rst @@ -5,7 +5,7 @@ File Formats ************ The modules described in this chapter parse various miscellaneous file formats -that aren't markup languages or are related to e-mail. +that aren't markup languages and are not related to e-mail. .. toctree:: diff --git a/Doc/library/fileinput.rst b/Doc/library/fileinput.rst index d98a198..ac44311 100644 --- a/Doc/library/fileinput.rst +++ b/Doc/library/fileinput.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Guido van Rossum <guido@python.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/fileinput.py` + +-------------- This module implements a helper class and functions to quickly write a loop over standard input or a list of files. If you just want to read or @@ -24,7 +27,7 @@ as the first argument to :func:`.input`. A single file name is also allowed. All files are opened in text mode by default, but you can override this by specifying the *mode* parameter in the call to :func:`.input` or -:class:`FileInput()`. If an I/O error occurs during opening or reading a file, +:class:`FileInput`. If an I/O error occurs during opening or reading a file, :exc:`IOError` is raised. If ``sys.stdin`` is used more than once, the second and further use will return @@ -54,6 +57,17 @@ The following function is the primary interface of this module: during iteration. The parameters to this function will be passed along to the constructor of the :class:`FileInput` class. + The :class:`FileInput` instance can be used as a context manager in the + :keyword:`with` statement. In this example, *input* is closed after the + :keyword:`with` statement is exited, even if an exception occurs:: + + with fileinput.input(files=('spam.txt', 'eggs.txt')) as f: + for line in f: + process(line) + + .. versionchanged:: 3.2 + Can be used as a context manager. + The following functions use the global state created by :func:`fileinput.input`; if there is no active state, :exc:`RuntimeError` is raised. @@ -132,13 +146,23 @@ available for subclassing as well: *filename* and *mode*, and returns an accordingly opened file-like object. You cannot use *inplace* and *openhook* together. + A :class:`FileInput` instance can be used as a context manager in the + :keyword:`with` statement. In this example, *input* is closed after the + :keyword:`with` statement is exited, even if an exception occurs:: + + with FileInput(files=('spam.txt', 'eggs.txt')) as input: + process(input) + + .. versionchanged:: 3.2 + Can be used as a context manager. + -**Optional in-place filtering:** if the keyword argument ``inplace=1`` is passed -to :func:`fileinput.input` or to the :class:`FileInput` constructor, the file is -moved to a backup file and standard output is directed to the input file (if a -file of the same name as the backup file already exists, it will be replaced -silently). This makes it possible to write a filter that rewrites its input -file in place. If the *backup* parameter is given (typically as +**Optional in-place filtering:** if the keyword argument ``inplace=True`` is +passed to :func:`fileinput.input` or to the :class:`FileInput` constructor, the +file is moved to a backup file and standard output is directed to the input file +(if a file of the same name as the backup file already exists, it will be +replaced silently). This makes it possible to write a filter that rewrites its +input file in place. If the *backup* parameter is given (typically as ``backup='.<some extension>'``), it specifies the extension for the backup file, and the backup file remains around; by default, the extension is ``'.bak'`` and it is deleted when the output file is closed. In-place filtering is disabled diff --git a/Doc/library/fnmatch.rst b/Doc/library/fnmatch.rst index 7fa6148..4ba6b77 100644 --- a/Doc/library/fnmatch.rst +++ b/Doc/library/fnmatch.rst @@ -9,6 +9,10 @@ .. index:: module: re +**Source code:** :source:`Lib/fnmatch.py` + +-------------- + This module provides support for Unix shell-style wildcards, which are *not* the same as regular expressions (which are documented in the :mod:`re` module). The special characters used in shell-style wildcards are: @@ -70,6 +74,8 @@ patterns. Return the shell-style *pattern* converted to a regular expression. + Be aware there is no way to quote meta-characters. + Example: >>> import fnmatch, re @@ -86,4 +92,3 @@ patterns. Module :mod:`glob` Unix shell-style path expansion. - diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst index 7960026..a3ad44a 100644 --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Jeffrey Yasskin <jyasskin at gmail.com> .. sectionauthor:: Jeffrey Yasskin <jyasskin at gmail.com> +**Source code:** :source:`Lib/fractions.py` + +-------------- The :mod:`fractions` module provides support for rational number arithmetic. @@ -15,17 +18,24 @@ another rational number, or from a string. .. class:: Fraction(numerator=0, denominator=1) Fraction(other_fraction) + Fraction(float) + Fraction(decimal) Fraction(string) - The first version requires that *numerator* and *denominator* are - instances of :class:`numbers.Rational` and returns a new - :class:`Fraction` instance with value ``numerator/denominator``. If - *denominator* is :const:`0`, it raises a - :exc:`ZeroDivisionError`. The second version requires that - *other_fraction* is an instance of :class:`numbers.Rational` and - returns an :class:`Fraction` instance with the same value. The - last version of the constructor expects a string instance. The - usual form for this string is:: + The first version requires that *numerator* and *denominator* are instances + of :class:`numbers.Rational` and returns a new :class:`Fraction` instance + with value ``numerator/denominator``. If *denominator* is :const:`0`, it + raises a :exc:`ZeroDivisionError`. The second version requires that + *other_fraction* is an instance of :class:`numbers.Rational` and returns a + :class:`Fraction` instance with the same value. The next two versions accept + either a :class:`float` or a :class:`decimal.Decimal` instance, and return a + :class:`Fraction` instance with exactly the same value. Note that due to the + usual issues with binary floating-point (see :ref:`tut-fp-issues`), the + argument to ``Fraction(1.1)`` is not exactly equal to 11/10, and so + ``Fraction(1.1)`` does *not* return ``Fraction(11, 10)`` as one might expect. + (But see the documentation for the :meth:`limit_denominator` method below.) + The last version of the constructor expects a string or unicode instance. + The usual form for this instance is:: [sign] numerator ['/' denominator] @@ -55,6 +65,13 @@ another rational number, or from a string. Fraction(-1, 8) >>> Fraction('7e-6') Fraction(7, 1000000) + >>> Fraction(2.25) + Fraction(9, 4) + >>> Fraction(1.1) + Fraction(2476979795053773, 2251799813685248) + >>> from decimal import Decimal + >>> Fraction(Decimal('1.1')) + Fraction(11, 10) The :class:`Fraction` class inherits from the abstract base class @@ -63,6 +80,10 @@ another rational number, or from a string. and should be treated as immutable. In addition, :class:`Fraction` has the following methods: + .. versionchanged:: 3.2 + The :class:`Fraction` constructor now accepts :class:`float` and + :class:`decimal.Decimal` instances. + .. method:: from_float(flt) @@ -70,12 +91,19 @@ another rational number, or from a string. value of *flt*, which must be a :class:`float`. Beware that ``Fraction.from_float(0.3)`` is not the same value as ``Fraction(3, 10)`` + .. note:: From Python 3.2 onwards, you can also construct a + :class:`Fraction` instance directly from a :class:`float`. + .. method:: from_decimal(dec) This class method constructs a :class:`Fraction` representing the exact value of *dec*, which must be a :class:`decimal.Decimal` instance. + .. note:: From Python 3.2 onwards, you can also construct a + :class:`Fraction` instance directly from a :class:`decimal.Decimal` + instance. + .. method:: limit_denominator(max_denominator=1000000) @@ -90,10 +118,12 @@ another rational number, or from a string. or for recovering a rational number that's represented as a float: >>> from math import pi, cos - >>> Fraction.from_float(cos(pi/3)) + >>> Fraction(cos(pi/3)) Fraction(4503599627370497, 9007199254740992) - >>> Fraction.from_float(cos(pi/3)).limit_denominator() + >>> Fraction(cos(pi/3)).limit_denominator() Fraction(1, 2) + >>> Fraction(1.1).limit_denominator() + Fraction(11, 10) .. method:: __floor__() diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst index 5545505..5bbef4f 100644 --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -9,6 +9,10 @@ pair: FTP; protocol single: FTP; ftplib (standard module) +**Source code:** :source:`Lib/ftplib.py` + +-------------- + This module defines the class :class:`FTP` and a few related items. The :class:`FTP` class implements the client side of the FTP protocol. You can use this to write Python programs that perform a variety of automated FTP jobs, such @@ -33,8 +37,8 @@ Here's a sample session using the :mod:`ftplib` module:: '226 Transfer complete.' >>> ftp.quit() -The module defines the following items: +The module defines the following items: .. class:: FTP(host='', user='', passwd='', acct=''[, timeout]) @@ -46,6 +50,61 @@ The module defines the following items: connection attempt (if is not specified, the global default timeout setting will be used). + :class:`FTP` class supports the :keyword:`with` statement. Here is a sample + on how using it: + + >>> from ftplib import FTP + >>> with FTP("ftp1.at.proftpd.org") as ftp: + ... ftp.login() + ... ftp.dir() + ... + '230 Anonymous login ok, restrictions apply.' + dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 . + dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .. + dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS + dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora + >>> + + .. versionchanged:: 3.2 + Support for the :keyword:`with` statement was added. + + +.. class:: FTP_TLS(host='', user='', passwd='', acct='', [keyfile[, certfile[, context[, timeout]]]]) + + A :class:`FTP` subclass which adds TLS support to FTP as described in + :rfc:`4217`. + Connect as usual to port 21 implicitly securing the FTP control connection + before authenticating. Securing the data connection requires the user to + explicitly ask for it by calling the :meth:`prot_p` method. + *keyfile* and *certfile* are optional -- they can contain a PEM formatted + private key and certificate chain file name for the SSL connection. + *context* parameter is a :class:`ssl.SSLContext` object which allows + bundling SSL configuration options, certificates and private keys into a + single (potentially long-lived) structure. + + .. versionadded:: 3.2 + + Here's a sample session using the :class:`FTP_TLS` class: + + >>> from ftplib import FTP_TLS + >>> ftps = FTP_TLS('ftp.python.org') + >>> ftps.login() # login anonymously before securing control channel + >>> ftps.prot_p() # switch to secure data connection + >>> ftps.retrlines('LIST') # list directory content securely + total 9 + drwxr-xr-x 8 root wheel 1024 Jan 3 1994 . + drwxr-xr-x 8 root wheel 1024 Jan 3 1994 .. + drwxr-xr-x 2 root wheel 1024 Jan 3 1994 bin + drwxr-xr-x 2 root wheel 1024 Jan 3 1994 etc + d-wxrwxr-x 2 ftp wheel 1024 Sep 5 13:43 incoming + drwxr-xr-x 2 root wheel 1024 Nov 17 1993 lib + drwxr-xr-x 6 1094 wheel 1024 Sep 13 19:07 pub + drwxr-xr-x 3 root wheel 1024 Jan 3 1994 usr + -rw-r--r-- 1 root root 312 Aug 1 1994 welcome.msg + '226 Transfer complete.' + >>> ftps.quit() + >>> + .. exception:: error_reply @@ -197,14 +256,18 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. Passive mode is on by default. -.. method:: FTP.storbinary(cmd, file, blocksize=8192, callback=None) +.. method:: FTP.storbinary(cmd, file, blocksize=8192, callback=None, rest=None) Store a file in binary transfer mode. *cmd* should be an appropriate ``STOR`` command: ``"STOR filename"``. *file* is an open :term:`file object` which is read until EOF using its :meth:`read` method in blocks of size *blocksize* to provide the data to be stored. The *blocksize* argument defaults to 8192. *callback* is an optional single parameter callable that - is called on each block of data after it is sent. + is called on each block of data after it is sent. *rest* means the same thing + as in the :meth:`transfercmd` method. + + .. versionchanged:: 3.2 + *rest* parameter added. .. method:: FTP.storlines(cmd, file, callback=None) @@ -319,3 +382,26 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. :meth:`close` or :meth:`quit` you cannot reopen the connection by issuing another :meth:`login` method). + +FTP_TLS Objects +--------------- + +:class:`FTP_TLS` class inherits from :class:`FTP`, defining these additional objects: + +.. attribute:: FTP_TLS.ssl_version + + The SSL version to use (defaults to *TLSv1*). + +.. method:: FTP_TLS.auth() + + Set up secure control connection by using TLS or SSL, depending on what specified in :meth:`ssl_version` attribute. + +.. method:: FTP_TLS.prot_p() + + Set up secure data connection. + +.. method:: FTP_TLS.prot_c() + + Set up clear text data connection. + + diff --git a/Doc/library/functional.rst b/Doc/library/functional.rst new file mode 100644 index 0000000..5b6185a --- /dev/null +++ b/Doc/library/functional.rst @@ -0,0 +1,15 @@ +****************************** +Functional Programming Modules +****************************** + +The modules described in this chapter provide functions and classes that support +a functional programming style, and general operations on callables. + +The following modules are documented in this chapter: + + +.. toctree:: + + itertools.rst + functools.rst + operator.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 1547f6d..3020128 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -10,20 +10,20 @@ are always available. They are listed here in alphabetical order. =================== ================= ================== ================ ==================== .. .. Built-in Functions .. .. =================== ================= ================== ================ ==================== -:func:`abs` :func:`dir` :func:`hex` :func:`next` :func:`slice` -:func:`all` :func:`divmod` :func:`id` :func:`object` :func:`sorted` -:func:`any` :func:`enumerate` :func:`input` :func:`oct` :func:`staticmethod` -:func:`ascii` :func:`eval` :func:`int` :func:`open` :func:`str` -:func:`bin` :func:`exec` :func:`isinstance` :func:`ord` :func:`sum` -:func:`bool` :func:`filter` :func:`issubclass` :func:`pow` :func:`super` -:func:`bytearray` :func:`float` :func:`iter` :func:`print` :func:`tuple` -:func:`bytes` :func:`format` :func:`len` :func:`property` :func:`type` +:func:`abs` :func:`dict` :func:`help` :func:`min` :func:`setattr` +:func:`all` :func:`dir` :func:`hex` :func:`next` :func:`slice` +:func:`any` :func:`divmod` :func:`id` :func:`object` :func:`sorted` +:func:`ascii` :func:`enumerate` :func:`input` :func:`oct` :func:`staticmethod` +:func:`bin` :func:`eval` :func:`int` :func:`open` :func:`str` +:func:`bool` :func:`exec` :func:`isinstance` :func:`ord` :func:`sum` +:func:`bytearray` :func:`filter` :func:`issubclass` :func:`pow` :func:`super` +:func:`bytes` :func:`float` :func:`iter` :func:`print` :func:`tuple` +:func:`callable` :func:`format` :func:`len` :func:`property` :func:`type` :func:`chr` :func:`frozenset` :func:`list` :func:`range` :func:`vars` :func:`classmethod` :func:`getattr` :func:`locals` :func:`repr` :func:`zip` :func:`compile` :func:`globals` :func:`map` :func:`reversed` :func:`__import__` :func:`complex` :func:`hasattr` :func:`max` :func:`round` :func:`delattr` :func:`hash` :func:`memoryview` :func:`set` -:func:`dict` :func:`help` :func:`min` :func:`setattr` =================== ================= ================== ================ ==================== .. function:: abs(x) @@ -121,6 +121,19 @@ are always available. They are listed here in alphabetical order. Bytes objects can also be created with literals, see :ref:`strings`. +.. function:: callable(object) + + Return :const:`True` if the *object* argument appears callable, + :const:`False` if not. If this returns true, it is still possible that a + call fails, but if it is false, calling *object* will never succeed. + Note that classes are callable (calling a class returns a new instance); + instances are callable if their class has a :meth:`__call__` method. + + .. versionadded:: 3.2 + This function was first removed in Python 3.0 and then brought back + in Python 3.2. + + .. function:: chr(i) Return the string representing a character whose Unicode codepoint is the integer @@ -161,7 +174,7 @@ are always available. They are listed here in alphabetical order. type hierarchy in :ref:`types`. -.. function:: compile(source, filename, mode, flags=0, dont_inherit=False) +.. function:: compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1) Compile the *source* into a code or AST object. Code objects can be executed by :func:`exec` or :func:`eval`. *source* can either be a string or an AST @@ -193,16 +206,25 @@ are always available. They are listed here in alphabetical order. can be found as the :attr:`compiler_flag` attribute on the :class:`_Feature` instance in the :mod:`__future__` module. + The argument *optimize* specifies the optimization level of the compiler; the + default value of ``-1`` selects the optimization level of the interpreter as + given by :option:`-O` options. Explicit levels are ``0`` (no optimization; + ``__debug__`` is true), ``1`` (asserts are removed, ``__debug__`` is false) + or ``2`` (docstrings are removed too). + This function raises :exc:`SyntaxError` if the compiled source is invalid, and :exc:`TypeError` if the source contains null bytes. .. note:: - When compiling a string with multi-line statements, line endings must be - represented by a single newline character (``'\n'``), and the input must - be terminated by at least one newline character. If line endings are - represented by ``'\r\n'``, use :meth:`str.replace` to change them into - ``'\n'``. + When compiling a string with multi-line code in ``'single'`` or + ``'eval'`` mode, input must be terminated by at least one newline + character. This is to facilitate detection of incomplete and complete + statements in the :mod:`code` module. + + .. versionchanged:: 3.2 + Allowed use of Windows and Mac newlines. Also input in ``'exec'`` mode + does not have to end in a newline anymore. Added the *optimize* parameter. .. function:: complex([real[, imag]]) @@ -346,7 +368,7 @@ are always available. They are listed here in alphabetical order. This function can also be used to execute arbitrary code objects (such as those created by :func:`compile`). In this case pass a code object instead of a string. If the code object has been compiled with ``'exec'`` as the - *kind* argument, :func:`eval`\'s return value will be ``None``. + *mode* argument, :func:`eval`\'s return value will be ``None``. Hints: dynamic execution of statements is supported by the :func:`exec` function. The :func:`globals` and :func:`locals` functions @@ -414,26 +436,54 @@ are always available. They are listed here in alphabetical order. .. function:: float([x]) - Convert a string or a number to floating point. If the argument is a string, - it must contain a possibly signed decimal or floating point number, possibly - embedded in whitespace. The argument may also be ``'[+|-]nan'`` or - ``'[+|-]inf'``. Otherwise, the argument may be an integer or a floating - point number, and a floating point number with the same value (within - Python's floating point precision) is returned. If no argument is given, - ``0.0`` is returned. - - .. note:: - - .. index:: - single: NaN - single: Infinity - - When passing in a string, values for NaN and Infinity may be returned, - depending on the underlying C library. Float accepts the strings - ``'nan'``, ``'inf'`` and ``'-inf'`` for NaN and positive or negative - infinity. The case and a leading + are ignored as well as a leading - is - ignored for NaN. Float always represents NaN and infinity as ``nan``, - ``inf`` or ``-inf``. + .. index:: + single: NaN + single: Infinity + + Convert a string or a number to floating point. + + If the argument is a string, it should contain a decimal number, optionally + preceded by a sign, and optionally embedded in whitespace. The optional + sign may be ``'+'`` or ``'-'``; a ``'+'`` sign has no effect on the value + produced. The argument may also be a string representing a NaN + (not-a-number), or a positive or negative infinity. More precisely, the + input must conform to the following grammar after leading and trailing + whitespace characters are removed: + + .. productionlist:: + sign: "+" | "-" + infinity: "Infinity" | "inf" + nan: "nan" + numeric_value: `floatnumber` | `infinity` | `nan` + numeric_string: [`sign`] `numeric_value` + + Here ``floatnumber`` is the form of a Python floating-point literal, + described in :ref:`floating`. Case is not significant, so, for example, + "inf", "Inf", "INFINITY" and "iNfINity" are all acceptable spellings for + positive infinity. + + Otherwise, if the argument is an integer or a floating point number, a + floating point number with the same value (within Python's floating point + precision) is returned. If the argument is outside the range of a Python + float, an :exc:`OverflowError` will be raised. + + For a general Python object ``x``, ``float(x)`` delegates to + ``x.__float__()``. + + If no argument is given, ``0.0`` is returned. + + Examples:: + + >>> float('+1.23') + 1.23 + >>> float(' -12345\n') + -12345.0 + >>> float('1e-003') + 0.001 + >>> float('+1E6') + 1000000.0 + >>> float('-Infinity') + -inf The float type is described in :ref:`typesnumeric`. @@ -482,10 +532,10 @@ are always available. They are listed here in alphabetical order. .. function:: hasattr(object, name) - The arguments are an object and a string. The result is ``True`` if the string - is the name of one of the object's attributes, ``False`` if not. (This is - implemented by calling ``getattr(object, name)`` and seeing whether it raises an - exception or not.) + The arguments are an object and a string. The result is ``True`` if the + string is the name of one of the object's attributes, ``False`` if not. (This + is implemented by calling ``getattr(object, name)`` and seeing whether it + raises an :exc:`AttributeError` or not.) .. function:: hash(object) @@ -653,6 +703,10 @@ are always available. They are listed here in alphabetical order. The optional keyword-only *key* argument specifies a one-argument ordering function like that used for :meth:`list.sort`. + If multiple items are maximal, the function returns the first one + encountered. This is consistent with other sort-stability preserving tools + such as ``sorted(iterable, key=keyfunc, reverse=True)[0]`` and + ``heapq.nlargest(1, iterable, key=keyfunc)``. .. function:: memoryview(obj) :noindex: @@ -670,6 +724,10 @@ are always available. They are listed here in alphabetical order. The optional keyword-only *key* argument specifies a one-argument ordering function like that used for :meth:`list.sort`. + If multiple items are minimal, the function returns the first one + encountered. This is consistent with other sort-stability preserving tools + such as ``sorted(iterable, key=keyfunc)[0]`` and ``heapq.nsmallest(1, + iterable, key=keyfunc)``. .. function:: next(iterator[, default]) @@ -971,6 +1029,35 @@ are always available. They are listed here in alphabetical order. >>> list(range(1, 0)) [] + Range objects implement the :class:`collections.Sequence` ABC, and provide + features such as containment tests, element index lookup, slicing and + support for negative indices: + + >>> r = range(0, 20, 2) + >>> r + range(0, 20, 2) + >>> 11 in r + False + >>> 10 in r + True + >>> r.index(10) + 5 + >>> r[5] + 10 + >>> r[:5] + range(0, 10, 2) + >>> r[-1] + 18 + + Ranges containing absolute values larger than :data:`sys.maxsize` are permitted + but some features (such as :func:`len`) will raise :exc:`OverflowError`. + + .. versionchanged:: 3.2 + Implement the Sequence ABC. + Support slicing and negative indices. + Test integers for membership in constant time instead of iterating + through all items. + .. function:: repr(object) @@ -1050,14 +1137,14 @@ are always available. They are listed here in alphabetical order. Has two optional arguments which must be specified as keyword arguments. *key* specifies a function of one argument that is used to extract a comparison - key from each list element: ``key=str.lower``. The default value is ``None``. + key from each list element: ``key=str.lower``. The default value is ``None`` + (compare the elements directly). *reverse* is a boolean value. If set to ``True``, then the list elements are sorted as if each comparison were reversed. - To convert an old-style *cmp* function to a *key* function, see the - `CmpToKey recipe in the ASPN cookbook - <http://code.activestate.com/recipes/576653/>`_\. + Use :func:`functools.cmp_to_key` to convert an old-style *cmp* function to a + *key* function. For sorting examples and a brief sorting tutorial, see `Sorting HowTo <http://wiki.python.org/moin/HowTo/Sorting/>`_\. diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst index 570f4d2..2316e80 100644 --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -8,6 +8,9 @@ .. moduleauthor:: Nick Coghlan <ncoghlan@gmail.com> .. sectionauthor:: Peter Harris <scav@blueyonder.co.uk> +**Source code:** :source:`Lib/functools.py` + +-------------- The :mod:`functools` module is for higher-order functions: functions that act on or return other functions. In general, any callable object can be treated as a @@ -15,6 +18,123 @@ function for the purposes of this module. The :mod:`functools` module defines the following functions: +.. function:: cmp_to_key(func) + + Transform an old-style comparison function to a key-function. Used with + tools that accept key functions (such as :func:`sorted`, :func:`min`, + :func:`max`, :func:`heapq.nlargest`, :func:`heapq.nsmallest`, + :func:`itertools.groupby`). This function is primarily used as a transition + tool for programs being converted from Py2.x which supported the use of + comparison functions. + + A compare function is any callable that accept two arguments, compares them, + and returns a negative number for less-than, zero for equality, or a positive + number for greater-than. A key function is a callable that accepts one + argument and returns another value indicating the position in the desired + collation sequence. + + Example:: + + sorted(iterable, key=cmp_to_key(locale.strcoll)) # locale-aware sort order + + .. versionadded:: 3.2 + + +.. decorator:: lru_cache(maxsize=100) + + Decorator to wrap a function with a memoizing callable that saves up to the + *maxsize* most recent calls. It can save time when an expensive or I/O bound + function is periodically called with the same arguments. + + Since a dictionary is used to cache results, the positional and keyword + arguments to the function must be hashable. + + If *maxsize* is set to None, the LRU feature is disabled and the cache + can grow without bound. + + To help measure the effectiveness of the cache and tune the *maxsize* + parameter, the wrapped function is instrumented with a :func:`cache_info` + function that returns a :term:`named tuple` showing *hits*, *misses*, + *maxsize* and *currsize*. In a multi-threaded environment, the hits + and misses are approximate. + + The decorator also provides a :func:`cache_clear` function for clearing or + invalidating the cache. + + The original underlying function is accessible through the + :attr:`__wrapped__` attribute. This is useful for introspection, for + bypassing the cache, or for rewrapping the function with a different cache. + + An `LRU (least recently used) cache + <http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used>`_ works + best when more recent calls are the best predictors of upcoming calls (for + example, the most popular articles on a news server tend to change daily). + The cache's size limit assures that the cache does not grow without bound on + long-running processes such as web servers. + + Example of an LRU cache for static web content:: + + @lru_cache(maxsize=20) + def get_pep(num): + 'Retrieve text of a Python Enhancement Proposal' + resource = 'http://www.python.org/dev/peps/pep-%04d/' % num + try: + with urllib.request.urlopen(resource) as s: + return s.read() + except urllib.error.HTTPError: + return 'Not Found' + + >>> for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991: + ... pep = get_pep(n) + ... print(n, len(pep)) + + >>> print(get_pep.cache_info()) + CacheInfo(hits=3, misses=8, maxsize=20, currsize=8) + + Example of efficiently computing + `Fibonacci numbers <http://en.wikipedia.org/wiki/Fibonacci_number>`_ + using a cache to implement a + `dynamic programming <http://en.wikipedia.org/wiki/Dynamic_programming>`_ + technique:: + + @lru_cache(maxsize=None) + def fib(n): + if n < 2: + return n + return fib(n-1) + fib(n-2) + + >>> print([fib(n) for n in range(16)]) + [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610] + + >>> print(fib.cache_info()) + CacheInfo(hits=28, misses=16, maxsize=None, currsize=16) + + .. versionadded:: 3.2 + +.. decorator:: total_ordering + + Given a class defining one or more rich comparison ordering methods, this + class decorator supplies the rest. This simplifies the effort involved + in specifying all of the possible rich comparison operations: + + The class must define one of :meth:`__lt__`, :meth:`__le__`, + :meth:`__gt__`, or :meth:`__ge__`. + In addition, the class should supply an :meth:`__eq__` method. + + For example:: + + @total_ordering + class Student: + def __eq__(self, other): + return ((self.lastname.lower(), self.firstname.lower()) == + (other.lastname.lower(), other.firstname.lower())) + def __lt__(self, other): + return ((self.lastname.lower(), self.firstname.lower()) < + (other.lastname.lower(), other.firstname.lower())) + + .. versionadded:: 3.2 + + .. function:: partial(func, *args, **keywords) Return a new :class:`partial` object which when called will behave like *func* @@ -70,14 +190,34 @@ The :mod:`functools` module defines the following functions: documentation string) and *WRAPPER_UPDATES* (which updates the wrapper function's *__dict__*, i.e. the instance dictionary). + To allow access to the original function for introspection and other purposes + (e.g. bypassing a caching decorator such as :func:`lru_cache`), this function + automatically adds a __wrapped__ attribute to the wrapper that refers to + the original function. + The main intended use for this function is in :term:`decorator` functions which wrap the decorated function and return the wrapper. If the wrapper function is not updated, the metadata of the returned function will reflect the wrapper definition rather than the original function definition, which is typically less than helpful. + :func:`update_wrapper` may be used with callables other than functions. Any + attributes named in *assigned* or *updated* that are missing from the object + being wrapped are ignored (i.e. this function will not attempt to set them + on the wrapper function). :exc:`AttributeError` is still raised if the + wrapper function itself is missing any attributes named in *updated*. + + .. versionadded:: 3.2 + Automatic addition of the ``__wrapped__`` attribute. + + .. versionadded:: 3.2 + Copying of the ``__annotations__`` attribute by default. + + .. versionchanged:: 3.2 + Missing attributes no longer trigger an :exc:`AttributeError`. + -.. function:: wraps(wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES) +.. decorator:: wraps(wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES) This is a convenience function for invoking ``partial(update_wrapper, wrapped=wrapped, assigned=assigned, updated=updated)`` as a function decorator diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 34aba65..0281bb7 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -43,7 +43,7 @@ The :mod:`gc` module provides the following functions: :exc:`ValueError` is raised if the generation number is invalid. The number of unreachable objects found is returned. - The free lists maintained for a number of builtin types are cleared + The free lists maintained for a number of built-in types are cleared whenever a full collection or collection of the highest generation (2) is run. Not all items in some free lists may be freed due to the particular implementation, in particular :class:`float`. @@ -174,8 +174,15 @@ value but should not rebind it): with :meth:`__del__` methods, and *garbage* can be examined in that case to verify that no such cycles are being created. - If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be added to - this list rather than freed. + If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be added + to this list rather than freed. + + .. versionchanged:: 3.2 + If this list is non-empty at interpreter shutdown, a + :exc:`ResourceWarning` is emitted, which is silent by default. If + :const:`DEBUG_UNCOLLECTABLE` is set, in addition all uncollectable objects + are printed. + The following constants are provided for use with :func:`set_debug`: @@ -194,9 +201,12 @@ The following constants are provided for use with :func:`set_debug`: .. data:: DEBUG_UNCOLLECTABLE Print information of uncollectable objects found (objects which are not - reachable but cannot be freed by the collector). These objects will be added to - the ``garbage`` list. + reachable but cannot be freed by the collector). These objects will be added + to the ``garbage`` list. + .. versionchanged:: 3.2 + Also print the contents of the :data:`garbage` list at interpreter + shutdown, if it isn't empty. .. data:: DEBUG_SAVEALL diff --git a/Doc/library/getopt.rst b/Doc/library/getopt.rst index 6a95142..bcfc4b5 100644 --- a/Doc/library/getopt.rst +++ b/Doc/library/getopt.rst @@ -1,13 +1,23 @@ -:mod:`getopt` --- Parser for command line options -================================================= +:mod:`getopt` --- C-style parser for command line options +========================================================= .. module:: getopt :synopsis: Portable parser for command line options; support both short and long option names. +**Source code:** :source:`Lib/getopt.py` + +-------------- + +.. note:: + The :mod:`getopt` module is a parser for command line options whose API is + designed to be familiar to users of the C :c:func:`getopt` function. Users who + are unfamiliar with the C :c:func:`getopt` function or who would like to write + less code and get better help and error messages should consider using the + :mod:`argparse` module instead. This module helps scripts to parse the command line arguments in ``sys.argv``. -It supports the same conventions as the Unix :cfunc:`getopt` function (including +It supports the same conventions as the Unix :c:func:`getopt` function (including the special meanings of arguments of the form '``-``' and '``--``'). Long options similar to those supported by GNU software may be used as well via an optional third argument. @@ -25,11 +35,11 @@ exception: be parsed, without the leading reference to the running program. Typically, this means ``sys.argv[1:]``. *shortopts* is the string of option letters that the script wants to recognize, with options that require an argument followed by a - colon (``':'``; i.e., the same format that Unix :cfunc:`getopt` uses). + colon (``':'``; i.e., the same format that Unix :c:func:`getopt` uses). .. note:: - Unlike GNU :cfunc:`getopt`, after a non-option argument, all further + Unlike GNU :c:func:`getopt`, after a non-option argument, all further arguments are considered also non-options. This is similar to the way non-GNU Unix systems work. @@ -136,9 +146,21 @@ In a script, typical usage is something like this:: if __name__ == "__main__": main() +Note that an equivalent command line interface could be produced with less code +and more informative help and error messages by using the :mod:`argparse` module:: + + import argparse + + if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('-o', '--output') + parser.add_argument('-v', dest='verbose', action='store_true') + args = parser.parse_args() + # ... do something with args.output ... + # ... do something with args.verbose .. .. seealso:: - Module :mod:`optparse` - More object-oriented command line option parsing. + Module :mod:`argparse` + Alternative command line option and argument parsing library. diff --git a/Doc/library/gettext.rst b/Doc/library/gettext.rst index 9e1528b..bc825cc 100644 --- a/Doc/library/gettext.rst +++ b/Doc/library/gettext.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Barry A. Warsaw <barry@zope.com> .. sectionauthor:: Barry A. Warsaw <barry@zope.com> +**Source code:** :source:`Lib/gettext.py` + +-------------- The :mod:`gettext` module provides internationalization (I18N) and localization (L10N) services for your Python modules and applications. It supports both the diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst index 3e0322d..3d31c11 100644 --- a/Doc/library/glob.rst +++ b/Doc/library/glob.rst @@ -7,6 +7,10 @@ .. index:: single: filenames; pathname expansion +**Source code:** :source:`Lib/glob.py` + +-------------- + The :mod:`glob` module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell. No tilde expansion is done, but ``*``, ``?``, and character ranges expressed with ``[]`` will be correctly diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst index fdd8590..659a027 100644 --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -4,6 +4,10 @@ .. module:: gzip :synopsis: Interfaces for gzip compression and decompression using file objects. +**Source code:** :source:`Lib/gzip.py` + +-------------- + This module provides a simple interface to compress and decompress files just like the GNU programs :program:`gzip` and :program:`gunzip` would. @@ -25,10 +29,10 @@ The module defines the following items: .. class:: GzipFile(filename=None, mode=None, compresslevel=9, fileobj=None, mtime=None) - Constructor for the :class:`GzipFile` class, which simulates most of the methods - of a :term:`file object`, with the exception of the :meth:`readinto` and - :meth:`truncate` methods. At least one of *fileobj* and *filename* must be - given a non-trivial value. + Constructor for the :class:`GzipFile` class, which simulates most of the + methods of a :term:`file object`, with the exception of the :meth:`truncate` + method. At least one of *fileobj* and *filename* must be given a non-trivial + value. The new class instance is based on *fileobj*, which can be a regular file, a :class:`StringIO` object, or any other object which simulates a file. It @@ -62,15 +66,34 @@ The module defines the following items: Calling a :class:`GzipFile` object's :meth:`close` method does not close *fileobj*, since you might wish to append more material after the compressed - data. This also allows you to pass a :class:`StringIO` object opened for + data. This also allows you to pass a :class:`io.BytesIO` object opened for writing as *fileobj*, and retrieve the resulting memory buffer using the - :class:`StringIO` object's :meth:`getvalue` method. + :class:`io.BytesIO` object's :meth:`~io.BytesIO.getvalue` method. + + :class:`GzipFile` supports the :class:`io.BufferedIOBase` interface, + including iteration and the :keyword:`with` statement. Only the + :meth:`read1` and :meth:`truncate` methods aren't implemented. + + :class:`GzipFile` also provides the following method: - :class:`GzipFile` supports the :keyword:`with` statement. + .. method:: peek([n]) + + Read *n* uncompressed bytes without advancing the file position. + At most one single read on the compressed stream is done to satisfy + the call. The number of bytes returned may be more or less than + requested. + + .. versionadded:: 3.2 .. versionchanged:: 3.1 Support for the :keyword:`with` statement was added. + .. versionchanged:: 3.2 + Support for zero-padded files was added. + + .. versionchanged:: 3.2 + Support for unseekable files was added. + .. function:: open(filename, mode='rb', compresslevel=9) @@ -78,6 +101,21 @@ The module defines the following items: The *filename* argument is required; *mode* defaults to ``'rb'`` and *compresslevel* defaults to ``9``. +.. function:: compress(data, compresslevel=9) + + Compress the *data*, returning a :class:`bytes` object containing + the compressed data. *compresslevel* has the same meaning as in + the :class:`GzipFile` constructor above. + + .. versionadded:: 3.2 + +.. function:: decompress(data) + + Decompress the *data*, returning a :class:`bytes` object containing the + uncompressed data. + + .. versionadded:: 3.2 + .. _gzip-usage-examples: @@ -104,6 +142,11 @@ Example of how to GZIP compress an existing file:: with gzip.open('/home/joe/file.txt.gz', 'wb') as f_out: f_out.writelines(f_in) +Example of how to GZIP compress a binary string:: + + import gzip + s_in = b"Lots of content here" + s_out = gzip.compress(s_in) .. seealso:: diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index b73d753..bc8ab2c 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -11,6 +11,10 @@ single: message digest, MD5 single: secure hash algorithm, SHA1, SHA224, SHA256, SHA384, SHA512 +**Source code:** :source:`Lib/hashlib.py` + +-------------- + This module implements a common interface to many different secure hash and message digest algorithms. Included are the FIPS secure hash algorithms SHA1, SHA224, SHA256, SHA384, and SHA512 (defined in FIPS 180-2) as well as RSA's MD5 @@ -70,10 +74,13 @@ More condensed: >>> hashlib.sha224(b"Nobody inspects the spammish repetition").hexdigest() 'a4337bc45a8fc544c03f52dc550cd6e1e87021bc896588bd79e901e2' -A generic :func:`new` constructor that takes the string name of the desired -algorithm as its first parameter also exists to allow access to the above listed -hashes as well as any other algorithms that your OpenSSL library may offer. The -named constructors are much faster than :func:`new` and should be preferred. +.. function:: new(name[, data]) + + Is a generic constructor that takes the string name of the desired + algorithm as its first parameter. It also exists to allow access to the + above listed hashes as well as any other algorithms that your OpenSSL + library may offer. The named constructors are much faster than :func:`new` + and should be preferred. Using :func:`new` with an algorithm provided by OpenSSL: @@ -82,6 +89,25 @@ Using :func:`new` with an algorithm provided by OpenSSL: >>> h.hexdigest() 'cc4a5ce1b3df48aec5d22d1f16b894a0b894eccc' +Hashlib provides the following constant attributes: + +.. data:: algorithms_guaranteed + + Contains the names of the hash algorithms guaranteed to be supported + by this module on all platforms. + + .. versionadded:: 3.2 + +.. data:: algorithms_available + + Contains the names of the hash algorithms that are available + in the running Python interpreter. These names will be recognized + when passed to :func:`new`. :attr:`algorithms_guaranteed` + will always be a subset. Duplicate algorithms with different + name formats may appear in this set (thanks to OpenSSL). + + .. versionadded:: 3.2 + The following values are provided as constant attributes of the hash objects returned by the constructors: diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst index 7735365..c8634ba 100644 --- a/Doc/library/heapq.rst +++ b/Doc/library/heapq.rst @@ -8,6 +8,10 @@ .. sectionauthor:: François Pinard .. sectionauthor:: Raymond Hettinger +**Source code:** :source:`Lib/heapq.py` + +-------------- + This module provides an implementation of the heap queue algorithm, also known as the priority queue algorithm. diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst index b2bd98d..0ca3eda 100644 --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -7,6 +7,9 @@ .. moduleauthor:: Gerhard Häring <ghaering@users.sourceforge.net> .. sectionauthor:: Gerhard Häring <ghaering@users.sourceforge.net> +**Source code:** :source:`Lib/hmac.py` + +-------------- This module implements the HMAC algorithm as described by :rfc:`2104`. diff --git a/Doc/library/html.entities.rst b/Doc/library/html.entities.rst index aa67bae..239ae50 100644 --- a/Doc/library/html.entities.rst +++ b/Doc/library/html.entities.rst @@ -5,6 +5,9 @@ :synopsis: Definitions of HTML general entities. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/html/entities.py` + +-------------- This module defines three dictionaries, ``name2codepoint``, ``codepoint2name``, and ``entitydefs``. ``entitydefs`` is used to provide the :attr:`entitydefs` diff --git a/Doc/library/html.parser.rst b/Doc/library/html.parser.rst index 1fa11a2..06a3b1a 100644 --- a/Doc/library/html.parser.rst +++ b/Doc/library/html.parser.rst @@ -9,12 +9,20 @@ single: HTML single: XHTML +**Source code:** :source:`Lib/html/parser.py` + +-------------- + This module defines a class :class:`HTMLParser` which serves as the basis for parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. -.. class:: HTMLParser() +.. class:: HTMLParser(strict=True) - The :class:`HTMLParser` class is instantiated without arguments. + Create a parser instance. If *strict* is ``True`` (the default), invalid + html results in :exc:`~html.parser.HTMLParseError` exceptions [#]_. If + *strict* is ``False``, the parser uses heuristics to make a best guess at + the intention of any invalid html it encounters, similar to the way most + browsers do. An :class:`HTMLParser` instance is fed HTML data and calls handler functions when tags begin and end. The :class:`HTMLParser` class is meant to be overridden by the @@ -23,6 +31,8 @@ parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. This parser does not check that end tags match start tags or call the end-tag handler for elements which are closed implicitly by closing an outer element. + .. versionchanged:: 3.2 *strict* keyword added + An exception is defined as well: @@ -191,3 +201,8 @@ As a basic example, below is a very basic HTML parser that uses the Encountered a html end tag +.. rubric:: Footnotes + +.. [#] For backward compatibility reasons *strict* mode does not raise + exceptions for all non-compliant HTML. That is, some invalid HTML + is tolerated even in *strict* mode. diff --git a/Doc/library/html.rst b/Doc/library/html.rst new file mode 100644 index 0000000..0063db6 --- /dev/null +++ b/Doc/library/html.rst @@ -0,0 +1,21 @@ +:mod:`html` --- HyperText Markup Language support +================================================= + +.. module:: html + :synopsis: Helpers for manipulating HTML. + +.. versionadded:: 3.2 + +**Source code:** :source:`Lib/html/__init__.py` + +-------------- + +This module defines utilities to manipulate HTML. + +.. function:: escape(s, quote=True) + + Convert the characters ``&``, ``<`` and ``>`` in string *s* to HTML-safe + sequences. Use this if you need to display text that might contain such + characters in HTML. If the optional flag *quote* is true, the characters + (``"``) and (``'``) are also translated; this helps for inclusion in an HTML + attribute value delimited by quotes, as in ``<a href="...">``. diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index cbe4f05..704585b 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -11,30 +11,33 @@ .. index:: module: urllib.request +**Source code:** :source:`Lib/http/client.py` + +-------------- + This module defines classes which implement the client side of the HTTP and HTTPS protocols. It is normally not used directly --- the module :mod:`urllib.request` uses it to handle URLs that use HTTP and HTTPS. .. note:: - HTTPS support is only available if the :mod:`socket` module was compiled with - SSL support. + HTTPS support is only available if Python was compiled with SSL support + (through the :mod:`ssl` module). The module provides the following classes: -.. class:: HTTPConnection(host, port=None, strict=None[, timeout]) +.. class:: HTTPConnection(host, port=None[, strict[, timeout[, source_address]]]) An :class:`HTTPConnection` instance represents one transaction with an HTTP server. It should be instantiated passing it a host and optional port number. If no port number is passed, the port is extracted from the host string if it has the form ``host:port``, else the default HTTP port (80) is - used. When True, the optional parameter *strict* (which defaults to a false - value) causes ``BadStatusLine`` to - be raised if the status line can't be parsed as a valid HTTP/1.0 or 1.1 - status line. If the optional *timeout* parameter is given, blocking + used. If the optional *timeout* parameter is given, blocking operations (like connection attempts) will timeout after that many seconds (if it is not given, the global default timeout setting is used). + The optional *source_address* parameter may be a tuple of a (host, port) + to use as the source address the HTTP connection is made from. For example, the following calls all create instances that connect to the server at the same host and port:: @@ -44,24 +47,58 @@ The module provides the following classes: >>> h3 = http.client.HTTPConnection('www.cwi.nl', 80) >>> h3 = http.client.HTTPConnection('www.cwi.nl', 80, timeout=10) + .. versionchanged:: 3.2 + *source_address* was added. + + .. versionchanged:: 3.2 + The *strict* parameter is deprecated. HTTP 0.9-style "Simple Responses" + are not supported anymore. -.. class:: HTTPSConnection(host, port=None, key_file=None, cert_file=None, strict=None[, timeout]) + +.. class:: HTTPSConnection(host, port=None, key_file=None, cert_file=None[, strict[, timeout[, source_address]]], *, context=None, check_hostname=None) A subclass of :class:`HTTPConnection` that uses SSL for communication with - secure servers. Default port is ``443``. *key_file* is the name of a PEM - formatted file that contains your private key, and *cert_file* is a PEM - formatted certificate chain file; both can be used for authenticating - yourself against the server. + secure servers. Default port is ``443``. If *context* is specified, it + must be a :class:`ssl.SSLContext` instance describing the various SSL + options. If *context* is specified and has a :attr:`~ssl.SSLContext.verify_mode` + of either :data:`~ssl.CERT_OPTIONAL` or :data:`~ssl.CERT_REQUIRED`, then + by default *host* is matched against the host name(s) allowed by the + server's certificate. If you want to change that behaviour, you can + explicitly set *check_hostname* to False. + + *key_file* and *cert_file* are deprecated, please use + :meth:`ssl.SSLContext.load_cert_chain` instead. + + If you access arbitrary hosts on the Internet, it is recommended to + require certificate checking and feed the *context* with a set of + trusted CA certificates:: + + context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context.verify_mode = ssl.CERT_REQUIRED + context.load_verify_locations('/etc/pki/tls/certs/ca-bundle.crt') + h = client.HTTPSConnection('svn.python.org', 443, context=context) + + .. versionchanged:: 3.2 + *source_address*, *context* and *check_hostname* were added. + + .. versionchanged:: 3.2 + This class now supports HTTPS virtual hosts if possible (that is, + if :data:`ssl.HAS_SNI` is true). - .. warning:: - This does not do any verification of the server's certificate. + .. versionchanged:: 3.2 + The *strict* parameter is deprecated. HTTP 0.9-style "Simple Responses" + are not supported anymore. -.. class:: HTTPResponse(sock, debuglevel=0, strict=0, method=None, url=None) +.. class:: HTTPResponse(sock, debuglevel=0[, strict], method=None, url=None) Class whose instances are returned upon successful connection. Not instantiated directly by user. + .. versionchanged:: 3.2 + The *strict* parameter is deprecated. HTTP 0.9-style "Simple Responses" + are not supported anymore. + The following exceptions are raised as appropriate: @@ -360,14 +397,18 @@ HTTPConnection Objects string. The *body* may also be an open :term:`file object`, in which case the - contents of the file is sent; this file object should support - ``fileno()`` and ``read()`` methods. The header Content-Length is - automatically set to the length of the file as reported by - stat. + contents of the file is sent; this file object should support ``fileno()`` + and ``read()`` methods. The header Content-Length is automatically set to + the length of the file as reported by stat. The *body* argument may also be + an iterable and Content-Length header should be explicitly provided when the + body is an iterable. The *headers* argument should be a mapping of extra HTTP headers to send with the request. + .. versionadded:: 3.2 + *body* can now be an iterable. + .. method:: HTTPConnection.getresponse() Should be called after a request is sent to get the response from the server. @@ -389,6 +430,17 @@ HTTPConnection Objects .. versionadded:: 3.1 +.. method:: HTTPConnection.set_tunnel(host, port=None, headers=None) + + Set the host and the port for HTTP Connect Tunnelling. Normally used when it + is required to a HTTPS Connection through a proxy server. + + The headers argument should be a mapping of extra HTTP headers to to sent + with the CONNECT request. + + .. versionadded:: 3.2 + + .. method:: HTTPConnection.connect() Connect to the server specified when the object was created. @@ -511,9 +563,8 @@ Here is an example session that uses the ``GET`` method:: >>> data2 = r2.read() >>> conn.close() -Here is an example session that uses ``HEAD`` method. Note that ``HEAD`` method -never returns any data. :: - +Here is an example session that uses the ``HEAD`` method. Note that the +``HEAD`` method never returns any data. :: >>> import http.client >>> conn = http.client.HTTPConnection("www.python.org") diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst index 74d8d16..9771496 100644 --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -6,6 +6,9 @@ .. moduleauthor:: John J. Lee <jjl@pobox.com> .. sectionauthor:: John J. Lee <jjl@pobox.com> +**Source code:** :source:`Lib/http/cookiejar.py` + +-------------- The :mod:`http.cookiejar` module defines classes for automatic handling of HTTP cookies. It is useful for accessing web sites that require small pieces of data diff --git a/Doc/library/http.cookies.rst b/Doc/library/http.cookies.rst index 472ddcf..d8a437b 100644 --- a/Doc/library/http.cookies.rst +++ b/Doc/library/http.cookies.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Timothy O'Malley <timo@alum.mit.edu> .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/http/cookies.py` + +-------------- The :mod:`http.cookies` module defines classes for abstracting the concept of cookies, an HTTP state management mechanism. It supports both simple string-only diff --git a/Doc/library/http.server.rst b/Doc/library/http.server.rst index 1ca1620..e3a3a10 100644 --- a/Doc/library/http.server.rst +++ b/Doc/library/http.server.rst @@ -11,6 +11,10 @@ single: URL single: httpd +**Source code:** :source:`Lib/http/server.py` + +-------------- + This module defines classes for implementing HTTP servers (Web servers). One class, :class:`HTTPServer`, is a :class:`socketserver.TCPServer` subclass. @@ -155,6 +159,17 @@ of which this module provides three different variants: This method will parse and dispatch the request to the appropriate :meth:`do_\*` method. You should never need to override it. + .. method:: handle_expect_100() + + When a HTTP/1.1 compliant server receives a ``Expect: 100-continue`` + request header it responds back with a ``100 Continue`` followed by ``200 + OK`` headers. + This method can be overridden to raise an error if the server does not + want the client to continue. For e.g. server can chose to send ``417 + Expectation Failed`` as a response header and ``return False``. + + .. versionadded:: 3.2 + .. method:: send_error(code, message=None) Sends and logs a complete error reply to the client. The numeric *code* @@ -171,13 +186,29 @@ of which this module provides three different variants: .. method:: send_header(keyword, value) - Writes a specific HTTP header to the output stream. *keyword* should - specify the header keyword, with *value* specifying its value. + Stores the HTTP header to an internal buffer which will be written to the + output stream when :meth:`end_headers` method is invoked. + *keyword* should specify the header keyword, with *value* + specifying its value. + + .. versionchanged:: 3.2 Storing the headers in an internal buffer + + + .. method:: send_response_only(code, message=None) + + Sends the reponse header only, used for the purposes when ``100 + Continue`` response is sent by the server to the client. The headers not + buffered and sent directly the output stream.If the *message* is not + specified, the HTTP message corresponding the response *code* is sent. + + .. versionadded:: 3.2 .. method:: end_headers() - Sends a blank line, indicating the end of the HTTP headers in the - response. + Write the buffered HTTP headers to the output stream and send a blank + line, indicating the end of the HTTP headers in the response. + + .. versionchanged:: 3.2 Writing the buffered headers to the output stream. .. method:: log_request(code='-', size='-') diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index 04088ac..1d92fe5 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -16,6 +16,10 @@ pair: IMAP4_SSL; protocol pair: IMAP4_stream; protocol +**Source code:** :source:`Lib/imaplib.py` + +-------------- + This module defines three classes, :class:`IMAP4`, :class:`IMAP4_SSL` and :class:`IMAP4_stream`, which encapsulate a connection to an IMAP4 server and implement a large subset of the IMAP4rev1 client protocol as defined in @@ -56,6 +60,7 @@ Three exceptions are defined as attributes of the :class:`IMAP4` class: write permission, and the mailbox will need to be re-opened to re-obtain write permission. + There's also a subclass for secure connections: @@ -68,6 +73,7 @@ There's also a subclass for secure connections: and *certfile* are also optional - they can contain a PEM formatted private key and certificate chain file for the SSL connection. + The second subclass allows for connections created by a child process: @@ -83,9 +89,9 @@ The following utility functions are defined: .. function:: Internaldate2tuple(datestr) - Converts an IMAP4 INTERNALDATE string to Coordinated Universal Time. Returns a - :mod:`time` module tuple. - + Parse an IMAP4 ``INTERNALDATE`` string and return corresponding local + time. The return value is a :class:`time.struct_time` tuple or + None if the string has wrong format. .. function:: Int2AP(num) @@ -100,9 +106,13 @@ The following utility functions are defined: .. function:: Time2Internaldate(date_time) - Converts a :mod:`time` module tuple to an IMAP4 ``INTERNALDATE`` representation. - Returns a string in the form: ``"DD-Mmm-YYYY HH:MM:SS +HHMM"`` (including - double-quotes). + Convert *date_time* to an IMAP4 ``INTERNALDATE`` representation. The + return value is a string in the form: ``"DD-Mmm-YYYY HH:MM:SS + +HHMM"`` (including double-quotes). The *date_time* argument can be a + number (int or float) represening seconds since epoch (as returned + by :func:`time.time`), a 9-tuple representing local time (as returned by + :func:`time.localtime`), or a double-quoted string. In the last case, it + is assumed to already be in the correct format. Note that IMAP4 message numbers change as the mailbox changes; in particular, after an ``EXPUNGE`` command performs deletions the remaining messages are @@ -408,6 +418,15 @@ An :class:`IMAP4` instance has the following methods: This is an ``IMAP4rev1`` extension command. +.. method:: IMAP4.starttls(ssl_context=None) + + Send a ``STARTTLS`` command. The *ssl_context* argument is optional + and should be a :class:`ssl.SSLContext` object. This will enable + encryption on the IMAP connection. + + .. versionadded:: 3.2 + + .. method:: IMAP4.status(mailbox, names) Request named status conditions for *mailbox*. diff --git a/Doc/library/imghdr.rst b/Doc/library/imghdr.rst index 0c0722d..32ec9cf 100644 --- a/Doc/library/imghdr.rst +++ b/Doc/library/imghdr.rst @@ -4,6 +4,9 @@ .. module:: imghdr :synopsis: Determine the type of image contained in a file or byte stream. +**Source code:** :source:`Lib/imghdr.py` + +-------------- The :mod:`imghdr` module determines the type of image contained in a file or byte stream. diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst index 2d83893..6e9845e 100644 --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -190,8 +190,43 @@ This module provides an interface to the mechanisms used to implement the continue to use the old class definition. The same is true for derived classes. -The following constants with integer values, defined in this module, are used to -indicate the search result of :func:`find_module`. +The following functions are conveniences for handling :pep:`3147` byte-compiled +file paths. + +.. versionadded:: 3.2 + +.. function:: cache_from_source(path, debug_override=None) + + Return the :pep:`3147` path to the byte-compiled file associated with the + source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return + value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. + The ``cpython-32`` string comes from the current magic tag (see + :func:`get_tag`). The returned path will end in ``.pyc`` when + ``__debug__`` is True or ``.pyo`` for an optimized Python + (i.e. ``__debug__`` is False). By passing in True or False for + *debug_override* you can override the system's value for ``__debug__`` for + extension selection. + + *path* need not exist. + + +.. function:: source_from_cache(path) + + Given the *path* to a :pep:`3147` file name, return the associated source code + file path. For example, if *path* is + ``/foo/bar/__pycache__/baz.cpython-32.pyc`` the returned path would be + ``/foo/bar/baz.py``. *path* need not exist, however if it does not conform + to :pep:`3147` format, a ``ValueError`` is raised. + + +.. function:: get_tag() + + Return the :pep:`3147` magic tag string matching this version of Python's + magic number, as returned by :func:`get_magic`. + + +The following constants with integer values, defined in this module, are used +to indicate the search result of :func:`find_module`. .. data:: PY_SOURCE @@ -273,10 +308,3 @@ in that version, since :func:`find_module` has been extended and # Since we may exit via an exception, close fp explicitly. if fp: fp.close() - -.. index:: module: knee - -A more complete example that implements hierarchical module names and includes a -:func:`reload` function can be found in the module :mod:`knee`. The :mod:`knee` -module can be found in :file:`Demo/imputil/` in the Python source distribution. - diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index cf13ba3..c9f742a 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -18,12 +18,12 @@ implementation of the :keyword:`import` statement (and thus, by extension, the :func:`__import__` function) in Python source code. This provides an implementation of :keyword:`import` which is portable to any Python interpreter. This also provides a reference implementation which is easier to -comprehend than one in a programming language other than Python. +comprehend than one implemented in a programming language other than Python. -Two, the components to implement :keyword:`import` can be exposed in this +Two, the components to implement :keyword:`import` are exposed in this package, making it easier for users to create their own custom objects (known generically as an :term:`importer`) to participate in the import process. -Details on providing custom importers can be found in :pep:`302`. +Details on custom importers can be found in :pep:`302`. .. seealso:: @@ -32,12 +32,11 @@ Details on providing custom importers can be found in :pep:`302`. `Packages specification <http://www.python.org/doc/essays/packages.html>`__ Original specification of packages. Some semantics have changed since - the writing of this document (e.g. redirecting based on :keyword:`None` + the writing of this document (e.g. redirecting based on ``None`` in :data:`sys.modules`). The :func:`.__import__` function - The built-in function for which the :keyword:`import` statement is - syntactic sugar. + The :keyword:`import` statement is syntactic sugar for this function. :pep:`235` Import on Case-Insensitive Platforms @@ -46,7 +45,7 @@ Details on providing custom importers can be found in :pep:`302`. Defining Python Source Code Encodings :pep:`302` - New Import Hooks. + New Import Hooks :pep:`328` Imports: Multi-Line and Absolute/Relative @@ -57,14 +56,16 @@ Details on providing custom importers can be found in :pep:`302`. :pep:`3120` Using UTF-8 as the Default Source Encoding + :pep:`3147` + PYC Repository Directories + Functions --------- .. function:: __import__(name, globals={}, locals={}, fromlist=list(), level=0) - An implementation of the built-in :func:`__import__` function. See the - built-in function's documentation for usage instructions. + An implementation of the built-in :func:`__import__` function. .. function:: import_module(name, package=None) @@ -108,7 +109,7 @@ are also provided to help in implementing the core ABCs. module. If the :term:`finder` is found on :data:`sys.meta_path` and the module to be searched for is a subpackage or module then *path* will be the value of :attr:`__path__` from the parent package. If a loader - cannot be found, :keyword:`None` is returned. + cannot be found, ``None`` is returned. .. class:: Loader @@ -184,14 +185,14 @@ are also provided to help in implementing the core ABCs. .. method:: get_code(fullname) An abstract method to return the :class:`code` object for a module. - :keyword:`None` is returned if the module does not have a code object + ``None`` is returned if the module does not have a code object (e.g. built-in module). :exc:`ImportError` is raised if loader cannot find the requested module. .. method:: get_source(fullname) An abstract method to return the source of a module. It is returned as - a text string with universal newlines. Returns :keyword:`None` if no + a text string with universal newlines. Returns ``None`` if no source is available (e.g. a built-in module). Raises :exc:`ImportError` if the loader cannot find the module specified. @@ -202,21 +203,133 @@ are also provided to help in implementing the core ABCs. :term:`loader` cannot find the module. +.. class:: ExecutionLoader + + An abstract base class which inherits from :class:`InspectLoader` that, + when implemented, helps a module to be executed as a script. The ABC + represents an optional :pep:`302` protocol. + + .. method:: get_filename(fullname) + + An abstract method that is to return the value of :attr:`__file__` for + the specified module. If no path is available, :exc:`ImportError` is + raised. + + If source code is available, then the method should return the path to + the source file, regardless of whether a bytecode was used to load the + module. + + +.. class:: SourceLoader + + An abstract base class for implementing source (and optionally bytecode) + file loading. The class inherits from both :class:`ResourceLoader` and + :class:`ExecutionLoader`, requiring the implementation of: + + * :meth:`ResourceLoader.get_data` + * :meth:`ExecutionLoader.get_filename` + Should only return the path to the source file; sourceless + loading is not supported. + + The abstract methods defined by this class are to add optional bytecode + file support. Not implementing these optional methods causes the loader to + only work with source code. Implementing the methods allows the loader to + work with source *and* bytecode files; it does not allow for *sourceless* + loading where only bytecode is provided. Bytecode files are an + optimization to speed up loading by removing the parsing step of Python's + compiler, and so no bytecode-specific API is exposed. + + .. method:: path_mtime(self, path) + + Optional abstract method which returns the modification time for the + specified path. + + .. method:: set_data(self, path, data) + + Optional abstract method which writes the specified bytes to a file + path. Any intermediate directories which do not exist are to be created + automatically. + + When writing to the path fails because the path is read-only + (:attr:`errno.EACCES`), do not propagate the exception. + + .. method:: get_code(self, fullname) + + Concrete implementation of :meth:`InspectLoader.get_code`. + + .. method:: load_module(self, fullname) + + Concrete implementation of :meth:`Loader.load_module`. + + .. method:: get_source(self, fullname) + + Concrete implementation of :meth:`InspectLoader.get_source`. + + .. method:: is_package(self, fullname) + + Concrete implementation of :meth:`InspectLoader.is_package`. A module + is determined to be a package if its file path is a file named + ``__init__`` when the file extension is removed. + + .. class:: PyLoader - An abstract base class inheriting from :class:`importlib.abc.InspectLoader` - and :class:`importlib.abc.ResourceLoader` designed to ease the loading of + An abstract base class inheriting from + :class:`ExecutionLoader` and + :class:`ResourceLoader` designed to ease the loading of Python source modules (bytecode is not handled; see - :class:`importlib.abc.PyPycLoader` for a source/bytecode ABC). A subclass + :class:`SourceLoader` for a source/bytecode ABC). A subclass implementing this ABC will only need to worry about exposing how the source code is stored; all other details for loading Python source code will be handled by the concrete implementations of key methods. + .. deprecated:: 3.2 + This class has been deprecated in favor of :class:`SourceLoader` and is + slated for removal in Python 3.4. See below for how to create a + subclass that is compatible with Python 3.1 onwards. + + If compatibility with Python 3.1 is required, then use the following idiom + to implement a subclass that will work with Python 3.1 onwards (make sure + to implement :meth:`ExecutionLoader.get_filename`):: + + try: + from importlib.abc import SourceLoader + except ImportError: + from importlib.abc import PyLoader as SourceLoader + + + class CustomLoader(SourceLoader): + def get_filename(self, fullname): + """Return the path to the source file.""" + # Implement ... + + def source_path(self, fullname): + """Implement source_path in terms of get_filename.""" + try: + return self.get_filename(fullname) + except ImportError: + return None + + def is_package(self, fullname): + """Implement is_package by looking for an __init__ file + name as returned by get_filename.""" + filename = os.path.basename(self.get_filename(fullname)) + return os.path.splitext(filename)[0] == '__init__' + + .. method:: source_path(fullname) An abstract method that returns the path to the source code for a - module. Should return :keyword:`None` if there is no source code. - :exc:`ImportError` if the module cannot be found. + module. Should return ``None`` if there is no source code. + Raises :exc:`ImportError` if the loader knows it cannot handle the + module. + + .. method:: get_filename(fullname) + + A concrete implementation of + :meth:`importlib.abc.ExecutionLoader.get_filename` that + relies on :meth:`source_path`. If :meth:`source_path` returns + ``None``, then :exc:`ImportError` is raised. .. method:: load_module(fullname) @@ -231,43 +344,62 @@ are also provided to help in implementing the core ABCs. A concrete implementation of :meth:`importlib.abc.InspectLoader.get_code` that creates code objects from Python source code, by requesting the source code (using - :meth:`source_path` and :meth:`get_data`), converting it to standard - newlines, and compiling it with the built-in :func:`compile` function. + :meth:`source_path` and :meth:`get_data`) and compiling it with the + built-in :func:`compile` function. .. method:: get_source(fullname) A concrete implementation of :meth:`importlib.abc.InspectLoader.get_source`. Uses - :meth:`importlib.abc.ResourceLoader.get_data` and :meth:`source_path` to - get the source code. It tries to guess the source encoding using + :meth:`importlib.abc.ResourceLoader.get_data` and :meth:`source_path` + to get the source code. It tries to guess the source encoding using :func:`tokenize.detect_encoding`. .. class:: PyPycLoader - An abstract base class inheriting from :class:`importlib.abc.PyLoader`. + An abstract base class inheriting from :class:`PyLoader`. This ABC is meant to help in creating loaders that support both Python source and bytecode. + .. deprecated:: 3.2 + This class has been deprecated in favor of :class:`SourceLoader` and to + properly support :pep:`3147`. If compatibility is required with + Python 3.1, implement both :class:`SourceLoader` and :class:`PyLoader`; + instructions on how to do so are included in the documentation for + :class:`PyLoader`. Do note that this solution will not support + sourceless/bytecode-only loading; only source *and* bytecode loading. + .. method:: source_mtime(fullname) An abstract method which returns the modification time for the source code of the specified module. The modification time should be an - integer. If there is no source code, return :keyword:`None`. If the + integer. If there is no source code, return ``None``. If the module cannot be found then :exc:`ImportError` is raised. .. method:: bytecode_path(fullname) An abstract method which returns the path to the bytecode for the - specified module, if it exists. It returns :keyword:`None` + specified module, if it exists. It returns ``None`` if no bytecode exists (yet). - Raises :exc:`ImportError` if the module is not found. + Raises :exc:`ImportError` if the loader knows it cannot handle the + module. + + .. method:: get_filename(fullname) + + A concrete implementation of + :meth:`ExecutionLoader.get_filename` that relies on + :meth:`PyLoader.source_path` and :meth:`bytecode_path`. + If :meth:`source_path` returns a path, then that value is returned. + Else if :meth:`bytecode_path` returns a path, that path will be + returned. If a path is not available from both methods, + :exc:`ImportError` is raised. .. method:: write_bytecode(fullname, bytecode) An abstract method which has the loader write *bytecode* for future - use. If the bytecode is written, return :keyword:`True`. Return - :keyword:`False` if the bytecode could not be written. This method + use. If the bytecode is written, return ``True``. Return + ``False`` if the bytecode could not be written. This method should not be called if :data:`sys.dont_write_bytecode` is true. The *bytecode* argument should be a bytes string or bytes array. @@ -311,7 +443,7 @@ find and load modules. terms of :data:`sys.path`. No implicit path hooks are assumed for simplification of the class and its semantics. - Only class method are defined by this class to alleviate the need for + Only class methods are defined by this class to alleviate the need for instantiation. .. classmethod:: find_module(fullname, path=None) @@ -325,7 +457,7 @@ find and load modules. :data:`sys.path_importer_cache`, then :data:`sys.path_hooks` is searched for a finder for the path entry and, if found, is stored in :data:`sys.path_importer_cache` along with being queried about the - module. If no finder is ever found then :keyword:`None` is returned. + module. If no finder is ever found then ``None`` is returned. :mod:`importlib.util` -- Utility code for importers @@ -337,7 +469,7 @@ find and load modules. This module contains the various objects that help in the construction of an :term:`importer`. -.. function:: module_for_loader(method) +.. decorator:: module_for_loader A :term:`decorator` for a :term:`loader` method, to handle selecting the proper @@ -362,7 +494,7 @@ an :term:`importer`. Use of this decorator handles all the details of which module object a loader should initialize as specified by :pep:`302`. -.. function:: set_loader(fxn) +.. decorator:: set_loader A :term:`decorator` for a :term:`loader` method, to set the :attr:`__loader__` @@ -370,11 +502,11 @@ an :term:`importer`. does nothing. It is assumed that the first positional argument to the wrapped method is what :attr:`__loader__` should be set to. -.. function:: set_package(fxn) +.. decorator:: set_package A :term:`decorator` for a :term:`loader` to set the :attr:`__package__` attribute on the module returned by the loader. If :attr:`__package__` is - set and has a value other than :keyword:`None` it will not be changed. + set and has a value other than ``None`` it will not be changed. Note that the module returned by the loader is what has the attribute set on and not the module found in :data:`sys.modules`. @@ -384,100 +516,3 @@ an :term:`importer`. attribute to be used at the global level of the module during initialization. - -Example -------- - -Below is an example meta path importer that uses a dict for back-end storage -for source code. While not an optimal solution -- manipulations of -:attr:`__path__` on packages does not influence import -- it does illustrate -what little is required to implement an importer. - -.. testcode:: - - """An importer where source is stored in a dict.""" - from importlib import abc - - - class DictImporter(abc.Finder, abc.PyLoader): - - """A meta path importer that stores source code in a dict. - - The keys are the module names -- packages must end in ``.__init__``. - The values must be something that can be passed to 'bytes'. - - """ - - def __init__(self, memory): - """Store the dict.""" - self.memory = memory - - def contains(self, name): - """See if a module or package is in the dict.""" - if name in self.memory: - return name - package_name = '{}.__init__'.format(name) - if package_name in self.memory: - return package_name - return False - - __contains__ = contains # Convenience. - - def find_module(self, fullname, path=None): - """Find the module in the dict.""" - if fullname in self: - return self - return None - - def source_path(self, fullname): - """Return the module name if the module is in the dict.""" - if not fullname in self: - raise ImportError - return fullname - - def get_data(self, path): - """Return the bytes for the source. - - The value found in the dict is passed through 'bytes' before being - returned. - - """ - name = self.contains(path) - if not name: - raise IOError - return bytes(self.memory[name]) - - def is_package(self, fullname): - """Tell if module is a package based on whether the dict contains the - name with ``.__init__`` appended to it.""" - if fullname not in self: - raise ImportError - if fullname in self.memory: - return False - # If name is in this importer but not as it is then it must end in - # ``__init__``. - else: - return True - -.. testcode:: - :hide: - - import importlib - import sys - - - # Build the dict; keys of name, value of __package__. - names = {'_top_level': '', '_pkg.__init__': '_pkg', '_pkg.mod': '_pkg'} - source = {name: "name = {!r}".format(name).encode() for name in names} - - # Register the meta path importer. - importer = DictImporter(source) - sys.meta_path.append(importer) - - # Sanity check. - for name in names: - module = importlib.import_module(name) - assert module.__name__ == name - assert getattr(module, 'name') == name - assert module.__loader__ is importer - assert module.__package__ == names[name] diff --git a/Doc/library/index.rst b/Doc/library/index.rst index aa582de..9ac688c 100644 --- a/Doc/library/index.rst +++ b/Doc/library/index.rst @@ -43,13 +43,13 @@ the `Python Package Index <http://pypi.python.org/pypi>`_. intro.rst functions.rst constants.rst - objects.rst stdtypes.rst exceptions.rst strings.rst datatypes.rst numeric.rst + functional.rst filesys.rst persistence.rst archiving.rst diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 4def286..7a57a0d 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Ka-Ping Yee <ping@lfw.org> .. sectionauthor:: Ka-Ping Yee <ping@lfw.org> +**Source code:** :source:`Lib/inspect.py` + +-------------- The :mod:`inspect` module provides several useful functions to help get information about live objects such as modules, classes, methods, functions, @@ -87,7 +90,7 @@ attributes: | frame | f_back | next outer frame object | | | | (this frame's caller) | +-----------+-----------------+---------------------------+ -| | f_builtins | built-in namespace seen | +| | f_builtins | builtins namespace seen | | | | by this frame | +-----------+-----------------+---------------------------+ | | f_code | code object being | @@ -295,7 +298,7 @@ attributes: .. impl-detail:: getsets are attributes defined in extension modules via - :ctype:`PyGetSetDef` structures. For Python implementations without such + :c:type:`PyGetSetDef` structures. For Python implementations without such types, this method will always return ``False``. @@ -306,7 +309,7 @@ attributes: .. impl-detail:: Member descriptors are attributes defined in extension modules via - :ctype:`PyMemberDef` structures. For Python implementations without such + :c:type:`PyMemberDef` structures. For Python implementations without such types, this method will always return ``False``. @@ -451,6 +454,32 @@ Classes and functions metatype is in use, cls will be the first element of the tuple. +.. function:: getcallargs(func[, *args][, **kwds]) + + Bind the *args* and *kwds* to the argument names of the Python function or + method *func*, as if it was called with them. For bound methods, bind also the + first argument (typically named ``self``) to the associated instance. A dict + is returned, mapping the argument names (including the names of the ``*`` and + ``**`` arguments, if any) to their values from *args* and *kwds*. In case of + invoking *func* incorrectly, i.e. whenever ``func(*args, **kwds)`` would raise + an exception because of incompatible signature, an exception of the same type + and the same or similar message is raised. For example:: + + >>> from inspect import getcallargs + >>> def f(a, b=1, *pos, **named): + ... pass + >>> getcallargs(f, 1, 2, 3) + {'a': 1, 'named': {}, 'b': 2, 'pos': (3,)} + >>> getcallargs(f, a=2, x=4) + {'a': 2, 'named': {'x': 4}, 'b': 1, 'pos': ()} + >>> getcallargs(f) + Traceback (most recent call last): + ... + TypeError: f() takes at least 1 argument (0 given) + + .. versionadded:: 3.2 + + .. _inspect-stack: The interpreter stack @@ -536,3 +565,81 @@ line. entry in the list represents the caller; the last entry represents where the exception was raised. + +Fetching attributes statically +------------------------------ + +Both :func:`getattr` and :func:`hasattr` can trigger code execution when +fetching or checking for the existence of attributes. Descriptors, like +properties, will be invoked and :meth:`__getattr__` and :meth:`__getattribute__` +may be called. + +For cases where you want passive introspection, like documentation tools, this +can be inconvenient. `getattr_static` has the same signature as :func:`getattr` +but avoids executing code when it fetches attributes. + +.. function:: getattr_static(obj, attr, default=None) + + Retrieve attributes without triggering dynamic lookup via the + descriptor protocol, `__getattr__` or `__getattribute__`. + + Note: this function may not be able to retrieve all attributes + that getattr can fetch (like dynamically created attributes) + and may find attributes that getattr can't (like descriptors + that raise AttributeError). It can also return descriptors objects + instead of instance members. + + If the instance `__dict__` is shadowed by another member (for example a + property) then this function will be unable to find instance members. + + .. versionadded:: 3.2 + +`getattr_static` does not resolve descriptors, for example slot descriptors or +getset descriptors on objects implemented in C. The descriptor object +is returned instead of the underlying attribute. + +You can handle these with code like the following. Note that +for arbitrary getset descriptors invoking these may trigger +code execution:: + + # example code for resolving the builtin descriptor types + class _foo: + __slots__ = ['foo'] + + slot_descriptor = type(_foo.foo) + getset_descriptor = type(type(open(__file__)).name) + wrapper_descriptor = type(str.__dict__['__add__']) + descriptor_types = (slot_descriptor, getset_descriptor, wrapper_descriptor) + + result = getattr_static(some_object, 'foo') + if type(result) in descriptor_types: + try: + result = result.__get__() + except AttributeError: + # descriptors can raise AttributeError to + # indicate there is no underlying value + # in which case the descriptor itself will + # have to do + pass + + +Current State of a Generator +---------------------------- + +When implementing coroutine schedulers and for other advanced uses of +generators, it is useful to determine whether a generator is currently +executing, is waiting to start or resume or execution, or has already +terminated. :func:`getgeneratorstate` allows the current state of a +generator to be determined easily. + +.. function:: getgeneratorstate(generator) + + Get current state of a generator-iterator. + + Possible states are: + * GEN_CREATED: Waiting to start execution. + * GEN_RUNNING: Currently being executed by the interpreter. + * GEN_SUSPENDED: Currently suspended at a yield expression. + * GEN_CLOSED: Execution has completed. + + .. versionadded:: 3.2 diff --git a/Doc/library/io.rst b/Doc/library/io.rst index bbe5112..0584b70 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -513,6 +513,24 @@ than raw I/O does. :class:`BytesIO` provides or overrides these methods in addition to those from :class:`BufferedIOBase` and :class:`IOBase`: + .. method:: getbuffer() + + Return a readable and writable view over the contents of the buffer + without copying them. Also, mutating the view will transparently + update the contents of the buffer:: + + >>> b = io.BytesIO(b"abcdef") + >>> view = b.getbuffer() + >>> view[2:4] = b"56" + >>> b.getvalue() + b'ab56ef' + + .. note:: + As long as the view exists, the :class:`BytesIO` object cannot be + resized. + + .. versionadded:: 3.2 + .. method:: getvalue() Return ``bytes`` containing the entire contents of the buffer. @@ -767,37 +785,33 @@ Text I/O inherits :class:`codecs.IncrementalDecoder`. -Advanced topics ---------------- - -Here we will discuss several advanced topics pertaining to the concrete -I/O implementations described above. - Performance -^^^^^^^^^^^ +----------- + +This section discusses the performance of the provided concrete I/O +implementations. Binary I/O -"""""""""" - -By reading and writing only large chunks of data even when the user asks -for a single byte, buffered I/O is designed to hide any inefficiency in -calling and executing the operating system's unbuffered I/O routines. The -gain will vary very much depending on the OS and the kind of I/O which is -performed (for example, on some contemporary OSes such as Linux, unbuffered -disk I/O can be as fast as buffered I/O). The bottom line, however, is -that buffered I/O will offer you predictable performance regardless of the -platform and the backing device. Therefore, it is most always preferable to -use buffered I/O rather than unbuffered I/O. +^^^^^^^^^^ + +By reading and writing only large chunks of data even when the user asks for a +single byte, buffered I/O hides any inefficiency in calling and executing the +operating system's unbuffered I/O routines. The gain depends on the OS and the +kind of I/O which is performed. For example, on some modern OSes such as Linux, +unbuffered disk I/O can be as fast as buffered I/O. The bottom line, however, +is that buffered I/O offers predictable performance regardless of the platform +and the backing device. Therefore, it is most always preferable to use buffered +I/O rather than unbuffered I/O for binary datal Text I/O -"""""""" +^^^^^^^^ Text I/O over a binary storage (such as a file) is significantly slower than -binary I/O over the same storage, because it implies conversions from -unicode to binary data using a character codec. This can become noticeable -if you handle huge amounts of text data (for example very large log files). -Also, :meth:`TextIOWrapper.tell` and :meth:`TextIOWrapper.seek` are both -quite slow due to the reconstruction algorithm used. +binary I/O over the same storage, because it requires conversions between +unicode and binary data using a character codec. This can become noticeable +handling huge amounts of text data like large log files. Also, +:meth:`TextIOWrapper.tell` and :meth:`TextIOWrapper.seek` are both quite slow +due to the reconstruction algorithm used. :class:`StringIO`, however, is a native in-memory unicode container and will exhibit similar speed to :class:`BytesIO`. @@ -805,9 +819,8 @@ exhibit similar speed to :class:`BytesIO`. Multi-threading ^^^^^^^^^^^^^^^ -:class:`FileIO` objects are thread-safe to the extent that the operating -system calls (such as ``read(2)`` under Unix) they are wrapping are thread-safe -too. +:class:`FileIO` objects are thread-safe to the extent that the operating system +calls (such as ``read(2)`` under Unix) they wrap are thread-safe too. Binary buffered objects (instances of :class:`BufferedReader`, :class:`BufferedWriter`, :class:`BufferedRandom` and :class:`BufferedRWPair`) @@ -822,12 +835,13 @@ Reentrancy Binary buffered objects (instances of :class:`BufferedReader`, :class:`BufferedWriter`, :class:`BufferedRandom` and :class:`BufferedRWPair`) are not reentrant. While reentrant calls will not happen in normal situations, -they can arise if you are doing I/O in a :mod:`signal` handler. If it is -attempted to enter a buffered object again while already being accessed -*from the same thread*, then a :exc:`RuntimeError` is raised. - -The above implicitly extends to text files, since the :func:`open()` -function will wrap a buffered object inside a :class:`TextIOWrapper`. This -includes standard streams and therefore affects the built-in function -:func:`print()` as well. +they can arise from doing I/O in a :mod:`signal` handler. If a thread tries to +renter a buffered object which it is already accessing, a :exc:`RuntimeError` is +raised. Note this doesn't prohibit a different thread from entering the +buffered object. + +The above implicitly extends to text files, since the :func:`open()` function +will wrap a buffered object inside a :class:`TextIOWrapper`. This includes +standard streams and therefore affects the built-in function :func:`print()` as +well. diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index add2091..757823d 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -46,6 +46,7 @@ Iterator Arguments Results ==================== ============================ ================================================= ============================================================= Iterator Arguments Results Example ==================== ============================ ================================================= ============================================================= +:func:`accumulate` p p0, p0+p1, p0+p1+p2, ... ``accumulate([1,2,3,4,5]) --> 1 3 6 10 15`` :func:`chain` p, q, ... p0, p1, ... plast, q0, q1, ... ``chain('ABC', 'DEF') --> A B C D E F`` :func:`compress` data, selectors (d[0] if s[0]), (d[1] if s[1]), ... ``compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F`` :func:`dropwhile` pred, seq seq[n], seq[n+1], starting when pred fails ``dropwhile(lambda x: x<5, [1,4,6,4,1]) --> 6 4 1`` @@ -83,6 +84,22 @@ The following module functions all construct and return iterators. Some provide streams of infinite length, so they should only be accessed by functions or loops that truncate the stream. +.. function:: accumulate(iterable) + + Make an iterator that returns accumulated sums. Elements may be any addable + type including :class:`Decimal` or :class:`Fraction`. Equivalent to:: + + def accumulate(iterable): + 'Return running totals' + # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 + it = iter(iterable) + total = next(it) + yield total + for element in it: + total = total + element + yield total + + .. versionadded:: 3.2 .. function:: chain(*iterables) @@ -560,8 +577,8 @@ loops that truncate the stream. .. _itertools-recipes: -Recipes -------- +Itertools Recipes +----------------- This section shows recipes for creating an extended toolset using the existing itertools as building blocks. @@ -653,6 +670,12 @@ which incur interpreter overhead. pending -= 1 nexts = cycle(islice(nexts, pending)) + def partition(pred, iterable): + 'Use a predicate to partition entries into false entries and true entries' + # partition(is_odd, range(10)) --> 0 2 4 6 8 and 1 3 5 7 9 + t1, t2 = tee(iterable) + return filterfalse(pred, t1), filter(pred, t2) + def powerset(iterable): "powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)" s = list(iterable) diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 48acaf1..4ee17f2 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -6,7 +6,7 @@ .. moduleauthor:: Bob Ippolito <bob@redivi.com> .. sectionauthor:: Bob Ippolito <bob@redivi.com> -JSON (JavaScript Object Notation) <http://json.org> is a subset of JavaScript +`JSON (JavaScript Object Notation) <http://json.org>`_ is a subset of JavaScript syntax (ECMA-262 3rd edition) used as a lightweight data interchange format. :mod:`json` exposes an API familiar to users of the standard library @@ -134,10 +134,12 @@ Basic Usage ``inf``, ``-inf``) in strict compliance of the JSON specification, instead of using the JavaScript equivalents (``NaN``, ``Infinity``, ``-Infinity``). - If *indent* is a non-negative integer, then JSON array elements and object - members will be pretty-printed with that indent level. An indent level of 0, - or negative, will only insert newlines. ``None`` (the default) selects the - most compact representation. + If *indent* is a non-negative integer or string, then JSON array elements and + object members will be pretty-printed with that indent level. An indent level + of 0, negative, or ``""`` will only insert newlines. ``None`` (the default) + selects the most compact representation. Using a positive integer indent + indents that many spaces per level. If *indent* is a string (such at '\t'), + that string is used to indent each level. If *separators* is an ``(item_separator, dict_separator)`` tuple, then it will be used instead of the default ``(', ', ': ')`` separators. ``(',', diff --git a/Doc/library/keyword.rst b/Doc/library/keyword.rst index a4bfa50..173db23 100644 --- a/Doc/library/keyword.rst +++ b/Doc/library/keyword.rst @@ -4,6 +4,9 @@ .. module:: keyword :synopsis: Test whether a string is a keyword in Python. +**Source code:** :source:`Lib/keyword.py` + +-------------- This module allows a Python program to determine if a string is a keyword. @@ -18,4 +21,3 @@ This module allows a Python program to determine if a string is a keyword. Sequence containing all the keywords defined for the interpreter. If any keywords are defined to only be active when particular :mod:`__future__` statements are in effect, these will be included as well. - diff --git a/Doc/library/language.rst b/Doc/library/language.rst index 3d907a0..1eac32e 100644 --- a/Doc/library/language.rst +++ b/Doc/library/language.rst @@ -26,4 +26,3 @@ These modules include: compileall.rst dis.rst pickletools.rst - distutils.rst diff --git a/Doc/library/linecache.rst b/Doc/library/linecache.rst index 688e297..dacf8aa 100644 --- a/Doc/library/linecache.rst +++ b/Doc/library/linecache.rst @@ -5,6 +5,9 @@ :synopsis: This module provides random access to individual lines from text files. .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/linecache.py` + +-------------- The :mod:`linecache` module allows one to get any line from any file, while attempting to optimize internally, using a cache, the common case where many diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index 5b2941e..dd6f954 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -215,7 +215,7 @@ The :mod:`locale` module defines the following exception and functions: .. note:: - The expression is in the syntax suitable for the :cfunc:`regex` function + The expression is in the syntax suitable for the :c:func:`regex` function from the C library, which might differ from the syntax used in :mod:`re`. .. data:: NOEXPR @@ -535,7 +535,7 @@ catalogs, and the C library's search algorithms for locating message catalogs. Python applications should normally find no need to invoke these functions, and should use :mod:`gettext` instead. A known exception to this rule are applications that link with additional C libraries which internally invoke -:cfunc:`gettext` or :func:`dcgettext`. For these applications, it may be +:c:func:`gettext` or :func:`dcgettext`. For these applications, it may be necessary to bind the text domain, so that the libraries can properly locate their message catalogs. diff --git a/Doc/library/logging.config.rst b/Doc/library/logging.config.rst new file mode 100644 index 0000000..bb80a7f --- /dev/null +++ b/Doc/library/logging.config.rst @@ -0,0 +1,680 @@ +:mod:`logging.config` --- Logging configuration +=============================================== + +.. module:: logging.config + :synopsis: Configuration of the logging module. + + +.. moduleauthor:: Vinay Sajip <vinay_sajip@red-dove.com> +.. sectionauthor:: Vinay Sajip <vinay_sajip@red-dove.com> + +.. sidebar:: Important + + This page contains only reference information. For tutorials, + please see + + * :ref:`Basic Tutorial <logging-basic-tutorial>` + * :ref:`Advanced Tutorial <logging-advanced-tutorial>` + * :ref:`Logging Cookbook <logging-cookbook>` + +This section describes the API for configuring the logging module. + +.. _logging-config-api: + +Configuration functions +^^^^^^^^^^^^^^^^^^^^^^^ + +The following functions configure the logging module. They are located in the +:mod:`logging.config` module. Their use is optional --- you can configure the +logging module using these functions or by making calls to the main API (defined +in :mod:`logging` itself) and defining handlers which are declared either in +:mod:`logging` or :mod:`logging.handlers`. + +.. function:: dictConfig(config) + + Takes the logging configuration from a dictionary. The contents of + this dictionary are described in :ref:`logging-config-dictschema` + below. + + If an error is encountered during configuration, this function will + raise a :exc:`ValueError`, :exc:`TypeError`, :exc:`AttributeError` + or :exc:`ImportError` with a suitably descriptive message. The + following is a (possibly incomplete) list of conditions which will + raise an error: + + * A ``level`` which is not a string or which is a string not + corresponding to an actual logging level. + * A ``propagate`` value which is not a boolean. + * An id which does not have a corresponding destination. + * A non-existent handler id found during an incremental call. + * An invalid logger name. + * Inability to resolve to an internal or external object. + + Parsing is performed by the :class:`DictConfigurator` class, whose + constructor is passed the dictionary used for configuration, and + has a :meth:`configure` method. The :mod:`logging.config` module + has a callable attribute :attr:`dictConfigClass` + which is initially set to :class:`DictConfigurator`. + You can replace the value of :attr:`dictConfigClass` with a + suitable implementation of your own. + + :func:`dictConfig` calls :attr:`dictConfigClass` passing + the specified dictionary, and then calls the :meth:`configure` method on + the returned object to put the configuration into effect:: + + def dictConfig(config): + dictConfigClass(config).configure() + + For example, a subclass of :class:`DictConfigurator` could call + ``DictConfigurator.__init__()`` in its own :meth:`__init__()`, then + set up custom prefixes which would be usable in the subsequent + :meth:`configure` call. :attr:`dictConfigClass` would be bound to + this new subclass, and then :func:`dictConfig` could be called exactly as + in the default, uncustomized state. + + .. versionadded:: 3.2 + +.. function:: fileConfig(fname, defaults=None, disable_existing_loggers=True) + + Reads the logging configuration from a :mod:`configparser`\-format file + named *fname*. This function can be called several times from an + application, allowing an end user to select from various pre-canned + configurations (if the developer provides a mechanism to present the choices + and load the chosen configuration). + + :param defaults: Defaults to be passed to the ConfigParser can be specified + in this argument. + + :param disable_existing_loggers: If specified as ``False``, loggers which + exist when this call is made are left + alone. The default is ``True`` because this + enables old behaviour in a backward- + compatible way. This behaviour is to + disable any existing loggers unless they or + their ancestors are explicitly named in the + logging configuration. + + +.. function:: listen(port=DEFAULT_LOGGING_CONFIG_PORT) + + Starts up a socket server on the specified port, and listens for new + configurations. If no port is specified, the module's default + :const:`DEFAULT_LOGGING_CONFIG_PORT` is used. Logging configurations will be + sent as a file suitable for processing by :func:`fileConfig`. Returns a + :class:`Thread` instance on which you can call :meth:`start` to start the + server, and which you can :meth:`join` when appropriate. To stop the server, + call :func:`stopListening`. + + To send a configuration to the socket, read in the configuration file and + send it to the socket as a string of bytes preceded by a four-byte length + string packed in binary using ``struct.pack('>L', n)``. + + +.. function:: stopListening() + + Stops the listening server which was created with a call to :func:`listen`. + This is typically called before calling :meth:`join` on the return value from + :func:`listen`. + + +.. _logging-config-dictschema: + +Configuration dictionary schema +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Describing a logging configuration requires listing the various +objects to create and the connections between them; for example, you +may create a handler named 'console' and then say that the logger +named 'startup' will send its messages to the 'console' handler. +These objects aren't limited to those provided by the :mod:`logging` +module because you might write your own formatter or handler class. +The parameters to these classes may also need to include external +objects such as ``sys.stderr``. The syntax for describing these +objects and connections is defined in :ref:`logging-config-dict-connections` +below. + +Dictionary Schema Details +""""""""""""""""""""""""" + +The dictionary passed to :func:`dictConfig` must contain the following +keys: + +* *version* - to be set to an integer value representing the schema + version. The only valid value at present is 1, but having this key + allows the schema to evolve while still preserving backwards + compatibility. + +All other keys are optional, but if present they will be interpreted +as described below. In all cases below where a 'configuring dict' is +mentioned, it will be checked for the special ``'()'`` key to see if a +custom instantiation is required. If so, the mechanism described in +:ref:`logging-config-dict-userdef` below is used to create an instance; +otherwise, the context is used to determine what to instantiate. + +* *formatters* - the corresponding value will be a dict in which each + key is a formatter id and each value is a dict describing how to + configure the corresponding Formatter instance. + + The configuring dict is searched for keys ``format`` and ``datefmt`` + (with defaults of ``None``) and these are used to construct a + :class:`logging.Formatter` instance. + +* *filters* - the corresponding value will be a dict in which each key + is a filter id and each value is a dict describing how to configure + the corresponding Filter instance. + + The configuring dict is searched for the key ``name`` (defaulting to the + empty string) and this is used to construct a :class:`logging.Filter` + instance. + +* *handlers* - the corresponding value will be a dict in which each + key is a handler id and each value is a dict describing how to + configure the corresponding Handler instance. + + The configuring dict is searched for the following keys: + + * ``class`` (mandatory). This is the fully qualified name of the + handler class. + + * ``level`` (optional). The level of the handler. + + * ``formatter`` (optional). The id of the formatter for this + handler. + + * ``filters`` (optional). A list of ids of the filters for this + handler. + + All *other* keys are passed through as keyword arguments to the + handler's constructor. For example, given the snippet:: + + handlers: + console: + class : logging.StreamHandler + formatter: brief + level : INFO + filters: [allow_foo] + stream : ext://sys.stdout + file: + class : logging.handlers.RotatingFileHandler + formatter: precise + filename: logconfig.log + maxBytes: 1024 + backupCount: 3 + + the handler with id ``console`` is instantiated as a + :class:`logging.StreamHandler`, using ``sys.stdout`` as the underlying + stream. The handler with id ``file`` is instantiated as a + :class:`logging.handlers.RotatingFileHandler` with the keyword arguments + ``filename='logconfig.log', maxBytes=1024, backupCount=3``. + +* *loggers* - the corresponding value will be a dict in which each key + is a logger name and each value is a dict describing how to + configure the corresponding Logger instance. + + The configuring dict is searched for the following keys: + + * ``level`` (optional). The level of the logger. + + * ``propagate`` (optional). The propagation setting of the logger. + + * ``filters`` (optional). A list of ids of the filters for this + logger. + + * ``handlers`` (optional). A list of ids of the handlers for this + logger. + + The specified loggers will be configured according to the level, + propagation, filters and handlers specified. + +* *root* - this will be the configuration for the root logger. + Processing of the configuration will be as for any logger, except + that the ``propagate`` setting will not be applicable. + +* *incremental* - whether the configuration is to be interpreted as + incremental to the existing configuration. This value defaults to + ``False``, which means that the specified configuration replaces the + existing configuration with the same semantics as used by the + existing :func:`fileConfig` API. + + If the specified value is ``True``, the configuration is processed + as described in the section on :ref:`logging-config-dict-incremental`. + +* *disable_existing_loggers* - whether any existing loggers are to be + disabled. This setting mirrors the parameter of the same name in + :func:`fileConfig`. If absent, this parameter defaults to ``True``. + This value is ignored if *incremental* is ``True``. + +.. _logging-config-dict-incremental: + +Incremental Configuration +""""""""""""""""""""""""" + +It is difficult to provide complete flexibility for incremental +configuration. For example, because objects such as filters +and formatters are anonymous, once a configuration is set up, it is +not possible to refer to such anonymous objects when augmenting a +configuration. + +Furthermore, there is not a compelling case for arbitrarily altering +the object graph of loggers, handlers, filters, formatters at +run-time, once a configuration is set up; the verbosity of loggers and +handlers can be controlled just by setting levels (and, in the case of +loggers, propagation flags). Changing the object graph arbitrarily in +a safe way is problematic in a multi-threaded environment; while not +impossible, the benefits are not worth the complexity it adds to the +implementation. + +Thus, when the ``incremental`` key of a configuration dict is present +and is ``True``, the system will completely ignore any ``formatters`` and +``filters`` entries, and process only the ``level`` +settings in the ``handlers`` entries, and the ``level`` and +``propagate`` settings in the ``loggers`` and ``root`` entries. + +Using a value in the configuration dict lets configurations to be sent +over the wire as pickled dicts to a socket listener. Thus, the logging +verbosity of a long-running application can be altered over time with +no need to stop and restart the application. + +.. _logging-config-dict-connections: + +Object connections +"""""""""""""""""" + +The schema describes a set of logging objects - loggers, +handlers, formatters, filters - which are connected to each other in +an object graph. Thus, the schema needs to represent connections +between the objects. For example, say that, once configured, a +particular logger has attached to it a particular handler. For the +purposes of this discussion, we can say that the logger represents the +source, and the handler the destination, of a connection between the +two. Of course in the configured objects this is represented by the +logger holding a reference to the handler. In the configuration dict, +this is done by giving each destination object an id which identifies +it unambiguously, and then using the id in the source object's +configuration to indicate that a connection exists between the source +and the destination object with that id. + +So, for example, consider the following YAML snippet:: + + formatters: + brief: + # configuration for formatter with id 'brief' goes here + precise: + # configuration for formatter with id 'precise' goes here + handlers: + h1: #This is an id + # configuration of handler with id 'h1' goes here + formatter: brief + h2: #This is another id + # configuration of handler with id 'h2' goes here + formatter: precise + loggers: + foo.bar.baz: + # other configuration for logger 'foo.bar.baz' + handlers: [h1, h2] + +(Note: YAML used here because it's a little more readable than the +equivalent Python source form for the dictionary.) + +The ids for loggers are the logger names which would be used +programmatically to obtain a reference to those loggers, e.g. +``foo.bar.baz``. The ids for Formatters and Filters can be any string +value (such as ``brief``, ``precise`` above) and they are transient, +in that they are only meaningful for processing the configuration +dictionary and used to determine connections between objects, and are +not persisted anywhere when the configuration call is complete. + +The above snippet indicates that logger named ``foo.bar.baz`` should +have two handlers attached to it, which are described by the handler +ids ``h1`` and ``h2``. The formatter for ``h1`` is that described by id +``brief``, and the formatter for ``h2`` is that described by id +``precise``. + + +.. _logging-config-dict-userdef: + +User-defined objects +"""""""""""""""""""" + +The schema supports user-defined objects for handlers, filters and +formatters. (Loggers do not need to have different types for +different instances, so there is no support in this configuration +schema for user-defined logger classes.) + +Objects to be configured are described by dictionaries +which detail their configuration. In some places, the logging system +will be able to infer from the context how an object is to be +instantiated, but when a user-defined object is to be instantiated, +the system will not know how to do this. In order to provide complete +flexibility for user-defined object instantiation, the user needs +to provide a 'factory' - a callable which is called with a +configuration dictionary and which returns the instantiated object. +This is signalled by an absolute import path to the factory being +made available under the special key ``'()'``. Here's a concrete +example:: + + formatters: + brief: + format: '%(message)s' + default: + format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s' + datefmt: '%Y-%m-%d %H:%M:%S' + custom: + (): my.package.customFormatterFactory + bar: baz + spam: 99.9 + answer: 42 + +The above YAML snippet defines three formatters. The first, with id +``brief``, is a standard :class:`logging.Formatter` instance with the +specified format string. The second, with id ``default``, has a +longer format and also defines the time format explicitly, and will +result in a :class:`logging.Formatter` initialized with those two format +strings. Shown in Python source form, the ``brief`` and ``default`` +formatters have configuration sub-dictionaries:: + + { + 'format' : '%(message)s' + } + +and:: + + { + 'format' : '%(asctime)s %(levelname)-8s %(name)-15s %(message)s', + 'datefmt' : '%Y-%m-%d %H:%M:%S' + } + +respectively, and as these dictionaries do not contain the special key +``'()'``, the instantiation is inferred from the context: as a result, +standard :class:`logging.Formatter` instances are created. The +configuration sub-dictionary for the third formatter, with id +``custom``, is:: + + { + '()' : 'my.package.customFormatterFactory', + 'bar' : 'baz', + 'spam' : 99.9, + 'answer' : 42 + } + +and this contains the special key ``'()'``, which means that +user-defined instantiation is wanted. In this case, the specified +factory callable will be used. If it is an actual callable it will be +used directly - otherwise, if you specify a string (as in the example) +the actual callable will be located using normal import mechanisms. +The callable will be called with the **remaining** items in the +configuration sub-dictionary as keyword arguments. In the above +example, the formatter with id ``custom`` will be assumed to be +returned by the call:: + + my.package.customFormatterFactory(bar='baz', spam=99.9, answer=42) + +The key ``'()'`` has been used as the special key because it is not a +valid keyword parameter name, and so will not clash with the names of +the keyword arguments used in the call. The ``'()'`` also serves as a +mnemonic that the corresponding value is a callable. + + +.. _logging-config-dict-externalobj: + +Access to external objects +"""""""""""""""""""""""""" + +There are times where a configuration needs to refer to objects +external to the configuration, for example ``sys.stderr``. If the +configuration dict is constructed using Python code, this is +straightforward, but a problem arises when the configuration is +provided via a text file (e.g. JSON, YAML). In a text file, there is +no standard way to distinguish ``sys.stderr`` from the literal string +``'sys.stderr'``. To facilitate this distinction, the configuration +system looks for certain special prefixes in string values and +treat them specially. For example, if the literal string +``'ext://sys.stderr'`` is provided as a value in the configuration, +then the ``ext://`` will be stripped off and the remainder of the +value processed using normal import mechanisms. + +The handling of such prefixes is done in a way analogous to protocol +handling: there is a generic mechanism to look for prefixes which +match the regular expression ``^(?P<prefix>[a-z]+)://(?P<suffix>.*)$`` +whereby, if the ``prefix`` is recognised, the ``suffix`` is processed +in a prefix-dependent manner and the result of the processing replaces +the string value. If the prefix is not recognised, then the string +value will be left as-is. + + +.. _logging-config-dict-internalobj: + +Access to internal objects +"""""""""""""""""""""""""" + +As well as external objects, there is sometimes also a need to refer +to objects in the configuration. This will be done implicitly by the +configuration system for things that it knows about. For example, the +string value ``'DEBUG'`` for a ``level`` in a logger or handler will +automatically be converted to the value ``logging.DEBUG``, and the +``handlers``, ``filters`` and ``formatter`` entries will take an +object id and resolve to the appropriate destination object. + +However, a more generic mechanism is needed for user-defined +objects which are not known to the :mod:`logging` module. For +example, consider :class:`logging.handlers.MemoryHandler`, which takes +a ``target`` argument which is another handler to delegate to. Since +the system already knows about this class, then in the configuration, +the given ``target`` just needs to be the object id of the relevant +target handler, and the system will resolve to the handler from the +id. If, however, a user defines a ``my.package.MyHandler`` which has +an ``alternate`` handler, the configuration system would not know that +the ``alternate`` referred to a handler. To cater for this, a generic +resolution system allows the user to specify:: + + handlers: + file: + # configuration of file handler goes here + + custom: + (): my.package.MyHandler + alternate: cfg://handlers.file + +The literal string ``'cfg://handlers.file'`` will be resolved in an +analogous way to strings with the ``ext://`` prefix, but looking +in the configuration itself rather than the import namespace. The +mechanism allows access by dot or by index, in a similar way to +that provided by ``str.format``. Thus, given the following snippet:: + + handlers: + email: + class: logging.handlers.SMTPHandler + mailhost: localhost + fromaddr: my_app@domain.tld + toaddrs: + - support_team@domain.tld + - dev_team@domain.tld + subject: Houston, we have a problem. + +in the configuration, the string ``'cfg://handlers'`` would resolve to +the dict with key ``handlers``, the string ``'cfg://handlers.email`` +would resolve to the dict with key ``email`` in the ``handlers`` dict, +and so on. The string ``'cfg://handlers.email.toaddrs[1]`` would +resolve to ``'dev_team.domain.tld'`` and the string +``'cfg://handlers.email.toaddrs[0]'`` would resolve to the value +``'support_team@domain.tld'``. The ``subject`` value could be accessed +using either ``'cfg://handlers.email.subject'`` or, equivalently, +``'cfg://handlers.email[subject]'``. The latter form only needs to be +used if the key contains spaces or non-alphanumeric characters. If an +index value consists only of decimal digits, access will be attempted +using the corresponding integer value, falling back to the string +value if needed. + +Given a string ``cfg://handlers.myhandler.mykey.123``, this will +resolve to ``config_dict['handlers']['myhandler']['mykey']['123']``. +If the string is specified as ``cfg://handlers.myhandler.mykey[123]``, +the system will attempt to retrieve the value from +``config_dict['handlers']['myhandler']['mykey'][123]``, and fall back +to ``config_dict['handlers']['myhandler']['mykey']['123']`` if that +fails. + +.. _logging-config-fileformat: + +Configuration file format +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The configuration file format understood by :func:`fileConfig` is based on +:mod:`configparser` functionality. The file must contain sections called +``[loggers]``, ``[handlers]`` and ``[formatters]`` which identify by name the +entities of each type which are defined in the file. For each such entity, there +is a separate section which identifies how that entity is configured. Thus, for +a logger named ``log01`` in the ``[loggers]`` section, the relevant +configuration details are held in a section ``[logger_log01]``. Similarly, a +handler called ``hand01`` in the ``[handlers]`` section will have its +configuration held in a section called ``[handler_hand01]``, while a formatter +called ``form01`` in the ``[formatters]`` section will have its configuration +specified in a section called ``[formatter_form01]``. The root logger +configuration must be specified in a section called ``[logger_root]``. + +Examples of these sections in the file are given below. :: + + [loggers] + keys=root,log02,log03,log04,log05,log06,log07 + + [handlers] + keys=hand01,hand02,hand03,hand04,hand05,hand06,hand07,hand08,hand09 + + [formatters] + keys=form01,form02,form03,form04,form05,form06,form07,form08,form09 + +The root logger must specify a level and a list of handlers. An example of a +root logger section is given below. :: + + [logger_root] + level=NOTSET + handlers=hand01 + +The ``level`` entry can be one of ``DEBUG, INFO, WARNING, ERROR, CRITICAL`` or +``NOTSET``. For the root logger only, ``NOTSET`` means that all messages will be +logged. Level values are :func:`eval`\ uated in the context of the ``logging`` +package's namespace. + +The ``handlers`` entry is a comma-separated list of handler names, which must +appear in the ``[handlers]`` section. These names must appear in the +``[handlers]`` section and have corresponding sections in the configuration +file. + +For loggers other than the root logger, some additional information is required. +This is illustrated by the following example. :: + + [logger_parser] + level=DEBUG + handlers=hand01 + propagate=1 + qualname=compiler.parser + +The ``level`` and ``handlers`` entries are interpreted as for the root logger, +except that if a non-root logger's level is specified as ``NOTSET``, the system +consults loggers higher up the hierarchy to determine the effective level of the +logger. The ``propagate`` entry is set to 1 to indicate that messages must +propagate to handlers higher up the logger hierarchy from this logger, or 0 to +indicate that messages are **not** propagated to handlers up the hierarchy. The +``qualname`` entry is the hierarchical channel name of the logger, that is to +say the name used by the application to get the logger. + +Sections which specify handler configuration are exemplified by the following. +:: + + [handler_hand01] + class=StreamHandler + level=NOTSET + formatter=form01 + args=(sys.stdout,) + +The ``class`` entry indicates the handler's class (as determined by :func:`eval` +in the ``logging`` package's namespace). The ``level`` is interpreted as for +loggers, and ``NOTSET`` is taken to mean 'log everything'. + +The ``formatter`` entry indicates the key name of the formatter for this +handler. If blank, a default formatter (``logging._defaultFormatter``) is used. +If a name is specified, it must appear in the ``[formatters]`` section and have +a corresponding section in the configuration file. + +The ``args`` entry, when :func:`eval`\ uated in the context of the ``logging`` +package's namespace, is the list of arguments to the constructor for the handler +class. Refer to the constructors for the relevant handlers, or to the examples +below, to see how typical entries are constructed. :: + + [handler_hand02] + class=FileHandler + level=DEBUG + formatter=form02 + args=('python.log', 'w') + + [handler_hand03] + class=handlers.SocketHandler + level=INFO + formatter=form03 + args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT) + + [handler_hand04] + class=handlers.DatagramHandler + level=WARN + formatter=form04 + args=('localhost', handlers.DEFAULT_UDP_LOGGING_PORT) + + [handler_hand05] + class=handlers.SysLogHandler + level=ERROR + formatter=form05 + args=(('localhost', handlers.SYSLOG_UDP_PORT), handlers.SysLogHandler.LOG_USER) + + [handler_hand06] + class=handlers.NTEventLogHandler + level=CRITICAL + formatter=form06 + args=('Python Application', '', 'Application') + + [handler_hand07] + class=handlers.SMTPHandler + level=WARN + formatter=form07 + args=('localhost', 'from@abc', ['user1@abc', 'user2@xyz'], 'Logger Subject') + + [handler_hand08] + class=handlers.MemoryHandler + level=NOTSET + formatter=form08 + target= + args=(10, ERROR) + + [handler_hand09] + class=handlers.HTTPHandler + level=NOTSET + formatter=form09 + args=('localhost:9022', '/log', 'GET') + +Sections which specify formatter configuration are typified by the following. :: + + [formatter_form01] + format=F1 %(asctime)s %(levelname)s %(message)s + datefmt= + class=logging.Formatter + +The ``format`` entry is the overall format string, and the ``datefmt`` entry is +the :func:`strftime`\ -compatible date/time format string. If empty, the +package substitutes ISO8601 format date/times, which is almost equivalent to +specifying the date format string ``'%Y-%m-%d %H:%M:%S'``. The ISO8601 format +also specifies milliseconds, which are appended to the result of using the above +format string, with a comma separator. An example time in ISO8601 format is +``2003-01-23 00:29:50,411``. + +The ``class`` entry is optional. It indicates the name of the formatter's class +(as a dotted module and class name.) This option is useful for instantiating a +:class:`Formatter` subclass. Subclasses of :class:`Formatter` can present +exception tracebacks in an expanded or condensed format. + +.. seealso:: + + Module :mod:`logging` + API reference for the logging module. + + Module :mod:`logging.handlers` + Useful handlers included with the logging module. + + diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst new file mode 100644 index 0000000..eda9302 --- /dev/null +++ b/Doc/library/logging.handlers.rst @@ -0,0 +1,850 @@ +:mod:`logging.handlers` --- Logging handlers +============================================ + +.. module:: logging.handlers + :synopsis: Handlers for the logging module. + + +.. moduleauthor:: Vinay Sajip <vinay_sajip@red-dove.com> +.. sectionauthor:: Vinay Sajip <vinay_sajip@red-dove.com> + +.. sidebar:: Important + + This page contains only reference information. For tutorials, + please see + + * :ref:`Basic Tutorial <logging-basic-tutorial>` + * :ref:`Advanced Tutorial <logging-advanced-tutorial>` + * :ref:`Logging Cookbook <logging-cookbook>` + +.. currentmodule:: logging + +The following useful handlers are provided in the package. Note that three of +the handlers (:class:`StreamHandler`, :class:`FileHandler` and +:class:`NullHandler`) are actually defined in the :mod:`logging` module itself, +but have been documented here along with the other handlers. + +.. _stream-handler: + +StreamHandler +^^^^^^^^^^^^^ + +The :class:`StreamHandler` class, located in the core :mod:`logging` package, +sends logging output to streams such as *sys.stdout*, *sys.stderr* or any +file-like object (or, more precisely, any object which supports :meth:`write` +and :meth:`flush` methods). + + +.. class:: StreamHandler(stream=None) + + Returns a new instance of the :class:`StreamHandler` class. If *stream* is + specified, the instance will use it for logging output; otherwise, *sys.stderr* + will be used. + + + .. method:: emit(record) + + If a formatter is specified, it is used to format the record. The record + is then written to the stream with a terminator. If exception information + is present, it is formatted using :func:`traceback.print_exception` and + appended to the stream. + + + .. method:: flush() + + Flushes the stream by calling its :meth:`flush` method. Note that the + :meth:`close` method is inherited from :class:`Handler` and so does + no output, so an explicit :meth:`flush` call may be needed at times. + +.. versionchanged:: 3.2 + The ``StreamHandler`` class now has a ``terminator`` attribute, default + value ``'\n'``, which is used as the terminator when writing a formatted + record to a stream. If you don't want this newline termination, you can + set the handler instance's ``terminator`` attribute to the empty string. + In earlier versions, the terminator was hardcoded as ``'\n'``. + +.. _file-handler: + +FileHandler +^^^^^^^^^^^ + +The :class:`FileHandler` class, located in the core :mod:`logging` package, +sends logging output to a disk file. It inherits the output functionality from +:class:`StreamHandler`. + + +.. class:: FileHandler(filename, mode='a', encoding=None, delay=False) + + Returns a new instance of the :class:`FileHandler` class. The specified file is + opened and used as the stream for logging. If *mode* is not specified, + :const:`'a'` is used. If *encoding* is not *None*, it is used to open the file + with that encoding. If *delay* is true, then file opening is deferred until the + first call to :meth:`emit`. By default, the file grows indefinitely. + + + .. method:: close() + + Closes the file. + + + .. method:: emit(record) + + Outputs the record to the file. + + +.. _null-handler: + +NullHandler +^^^^^^^^^^^ + +.. versionadded:: 3.1 + +The :class:`NullHandler` class, located in the core :mod:`logging` package, +does not do any formatting or output. It is essentially a 'no-op' handler +for use by library developers. + +.. class:: NullHandler() + + Returns a new instance of the :class:`NullHandler` class. + + .. method:: emit(record) + + This method does nothing. + + .. method:: handle(record) + + This method does nothing. + + .. method:: createLock() + + This method returns ``None`` for the lock, since there is no + underlying I/O to which access needs to be serialized. + + +See :ref:`library-config` for more information on how to use +:class:`NullHandler`. + +.. _watched-file-handler: + +WatchedFileHandler +^^^^^^^^^^^^^^^^^^ + +.. currentmodule:: logging.handlers + +The :class:`WatchedFileHandler` class, located in the :mod:`logging.handlers` +module, is a :class:`FileHandler` which watches the file it is logging to. If +the file changes, it is closed and reopened using the file name. + +A file change can happen because of usage of programs such as *newsyslog* and +*logrotate* which perform log file rotation. This handler, intended for use +under Unix/Linux, watches the file to see if it has changed since the last emit. +(A file is deemed to have changed if its device or inode have changed.) If the +file has changed, the old file stream is closed, and the file opened to get a +new stream. + +This handler is not appropriate for use under Windows, because under Windows +open log files cannot be moved or renamed - logging opens the files with +exclusive locks - and so there is no need for such a handler. Furthermore, +*ST_INO* is not supported under Windows; :func:`stat` always returns zero for +this value. + + +.. class:: WatchedFileHandler(filename[,mode[, encoding[, delay]]]) + + Returns a new instance of the :class:`WatchedFileHandler` class. The specified + file is opened and used as the stream for logging. If *mode* is not specified, + :const:`'a'` is used. If *encoding* is not *None*, it is used to open the file + with that encoding. If *delay* is true, then file opening is deferred until the + first call to :meth:`emit`. By default, the file grows indefinitely. + + + .. method:: emit(record) + + Outputs the record to the file, but first checks to see if the file has + changed. If it has, the existing stream is flushed and closed and the + file opened again, before outputting the record to the file. + +.. _rotating-file-handler: + +RotatingFileHandler +^^^^^^^^^^^^^^^^^^^ + +The :class:`RotatingFileHandler` class, located in the :mod:`logging.handlers` +module, supports rotation of disk log files. + + +.. class:: RotatingFileHandler(filename, mode='a', maxBytes=0, backupCount=0, encoding=None, delay=0) + + Returns a new instance of the :class:`RotatingFileHandler` class. The specified + file is opened and used as the stream for logging. If *mode* is not specified, + ``'a'`` is used. If *encoding* is not *None*, it is used to open the file + with that encoding. If *delay* is true, then file opening is deferred until the + first call to :meth:`emit`. By default, the file grows indefinitely. + + You can use the *maxBytes* and *backupCount* values to allow the file to + :dfn:`rollover` at a predetermined size. When the size is about to be exceeded, + the file is closed and a new file is silently opened for output. Rollover occurs + whenever the current log file is nearly *maxBytes* in length; if *maxBytes* is + zero, rollover never occurs. If *backupCount* is non-zero, the system will save + old log files by appending the extensions '.1', '.2' etc., to the filename. For + example, with a *backupCount* of 5 and a base file name of :file:`app.log`, you + would get :file:`app.log`, :file:`app.log.1`, :file:`app.log.2`, up to + :file:`app.log.5`. The file being written to is always :file:`app.log`. When + this file is filled, it is closed and renamed to :file:`app.log.1`, and if files + :file:`app.log.1`, :file:`app.log.2`, etc. exist, then they are renamed to + :file:`app.log.2`, :file:`app.log.3` etc. respectively. + + + .. method:: doRollover() + + Does a rollover, as described above. + + + .. method:: emit(record) + + Outputs the record to the file, catering for rollover as described + previously. + +.. _timed-rotating-file-handler: + +TimedRotatingFileHandler +^^^^^^^^^^^^^^^^^^^^^^^^ + +The :class:`TimedRotatingFileHandler` class, located in the +:mod:`logging.handlers` module, supports rotation of disk log files at certain +timed intervals. + + +.. class:: TimedRotatingFileHandler(filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False) + + Returns a new instance of the :class:`TimedRotatingFileHandler` class. The + specified file is opened and used as the stream for logging. On rotating it also + sets the filename suffix. Rotating happens based on the product of *when* and + *interval*. + + You can use the *when* to specify the type of *interval*. The list of possible + values is below. Note that they are not case sensitive. + + +----------------+-----------------------+ + | Value | Type of interval | + +================+=======================+ + | ``'S'`` | Seconds | + +----------------+-----------------------+ + | ``'M'`` | Minutes | + +----------------+-----------------------+ + | ``'H'`` | Hours | + +----------------+-----------------------+ + | ``'D'`` | Days | + +----------------+-----------------------+ + | ``'W'`` | Week day (0=Monday) | + +----------------+-----------------------+ + | ``'midnight'`` | Roll over at midnight | + +----------------+-----------------------+ + + The system will save old log files by appending extensions to the filename. + The extensions are date-and-time based, using the strftime format + ``%Y-%m-%d_%H-%M-%S`` or a leading portion thereof, depending on the + rollover interval. + + When computing the next rollover time for the first time (when the handler + is created), the last modification time of an existing log file, or else + the current time, is used to compute when the next rotation will occur. + + If the *utc* argument is true, times in UTC will be used; otherwise + local time is used. + + If *backupCount* is nonzero, at most *backupCount* files + will be kept, and if more would be created when rollover occurs, the oldest + one is deleted. The deletion logic uses the interval to determine which + files to delete, so changing the interval may leave old files lying around. + + If *delay* is true, then file opening is deferred until the first call to + :meth:`emit`. + + + .. method:: doRollover() + + Does a rollover, as described above. + + + .. method:: emit(record) + + Outputs the record to the file, catering for rollover as described above. + + +.. _socket-handler: + +SocketHandler +^^^^^^^^^^^^^ + +The :class:`SocketHandler` class, located in the :mod:`logging.handlers` module, +sends logging output to a network socket. The base class uses a TCP socket. + + +.. class:: SocketHandler(host, port) + + Returns a new instance of the :class:`SocketHandler` class intended to + communicate with a remote machine whose address is given by *host* and *port*. + + + .. method:: close() + + Closes the socket. + + + .. method:: emit() + + Pickles the record's attribute dictionary and writes it to the socket in + binary format. If there is an error with the socket, silently drops the + packet. If the connection was previously lost, re-establishes the + connection. To unpickle the record at the receiving end into a + :class:`LogRecord`, use the :func:`makeLogRecord` function. + + + .. method:: handleError() + + Handles an error which has occurred during :meth:`emit`. The most likely + cause is a lost connection. Closes the socket so that we can retry on the + next event. + + + .. method:: makeSocket() + + This is a factory method which allows subclasses to define the precise + type of socket they want. The default implementation creates a TCP socket + (:const:`socket.SOCK_STREAM`). + + + .. method:: makePickle(record) + + Pickles the record's attribute dictionary in binary format with a length + prefix, and returns it ready for transmission across the socket. + + Note that pickles aren't completely secure. If you are concerned about + security, you may want to override this method to implement a more secure + mechanism. For example, you can sign pickles using HMAC and then verify + them on the receiving end, or alternatively you can disable unpickling of + global objects on the receiving end. + + + .. method:: send(packet) + + Send a pickled string *packet* to the socket. This function allows for + partial sends which can happen when the network is busy. + + + .. method:: createSocket() + + Tries to create a socket; on failure, uses an exponential back-off + algorithm. On intial failure, the handler will drop the message it was + trying to send. When subsequent messages are handled by the same + instance, it will not try connecting until some time has passed. The + default parameters are such that the initial delay is one second, and if + after that delay the connection still can't be made, the handler will + double the delay each time up to a maximum of 30 seconds. + + This behaviour is controlled by the following handler attributes: + + * ``retryStart`` (initial delay, defaulting to 1.0 seconds). + * ``retryFactor`` (multiplier, defaulting to 2.0). + * ``retryMax`` (maximum delay, defaulting to 30.0 seconds). + + This means that if the remote listener starts up *after* the handler has + been used, you could lose messages (since the handler won't even attempt + a connection until the delay has elapsed, but just silently drop messages + during the delay period). + + +.. _datagram-handler: + +DatagramHandler +^^^^^^^^^^^^^^^ + +The :class:`DatagramHandler` class, located in the :mod:`logging.handlers` +module, inherits from :class:`SocketHandler` to support sending logging messages +over UDP sockets. + + +.. class:: DatagramHandler(host, port) + + Returns a new instance of the :class:`DatagramHandler` class intended to + communicate with a remote machine whose address is given by *host* and *port*. + + + .. method:: emit() + + Pickles the record's attribute dictionary and writes it to the socket in + binary format. If there is an error with the socket, silently drops the + packet. To unpickle the record at the receiving end into a + :class:`LogRecord`, use the :func:`makeLogRecord` function. + + + .. method:: makeSocket() + + The factory method of :class:`SocketHandler` is here overridden to create + a UDP socket (:const:`socket.SOCK_DGRAM`). + + + .. method:: send(s) + + Send a pickled string to a socket. + + +.. _syslog-handler: + +SysLogHandler +^^^^^^^^^^^^^ + +The :class:`SysLogHandler` class, located in the :mod:`logging.handlers` module, +supports sending logging messages to a remote or local Unix syslog. + + +.. class:: SysLogHandler(address=('localhost', SYSLOG_UDP_PORT), facility=LOG_USER, socktype=socket.SOCK_DGRAM) + + Returns a new instance of the :class:`SysLogHandler` class intended to + communicate with a remote Unix machine whose address is given by *address* in + the form of a ``(host, port)`` tuple. If *address* is not specified, + ``('localhost', 514)`` is used. The address is used to open a socket. An + alternative to providing a ``(host, port)`` tuple is providing an address as a + string, for example '/dev/log'. In this case, a Unix domain socket is used to + send the message to the syslog. If *facility* is not specified, + :const:`LOG_USER` is used. The type of socket opened depends on the + *socktype* argument, which defaults to :const:`socket.SOCK_DGRAM` and thus + opens a UDP socket. To open a TCP socket (for use with the newer syslog + daemons such as rsyslog), specify a value of :const:`socket.SOCK_STREAM`. + + Note that if your server is not listening on UDP port 514, + :class:`SysLogHandler` may appear not to work. In that case, check what + address you should be using for a domain socket - it's system dependent. + For example, on Linux it's usually '/dev/log' but on OS/X it's + '/var/run/syslog'. You'll need to check your platform and use the + appropriate address (you may need to do this check at runtime if your + application needs to run on several platforms). On Windows, you pretty + much have to use the UDP option. + + .. versionchanged:: 3.2 + *socktype* was added. + + + .. method:: close() + + Closes the socket to the remote host. + + + .. method:: emit(record) + + The record is formatted, and then sent to the syslog server. If exception + information is present, it is *not* sent to the server. + + + .. method:: encodePriority(facility, priority) + + Encodes the facility and priority into an integer. You can pass in strings + or integers - if strings are passed, internal mapping dictionaries are + used to convert them to integers. + + The symbolic ``LOG_`` values are defined in :class:`SysLogHandler` and + mirror the values defined in the ``sys/syslog.h`` header file. + + **Priorities** + + +--------------------------+---------------+ + | Name (string) | Symbolic value| + +==========================+===============+ + | ``alert`` | LOG_ALERT | + +--------------------------+---------------+ + | ``crit`` or ``critical`` | LOG_CRIT | + +--------------------------+---------------+ + | ``debug`` | LOG_DEBUG | + +--------------------------+---------------+ + | ``emerg`` or ``panic`` | LOG_EMERG | + +--------------------------+---------------+ + | ``err`` or ``error`` | LOG_ERR | + +--------------------------+---------------+ + | ``info`` | LOG_INFO | + +--------------------------+---------------+ + | ``notice`` | LOG_NOTICE | + +--------------------------+---------------+ + | ``warn`` or ``warning`` | LOG_WARNING | + +--------------------------+---------------+ + + **Facilities** + + +---------------+---------------+ + | Name (string) | Symbolic value| + +===============+===============+ + | ``auth`` | LOG_AUTH | + +---------------+---------------+ + | ``authpriv`` | LOG_AUTHPRIV | + +---------------+---------------+ + | ``cron`` | LOG_CRON | + +---------------+---------------+ + | ``daemon`` | LOG_DAEMON | + +---------------+---------------+ + | ``ftp`` | LOG_FTP | + +---------------+---------------+ + | ``kern`` | LOG_KERN | + +---------------+---------------+ + | ``lpr`` | LOG_LPR | + +---------------+---------------+ + | ``mail`` | LOG_MAIL | + +---------------+---------------+ + | ``news`` | LOG_NEWS | + +---------------+---------------+ + | ``syslog`` | LOG_SYSLOG | + +---------------+---------------+ + | ``user`` | LOG_USER | + +---------------+---------------+ + | ``uucp`` | LOG_UUCP | + +---------------+---------------+ + | ``local0`` | LOG_LOCAL0 | + +---------------+---------------+ + | ``local1`` | LOG_LOCAL1 | + +---------------+---------------+ + | ``local2`` | LOG_LOCAL2 | + +---------------+---------------+ + | ``local3`` | LOG_LOCAL3 | + +---------------+---------------+ + | ``local4`` | LOG_LOCAL4 | + +---------------+---------------+ + | ``local5`` | LOG_LOCAL5 | + +---------------+---------------+ + | ``local6`` | LOG_LOCAL6 | + +---------------+---------------+ + | ``local7`` | LOG_LOCAL7 | + +---------------+---------------+ + + .. method:: mapPriority(levelname) + + Maps a logging level name to a syslog priority name. + You may need to override this if you are using custom levels, or + if the default algorithm is not suitable for your needs. The + default algorithm maps ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR`` and + ``CRITICAL`` to the equivalent syslog names, and all other level + names to 'warning'. + +.. _nt-eventlog-handler: + +NTEventLogHandler +^^^^^^^^^^^^^^^^^ + +The :class:`NTEventLogHandler` class, located in the :mod:`logging.handlers` +module, supports sending logging messages to a local Windows NT, Windows 2000 or +Windows XP event log. Before you can use it, you need Mark Hammond's Win32 +extensions for Python installed. + + +.. class:: NTEventLogHandler(appname, dllname=None, logtype='Application') + + Returns a new instance of the :class:`NTEventLogHandler` class. The *appname* is + used to define the application name as it appears in the event log. An + appropriate registry entry is created using this name. The *dllname* should give + the fully qualified pathname of a .dll or .exe which contains message + definitions to hold in the log (if not specified, ``'win32service.pyd'`` is used + - this is installed with the Win32 extensions and contains some basic + placeholder message definitions. Note that use of these placeholders will make + your event logs big, as the entire message source is held in the log. If you + want slimmer logs, you have to pass in the name of your own .dll or .exe which + contains the message definitions you want to use in the event log). The + *logtype* is one of ``'Application'``, ``'System'`` or ``'Security'``, and + defaults to ``'Application'``. + + + .. method:: close() + + At this point, you can remove the application name from the registry as a + source of event log entries. However, if you do this, you will not be able + to see the events as you intended in the Event Log Viewer - it needs to be + able to access the registry to get the .dll name. The current version does + not do this. + + + .. method:: emit(record) + + Determines the message ID, event category and event type, and then logs + the message in the NT event log. + + + .. method:: getEventCategory(record) + + Returns the event category for the record. Override this if you want to + specify your own categories. This version returns 0. + + + .. method:: getEventType(record) + + Returns the event type for the record. Override this if you want to + specify your own types. This version does a mapping using the handler's + typemap attribute, which is set up in :meth:`__init__` to a dictionary + which contains mappings for :const:`DEBUG`, :const:`INFO`, + :const:`WARNING`, :const:`ERROR` and :const:`CRITICAL`. If you are using + your own levels, you will either need to override this method or place a + suitable dictionary in the handler's *typemap* attribute. + + + .. method:: getMessageID(record) + + Returns the message ID for the record. If you are using your own messages, + you could do this by having the *msg* passed to the logger being an ID + rather than a format string. Then, in here, you could use a dictionary + lookup to get the message ID. This version returns 1, which is the base + message ID in :file:`win32service.pyd`. + +.. _smtp-handler: + +SMTPHandler +^^^^^^^^^^^ + +The :class:`SMTPHandler` class, located in the :mod:`logging.handlers` module, +supports sending logging messages to an email address via SMTP. + + +.. class:: SMTPHandler(mailhost, fromaddr, toaddrs, subject, credentials=None) + + Returns a new instance of the :class:`SMTPHandler` class. The instance is + initialized with the from and to addresses and subject line of the email. The + *toaddrs* should be a list of strings. To specify a non-standard SMTP port, use + the (host, port) tuple format for the *mailhost* argument. If you use a string, + the standard SMTP port is used. If your SMTP server requires authentication, you + can specify a (username, password) tuple for the *credentials* argument. + + + .. method:: emit(record) + + Formats the record and sends it to the specified addressees. + + + .. method:: getSubject(record) + + If you want to specify a subject line which is record-dependent, override + this method. + +.. _memory-handler: + +MemoryHandler +^^^^^^^^^^^^^ + +The :class:`MemoryHandler` class, located in the :mod:`logging.handlers` module, +supports buffering of logging records in memory, periodically flushing them to a +:dfn:`target` handler. Flushing occurs whenever the buffer is full, or when an +event of a certain severity or greater is seen. + +:class:`MemoryHandler` is a subclass of the more general +:class:`BufferingHandler`, which is an abstract class. This buffers logging +records in memory. Whenever each record is added to the buffer, a check is made +by calling :meth:`shouldFlush` to see if the buffer should be flushed. If it +should, then :meth:`flush` is expected to do the needful. + + +.. class:: BufferingHandler(capacity) + + Initializes the handler with a buffer of the specified capacity. + + + .. method:: emit(record) + + Appends the record to the buffer. If :meth:`shouldFlush` returns true, + calls :meth:`flush` to process the buffer. + + + .. method:: flush() + + You can override this to implement custom flushing behavior. This version + just zaps the buffer to empty. + + + .. method:: shouldFlush(record) + + Returns true if the buffer is up to capacity. This method can be + overridden to implement custom flushing strategies. + + +.. class:: MemoryHandler(capacity, flushLevel=ERROR, target=None) + + Returns a new instance of the :class:`MemoryHandler` class. The instance is + initialized with a buffer size of *capacity*. If *flushLevel* is not specified, + :const:`ERROR` is used. If no *target* is specified, the target will need to be + set using :meth:`setTarget` before this handler does anything useful. + + + .. method:: close() + + Calls :meth:`flush`, sets the target to :const:`None` and clears the + buffer. + + + .. method:: flush() + + For a :class:`MemoryHandler`, flushing means just sending the buffered + records to the target, if there is one. The buffer is also cleared when + this happens. Override if you want different behavior. + + + .. method:: setTarget(target) + + Sets the target handler for this handler. + + + .. method:: shouldFlush(record) + + Checks for buffer full or a record at the *flushLevel* or higher. + + +.. _http-handler: + +HTTPHandler +^^^^^^^^^^^ + +The :class:`HTTPHandler` class, located in the :mod:`logging.handlers` module, +supports sending logging messages to a Web server, using either ``GET`` or +``POST`` semantics. + + +.. class:: HTTPHandler(host, url, method='GET', secure=False, credentials=None) + + Returns a new instance of the :class:`HTTPHandler` class. The *host* can be + of the form ``host:port``, should you need to use a specific port number. + If no *method* is specified, ``GET`` is used. If *secure* is True, an HTTPS + connection will be used. If *credentials* is specified, it should be a + 2-tuple consisting of userid and password, which will be placed in an HTTP + 'Authorization' header using Basic authentication. If you specify + credentials, you should also specify secure=True so that your userid and + password are not passed in cleartext across the wire. + + + .. method:: emit(record) + + Sends the record to the Web server as a percent-encoded dictionary. + + +.. _queue-handler: + + +QueueHandler +^^^^^^^^^^^^ + +.. versionadded:: 3.2 + +The :class:`QueueHandler` class, located in the :mod:`logging.handlers` module, +supports sending logging messages to a queue, such as those implemented in the +:mod:`queue` or :mod:`multiprocessing` modules. + +Along with the :class:`QueueListener` class, :class:`QueueHandler` can be used +to let handlers do their work on a separate thread from the one which does the +logging. This is important in Web applications and also other service +applications where threads servicing clients need to respond as quickly as +possible, while any potentially slow operations (such as sending an email via +:class:`SMTPHandler`) are done on a separate thread. + +.. class:: QueueHandler(queue) + + Returns a new instance of the :class:`QueueHandler` class. The instance is + initialized with the queue to send messages to. The queue can be any queue- + like object; it's used as-is by the :meth:`enqueue` method, which needs + to know how to send messages to it. + + + .. method:: emit(record) + + Enqueues the result of preparing the LogRecord. + + .. method:: prepare(record) + + Prepares a record for queuing. The object returned by this + method is enqueued. + + The base implementation formats the record to merge the message + and arguments, and removes unpickleable items from the record + in-place. + + You might want to override this method if you want to convert + the record to a dict or JSON string, or send a modified copy + of the record while leaving the original intact. + + .. method:: enqueue(record) + + Enqueues the record on the queue using ``put_nowait()``; you may + want to override this if you want to use blocking behaviour, or a + timeout, or a customised queue implementation. + + + +.. queue-listener: + +QueueListener +^^^^^^^^^^^^^ + +.. versionadded:: 3.2 + +The :class:`QueueListener` class, located in the :mod:`logging.handlers` +module, supports receiving logging messages from a queue, such as those +implemented in the :mod:`queue` or :mod:`multiprocessing` modules. The +messages are received from a queue in an internal thread and passed, on +the same thread, to one or more handlers for processing. While +:class:`QueueListener` is not itself a handler, it is documented here +because it works hand-in-hand with :class:`QueueHandler`. + +Along with the :class:`QueueHandler` class, :class:`QueueListener` can be used +to let handlers do their work on a separate thread from the one which does the +logging. This is important in Web applications and also other service +applications where threads servicing clients need to respond as quickly as +possible, while any potentially slow operations (such as sending an email via +:class:`SMTPHandler`) are done on a separate thread. + +.. class:: QueueListener(queue, *handlers) + + Returns a new instance of the :class:`QueueListener` class. The instance is + initialized with the queue to send messages to and a list of handlers which + will handle entries placed on the queue. The queue can be any queue- + like object; it's passed as-is to the :meth:`dequeue` method, which needs + to know how to get messages from it. + + .. method:: dequeue(block) + + Dequeues a record and return it, optionally blocking. + + The base implementation uses ``get()``. You may want to override this + method if you want to use timeouts or work with custom queue + implementations. + + .. method:: prepare(record) + + Prepare a record for handling. + + This implementation just returns the passed-in record. You may want to + override this method if you need to do any custom marshalling or + manipulation of the record before passing it to the handlers. + + .. method:: handle(record) + + Handle a record. + + This just loops through the handlers offering them the record + to handle. The actual object passed to the handlers is that which + is returned from :meth:`prepare`. + + .. method:: start() + + Starts the listener. + + This starts up a background thread to monitor the queue for + LogRecords to process. + + .. method:: stop() + + Stops the listener. + + This asks the thread to terminate, and then waits for it to do so. + Note that if you don't call this before your application exits, there + may be some records still left on the queue, which won't be processed. + + +.. seealso:: + + Module :mod:`logging` + API reference for the logging module. + + Module :mod:`logging.config` + Configuration API for the logging module. + + diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 46b643f..32f762d 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -2,7 +2,7 @@ ============================================== .. module:: logging - :synopsis: Flexible error logging system for applications. + :synopsis: Flexible event logging system for applications. .. moduleauthor:: Vinay Sajip <vinay_sajip@red-dove.com> @@ -11,833 +11,38 @@ .. index:: pair: Errors; logging -This module defines functions and classes which implement a flexible error -logging system for applications. +.. sidebar:: Important -Logging is performed by calling methods on instances of the :class:`Logger` -class (hereafter called :dfn:`loggers`). Each instance has a name, and they are -conceptually arranged in a namespace hierarchy using dots (periods) as -separators. For example, a logger named "scan" is the parent of loggers -"scan.text", "scan.html" and "scan.pdf". Logger names can be anything you want, -and indicate the area of an application in which a logged message originates. + This page contains the API reference information. For tutorial + information and discussion of more advanced topics, see -Logged messages also have levels of importance associated with them. The default -levels provided are :const:`DEBUG`, :const:`INFO`, :const:`WARNING`, -:const:`ERROR` and :const:`CRITICAL`. As a convenience, you indicate the -importance of a logged message by calling an appropriate method of -:class:`Logger`. The methods are :meth:`debug`, :meth:`info`, :meth:`warning`, -:meth:`error` and :meth:`critical`, which mirror the default levels. You are not -constrained to use these levels: you can specify your own and use a more general -:class:`Logger` method, :meth:`log`, which takes an explicit level argument. + * :ref:`Basic Tutorial <logging-basic-tutorial>` + * :ref:`Advanced Tutorial <logging-advanced-tutorial>` + * :ref:`Logging Cookbook <logging-cookbook>` -Logging tutorial ----------------- +This module defines functions and classes which implement a flexible event +logging system for applications and libraries. The key benefit of having the logging API provided by a standard library module is that all Python modules can participate in logging, so your application log -can include messages from third-party modules. +can include your own messages integrated with messages from third-party +modules. -It is, of course, possible to log messages with different verbosity levels or to -different destinations. Support for writing log messages to files, HTTP -GET/POST locations, email via SMTP, generic sockets, or OS-specific logging -mechanisms are all supported by the standard module. You can also create your -own log destination class if you have special requirements not met by any of the -built-in classes. +The module provides a lot of functionality and flexibility. If you are +unfamiliar with logging, the best way to get to grips with it is to see the +tutorials (see the links on the right). -Simple examples -^^^^^^^^^^^^^^^ - -.. sectionauthor:: Doug Hellmann -.. (see <http://blog.doughellmann.com/2007/05/pymotw-logging.html>) +The basic classes defined by the module, together with their functions, are +listed below. -Most applications are probably going to want to log to a file, so let's start -with that case. Using the :func:`basicConfig` function, we can set up the -default handler so that debug messages are written to a file (in the example, -we assume that you have the appropriate permissions to create a file called -*example.log* in the current directory):: - - import logging - LOG_FILENAME = 'example.log' - logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG) - - logging.debug('This message should go to the log file') - -And now if we open the file and look at what we have, we should find the log -message:: - - DEBUG:root:This message should go to the log file - -If you run the script repeatedly, the additional log messages are appended to -the file. To create a new file each time, you can pass a *filemode* argument to -:func:`basicConfig` with a value of ``'w'``. Rather than managing the file size -yourself, though, it is simpler to use a :class:`RotatingFileHandler`:: - - import glob - import logging - import logging.handlers - - LOG_FILENAME = 'logging_rotatingfile_example.out' - - # Set up a specific logger with our desired output level - my_logger = logging.getLogger('MyLogger') - my_logger.setLevel(logging.DEBUG) - - # Add the log message handler to the logger - handler = logging.handlers.RotatingFileHandler( - LOG_FILENAME, maxBytes=20, backupCount=5) +* Loggers expose the interface that application code directly uses. +* Handlers send the log records (created by loggers) to the appropriate + destination. +* Filters provide a finer grained facility for determining which log records + to output. +* Formatters specify the layout of log records in the final output. - my_logger.addHandler(handler) - - # Log some messages - for i in range(20): - my_logger.debug('i = %d' % i) - - # See what files are created - logfiles = glob.glob('%s*' % LOG_FILENAME) - - for filename in logfiles: - print(filename) - -The result should be 6 separate files, each with part of the log history for the -application:: - - logging_rotatingfile_example.out - logging_rotatingfile_example.out.1 - logging_rotatingfile_example.out.2 - logging_rotatingfile_example.out.3 - logging_rotatingfile_example.out.4 - logging_rotatingfile_example.out.5 - -The most current file is always :file:`logging_rotatingfile_example.out`, -and each time it reaches the size limit it is renamed with the suffix -``.1``. Each of the existing backup files is renamed to increment the suffix -(``.1`` becomes ``.2``, etc.) and the ``.6`` file is erased. - -Obviously this example sets the log length much much too small as an extreme -example. You would want to set *maxBytes* to an appropriate value. - -Another useful feature of the logging API is the ability to produce different -messages at different log levels. This allows you to instrument your code with -debug messages, for example, but turning the log level down so that those debug -messages are not written for your production system. The default levels are -``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG`` and ``NOTSET``. - -The logger, handler, and log message call each specify a level. The log message -is only emitted if the handler and logger are configured to emit messages of -that level or lower. For example, if a message is ``CRITICAL``, and the logger -is set to ``ERROR``, the message is emitted. If a message is a ``WARNING``, and -the logger is set to produce only ``ERROR``\s, the message is not emitted:: - - import logging - import sys - - LEVELS = {'debug': logging.DEBUG, - 'info': logging.INFO, - 'warning': logging.WARNING, - 'error': logging.ERROR, - 'critical': logging.CRITICAL} - - if len(sys.argv) > 1: - level_name = sys.argv[1] - level = LEVELS.get(level_name, logging.NOTSET) - logging.basicConfig(level=level) - - logging.debug('This is a debug message') - logging.info('This is an info message') - logging.warning('This is a warning message') - logging.error('This is an error message') - logging.critical('This is a critical error message') - -Run the script with an argument like 'debug' or 'warning' to see which messages -show up at different levels:: - - $ python logging_level_example.py debug - DEBUG:root:This is a debug message - INFO:root:This is an info message - WARNING:root:This is a warning message - ERROR:root:This is an error message - CRITICAL:root:This is a critical error message - - $ python logging_level_example.py info - INFO:root:This is an info message - WARNING:root:This is a warning message - ERROR:root:This is an error message - CRITICAL:root:This is a critical error message - -You will notice that these log messages all have ``root`` embedded in them. The -logging module supports a hierarchy of loggers with different names. An easy -way to tell where a specific log message comes from is to use a separate logger -object for each of your modules. Each new logger "inherits" the configuration -of its parent, and log messages sent to a logger include the name of that -logger. Optionally, each logger can be configured differently, so that messages -from different modules are handled in different ways. Let's look at a simple -example of how to log from different modules so it is easy to trace the source -of the message:: - - import logging - - logging.basicConfig(level=logging.WARNING) - - logger1 = logging.getLogger('package1.module1') - logger2 = logging.getLogger('package2.module2') - - logger1.warning('This message comes from one module') - logger2.warning('And this message comes from another module') - -And the output:: - - $ python logging_modules_example.py - WARNING:package1.module1:This message comes from one module - WARNING:package2.module2:And this message comes from another module - -There are many more options for configuring logging, including different log -message formatting options, having messages delivered to multiple destinations, -and changing the configuration of a long-running application on the fly using a -socket interface. All of these options are covered in depth in the library -module documentation. - -Loggers -^^^^^^^ - -The logging library takes a modular approach and offers the several categories -of components: loggers, handlers, filters, and formatters. Loggers expose the -interface that application code directly uses. Handlers send the log records to -the appropriate destination. Filters provide a finer grained facility for -determining which log records to send on to a handler. Formatters specify the -layout of the resultant log record. - -:class:`Logger` objects have a threefold job. First, they expose several -methods to application code so that applications can log messages at runtime. -Second, logger objects determine which log messages to act upon based upon -severity (the default filtering facility) or filter objects. Third, logger -objects pass along relevant log messages to all interested log handlers. - -The most widely used methods on logger objects fall into two categories: -configuration and message sending. - -* :meth:`Logger.setLevel` specifies the lowest-severity log message a logger - will handle, where debug is the lowest built-in severity level and critical is - the highest built-in severity. For example, if the severity level is info, - the logger will handle only info, warning, error, and critical messages and - will ignore debug messages. - -* :meth:`Logger.addFilter` and :meth:`Logger.removeFilter` add and remove filter - objects from the logger object. This tutorial does not address filters. - -With the logger object configured, the following methods create log messages: - -* :meth:`Logger.debug`, :meth:`Logger.info`, :meth:`Logger.warning`, - :meth:`Logger.error`, and :meth:`Logger.critical` all create log records with - a message and a level that corresponds to their respective method names. The - message is actually a format string, which may contain the standard string - substitution syntax of :const:`%s`, :const:`%d`, :const:`%f`, and so on. The - rest of their arguments is a list of objects that correspond with the - substitution fields in the message. With regard to :const:`**kwargs`, the - logging methods care only about a keyword of :const:`exc_info` and use it to - determine whether to log exception information. - -* :meth:`Logger.exception` creates a log message similar to - :meth:`Logger.error`. The difference is that :meth:`Logger.exception` dumps a - stack trace along with it. Call this method only from an exception handler. - -* :meth:`Logger.log` takes a log level as an explicit argument. This is a - little more verbose for logging messages than using the log level convenience - methods listed above, but this is how to log at custom log levels. - -:func:`getLogger` returns a reference to a logger instance with the specified -name if it is provided, or ``root`` if not. The names are period-separated -hierarchical structures. Multiple calls to :func:`getLogger` with the same name -will return a reference to the same logger object. Loggers that are further -down in the hierarchical list are children of loggers higher up in the list. -For example, given a logger with a name of ``foo``, loggers with names of -``foo.bar``, ``foo.bar.baz``, and ``foo.bam`` are all descendants of ``foo``. -Child loggers propagate messages up to the handlers associated with their -ancestor loggers. Because of this, it is unnecessary to define and configure -handlers for all the loggers an application uses. It is sufficient to -configure handlers for a top-level logger and create child loggers as needed. - - -Handlers -^^^^^^^^ - -:class:`Handler` objects are responsible for dispatching the appropriate log -messages (based on the log messages' severity) to the handler's specified -destination. Logger objects can add zero or more handler objects to themselves -with an :func:`addHandler` method. As an example scenario, an application may -want to send all log messages to a log file, all log messages of error or higher -to stdout, and all messages of critical to an email address. This scenario -requires three individual handlers where each handler is responsible for sending -messages of a specific severity to a specific location. - -The standard library includes quite a few handler types; this tutorial uses only -:class:`StreamHandler` and :class:`FileHandler` in its examples. - -There are very few methods in a handler for application developers to concern -themselves with. The only handler methods that seem relevant for application -developers who are using the built-in handler objects (that is, not creating -custom handlers) are the following configuration methods: - -* The :meth:`Handler.setLevel` method, just as in logger objects, specifies the - lowest severity that will be dispatched to the appropriate destination. Why - are there two :func:`setLevel` methods? The level set in the logger - determines which severity of messages it will pass to its handlers. The level - set in each handler determines which messages that handler will send on. - -* :func:`setFormatter` selects a Formatter object for this handler to use. - -* :func:`addFilter` and :func:`removeFilter` respectively configure and - deconfigure filter objects on handlers. - -Application code should not directly instantiate and use instances of -:class:`Handler`. Instead, the :class:`Handler` class is a base class that -defines the interface that all handlers should have and establishes some -default behavior that child classes can use (or override). - - -Formatters -^^^^^^^^^^ - -Formatter objects configure the final order, structure, and contents of the log -message. Unlike the base :class:`logging.Handler` class, application code may -instantiate formatter classes, although you could likely subclass the formatter -if your application needs special behavior. The constructor takes two optional -arguments: a message format string and a date format string. If there is no -message format string, the default is to use the raw message. If there is no -date format string, the default date format is:: - - %Y-%m-%d %H:%M:%S - -with the milliseconds tacked on at the end. - -The message format string uses ``%(<dictionary key>)s`` styled string -substitution; the possible keys are documented in :ref:`formatter-objects`. - -The following message format string will log the time in a human-readable -format, the severity of the message, and the contents of the message, in that -order:: - - "%(asctime)s - %(levelname)s - %(message)s" - -Formatters use a user-configurable function to convert the creation time of a -record to a tuple. By default, :func:`time.localtime` is used; to change this -for a particular formatter instance, set the ``converter`` attribute of the -instance to a function with the same signature as :func:`time.localtime` or -:func:`time.gmtime`. To change it for all formatters, for example if you want -all logging times to be shown in GMT, set the ``converter`` attribute in the -Formatter class (to ``time.gmtime`` for GMT display). - - -Configuring Logging -^^^^^^^^^^^^^^^^^^^ - -Programmers can configure logging either by creating loggers, handlers, and -formatters explicitly in a main module with the configuration methods listed -above (using Python code), or by creating a logging config file. The following -code is an example of configuring a very simple logger, a console handler, and a -simple formatter in a Python module:: - - import logging - - # create logger - logger = logging.getLogger("simple_example") - logger.setLevel(logging.DEBUG) - # create console handler and set level to debug - ch = logging.StreamHandler() - ch.setLevel(logging.DEBUG) - # create formatter - formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") - # add formatter to ch - ch.setFormatter(formatter) - # add ch to logger - logger.addHandler(ch) - - # "application" code - logger.debug("debug message") - logger.info("info message") - logger.warn("warn message") - logger.error("error message") - logger.critical("critical message") - -Running this module from the command line produces the following output:: - - $ python simple_logging_module.py - 2005-03-19 15:10:26,618 - simple_example - DEBUG - debug message - 2005-03-19 15:10:26,620 - simple_example - INFO - info message - 2005-03-19 15:10:26,695 - simple_example - WARNING - warn message - 2005-03-19 15:10:26,697 - simple_example - ERROR - error message - 2005-03-19 15:10:26,773 - simple_example - CRITICAL - critical message - -The following Python module creates a logger, handler, and formatter nearly -identical to those in the example listed above, with the only difference being -the names of the objects:: - - import logging - import logging.config - - logging.config.fileConfig("logging.conf") - - # create logger - logger = logging.getLogger("simpleExample") - - # "application" code - logger.debug("debug message") - logger.info("info message") - logger.warn("warn message") - logger.error("error message") - logger.critical("critical message") - -Here is the logging.conf file:: - - [loggers] - keys=root,simpleExample - - [handlers] - keys=consoleHandler - - [formatters] - keys=simpleFormatter - - [logger_root] - level=DEBUG - handlers=consoleHandler - - [logger_simpleExample] - level=DEBUG - handlers=consoleHandler - qualname=simpleExample - propagate=0 - - [handler_consoleHandler] - class=StreamHandler - level=DEBUG - formatter=simpleFormatter - args=(sys.stdout,) - - [formatter_simpleFormatter] - format=%(asctime)s - %(name)s - %(levelname)s - %(message)s - datefmt= - -The output is nearly identical to that of the non-config-file-based example:: - - $ python simple_logging_config.py - 2005-03-19 15:38:55,977 - simpleExample - DEBUG - debug message - 2005-03-19 15:38:55,979 - simpleExample - INFO - info message - 2005-03-19 15:38:56,054 - simpleExample - WARNING - warn message - 2005-03-19 15:38:56,055 - simpleExample - ERROR - error message - 2005-03-19 15:38:56,130 - simpleExample - CRITICAL - critical message - -You can see that the config file approach has a few advantages over the Python -code approach, mainly separation of configuration and code and the ability of -noncoders to easily modify the logging properties. - -.. _library-config: - -Configuring Logging for a Library -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When developing a library which uses logging, some consideration needs to be -given to its configuration. If the using application does not use logging, and -library code makes logging calls, then a one-off message "No handlers could be -found for logger X.Y.Z" is printed to the console. This message is intended -to catch mistakes in logging configuration, but will confuse an application -developer who is not aware of logging by the library. - -In addition to documenting how a library uses logging, a good way to configure -library logging so that it does not cause a spurious message is to add a -handler which does nothing. This avoids the message being printed, since a -handler will be found: it just doesn't produce any output. If the library user -configures logging for application use, presumably that configuration will add -some handlers, and if levels are suitably configured then logging calls made -in library code will send output to those handlers, as normal. - -A do-nothing handler can be simply defined as follows:: - - import logging - - class NullHandler(logging.Handler): - def emit(self, record): - pass - -An instance of this handler should be added to the top-level logger of the -logging namespace used by the library. If all logging by a library *foo* is -done using loggers with names matching "foo.x.y", then the code:: - - import logging - - h = NullHandler() - logging.getLogger("foo").addHandler(h) - -should have the desired effect. If an organisation produces a number of -libraries, then the logger name specified can be "orgname.foo" rather than -just "foo". - -.. versionadded:: 3.1 - The :class:`NullHandler` class. - - -Logging Levels --------------- - -The numeric values of logging levels are given in the following table. These are -primarily of interest if you want to define your own levels, and need them to -have specific values relative to the predefined levels. If you define a level -with the same numeric value, it overwrites the predefined value; the predefined -name is lost. - -+--------------+---------------+ -| Level | Numeric value | -+==============+===============+ -| ``CRITICAL`` | 50 | -+--------------+---------------+ -| ``ERROR`` | 40 | -+--------------+---------------+ -| ``WARNING`` | 30 | -+--------------+---------------+ -| ``INFO`` | 20 | -+--------------+---------------+ -| ``DEBUG`` | 10 | -+--------------+---------------+ -| ``NOTSET`` | 0 | -+--------------+---------------+ - -Levels can also be associated with loggers, being set either by the developer or -through loading a saved logging configuration. When a logging method is called -on a logger, the logger compares its own level with the level associated with -the method call. If the logger's level is higher than the method call's, no -logging message is actually generated. This is the basic mechanism controlling -the verbosity of logging output. - -Logging messages are encoded as instances of the :class:`LogRecord` class. When -a logger decides to actually log an event, a :class:`LogRecord` instance is -created from the logging message. - -Logging messages are subjected to a dispatch mechanism through the use of -:dfn:`handlers`, which are instances of subclasses of the :class:`Handler` -class. Handlers are responsible for ensuring that a logged message (in the form -of a :class:`LogRecord`) ends up in a particular location (or set of locations) -which is useful for the target audience for that message (such as end users, -support desk staff, system administrators, developers). Handlers are passed -:class:`LogRecord` instances intended for particular destinations. Each logger -can have zero, one or more handlers associated with it (via the -:meth:`addHandler` method of :class:`Logger`). In addition to any handlers -directly associated with a logger, *all handlers associated with all ancestors -of the logger* are called to dispatch the message (unless the *propagate* flag -for a logger is set to a false value, at which point the passing to ancestor -handlers stops). - -Just as for loggers, handlers can have levels associated with them. A handler's -level acts as a filter in the same way as a logger's level does. If a handler -decides to actually dispatch an event, the :meth:`emit` method is used to send -the message to its destination. Most user-defined subclasses of :class:`Handler` -will need to override this :meth:`emit`. - -Useful Handlers ---------------- - -In addition to the base :class:`Handler` class, many useful subclasses are -provided: - -#. :class:`StreamHandler` instances send messages to streams (file-like - objects). - -#. :class:`FileHandler` instances send messages to disk files. - -.. module:: logging.handlers - -#. :class:`BaseRotatingHandler` is the base class for handlers that - rotate log files at a certain point. It is not meant to be instantiated - directly. Instead, use :class:`RotatingFileHandler` or - :class:`TimedRotatingFileHandler`. - -#. :class:`RotatingFileHandler` instances send messages to disk - files, with support for maximum log file sizes and log file rotation. - -#. :class:`TimedRotatingFileHandler` instances send messages to - disk files, rotating the log file at certain timed intervals. - -#. :class:`SocketHandler` instances send messages to TCP/IP - sockets. - -#. :class:`DatagramHandler` instances send messages to UDP - sockets. - -#. :class:`SMTPHandler` instances send messages to a designated - email address. - -#. :class:`SysLogHandler` instances send messages to a Unix - syslog daemon, possibly on a remote machine. - -#. :class:`NTEventLogHandler` instances send messages to a - Windows NT/2000/XP event log. - -#. :class:`MemoryHandler` instances send messages to a buffer - in memory, which is flushed whenever specific criteria are met. - -#. :class:`HTTPHandler` instances send messages to an HTTP - server using either ``GET`` or ``POST`` semantics. - -#. :class:`WatchedFileHandler` instances watch the file they are - logging to. If the file changes, it is closed and reopened using the file - name. This handler is only useful on Unix-like systems; Windows does not - support the underlying mechanism used. - -.. currentmodule:: logging - -#. :class:`NullHandler` instances do nothing with error messages. They are used - by library developers who want to use logging, but want to avoid the "No - handlers could be found for logger XXX" message which can be displayed if - the library user has not configured logging. See :ref:`library-config` for - more information. - -.. versionadded:: 3.1 - The :class:`NullHandler` class. - -The :class:`NullHandler`, :class:`StreamHandler` and :class:`FileHandler` -classes are defined in the core logging package. The other handlers are -defined in a sub- module, :mod:`logging.handlers`. (There is also another -sub-module, :mod:`logging.config`, for configuration functionality.) - -Logged messages are formatted for presentation through instances of the -:class:`Formatter` class. They are initialized with a format string suitable for -use with the % operator and a dictionary. - -For formatting multiple messages in a batch, instances of -:class:`BufferingFormatter` can be used. In addition to the format string (which -is applied to each message in the batch), there is provision for header and -trailer format strings. - -When filtering based on logger level and/or handler level is not enough, -instances of :class:`Filter` can be added to both :class:`Logger` and -:class:`Handler` instances (through their :meth:`addFilter` method). Before -deciding to process a message further, both loggers and handlers consult all -their filters for permission. If any filter returns a false value, the message -is not processed further. - -The basic :class:`Filter` functionality allows filtering by specific logger -name. If this feature is used, messages sent to the named logger and its -children are allowed through the filter, and all others dropped. - -Module-Level Functions ----------------------- - -In addition to the classes described above, there are a number of module- level -functions. - - -.. function:: getLogger(name=None) - - Return a logger with the specified name or, if name is ``None``, return a - logger which is the root logger of the hierarchy. If specified, the name is - typically a dot-separated hierarchical name like *"a"*, *"a.b"* or *"a.b.c.d"*. - Choice of these names is entirely up to the developer who is using logging. - - All calls to this function with a given name return the same logger instance. - This means that logger instances never need to be passed between different parts - of an application. - - -.. function:: getLoggerClass() - - Return either the standard :class:`Logger` class, or the last class passed to - :func:`setLoggerClass`. This function may be called from within a new class - definition, to ensure that installing a customised :class:`Logger` class will - not undo customisations already applied by other code. For example:: - - class MyLogger(logging.getLoggerClass()): - # ... override behaviour here - - -.. function:: debug(msg, *args, **kwargs) - - Logs a message with level :const:`DEBUG` on the root logger. The *msg* is the - message format string, and the *args* are the arguments which are merged into - *msg* using the string formatting operator. (Note that this means that you can - use keywords in the format string, together with a single dictionary argument.) - - There are two keyword arguments in *kwargs* which are inspected: *exc_info* - which, if it does not evaluate as false, causes exception information to be - added to the logging message. If an exception tuple (in the format returned by - :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` - is called to get the exception information. - - The other optional keyword argument is *extra* which can be used to pass a - dictionary which is used to populate the __dict__ of the LogRecord created for - the logging event with user-defined attributes. These custom attributes can then - be used as you like. For example, they could be incorporated into logged - messages. For example:: - - FORMAT = "%(asctime)-15s %(clientip)s %(user)-8s %(message)s" - logging.basicConfig(format=FORMAT) - d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} - logging.warning("Protocol problem: %s", "connection reset", extra=d) - - would print something like:: - - 2006-02-08 22:20:02,165 192.168.0.1 fbloggs Protocol problem: connection reset - - The keys in the dictionary passed in *extra* should not clash with the keys used - by the logging system. (See the :class:`Formatter` documentation for more - information on which keys are used by the logging system.) - - If you choose to use these attributes in logged messages, you need to exercise - some care. In the above example, for instance, the :class:`Formatter` has been - set up with a format string which expects 'clientip' and 'user' in the attribute - dictionary of the LogRecord. If these are missing, the message will not be - logged because a string formatting exception will occur. So in this case, you - always need to pass the *extra* dictionary with these keys. - - While this might be annoying, this feature is intended for use in specialized - circumstances, such as multi-threaded servers where the same code executes in - many contexts, and interesting conditions which arise are dependent on this - context (such as remote client IP address and authenticated user name, in the - above example). In such circumstances, it is likely that specialized - :class:`Formatter`\ s would be used with particular :class:`Handler`\ s. - - -.. function:: info(msg, *args, **kwargs) - - Logs a message with level :const:`INFO` on the root logger. The arguments are - interpreted as for :func:`debug`. - - -.. function:: warning(msg, *args, **kwargs) - - Logs a message with level :const:`WARNING` on the root logger. The arguments are - interpreted as for :func:`debug`. - - -.. function:: error(msg, *args, **kwargs) - - Logs a message with level :const:`ERROR` on the root logger. The arguments are - interpreted as for :func:`debug`. - - -.. function:: critical(msg, *args, **kwargs) - - Logs a message with level :const:`CRITICAL` on the root logger. The arguments - are interpreted as for :func:`debug`. - - -.. function:: exception(msg, *args) - - Logs a message with level :const:`ERROR` on the root logger. The arguments are - interpreted as for :func:`debug`. Exception info is added to the logging - message. This function should only be called from an exception handler. - - -.. function:: log(level, msg, *args, **kwargs) - - Logs a message with level *level* on the root logger. The other arguments are - interpreted as for :func:`debug`. - - -.. function:: disable(lvl) - - Provides an overriding level *lvl* for all loggers which takes precedence over - the logger's own level. When the need arises to temporarily throttle logging - output down across the whole application, this function can be useful. Its - effect is to disable all logging calls of severity *lvl* and below, so that - if you call it with a value of INFO, then all INFO and DEBUG events would be - discarded, whereas those of severity WARNING and above would be processed - according to the logger's effective level. - - -.. function:: addLevelName(lvl, levelName) - - Associates level *lvl* with text *levelName* in an internal dictionary, which is - used to map numeric levels to a textual representation, for example when a - :class:`Formatter` formats a message. This function can also be used to define - your own levels. The only constraints are that all levels used must be - registered using this function, levels should be positive integers and they - should increase in increasing order of severity. - - -.. function:: getLevelName(lvl) - - Returns the textual representation of logging level *lvl*. If the level is one - of the predefined levels :const:`CRITICAL`, :const:`ERROR`, :const:`WARNING`, - :const:`INFO` or :const:`DEBUG` then you get the corresponding string. If you - have associated levels with names using :func:`addLevelName` then the name you - have associated with *lvl* is returned. If a numeric value corresponding to one - of the defined levels is passed in, the corresponding string representation is - returned. Otherwise, the string "Level %s" % lvl is returned. - - -.. function:: makeLogRecord(attrdict) - - Creates and returns a new :class:`LogRecord` instance whose attributes are - defined by *attrdict*. This function is useful for taking a pickled - :class:`LogRecord` attribute dictionary, sent over a socket, and reconstituting - it as a :class:`LogRecord` instance at the receiving end. - - -.. function:: basicConfig(**kwargs) - - Does basic configuration for the logging system by creating a - :class:`StreamHandler` with a default :class:`Formatter` and adding it to the - root logger. The functions :func:`debug`, :func:`info`, :func:`warning`, - :func:`error` and :func:`critical` will call :func:`basicConfig` automatically - if no handlers are defined for the root logger. - - This function does nothing if the root logger already has handlers - configured for it. - - The following keyword arguments are supported. - - +--------------+---------------------------------------------+ - | Format | Description | - +==============+=============================================+ - | ``filename`` | Specifies that a FileHandler be created, | - | | using the specified filename, rather than a | - | | StreamHandler. | - +--------------+---------------------------------------------+ - | ``filemode`` | Specifies the mode to open the file, if | - | | filename is specified (if filemode is | - | | unspecified, it defaults to 'a'). | - +--------------+---------------------------------------------+ - | ``format`` | Use the specified format string for the | - | | handler. | - +--------------+---------------------------------------------+ - | ``datefmt`` | Use the specified date/time format. | - +--------------+---------------------------------------------+ - | ``level`` | Set the root logger level to the specified | - | | level. | - +--------------+---------------------------------------------+ - | ``stream`` | Use the specified stream to initialize the | - | | StreamHandler. Note that this argument is | - | | incompatible with 'filename' - if both are | - | | present, 'stream' is ignored. | - +--------------+---------------------------------------------+ - - -.. function:: shutdown() - - Informs the logging system to perform an orderly shutdown by flushing and - closing all handlers. This should be called at application exit and no - further use of the logging system should be made after this call. - - -.. function:: setLoggerClass(klass) - - Tells the logging system to use the class *klass* when instantiating a logger. - The class should define :meth:`__init__` such that only a name argument is - required, and the :meth:`__init__` should call :meth:`Logger.__init__`. This - function is typically called before any loggers are instantiated by applications - which need to use custom logger behavior. - - -.. seealso:: - - :pep:`282` - A Logging System - The proposal which described this feature for inclusion in the Python standard - library. - - `Original Python logging package <http://www.red-dove.com/python_logging.html>`_ - This is the original source for the :mod:`logging` package. The version of the - package available from this site is suitable for use with Python 1.5.2, 2.1.x - and 2.2.x, which do not include the :mod:`logging` package in the standard - library. .. _logger: @@ -848,6 +53,7 @@ Loggers have the following attributes and methods. Note that Loggers are never instantiated directly, but always through the module-level function ``logging.getLogger(name)``. +.. class:: Logger .. attribute:: Logger.propagate @@ -864,7 +70,7 @@ instantiated directly, but always through the module-level function the root logger, or delegation to the parent when the logger is a non-root logger). Note that the root logger is created with level :const:`WARNING`. - The term "delegation to the parent" means that if a logger has a level of + The term 'delegation to the parent' means that if a logger has a level of NOTSET, its chain of ancestor loggers is traversed until either an ancestor with a level other than NOTSET is found, or the root is reached. @@ -892,6 +98,16 @@ instantiated directly, but always through the module-level function :const:`NOTSET` is found, and that value is returned. +.. method:: Logger.getChild(suffix) + + Returns a logger which is a descendant to this logger, as determined by the suffix. + Thus, ``logging.getLogger('abc').getChild('def.ghi')`` would return the same + logger as would be returned by ``logging.getLogger('abc.def.ghi')``. This is a + convenience method, useful when the parent logger is named using e.g. ``__name__`` + rather than a literal string. + + .. versionadded:: 3.2 + .. method:: Logger.debug(msg, *args, **kwargs) @@ -900,23 +116,41 @@ instantiated directly, but always through the module-level function *msg* using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.) - There are two keyword arguments in *kwargs* which are inspected: *exc_info* + There are three keyword arguments in *kwargs* which are inspected: *exc_info* which, if it does not evaluate as false, causes exception information to be added to the logging message. If an exception tuple (in the format returned by :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` is called to get the exception information. - The other optional keyword argument is *extra* which can be used to pass a + The second optional keyword argument is *stack_info*, which defaults to + False. If specified as True, stack information is added to the logging + message, including the actual logging call. Note that this is not the same + stack information as that displayed through specifying *exc_info*: The + former is stack frames from the bottom of the stack up to the logging call + in the current thread, whereas the latter is information about stack frames + which have been unwound, following an exception, while searching for + exception handlers. + + You can specify *stack_info* independently of *exc_info*, e.g. to just show + how you got to a certain point in your code, even when no exceptions were + raised. The stack frames are printed following a header line which says:: + + Stack (most recent call last): + + This mimics the `Traceback (most recent call last):` which is used when + displaying exception frames. + + The third keyword argument is *extra* which can be used to pass a dictionary which is used to populate the __dict__ of the LogRecord created for the logging event with user-defined attributes. These custom attributes can then be used as you like. For example, they could be incorporated into logged messages. For example:: - FORMAT = "%(asctime)-15s %(clientip)s %(user)-8s %(message)s" + FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' logging.basicConfig(format=FORMAT) d = { 'clientip' : '192.168.0.1', 'user' : 'fbloggs' } - logger = logging.getLogger("tcpserver") - logger.warning("Protocol problem: %s", "connection reset", extra=d) + logger = logging.getLogger('tcpserver') + logger.warning('Protocol problem: %s', 'connection reset', extra=d) would print something like :: @@ -940,6 +174,9 @@ instantiated directly, but always through the module-level function above example). In such circumstances, it is likely that specialized :class:`Formatter`\ s would be used with particular :class:`Handler`\ s. + .. versionadded:: 3.2 + The *stack_info* parameter was added. + .. method:: Logger.info(msg, *args, **kwargs) @@ -1004,10 +241,11 @@ instantiated directly, but always through the module-level function Removes the specified handler *hdlr* from this logger. -.. method:: Logger.findCaller() +.. method:: Logger.findCaller(stack_info=False) Finds the caller's source filename and line number. Returns the filename, line - number and function name as a 3-element tuple. + number, function name and stack information as a 4-element tuple. The stack + information is returned as *None* unless *stack_info* is *True*. .. method:: Logger.handle(record) @@ -1018,630 +256,22 @@ instantiated directly, but always through the module-level function Logger-level filtering is applied using :meth:`~Logger.filter`. -.. method:: Logger.makeRecord(name, lvl, fn, lno, msg, args, exc_info, func=None, extra=None) +.. method:: Logger.makeRecord(name, lvl, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None) This is a factory method which can be overridden in subclasses to create specialized :class:`LogRecord` instances. +.. method:: Logger.hasHandlers() -.. _minimal-example: - -Basic example -------------- - -The :mod:`logging` package provides a lot of flexibility, and its configuration -can appear daunting. This section demonstrates that simple use of the logging -package is possible. - -The simplest example shows logging to the console:: - - import logging - - logging.debug('A debug message') - logging.info('Some information') - logging.warning('A shot across the bows') - -If you run the above script, you'll see this:: - - WARNING:root:A shot across the bows - -Because no particular logger was specified, the system used the root logger. The -debug and info messages didn't appear because by default, the root logger is -configured to only handle messages with a severity of WARNING or above. The -message format is also a configuration default, as is the output destination of -the messages - ``sys.stderr``. The severity level, the message format and -destination can be easily changed, as shown in the example below:: - - import logging - - logging.basicConfig(level=logging.DEBUG, - format='%(asctime)s %(levelname)s %(message)s', - filename='myapp.log', - filemode='w') - logging.debug('A debug message') - logging.info('Some information') - logging.warning('A shot across the bows') + Checks to see if this logger has any handlers configured. This is done by + looking for handlers in this logger and its parents in the logger hierarchy. + Returns True if a handler was found, else False. The method stops searching + up the hierarchy whenever a logger with the 'propagate' attribute set to + False is found - that will be the last logger which is checked for the + existence of handlers. -The :meth:`basicConfig` method is used to change the configuration defaults, -which results in output (written to ``myapp.log``) which should look -something like the following:: - - 2004-07-02 13:00:08,743 DEBUG A debug message - 2004-07-02 13:00:08,743 INFO Some information - 2004-07-02 13:00:08,743 WARNING A shot across the bows - -This time, all messages with a severity of DEBUG or above were handled, and the -format of the messages was also changed, and output went to the specified file -rather than the console. - -.. XXX logging should probably be updated for new string formatting! - -Formatting uses the old Python string formatting - see section -:ref:`old-string-formatting`. The format string takes the following common -specifiers. For a complete list of specifiers, consult the :class:`Formatter` -documentation. - -+-------------------+-----------------------------------------------+ -| Format | Description | -+===================+===============================================+ -| ``%(name)s`` | Name of the logger (logging channel). | -+-------------------+-----------------------------------------------+ -| ``%(levelname)s`` | Text logging level for the message | -| | (``'DEBUG'``, ``'INFO'``, ``'WARNING'``, | -| | ``'ERROR'``, ``'CRITICAL'``). | -+-------------------+-----------------------------------------------+ -| ``%(asctime)s`` | Human-readable time when the | -| | :class:`LogRecord` was created. By default | -| | this is of the form "2003-07-08 16:49:45,896" | -| | (the numbers after the comma are millisecond | -| | portion of the time). | -+-------------------+-----------------------------------------------+ -| ``%(message)s`` | The logged message. | -+-------------------+-----------------------------------------------+ - -To change the date/time format, you can pass an additional keyword parameter, -*datefmt*, as in the following:: - - import logging - - logging.basicConfig(level=logging.DEBUG, - format='%(asctime)s %(levelname)-8s %(message)s', - datefmt='%a, %d %b %Y %H:%M:%S', - filename='/temp/myapp.log', - filemode='w') - logging.debug('A debug message') - logging.info('Some information') - logging.warning('A shot across the bows') - -which would result in output like :: + .. versionadded:: 3.2 - Fri, 02 Jul 2004 13:06:18 DEBUG A debug message - Fri, 02 Jul 2004 13:06:18 INFO Some information - Fri, 02 Jul 2004 13:06:18 WARNING A shot across the bows - -The date format string follows the requirements of :func:`strftime` - see the -documentation for the :mod:`time` module. - -If, instead of sending logging output to the console or a file, you'd rather use -a file-like object which you have created separately, you can pass it to -:func:`basicConfig` using the *stream* keyword argument. Note that if both -*stream* and *filename* keyword arguments are passed, the *stream* argument is -ignored. - -Of course, you can put variable information in your output. To do this, simply -have the message be a format string and pass in additional arguments containing -the variable information, as in the following example:: - - import logging - - logging.basicConfig(level=logging.DEBUG, - format='%(asctime)s %(levelname)-8s %(message)s', - datefmt='%a, %d %b %Y %H:%M:%S', - filename='/temp/myapp.log', - filemode='w') - logging.error('Pack my box with %d dozen %s', 5, 'liquor jugs') - -which would result in :: - - Wed, 21 Jul 2004 15:35:16 ERROR Pack my box with 5 dozen liquor jugs - - -.. _multiple-destinations: - -Logging to multiple destinations --------------------------------- - -Let's say you want to log to console and file with different message formats and -in differing circumstances. Say you want to log messages with levels of DEBUG -and higher to file, and those messages at level INFO and higher to the console. -Let's also assume that the file should contain timestamps, but the console -messages should not. Here's how you can achieve this:: - - import logging - - # set up logging to file - see previous section for more details - logging.basicConfig(level=logging.DEBUG, - format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', - datefmt='%m-%d %H:%M', - filename='/temp/myapp.log', - filemode='w') - # define a Handler which writes INFO messages or higher to the sys.stderr - console = logging.StreamHandler() - console.setLevel(logging.INFO) - # set a format which is simpler for console use - formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s') - # tell the handler to use this format - console.setFormatter(formatter) - # add the handler to the root logger - logging.getLogger('').addHandler(console) - - # Now, we can log to the root logger, or any other logger. First the root... - logging.info('Jackdaws love my big sphinx of quartz.') - - # Now, define a couple of other loggers which might represent areas in your - # application: - - logger1 = logging.getLogger('myapp.area1') - logger2 = logging.getLogger('myapp.area2') - - logger1.debug('Quick zephyrs blow, vexing daft Jim.') - logger1.info('How quickly daft jumping zebras vex.') - logger2.warning('Jail zesty vixen who grabbed pay from quack.') - logger2.error('The five boxing wizards jump quickly.') - -When you run this, on the console you will see :: - - root : INFO Jackdaws love my big sphinx of quartz. - myapp.area1 : INFO How quickly daft jumping zebras vex. - myapp.area2 : WARNING Jail zesty vixen who grabbed pay from quack. - myapp.area2 : ERROR The five boxing wizards jump quickly. - -and in the file you will see something like :: - - 10-22 22:19 root INFO Jackdaws love my big sphinx of quartz. - 10-22 22:19 myapp.area1 DEBUG Quick zephyrs blow, vexing daft Jim. - 10-22 22:19 myapp.area1 INFO How quickly daft jumping zebras vex. - 10-22 22:19 myapp.area2 WARNING Jail zesty vixen who grabbed pay from quack. - 10-22 22:19 myapp.area2 ERROR The five boxing wizards jump quickly. - -As you can see, the DEBUG message only shows up in the file. The other messages -are sent to both destinations. - -This example uses console and file handlers, but you can use any number and -combination of handlers you choose. - -.. _logging-exceptions: - -Exceptions raised during logging --------------------------------- - -The logging package is designed to swallow exceptions which occur while logging -in production. This is so that errors which occur while handling logging events -- such as logging misconfiguration, network or other similar errors - do not -cause the application using logging to terminate prematurely. - -:class:`SystemExit` and :class:`KeyboardInterrupt` exceptions are never -swallowed. Other exceptions which occur during the :meth:`emit` method of a -:class:`Handler` subclass are passed to its :meth:`handleError` method. - -The default implementation of :meth:`handleError` in :class:`Handler` checks -to see if a module-level variable, :data:`raiseExceptions`, is set. If set, a -traceback is printed to :data:`sys.stderr`. If not set, the exception is swallowed. - -**Note:** The default value of :data:`raiseExceptions` is ``True``. This is because -during development, you typically want to be notified of any exceptions that -occur. It's advised that you set :data:`raiseExceptions` to ``False`` for production -usage. - -.. _context-info: - -Adding contextual information to your logging output ----------------------------------------------------- - -Sometimes you want logging output to contain contextual information in -addition to the parameters passed to the logging call. For example, in a -networked application, it may be desirable to log client-specific information -in the log (e.g. remote client's username, or IP address). Although you could -use the *extra* parameter to achieve this, it's not always convenient to pass -the information in this way. While it might be tempting to create -:class:`Logger` instances on a per-connection basis, this is not a good idea -because these instances are not garbage collected. While this is not a problem -in practice, when the number of :class:`Logger` instances is dependent on the -level of granularity you want to use in logging an application, it could -be hard to manage if the number of :class:`Logger` instances becomes -effectively unbounded. - - -Using LoggerAdapters to impart contextual information -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -An easy way in which you can pass contextual information to be output along -with logging event information is to use the :class:`LoggerAdapter` class. -This class is designed to look like a :class:`Logger`, so that you can call -:meth:`debug`, :meth:`info`, :meth:`warning`, :meth:`error`, -:meth:`exception`, :meth:`critical` and :meth:`log`. These methods have the -same signatures as their counterparts in :class:`Logger`, so you can use the -two types of instances interchangeably. - -When you create an instance of :class:`LoggerAdapter`, you pass it a -:class:`Logger` instance and a dict-like object which contains your contextual -information. When you call one of the logging methods on an instance of -:class:`LoggerAdapter`, it delegates the call to the underlying instance of -:class:`Logger` passed to its constructor, and arranges to pass the contextual -information in the delegated call. Here's a snippet from the code of -:class:`LoggerAdapter`:: - - def debug(self, msg, *args, **kwargs): - """ - Delegate a debug call to the underlying logger, after adding - contextual information from this adapter instance. - """ - msg, kwargs = self.process(msg, kwargs) - self.logger.debug(msg, *args, **kwargs) - -The :meth:`process` method of :class:`LoggerAdapter` is where the contextual -information is added to the logging output. It's passed the message and -keyword arguments of the logging call, and it passes back (potentially) -modified versions of these to use in the call to the underlying logger. The -default implementation of this method leaves the message alone, but inserts -an "extra" key in the keyword argument whose value is the dict-like object -passed to the constructor. Of course, if you had passed an "extra" keyword -argument in the call to the adapter, it will be silently overwritten. - -The advantage of using "extra" is that the values in the dict-like object are -merged into the :class:`LogRecord` instance's __dict__, allowing you to use -customized strings with your :class:`Formatter` instances which know about -the keys of the dict-like object. If you need a different method, e.g. if you -want to prepend or append the contextual information to the message string, -you just need to subclass :class:`LoggerAdapter` and override :meth:`process` -to do what you need. Here's an example script which uses this class, which -also illustrates what dict-like behaviour is needed from an arbitrary -"dict-like" object for use in the constructor:: - - import logging - - class ConnInfo: - """ - An example class which shows how an arbitrary class can be used as - the 'extra' context information repository passed to a LoggerAdapter. - """ - - def __getitem__(self, name): - """ - To allow this instance to look like a dict. - """ - from random import choice - if name == "ip": - result = choice(["127.0.0.1", "192.168.0.1"]) - elif name == "user": - result = choice(["jim", "fred", "sheila"]) - else: - result = self.__dict__.get(name, "?") - return result - - def __iter__(self): - """ - To allow iteration over keys, which will be merged into - the LogRecord dict before formatting and output. - """ - keys = ["ip", "user"] - keys.extend(self.__dict__.keys()) - return keys.__iter__() - - if __name__ == "__main__": - from random import choice - levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) - a1 = logging.LoggerAdapter(logging.getLogger("a.b.c"), - { "ip" : "123.231.231.123", "user" : "sheila" }) - logging.basicConfig(level=logging.DEBUG, - format="%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s") - a1.debug("A debug message") - a1.info("An info message with %s", "some parameters") - a2 = logging.LoggerAdapter(logging.getLogger("d.e.f"), ConnInfo()) - for x in range(10): - lvl = choice(levels) - lvlname = logging.getLevelName(lvl) - a2.log(lvl, "A message at %s level with %d %s", lvlname, 2, "parameters") - -When this script is run, the output should look something like this:: - - 2008-01-18 14:49:54,023 a.b.c DEBUG IP: 123.231.231.123 User: sheila A debug message - 2008-01-18 14:49:54,023 a.b.c INFO IP: 123.231.231.123 User: sheila An info message with some parameters - 2008-01-18 14:49:54,023 d.e.f CRITICAL IP: 192.168.0.1 User: jim A message at CRITICAL level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f INFO IP: 192.168.0.1 User: jim A message at INFO level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: sheila A message at WARNING level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f ERROR IP: 127.0.0.1 User: fred A message at ERROR level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f ERROR IP: 127.0.0.1 User: sheila A message at ERROR level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: sheila A message at WARNING level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: jim A message at WARNING level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f INFO IP: 192.168.0.1 User: fred A message at INFO level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f WARNING IP: 192.168.0.1 User: sheila A message at WARNING level with 2 parameters - 2008-01-18 14:49:54,033 d.e.f WARNING IP: 127.0.0.1 User: jim A message at WARNING level with 2 parameters - - -Using Filters to impart contextual information -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -You can also add contextual information to log output using a user-defined -:class:`Filter`. ``Filter`` instances are allowed to modify the ``LogRecords`` -passed to them, including adding additional attributes which can then be output -using a suitable format string, or if needed a custom :class:`Formatter`. - -For example in a web application, the request being processed (or at least, -the interesting parts of it) can be stored in a threadlocal -(:class:`threading.local`) variable, and then accessed from a ``Filter`` to -add, say, information from the request - say, the remote IP address and remote -user's username - to the ``LogRecord``, using the attribute names 'ip' and -'user' as in the ``LoggerAdapter`` example above. In that case, the same format -string can be used to get similar output to that shown above. Here's an example -script:: - - import logging - from random import choice - - class ContextFilter(logging.Filter): - """ - This is a filter which injects contextual information into the log. - - Rather than use actual contextual information, we just use random - data in this demo. - """ - - USERS = ['jim', 'fred', 'sheila'] - IPS = ['123.231.231.123', '127.0.0.1', '192.168.0.1'] - - def filter(self, record): - - record.ip = choice(ContextFilter.IPS) - record.user = choice(ContextFilter.USERS) - return True - - if __name__ == "__main__": - levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) - a1 = logging.LoggerAdapter(logging.getLogger("a.b.c"), - { "ip" : "123.231.231.123", "user" : "sheila" }) - logging.basicConfig(level=logging.DEBUG, - format="%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s") - a1 = logging.getLogger("a.b.c") - a2 = logging.getLogger("d.e.f") - - f = ContextFilter() - a1.addFilter(f) - a2.addFilter(f) - a1.debug("A debug message") - a1.info("An info message with %s", "some parameters") - for x in range(10): - lvl = choice(levels) - lvlname = logging.getLevelName(lvl) - a2.log(lvl, "A message at %s level with %d %s", lvlname, 2, "parameters") - -which, when run, produces something like:: - - 2010-09-06 22:38:15,292 a.b.c DEBUG IP: 123.231.231.123 User: fred A debug message - 2010-09-06 22:38:15,300 a.b.c INFO IP: 192.168.0.1 User: sheila An info message with some parameters - 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f ERROR IP: 127.0.0.1 User: jim A message at ERROR level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f DEBUG IP: 127.0.0.1 User: sheila A message at DEBUG level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f ERROR IP: 123.231.231.123 User: fred A message at ERROR level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 192.168.0.1 User: jim A message at CRITICAL level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL level with 2 parameters - 2010-09-06 22:38:15,300 d.e.f DEBUG IP: 192.168.0.1 User: jim A message at DEBUG level with 2 parameters - 2010-09-06 22:38:15,301 d.e.f ERROR IP: 127.0.0.1 User: sheila A message at ERROR level with 2 parameters - 2010-09-06 22:38:15,301 d.e.f DEBUG IP: 123.231.231.123 User: fred A message at DEBUG level with 2 parameters - 2010-09-06 22:38:15,301 d.e.f INFO IP: 123.231.231.123 User: fred A message at INFO level with 2 parameters - - -.. _multiple-processes: - -Logging to a single file from multiple processes ------------------------------------------------- - -Although logging is thread-safe, and logging to a single file from multiple -threads in a single process *is* supported, logging to a single file from -*multiple processes* is *not* supported, because there is no standard way to -serialize access to a single file across multiple processes in Python. If you -need to log to a single file from multiple processes, one way of doing this is -to have all the processes log to a :class:`SocketHandler`, and have a separate -process which implements a socket server which reads from the socket and logs -to file. (If you prefer, you can dedicate one thread in one of the existing -processes to perform this function.) The following section documents this -approach in more detail and includes a working socket receiver which can be -used as a starting point for you to adapt in your own applications. - -If you are using a recent version of Python which includes the -:mod:`multiprocessing` module, you could write your own handler which uses the -:class:`Lock` class from this module to serialize access to the file from -your processes. The existing :class:`FileHandler` and subclasses do not make -use of :mod:`multiprocessing` at present, though they may do so in the future. -Note that at present, the :mod:`multiprocessing` module does not provide -working lock functionality on all platforms (see -http://bugs.python.org/issue3770). - - -.. _network-logging: - -Sending and receiving logging events across a network ------------------------------------------------------ - -Let's say you want to send logging events across a network, and handle them at -the receiving end. A simple way of doing this is attaching a -:class:`SocketHandler` instance to the root logger at the sending end:: - - import logging, logging.handlers - - rootLogger = logging.getLogger('') - rootLogger.setLevel(logging.DEBUG) - socketHandler = logging.handlers.SocketHandler('localhost', - logging.handlers.DEFAULT_TCP_LOGGING_PORT) - # don't bother with a formatter, since a socket handler sends the event as - # an unformatted pickle - rootLogger.addHandler(socketHandler) - - # Now, we can log to the root logger, or any other logger. First the root... - logging.info('Jackdaws love my big sphinx of quartz.') - - # Now, define a couple of other loggers which might represent areas in your - # application: - - logger1 = logging.getLogger('myapp.area1') - logger2 = logging.getLogger('myapp.area2') - - logger1.debug('Quick zephyrs blow, vexing daft Jim.') - logger1.info('How quickly daft jumping zebras vex.') - logger2.warning('Jail zesty vixen who grabbed pay from quack.') - logger2.error('The five boxing wizards jump quickly.') - -At the receiving end, you can set up a receiver using the :mod:`socketserver` -module. Here is a basic working example:: - - import pickle - import logging - import logging.handlers - import socketserver - import struct - - - class LogRecordStreamHandler(socketserver.StreamRequestHandler): - """Handler for a streaming logging request. - - This basically logs the record using whatever logging policy is - configured locally. - """ - - def handle(self): - """ - Handle multiple requests - each expected to be a 4-byte length, - followed by the LogRecord in pickle format. Logs the record - according to whatever policy is configured locally. - """ - while True: - chunk = self.connection.recv(4) - if len(chunk) < 4: - break - slen = struct.unpack(">L", chunk)[0] - chunk = self.connection.recv(slen) - while len(chunk) < slen: - chunk = chunk + self.connection.recv(slen - len(chunk)) - obj = self.unPickle(chunk) - record = logging.makeLogRecord(obj) - self.handleLogRecord(record) - - def unPickle(self, data): - return pickle.loads(data) - - def handleLogRecord(self, record): - # if a name is specified, we use the named logger rather than the one - # implied by the record. - if self.server.logname is not None: - name = self.server.logname - else: - name = record.name - logger = logging.getLogger(name) - # N.B. EVERY record gets logged. This is because Logger.handle - # is normally called AFTER logger-level filtering. If you want - # to do filtering, do it at the client end to save wasting - # cycles and network bandwidth! - logger.handle(record) - - class LogRecordSocketReceiver(socketserver.ThreadingTCPServer): - """simple TCP socket-based logging receiver suitable for testing. - """ - - allow_reuse_address = 1 - - def __init__(self, host='localhost', - port=logging.handlers.DEFAULT_TCP_LOGGING_PORT, - handler=LogRecordStreamHandler): - socketserver.ThreadingTCPServer.__init__(self, (host, port), handler) - self.abort = 0 - self.timeout = 1 - self.logname = None - - def serve_until_stopped(self): - import select - abort = 0 - while not abort: - rd, wr, ex = select.select([self.socket.fileno()], - [], [], - self.timeout) - if rd: - self.handle_request() - abort = self.abort - - def main(): - logging.basicConfig( - format="%(relativeCreated)5d %(name)-15s %(levelname)-8s %(message)s") - tcpserver = LogRecordSocketReceiver() - print("About to start TCP server...") - tcpserver.serve_until_stopped() - - if __name__ == "__main__": - main() - -First run the server, and then the client. On the client side, nothing is -printed on the console; on the server side, you should see something like:: - - About to start TCP server... - 59 root INFO Jackdaws love my big sphinx of quartz. - 59 myapp.area1 DEBUG Quick zephyrs blow, vexing daft Jim. - 69 myapp.area1 INFO How quickly daft jumping zebras vex. - 69 myapp.area2 WARNING Jail zesty vixen who grabbed pay from quack. - 69 myapp.area2 ERROR The five boxing wizards jump quickly. - -Note that there are some security issues with pickle in some scenarios. If -these affect you, you can use an alternative serialization scheme by overriding -the :meth:`makePickle` method and implementing your alternative there, as -well as adapting the above script to use your alternative serialization. - -.. _arbitrary-object-messages: - -Using arbitrary objects as messages ------------------------------------ - -In the preceding sections and examples, it has been assumed that the message -passed when logging the event is a string. However, this is not the only -possibility. You can pass an arbitrary object as a message, and its -:meth:`__str__` method will be called when the logging system needs to convert -it to a string representation. In fact, if you want to, you can avoid -computing a string representation altogether - for example, the -:class:`SocketHandler` emits an event by pickling it and sending it over the -wire. - -Optimization ------------- - -Formatting of message arguments is deferred until it cannot be avoided. -However, computing the arguments passed to the logging method can also be -expensive, and you may want to avoid doing it if the logger will just throw -away your event. To decide what to do, you can call the :meth:`isEnabledFor` -method which takes a level argument and returns true if the event would be -created by the Logger for that level of call. You can write code like this:: - - if logger.isEnabledFor(logging.DEBUG): - logger.debug("Message with %s, %s", expensive_func1(), - expensive_func2()) - -so that if the logger's threshold is set above ``DEBUG``, the calls to -:func:`expensive_func1` and :func:`expensive_func2` are never made. - -There are other optimizations which can be made for specific applications which -need more precise control over what logging information is collected. Here's a -list of things you can do to avoid processing during logging which you don't -need: - -+-----------------------------------------------+----------------------------------------+ -| What you don't want to collect | How to avoid collecting it | -+===============================================+========================================+ -| Information about where calls were made from. | Set ``logging._srcfile`` to ``None``. | -+-----------------------------------------------+----------------------------------------+ -| Threading information. | Set ``logging.logThreads`` to ``0``. | -+-----------------------------------------------+----------------------------------------+ -| Process information. | Set ``logging.logProcesses`` to ``0``. | -+-----------------------------------------------+----------------------------------------+ - -Also note that the core logging module only includes the basic handlers. If -you don't import :mod:`logging.handlers` and :mod:`logging.config`, they won't -take up any memory. .. _handler: @@ -1749,643 +379,7 @@ subclasses. However, the :meth:`__init__` method in subclasses needs to call is intended to be implemented by subclasses and so raises a :exc:`NotImplementedError`. - -.. _stream-handler: - -StreamHandler -^^^^^^^^^^^^^ - -The :class:`StreamHandler` class, located in the core :mod:`logging` package, -sends logging output to streams such as *sys.stdout*, *sys.stderr* or any -file-like object (or, more precisely, any object which supports :meth:`write` -and :meth:`flush` methods). - - -.. currentmodule:: logging - -.. class:: StreamHandler(stream=None) - - Returns a new instance of the :class:`StreamHandler` class. If *stream* is - specified, the instance will use it for logging output; otherwise, *sys.stderr* - will be used. - - - .. method:: emit(record) - - If a formatter is specified, it is used to format the record. The record - is then written to the stream with a trailing newline. If exception - information is present, it is formatted using - :func:`traceback.print_exception` and appended to the stream. - - - .. method:: flush() - - Flushes the stream by calling its :meth:`flush` method. Note that the - :meth:`close` method is inherited from :class:`Handler` and so does - no output, so an explicit :meth:`flush` call may be needed at times. - - -.. _file-handler: - -FileHandler -^^^^^^^^^^^ - -The :class:`FileHandler` class, located in the core :mod:`logging` package, -sends logging output to a disk file. It inherits the output functionality from -:class:`StreamHandler`. - - -.. class:: FileHandler(filename, mode='a', encoding=None, delay=0) - - Returns a new instance of the :class:`FileHandler` class. The specified file is - opened and used as the stream for logging. If *mode* is not specified, - :const:`'a'` is used. If *encoding* is not *None*, it is used to open the file - with that encoding. If *delay* is true, then file opening is deferred until the - first call to :meth:`emit`. By default, the file grows indefinitely. - - - .. method:: close() - - Closes the file. - - - .. method:: emit(record) - - Outputs the record to the file. - - -.. _null-handler: - -NullHandler -^^^^^^^^^^^ - -.. versionadded:: 3.1 - -The :class:`NullHandler` class, located in the core :mod:`logging` package, -does not do any formatting or output. It is essentially a "no-op" handler -for use by library developers. - -.. class:: NullHandler() - - Returns a new instance of the :class:`NullHandler` class. - - .. method:: emit(record) - - This method does nothing. - -See :ref:`library-config` for more information on how to use -:class:`NullHandler`. - -.. _watched-file-handler: - -WatchedFileHandler -^^^^^^^^^^^^^^^^^^ - -.. currentmodule:: logging.handlers - -The :class:`WatchedFileHandler` class, located in the :mod:`logging.handlers` -module, is a :class:`FileHandler` which watches the file it is logging to. If -the file changes, it is closed and reopened using the file name. - -A file change can happen because of usage of programs such as *newsyslog* and -*logrotate* which perform log file rotation. This handler, intended for use -under Unix/Linux, watches the file to see if it has changed since the last emit. -(A file is deemed to have changed if its device or inode have changed.) If the -file has changed, the old file stream is closed, and the file opened to get a -new stream. - -This handler is not appropriate for use under Windows, because under Windows -open log files cannot be moved or renamed - logging opens the files with -exclusive locks - and so there is no need for such a handler. Furthermore, -*ST_INO* is not supported under Windows; :func:`stat` always returns zero for -this value. - - -.. class:: WatchedFileHandler(filename[,mode[, encoding[, delay]]]) - - Returns a new instance of the :class:`WatchedFileHandler` class. The specified - file is opened and used as the stream for logging. If *mode* is not specified, - :const:`'a'` is used. If *encoding* is not *None*, it is used to open the file - with that encoding. If *delay* is true, then file opening is deferred until the - first call to :meth:`emit`. By default, the file grows indefinitely. - - - .. method:: emit(record) - - Outputs the record to the file, but first checks to see if the file has - changed. If it has, the existing stream is flushed and closed and the - file opened again, before outputting the record to the file. - -.. _rotating-file-handler: - -RotatingFileHandler -^^^^^^^^^^^^^^^^^^^ - -The :class:`RotatingFileHandler` class, located in the :mod:`logging.handlers` -module, supports rotation of disk log files. - - -.. class:: RotatingFileHandler(filename, mode='a', maxBytes=0, backupCount=0, encoding=None, delay=0) - - Returns a new instance of the :class:`RotatingFileHandler` class. The specified - file is opened and used as the stream for logging. If *mode* is not specified, - ``'a'`` is used. If *encoding* is not *None*, it is used to open the file - with that encoding. If *delay* is true, then file opening is deferred until the - first call to :meth:`emit`. By default, the file grows indefinitely. - - You can use the *maxBytes* and *backupCount* values to allow the file to - :dfn:`rollover` at a predetermined size. When the size is about to be exceeded, - the file is closed and a new file is silently opened for output. Rollover occurs - whenever the current log file is nearly *maxBytes* in length; if *maxBytes* is - zero, rollover never occurs. If *backupCount* is non-zero, the system will save - old log files by appending the extensions ".1", ".2" etc., to the filename. For - example, with a *backupCount* of 5 and a base file name of :file:`app.log`, you - would get :file:`app.log`, :file:`app.log.1`, :file:`app.log.2`, up to - :file:`app.log.5`. The file being written to is always :file:`app.log`. When - this file is filled, it is closed and renamed to :file:`app.log.1`, and if files - :file:`app.log.1`, :file:`app.log.2`, etc. exist, then they are renamed to - :file:`app.log.2`, :file:`app.log.3` etc. respectively. - - - .. method:: doRollover() - - Does a rollover, as described above. - - - .. method:: emit(record) - - Outputs the record to the file, catering for rollover as described - previously. - -.. _timed-rotating-file-handler: - -TimedRotatingFileHandler -^^^^^^^^^^^^^^^^^^^^^^^^ - -The :class:`TimedRotatingFileHandler` class, located in the -:mod:`logging.handlers` module, supports rotation of disk log files at certain -timed intervals. - - -.. class:: TimedRotatingFileHandler(filename, when='h', interval=1, backupCount=0, encoding=None, delay=0, utc=False) - - Returns a new instance of the :class:`TimedRotatingFileHandler` class. The - specified file is opened and used as the stream for logging. On rotating it also - sets the filename suffix. Rotating happens based on the product of *when* and - *interval*. - - You can use the *when* to specify the type of *interval*. The list of possible - values is below. Note that they are not case sensitive. - - +----------------+-----------------------+ - | Value | Type of interval | - +================+=======================+ - | ``'S'`` | Seconds | - +----------------+-----------------------+ - | ``'M'`` | Minutes | - +----------------+-----------------------+ - | ``'H'`` | Hours | - +----------------+-----------------------+ - | ``'D'`` | Days | - +----------------+-----------------------+ - | ``'W'`` | Week day (0=Monday) | - +----------------+-----------------------+ - | ``'midnight'`` | Roll over at midnight | - +----------------+-----------------------+ - - The system will save old log files by appending extensions to the filename. - The extensions are date-and-time based, using the strftime format - ``%Y-%m-%d_%H-%M-%S`` or a leading portion thereof, depending on the - rollover interval. - - When computing the next rollover time for the first time (when the handler - is created), the last modification time of an existing log file, or else - the current time, is used to compute when the next rotation will occur. - - If the *utc* argument is true, times in UTC will be used; otherwise - local time is used. - - If *backupCount* is nonzero, at most *backupCount* files - will be kept, and if more would be created when rollover occurs, the oldest - one is deleted. The deletion logic uses the interval to determine which - files to delete, so changing the interval may leave old files lying around. - - If *delay* is true, then file opening is deferred until the first call to - :meth:`emit`. - - - .. method:: doRollover() - - Does a rollover, as described above. - - - .. method:: emit(record) - - Outputs the record to the file, catering for rollover as described above. - - -.. _socket-handler: - -SocketHandler -^^^^^^^^^^^^^ - -The :class:`SocketHandler` class, located in the :mod:`logging.handlers` module, -sends logging output to a network socket. The base class uses a TCP socket. - - -.. class:: SocketHandler(host, port) - - Returns a new instance of the :class:`SocketHandler` class intended to - communicate with a remote machine whose address is given by *host* and *port*. - - - .. method:: close() - - Closes the socket. - - - .. method:: emit() - - Pickles the record's attribute dictionary and writes it to the socket in - binary format. If there is an error with the socket, silently drops the - packet. If the connection was previously lost, re-establishes the - connection. To unpickle the record at the receiving end into a - :class:`LogRecord`, use the :func:`makeLogRecord` function. - - - .. method:: handleError() - - Handles an error which has occurred during :meth:`emit`. The most likely - cause is a lost connection. Closes the socket so that we can retry on the - next event. - - - .. method:: makeSocket() - - This is a factory method which allows subclasses to define the precise - type of socket they want. The default implementation creates a TCP socket - (:const:`socket.SOCK_STREAM`). - - - .. method:: makePickle(record) - - Pickles the record's attribute dictionary in binary format with a length - prefix, and returns it ready for transmission across the socket. - - Note that pickles aren't completely secure. If you are concerned about - security, you may want to override this method to implement a more secure - mechanism. For example, you can sign pickles using HMAC and then verify - them on the receiving end, or alternatively you can disable unpickling of - global objects on the receiving end. - - .. method:: send(packet) - - Send a pickled string *packet* to the socket. This function allows for - partial sends which can happen when the network is busy. - - -.. _datagram-handler: - -DatagramHandler -^^^^^^^^^^^^^^^ - -The :class:`DatagramHandler` class, located in the :mod:`logging.handlers` -module, inherits from :class:`SocketHandler` to support sending logging messages -over UDP sockets. - - -.. class:: DatagramHandler(host, port) - - Returns a new instance of the :class:`DatagramHandler` class intended to - communicate with a remote machine whose address is given by *host* and *port*. - - - .. method:: emit() - - Pickles the record's attribute dictionary and writes it to the socket in - binary format. If there is an error with the socket, silently drops the - packet. To unpickle the record at the receiving end into a - :class:`LogRecord`, use the :func:`makeLogRecord` function. - - - .. method:: makeSocket() - - The factory method of :class:`SocketHandler` is here overridden to create - a UDP socket (:const:`socket.SOCK_DGRAM`). - - - .. method:: send(s) - - Send a pickled string to a socket. - - -.. _syslog-handler: - -SysLogHandler -^^^^^^^^^^^^^ - -The :class:`SysLogHandler` class, located in the :mod:`logging.handlers` module, -supports sending logging messages to a remote or local Unix syslog. - - -.. class:: SysLogHandler(address=('localhost', SYSLOG_UDP_PORT), facility=LOG_USER) - - Returns a new instance of the :class:`SysLogHandler` class intended to - communicate with a remote Unix machine whose address is given by *address* in - the form of a ``(host, port)`` tuple. If *address* is not specified, - ``('localhost', 514)`` is used. The address is used to open a UDP socket. An - alternative to providing a ``(host, port)`` tuple is providing an address as a - string, for example "/dev/log". In this case, a Unix domain socket is used to - send the message to the syslog. If *facility* is not specified, - :const:`LOG_USER` is used. - - - .. method:: close() - - Closes the socket to the remote host. - - - .. method:: emit(record) - - The record is formatted, and then sent to the syslog server. If exception - information is present, it is *not* sent to the server. - - - .. method:: encodePriority(facility, priority) - - Encodes the facility and priority into an integer. You can pass in strings - or integers - if strings are passed, internal mapping dictionaries are - used to convert them to integers. - - The symbolic ``LOG_`` values are defined in :class:`SysLogHandler` and - mirror the values defined in the ``sys/syslog.h`` header file. - - **Priorities** - - +--------------------------+---------------+ - | Name (string) | Symbolic value| - +==========================+===============+ - | ``alert`` | LOG_ALERT | - +--------------------------+---------------+ - | ``crit`` or ``critical`` | LOG_CRIT | - +--------------------------+---------------+ - | ``debug`` | LOG_DEBUG | - +--------------------------+---------------+ - | ``emerg`` or ``panic`` | LOG_EMERG | - +--------------------------+---------------+ - | ``err`` or ``error`` | LOG_ERR | - +--------------------------+---------------+ - | ``info`` | LOG_INFO | - +--------------------------+---------------+ - | ``notice`` | LOG_NOTICE | - +--------------------------+---------------+ - | ``warn`` or ``warning`` | LOG_WARNING | - +--------------------------+---------------+ - - **Facilities** - - +---------------+---------------+ - | Name (string) | Symbolic value| - +===============+===============+ - | ``auth`` | LOG_AUTH | - +---------------+---------------+ - | ``authpriv`` | LOG_AUTHPRIV | - +---------------+---------------+ - | ``cron`` | LOG_CRON | - +---------------+---------------+ - | ``daemon`` | LOG_DAEMON | - +---------------+---------------+ - | ``ftp`` | LOG_FTP | - +---------------+---------------+ - | ``kern`` | LOG_KERN | - +---------------+---------------+ - | ``lpr`` | LOG_LPR | - +---------------+---------------+ - | ``mail`` | LOG_MAIL | - +---------------+---------------+ - | ``news`` | LOG_NEWS | - +---------------+---------------+ - | ``syslog`` | LOG_SYSLOG | - +---------------+---------------+ - | ``user`` | LOG_USER | - +---------------+---------------+ - | ``uucp`` | LOG_UUCP | - +---------------+---------------+ - | ``local0`` | LOG_LOCAL0 | - +---------------+---------------+ - | ``local1`` | LOG_LOCAL1 | - +---------------+---------------+ - | ``local2`` | LOG_LOCAL2 | - +---------------+---------------+ - | ``local3`` | LOG_LOCAL3 | - +---------------+---------------+ - | ``local4`` | LOG_LOCAL4 | - +---------------+---------------+ - | ``local5`` | LOG_LOCAL5 | - +---------------+---------------+ - | ``local6`` | LOG_LOCAL6 | - +---------------+---------------+ - | ``local7`` | LOG_LOCAL7 | - +---------------+---------------+ - - .. method:: mapPriority(levelname) - - Maps a logging level name to a syslog priority name. - You may need to override this if you are using custom levels, or - if the default algorithm is not suitable for your needs. The - default algorithm maps ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR`` and - ``CRITICAL`` to the equivalent syslog names, and all other level - names to "warning". - -.. _nt-eventlog-handler: - -NTEventLogHandler -^^^^^^^^^^^^^^^^^ - -The :class:`NTEventLogHandler` class, located in the :mod:`logging.handlers` -module, supports sending logging messages to a local Windows NT, Windows 2000 or -Windows XP event log. Before you can use it, you need Mark Hammond's Win32 -extensions for Python installed. - - -.. class:: NTEventLogHandler(appname, dllname=None, logtype='Application') - - Returns a new instance of the :class:`NTEventLogHandler` class. The *appname* is - used to define the application name as it appears in the event log. An - appropriate registry entry is created using this name. The *dllname* should give - the fully qualified pathname of a .dll or .exe which contains message - definitions to hold in the log (if not specified, ``'win32service.pyd'`` is used - - this is installed with the Win32 extensions and contains some basic - placeholder message definitions. Note that use of these placeholders will make - your event logs big, as the entire message source is held in the log. If you - want slimmer logs, you have to pass in the name of your own .dll or .exe which - contains the message definitions you want to use in the event log). The - *logtype* is one of ``'Application'``, ``'System'`` or ``'Security'``, and - defaults to ``'Application'``. - - - .. method:: close() - - At this point, you can remove the application name from the registry as a - source of event log entries. However, if you do this, you will not be able - to see the events as you intended in the Event Log Viewer - it needs to be - able to access the registry to get the .dll name. The current version does - not do this. - - - .. method:: emit(record) - - Determines the message ID, event category and event type, and then logs - the message in the NT event log. - - - .. method:: getEventCategory(record) - - Returns the event category for the record. Override this if you want to - specify your own categories. This version returns 0. - - - .. method:: getEventType(record) - - Returns the event type for the record. Override this if you want to - specify your own types. This version does a mapping using the handler's - typemap attribute, which is set up in :meth:`__init__` to a dictionary - which contains mappings for :const:`DEBUG`, :const:`INFO`, - :const:`WARNING`, :const:`ERROR` and :const:`CRITICAL`. If you are using - your own levels, you will either need to override this method or place a - suitable dictionary in the handler's *typemap* attribute. - - - .. method:: getMessageID(record) - - Returns the message ID for the record. If you are using your own messages, - you could do this by having the *msg* passed to the logger being an ID - rather than a format string. Then, in here, you could use a dictionary - lookup to get the message ID. This version returns 1, which is the base - message ID in :file:`win32service.pyd`. - -.. _smtp-handler: - -SMTPHandler -^^^^^^^^^^^ - -The :class:`SMTPHandler` class, located in the :mod:`logging.handlers` module, -supports sending logging messages to an email address via SMTP. - - -.. class:: SMTPHandler(mailhost, fromaddr, toaddrs, subject, credentials=None) - - Returns a new instance of the :class:`SMTPHandler` class. The instance is - initialized with the from and to addresses and subject line of the email. The - *toaddrs* should be a list of strings. To specify a non-standard SMTP port, use - the (host, port) tuple format for the *mailhost* argument. If you use a string, - the standard SMTP port is used. If your SMTP server requires authentication, you - can specify a (username, password) tuple for the *credentials* argument. - - - .. method:: emit(record) - - Formats the record and sends it to the specified addressees. - - - .. method:: getSubject(record) - - If you want to specify a subject line which is record-dependent, override - this method. - -.. _memory-handler: - -MemoryHandler -^^^^^^^^^^^^^ - -The :class:`MemoryHandler` class, located in the :mod:`logging.handlers` module, -supports buffering of logging records in memory, periodically flushing them to a -:dfn:`target` handler. Flushing occurs whenever the buffer is full, or when an -event of a certain severity or greater is seen. - -:class:`MemoryHandler` is a subclass of the more general -:class:`BufferingHandler`, which is an abstract class. This buffers logging -records in memory. Whenever each record is added to the buffer, a check is made -by calling :meth:`shouldFlush` to see if the buffer should be flushed. If it -should, then :meth:`flush` is expected to do the needful. - - -.. class:: BufferingHandler(capacity) - - Initializes the handler with a buffer of the specified capacity. - - - .. method:: emit(record) - - Appends the record to the buffer. If :meth:`shouldFlush` returns true, - calls :meth:`flush` to process the buffer. - - - .. method:: flush() - - You can override this to implement custom flushing behavior. This version - just zaps the buffer to empty. - - - .. method:: shouldFlush(record) - - Returns true if the buffer is up to capacity. This method can be - overridden to implement custom flushing strategies. - - -.. class:: MemoryHandler(capacity, flushLevel=ERROR, target=None) - - Returns a new instance of the :class:`MemoryHandler` class. The instance is - initialized with a buffer size of *capacity*. If *flushLevel* is not specified, - :const:`ERROR` is used. If no *target* is specified, the target will need to be - set using :meth:`setTarget` before this handler does anything useful. - - - .. method:: close() - - Calls :meth:`flush`, sets the target to :const:`None` and clears the - buffer. - - - .. method:: flush() - - For a :class:`MemoryHandler`, flushing means just sending the buffered - records to the target, if there is one. Override if you want different - behavior. - - - .. method:: setTarget(target) - - Sets the target handler for this handler. - - - .. method:: shouldFlush(record) - - Checks for buffer full or a record at the *flushLevel* or higher. - - -.. _http-handler: - -HTTPHandler -^^^^^^^^^^^ - -The :class:`HTTPHandler` class, located in the :mod:`logging.handlers` module, -supports sending logging messages to a Web server, using either ``GET`` or -``POST`` semantics. - - -.. class:: HTTPHandler(host, url, method='GET') - - Returns a new instance of the :class:`HTTPHandler` class. The instance is - initialized with a host address, url and HTTP method. The *host* can be of the - form ``host:port``, should you need to use a specific port number. If no - *method* is specified, ``GET`` is used. - - - .. method:: emit(record) - - Sends the record to the Web server as a percent-encoded dictionary. - +For a list of handlers included as standard, see :mod:`logging.handlers`. .. _formatter-objects: @@ -2394,7 +388,7 @@ Formatter Objects .. currentmodule:: logging -:class:`Formatter`\ s have the following attributes and methods. They are +:class:`Formatter` objects have the following attributes and methods. They are responsible for converting a :class:`LogRecord` to (usually) a string which can be interpreted by either a human or an external system. The base :class:`Formatter` allows a formatting string to be specified. If none is @@ -2407,64 +401,11 @@ into a :class:`LogRecord`'s *message* attribute. This format string contains standard Python %-style mapping keys. See section :ref:`old-string-formatting` for more information on string formatting. -Currently, the useful mapping keys in a :class:`LogRecord` are: - -+-------------------------+-----------------------------------------------+ -| Format | Description | -+=========================+===============================================+ -| ``%(name)s`` | Name of the logger (logging channel). | -+-------------------------+-----------------------------------------------+ -| ``%(levelno)s`` | Numeric logging level for the message | -| | (:const:`DEBUG`, :const:`INFO`, | -| | :const:`WARNING`, :const:`ERROR`, | -| | :const:`CRITICAL`). | -+-------------------------+-----------------------------------------------+ -| ``%(levelname)s`` | Text logging level for the message | -| | (``'DEBUG'``, ``'INFO'``, ``'WARNING'``, | -| | ``'ERROR'``, ``'CRITICAL'``). | -+-------------------------+-----------------------------------------------+ -| ``%(pathname)s`` | Full pathname of the source file where the | -| | logging call was issued (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(filename)s`` | Filename portion of pathname. | -+-------------------------+-----------------------------------------------+ -| ``%(module)s`` | Module (name portion of filename). | -+-------------------------+-----------------------------------------------+ -| ``%(funcName)s`` | Name of function containing the logging call. | -+-------------------------+-----------------------------------------------+ -| ``%(lineno)d`` | Source line number where the logging call was | -| | issued (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(created)f`` | Time when the :class:`LogRecord` was created | -| | (as returned by :func:`time.time`). | -+-------------------------+-----------------------------------------------+ -| ``%(relativeCreated)d`` | Time in milliseconds when the LogRecord was | -| | created, relative to the time the logging | -| | module was loaded. | -+-------------------------+-----------------------------------------------+ -| ``%(asctime)s`` | Human-readable time when the | -| | :class:`LogRecord` was created. By default | -| | this is of the form "2003-07-08 16:49:45,896" | -| | (the numbers after the comma are millisecond | -| | portion of the time). | -+-------------------------+-----------------------------------------------+ -| ``%(msecs)d`` | Millisecond portion of the time when the | -| | :class:`LogRecord` was created. | -+-------------------------+-----------------------------------------------+ -| ``%(thread)d`` | Thread ID (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(threadName)s`` | Thread name (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(process)d`` | Process ID (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(processName)s`` | Process name (if available). | -+-------------------------+-----------------------------------------------+ -| ``%(message)s`` | The logged message, computed as ``msg % | -| | args``. | -+-------------------------+-----------------------------------------------+ - - -.. class:: Formatter(fmt=None, datefmt=None) +The useful mapping keys in a :class:`LogRecord` are given in the section on +:ref:`logrecord-attributes`. + + +.. class:: Formatter(fmt=None, datefmt=None, style='%') Returns a new instance of the :class:`Formatter` class. The instance is initialized with a format string for the message as a whole, as well as a @@ -2472,6 +413,14 @@ Currently, the useful mapping keys in a :class:`LogRecord` are: specified, ``'%(message)s'`` is used. If no *datefmt* is specified, the ISO8601 date format is used. + The *style* parameter can be one of '%', '{' or '$' and determines how + the format string will be merged with its data: using one of %-formatting, + :meth:`str.format` or :class:`string.Template`. + + .. versionchanged:: 3.2 + The *style* parameter was added. + + .. method:: format(record) The record's attribute dictionary is used as the operand to a string @@ -2490,6 +439,9 @@ Currently, the useful mapping keys in a :class:`LogRecord` are: formatter to handle the event doesn't use the cached value but recalculates it afresh. + If stack information is available, it's appended after the exception + information, using :meth:`formatStack` to transform it if necessary. + .. method:: formatTime(record, datefmt=None) @@ -2509,17 +461,23 @@ Currently, the useful mapping keys in a :class:`LogRecord` are: just uses :func:`traceback.print_exception`. The resulting string is returned. + .. method:: formatStack(stack_info) + + Formats the specified stack information (a string as returned by + :func:`traceback.print_stack`, but with the last newline removed) as a + string. This default implementation just returns the input value. + .. _filter: Filter Objects -------------- -:class:`Filter`\ s can be used by :class:`Handler`\ s and :class:`Logger`\ s for -more sophisticated filtering than is provided by levels. The base filter class -only allows events which are below a certain point in the logger hierarchy. For -example, a filter initialized with "A.B" will allow events logged by loggers -"A.B", "A.B.C", "A.B.C.D", "A.B.D" etc. but not "A.BB", "B.A.B" etc. If -initialized with the empty string, all events are passed. +``Filters`` can be used by ``Handlers`` and ``Loggers`` for more sophisticated +filtering than is provided by levels. The base filter class only allows events +which are below a certain point in the logger hierarchy. For example, a filter +initialized with 'A.B' will allow events logged by loggers 'A.B', 'A.B.C', +'A.B.C.D', 'A.B.D' etc. but not 'A.BB', 'B.A.B' etc. If initialized with the +empty string, all events are passed. .. class:: Filter(name='') @@ -2542,6 +500,28 @@ etc.) This means that events which have been generated by descendant loggers will not be filtered by a logger's filter setting, unless the filter has also been applied to those descendant loggers. +You don't actually need to subclass ``Filter``: you can pass any instance +which has a ``filter`` method with the same semantics. + +.. versionchanged:: 3.2 + You don't need to create specialized ``Filter`` classes, or use other + classes with a ``filter`` method: you can use a function (or other + callable) as a filter. The filtering logic will check to see if the filter + object has a ``filter`` attribute: if it does, it's assumed to be a + ``Filter`` and its :meth:`~Filter.filter` method is called. Otherwise, it's + assumed to be a callable and called with the record as the single + parameter. The returned value should conform to that returned by + :meth:`~Filter.filter`. + +Although filters are used primarily to filter records based on more +sophisticated criteria than levels, they get to see every record which is +processed by the handler or logger they're attached to: this can be useful if +you want to do things like counting how many records were processed by a +particular logger or handler, or adding, changing or removing attributes in +the LogRecord being processed. Obviously changing the LogRecord needs to be +done with some care, but it does allow the injection of contextual information +into logs (see :ref:`filters-contextual`). + .. _log-record: LogRecord Objects @@ -2553,7 +533,7 @@ every time something is logged, and can be created manually via wire). -.. class:: LogRecord(name, lvl, pathname, lineno, msg, args, exc_info, func=None) +.. class:: LogRecord(name, level, pathname, lineno, msg, args, exc_info, func=None, sinfo=None) Contains all the information pertinent to the event being logged. @@ -2561,49 +541,153 @@ wire). are combined using ``msg % args`` to create the :attr:`message` field of the record. - .. attribute:: args - - Tuple of arguments to be used in formatting :attr:`msg`. - - .. attribute:: exc_info - - Exception tuple (à la `sys.exc_info`) or `None` if no exception - information is available. - - .. attribute:: func - - Name of the function of origin (i.e. in which the logging call was made). - - .. attribute:: lineno - - Line number in the source file of origin. - - .. attribute:: lvl - - Numeric logging level. - - .. attribute:: message - - Bound to the result of :meth:`getMessage` when - :meth:`Formatter.format(record)<Formatter.format>` is invoked. - - .. attribute:: msg - - User-supplied :ref:`format string<string-formatting>` or arbitrary object - (see :ref:`arbitrary-object-messages`) used in :meth:`getMessage`. - - .. attribute:: name - - Name of the logger that emitted the record. - - .. attribute:: pathname - - Absolute pathname of the source file of origin. + :param name: The name of the logger used to log the event represented by + this LogRecord. + :param level: The numeric level of the logging event (one of DEBUG, INFO etc.) + :param pathname: The full pathname of the source file where the logging call + was made. + :param lineno: The line number in the source file where the logging call was + made. + :param msg: The event description message, possibly a format string with + placeholders for variable data. + :param args: Variable data to merge into the *msg* argument to obtain the + event description. + :param exc_info: An exception tuple with the current exception information, + or *None* if no exception information is available. + :param func: The name of the function or method from which the logging call + was invoked. + :param sinfo: A text string representing stack information from the base of + the stack in the current thread, up to the logging call. .. method:: getMessage() Returns the message for this :class:`LogRecord` instance after merging any - user-supplied arguments with the message. + user-supplied arguments with the message. If the user-supplied message + argument to the logging call is not a string, :func:`str` is called on it to + convert it to a string. This allows use of user-defined classes as + messages, whose ``__str__`` method can return the actual format string to + be used. + + .. versionchanged:: 3.2 + The creation of a ``LogRecord`` has been made more configurable by + providing a factory which is used to create the record. The factory can be + set using :func:`getLogRecordFactory` and :func:`setLogRecordFactory` + (see this for the factory's signature). + + This functionality can be used to inject your own values into a + LogRecord at creation time. You can use the following pattern:: + + old_factory = logging.getLogRecordFactory() + + def record_factory(*args, **kwargs): + record = old_factory(*args, **kwargs) + record.custom_attribute = 0xdecafbad + return record + + logging.setLogRecordFactory(record_factory) + + With this pattern, multiple factories could be chained, and as long + as they don't overwrite each other's attributes or unintentionally + overwrite the standard attributes listed above, there should be no + surprises. + + +.. _logrecord-attributes: + +LogRecord attributes +-------------------- + +The LogRecord has a number of attributes, most of which are derived from the +parameters to the constructor. (Note that the names do not always correspond +exactly between the LogRecord constructor parameters and the LogRecord +attributes.) These attributes can be used to merge data from the record into +the format string. The following table lists (in alphabetical order) the +attribute names, their meanings and the corresponding placeholder in a %-style +format string. + +If you are using {}-formatting (:func:`str.format`), you can use +``{attrname}`` as the placeholder in the format string. If you are using +$-formatting (:class:`string.Template`), use the form ``${attrname}``. In +both cases, of course, replace ``attrname`` with the actual attribute name +you want to use. + +In the case of {}-formatting, you can specify formatting flags by placing them +after the attribute name, separated from it with a colon. For example: a +placeholder of ``{msecs:03d}`` would format a millisecond value of ``4`` as +``004``. Refer to the :meth:`str.format` documentation for full details on +the options available to you. + ++----------------+-------------------------+-----------------------------------------------+ +| Attribute name | Format | Description | ++================+=========================+===============================================+ +| args | You shouldn't need to | The tuple of arguments merged into ``msg`` to | +| | format this yourself. | produce ``message``. | ++----------------+-------------------------+-----------------------------------------------+ +| asctime | ``%(asctime)s`` | Human-readable time when the | +| | | :class:`LogRecord` was created. By default | +| | | this is of the form '2003-07-08 16:49:45,896' | +| | | (the numbers after the comma are millisecond | +| | | portion of the time). | ++----------------+-------------------------+-----------------------------------------------+ +| created | ``%(created)f`` | Time when the :class:`LogRecord` was created | +| | | (as returned by :func:`time.time`). | ++----------------+-------------------------+-----------------------------------------------+ +| exc_info | You shouldn't need to | Exception tuple (à la ``sys.exc_info``) or, | +| | format this yourself. | if no exception has occurred, *None*. | ++----------------+-------------------------+-----------------------------------------------+ +| filename | ``%(filename)s`` | Filename portion of ``pathname``. | ++----------------+-------------------------+-----------------------------------------------+ +| funcName | ``%(funcName)s`` | Name of function containing the logging call. | ++----------------+-------------------------+-----------------------------------------------+ +| levelname | ``%(levelname)s`` | Text logging level for the message | +| | | (``'DEBUG'``, ``'INFO'``, ``'WARNING'``, | +| | | ``'ERROR'``, ``'CRITICAL'``). | ++----------------+-------------------------+-----------------------------------------------+ +| levelno | ``%(levelno)s`` | Numeric logging level for the message | +| | | (:const:`DEBUG`, :const:`INFO`, | +| | | :const:`WARNING`, :const:`ERROR`, | +| | | :const:`CRITICAL`). | ++----------------+-------------------------+-----------------------------------------------+ +| lineno | ``%(lineno)d`` | Source line number where the logging call was | +| | | issued (if available). | ++----------------+-------------------------+-----------------------------------------------+ +| module | ``%(module)s`` | Module (name portion of ``filename``). | ++----------------+-------------------------+-----------------------------------------------+ +| msecs | ``%(msecs)d`` | Millisecond portion of the time when the | +| | | :class:`LogRecord` was created. | ++----------------+-------------------------+-----------------------------------------------+ +| message | ``%(message)s`` | The logged message, computed as ``msg % | +| | | args``. This is set when | +| | | :meth:`Formatter.format` is invoked. | ++----------------+-------------------------+-----------------------------------------------+ +| msg | You shouldn't need to | The format string passed in the original | +| | format this yourself. | logging call. Merged with ``args`` to | +| | | produce ``message``, or an arbitrary object | +| | | (see :ref:`arbitrary-object-messages`). | ++----------------+-------------------------+-----------------------------------------------+ +| name | ``%(name)s`` | Name of the logger used to log the call. | ++----------------+-------------------------+-----------------------------------------------+ +| pathname | ``%(pathname)s`` | Full pathname of the source file where the | +| | | logging call was issued (if available). | ++----------------+-------------------------+-----------------------------------------------+ +| process | ``%(process)d`` | Process ID (if available). | ++----------------+-------------------------+-----------------------------------------------+ +| processName | ``%(processName)s`` | Process name (if available). | ++----------------+-------------------------+-----------------------------------------------+ +| relativeCreated| ``%(relativeCreated)d`` | Time in milliseconds when the LogRecord was | +| | | created, relative to the time the logging | +| | | module was loaded. | ++----------------+-------------------------+-----------------------------------------------+ +| stack_info | You shouldn't need to | Stack frame information (where available) | +| | format this yourself. | from the bottom of the stack in the current | +| | | thread, up to and including the stack frame | +| | | of the logging call which resulted in the | +| | | creation of this record. | ++----------------+-------------------------+-----------------------------------------------+ +| thread | ``%(thread)d`` | Thread ID (if available). | ++----------------+-------------------------+-----------------------------------------------+ +| threadName | ``%(threadName)s`` | Thread name (if available). | ++----------------+-------------------------+-----------------------------------------------+ .. _logger-adapter: @@ -2615,7 +699,6 @@ LoggerAdapter Objects information into logging calls. For a usage example , see the section on :ref:`adding contextual information to your logging output <context-info>`. - .. class:: LoggerAdapter(logger, extra) Returns an instance of :class:`LoggerAdapter` initialized with an @@ -2629,14 +712,18 @@ information into logging calls. For a usage example , see the section on 'extra'. The return value is a (*msg*, *kwargs*) tuple which has the (possibly modified) versions of the arguments passed in. -In addition to the above, :class:`LoggerAdapter` supports all the logging +In addition to the above, :class:`LoggerAdapter` supports the following methods of :class:`Logger`, i.e. :meth:`debug`, :meth:`info`, :meth:`warning`, -:meth:`error`, :meth:`exception`, :meth:`critical` and :meth:`log`. These -methods have the same signatures as their counterparts in :class:`Logger`, so -you can use the two types of instances interchangeably. +:meth:`error`, :meth:`exception`, :meth:`critical`, :meth:`log`, +:meth:`isEnabledFor`, :meth:`getEffectiveLevel`, :meth:`setLevel`, +:meth:`hasHandlers`. These methods have the same signatures as their +counterparts in :class:`Logger`, so you can use the two types of instances +interchangeably. - The :meth:`isEnabledFor` method was added to :class:`LoggerAdapter`. This - method delegates to the underlying logger. +.. versionchanged:: 3.2 + The :meth:`isEnabledFor`, :meth:`getEffectiveLevel`, :meth:`setLevel` and + :meth:`hasHandlers` methods were added to :class:`LoggerAdapter`. These + methods delegate to the underlying logger. Thread Safety @@ -2652,401 +739,338 @@ module, you may not be able to use logging from within such handlers. This is because lock implementations in the :mod:`threading` module are not always re-entrant, and so cannot be invoked from such signal handlers. -Configuration -------------- + +Module-Level Functions +---------------------- + +In addition to the classes described above, there are a number of module- level +functions. -.. _logging-config-api: +.. function:: getLogger(name=None) -Configuration functions -^^^^^^^^^^^^^^^^^^^^^^^ + Return a logger with the specified name or, if name is ``None``, return a + logger which is the root logger of the hierarchy. If specified, the name is + typically a dot-separated hierarchical name like *'a'*, *'a.b'* or *'a.b.c.d'*. + Choice of these names is entirely up to the developer who is using logging. + + All calls to this function with a given name return the same logger instance. + This means that logger instances never need to be passed between different parts + of an application. -The following functions configure the logging module. They are located in the -:mod:`logging.config` module. Their use is optional --- you can configure the -logging module using these functions or by making calls to the main API (defined -in :mod:`logging` itself) and defining handlers which are declared either in -:mod:`logging` or :mod:`logging.handlers`. +.. function:: getLoggerClass() -.. function:: fileConfig(fname, defaults=None, disable_existing_loggers=True) + Return either the standard :class:`Logger` class, or the last class passed to + :func:`setLoggerClass`. This function may be called from within a new class + definition, to ensure that installing a customised :class:`Logger` class will + not undo customisations already applied by other code. For example:: - Reads the logging configuration from a :mod:`configparser`\-format file named - *fname*. This function can be called several times from an application, - allowing an end user the ability to select from various pre-canned - configurations (if the developer provides a mechanism to present the choices - and load the chosen configuration). Defaults to be passed to the ConfigParser - can be specified in the *defaults* argument. + class MyLogger(logging.getLoggerClass()): + # ... override behaviour here - If *disable_existing_loggers* is true, any existing loggers that are not - children of named loggers will be disabled. +.. function:: getLogRecordFactory() -.. function:: listen(port=DEFAULT_LOGGING_CONFIG_PORT) + Return a callable which is used to create a :class:`LogRecord`. - Starts up a socket server on the specified port, and listens for new - configurations. If no port is specified, the module's default - :const:`DEFAULT_LOGGING_CONFIG_PORT` is used. Logging configurations will be - sent as a file suitable for processing by :func:`fileConfig`. Returns a - :class:`Thread` instance on which you can call :meth:`start` to start the - server, and which you can :meth:`join` when appropriate. To stop the server, - call :func:`stopListening`. + .. versionadded:: 3.2 + This function has been provided, along with :func:`setLogRecordFactory`, + to allow developers more control over how the :class:`LogRecord` + representing a logging event is constructed. - To send a configuration to the socket, read in the configuration file and - send it to the socket as a string of bytes preceded by a four-byte length - string packed in binary using ``struct.pack('>L', n)``. + See :func:`setLogRecordFactory` for more information about the how the + factory is called. +.. function:: debug(msg, *args, **kwargs) -.. function:: stopListening() + Logs a message with level :const:`DEBUG` on the root logger. The *msg* is the + message format string, and the *args* are the arguments which are merged into + *msg* using the string formatting operator. (Note that this means that you can + use keywords in the format string, together with a single dictionary argument.) - Stops the listening server which was created with a call to :func:`listen`. - This is typically called before calling :meth:`join` on the return value from - :func:`listen`. + There are three keyword arguments in *kwargs* which are inspected: *exc_info* + which, if it does not evaluate as false, causes exception information to be + added to the logging message. If an exception tuple (in the format returned by + :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` + is called to get the exception information. + The second optional keyword argument is *stack_info*, which defaults to + False. If specified as True, stack information is added to the logging + message, including the actual logging call. Note that this is not the same + stack information as that displayed through specifying *exc_info*: The + former is stack frames from the bottom of the stack up to the logging call + in the current thread, whereas the latter is information about stack frames + which have been unwound, following an exception, while searching for + exception handlers. -.. _logging-config-fileformat: + You can specify *stack_info* independently of *exc_info*, e.g. to just show + how you got to a certain point in your code, even when no exceptions were + raised. The stack frames are printed following a header line which says:: -Configuration file format -^^^^^^^^^^^^^^^^^^^^^^^^^ + Stack (most recent call last): -The configuration file format understood by :func:`fileConfig` is based on -:mod:`configparser` functionality. The file must contain sections called -``[loggers]``, ``[handlers]`` and ``[formatters]`` which identify by name the -entities of each type which are defined in the file. For each such entity, there -is a separate section which identifies how that entity is configured. Thus, for -a logger named ``log01`` in the ``[loggers]`` section, the relevant -configuration details are held in a section ``[logger_log01]``. Similarly, a -handler called ``hand01`` in the ``[handlers]`` section will have its -configuration held in a section called ``[handler_hand01]``, while a formatter -called ``form01`` in the ``[formatters]`` section will have its configuration -specified in a section called ``[formatter_form01]``. The root logger -configuration must be specified in a section called ``[logger_root]``. + This mimics the `Traceback (most recent call last):` which is used when + displaying exception frames. -Examples of these sections in the file are given below. :: - - [loggers] - keys=root,log02,log03,log04,log05,log06,log07 - - [handlers] - keys=hand01,hand02,hand03,hand04,hand05,hand06,hand07,hand08,hand09 - - [formatters] - keys=form01,form02,form03,form04,form05,form06,form07,form08,form09 - -The root logger must specify a level and a list of handlers. An example of a -root logger section is given below. :: - - [logger_root] - level=NOTSET - handlers=hand01 - -The ``level`` entry can be one of ``DEBUG, INFO, WARNING, ERROR, CRITICAL`` or -``NOTSET``. For the root logger only, ``NOTSET`` means that all messages will be -logged. Level values are :func:`eval`\ uated in the context of the ``logging`` -package's namespace. - -The ``handlers`` entry is a comma-separated list of handler names, which must -appear in the ``[handlers]`` section. These names must appear in the -``[handlers]`` section and have corresponding sections in the configuration -file. - -For loggers other than the root logger, some additional information is required. -This is illustrated by the following example. :: - - [logger_parser] - level=DEBUG - handlers=hand01 - propagate=1 - qualname=compiler.parser - -The ``level`` and ``handlers`` entries are interpreted as for the root logger, -except that if a non-root logger's level is specified as ``NOTSET``, the system -consults loggers higher up the hierarchy to determine the effective level of the -logger. The ``propagate`` entry is set to 1 to indicate that messages must -propagate to handlers higher up the logger hierarchy from this logger, or 0 to -indicate that messages are **not** propagated to handlers up the hierarchy. The -``qualname`` entry is the hierarchical channel name of the logger, that is to -say the name used by the application to get the logger. - -Sections which specify handler configuration are exemplified by the following. -:: - - [handler_hand01] - class=StreamHandler - level=NOTSET - formatter=form01 - args=(sys.stdout,) - -The ``class`` entry indicates the handler's class (as determined by :func:`eval` -in the ``logging`` package's namespace). The ``level`` is interpreted as for -loggers, and ``NOTSET`` is taken to mean "log everything". - -The ``formatter`` entry indicates the key name of the formatter for this -handler. If blank, a default formatter (``logging._defaultFormatter``) is used. -If a name is specified, it must appear in the ``[formatters]`` section and have -a corresponding section in the configuration file. - -The ``args`` entry, when :func:`eval`\ uated in the context of the ``logging`` -package's namespace, is the list of arguments to the constructor for the handler -class. Refer to the constructors for the relevant handlers, or to the examples -below, to see how typical entries are constructed. :: - - [handler_hand02] - class=FileHandler - level=DEBUG - formatter=form02 - args=('python.log', 'w') - - [handler_hand03] - class=handlers.SocketHandler - level=INFO - formatter=form03 - args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT) - - [handler_hand04] - class=handlers.DatagramHandler - level=WARN - formatter=form04 - args=('localhost', handlers.DEFAULT_UDP_LOGGING_PORT) - - [handler_hand05] - class=handlers.SysLogHandler - level=ERROR - formatter=form05 - args=(('localhost', handlers.SYSLOG_UDP_PORT), handlers.SysLogHandler.LOG_USER) - - [handler_hand06] - class=handlers.NTEventLogHandler - level=CRITICAL - formatter=form06 - args=('Python Application', '', 'Application') - - [handler_hand07] - class=handlers.SMTPHandler - level=WARN - formatter=form07 - args=('localhost', 'from@abc', ['user1@abc', 'user2@xyz'], 'Logger Subject') - - [handler_hand08] - class=handlers.MemoryHandler - level=NOTSET - formatter=form08 - target= - args=(10, ERROR) - - [handler_hand09] - class=handlers.HTTPHandler - level=NOTSET - formatter=form09 - args=('localhost:9022', '/log', 'GET') - -Sections which specify formatter configuration are typified by the following. :: - - [formatter_form01] - format=F1 %(asctime)s %(levelname)s %(message)s - datefmt= - class=logging.Formatter - -The ``format`` entry is the overall format string, and the ``datefmt`` entry is -the :func:`strftime`\ -compatible date/time format string. If empty, the -package substitutes ISO8601 format date/times, which is almost equivalent to -specifying the date format string ``"%Y-%m-%d %H:%M:%S"``. The ISO8601 format -also specifies milliseconds, which are appended to the result of using the above -format string, with a comma separator. An example time in ISO8601 format is -``2003-01-23 00:29:50,411``. - -The ``class`` entry is optional. It indicates the name of the formatter's class -(as a dotted module and class name.) This option is useful for instantiating a -:class:`Formatter` subclass. Subclasses of :class:`Formatter` can present -exception tracebacks in an expanded or condensed format. - - -Configuration server example -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Here is an example of a module using the logging configuration server:: - - import logging - import logging.config - import time - import os - - # read initial config file - logging.config.fileConfig("logging.conf") - - # create and start listener on port 9999 - t = logging.config.listen(9999) - t.start() - - logger = logging.getLogger("simpleExample") - - try: - # loop through logging calls to see the difference - # new configurations make, until Ctrl+C is pressed - while True: - logger.debug("debug message") - logger.info("info message") - logger.warn("warn message") - logger.error("error message") - logger.critical("critical message") - time.sleep(5) - except KeyboardInterrupt: - # cleanup - logging.config.stopListening() - t.join() - -And here is a script that takes a filename and sends that file to the server, -properly preceded with the binary-encoded length, as the new logging -configuration:: - - #!/usr/bin/env python - import socket, sys, struct - - data_to_send = open(sys.argv[1], "r").read() - - HOST = 'localhost' - PORT = 9999 - s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - print("connecting...") - s.connect((HOST, PORT)) - print("sending config...") - s.send(struct.pack(">L", len(data_to_send))) - s.send(data_to_send) - s.close() - print("complete") - - -More examples -------------- + The third optional keyword argument is *extra* which can be used to pass a + dictionary which is used to populate the __dict__ of the LogRecord created for + the logging event with user-defined attributes. These custom attributes can then + be used as you like. For example, they could be incorporated into logged + messages. For example:: -Multiple handlers and formatters -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Loggers are plain Python objects. The :func:`addHandler` method has no minimum -or maximum quota for the number of handlers you may add. Sometimes it will be -beneficial for an application to log all messages of all severities to a text -file while simultaneously logging errors or above to the console. To set this -up, simply configure the appropriate handlers. The logging calls in the -application code will remain unchanged. Here is a slight modification to the -previous simple module-based configuration example:: - - import logging - - logger = logging.getLogger("simple_example") - logger.setLevel(logging.DEBUG) - # create file handler which logs even debug messages - fh = logging.FileHandler("spam.log") - fh.setLevel(logging.DEBUG) - # create console handler with a higher log level - ch = logging.StreamHandler() - ch.setLevel(logging.ERROR) - # create formatter and add it to the handlers - formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") - ch.setFormatter(formatter) - fh.setFormatter(formatter) - # add the handlers to logger - logger.addHandler(ch) - logger.addHandler(fh) - - # "application" code - logger.debug("debug message") - logger.info("info message") - logger.warn("warn message") - logger.error("error message") - logger.critical("critical message") - -Notice that the "application" code does not care about multiple handlers. All -that changed was the addition and configuration of a new handler named *fh*. - -The ability to create new handlers with higher- or lower-severity filters can be -very helpful when writing and testing an application. Instead of using many -``print`` statements for debugging, use ``logger.debug``: Unlike the print -statements, which you will have to delete or comment out later, the logger.debug -statements can remain intact in the source code and remain dormant until you -need them again. At that time, the only change that needs to happen is to -modify the severity level of the logger and/or handler to debug. - - -Using logging in multiple modules -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -It was mentioned above that multiple calls to -``logging.getLogger('someLogger')`` return a reference to the same logger -object. This is true not only within the same module, but also across modules -as long as it is in the same Python interpreter process. It is true for -references to the same object; additionally, application code can define and -configure a parent logger in one module and create (but not configure) a child -logger in a separate module, and all logger calls to the child will pass up to -the parent. Here is a main module:: - - import logging - import auxiliary_module - - # create logger with "spam_application" - logger = logging.getLogger("spam_application") - logger.setLevel(logging.DEBUG) - # create file handler which logs even debug messages - fh = logging.FileHandler("spam.log") - fh.setLevel(logging.DEBUG) - # create console handler with a higher log level - ch = logging.StreamHandler() - ch.setLevel(logging.ERROR) - # create formatter and add it to the handlers - formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") - fh.setFormatter(formatter) - ch.setFormatter(formatter) - # add the handlers to the logger - logger.addHandler(fh) - logger.addHandler(ch) - - logger.info("creating an instance of auxiliary_module.Auxiliary") - a = auxiliary_module.Auxiliary() - logger.info("created an instance of auxiliary_module.Auxiliary") - logger.info("calling auxiliary_module.Auxiliary.do_something") - a.do_something() - logger.info("finished auxiliary_module.Auxiliary.do_something") - logger.info("calling auxiliary_module.some_function()") - auxiliary_module.some_function() - logger.info("done with auxiliary_module.some_function()") - -Here is the auxiliary module:: - - import logging - - # create logger - module_logger = logging.getLogger("spam_application.auxiliary") - - class Auxiliary: - def __init__(self): - self.logger = logging.getLogger("spam_application.auxiliary.Auxiliary") - self.logger.info("creating an instance of Auxiliary") - def do_something(self): - self.logger.info("doing something") - a = 1 + 1 - self.logger.info("done doing something") - - def some_function(): - module_logger.info("received a call to \"some_function\"") - -The output looks like this:: - - 2005-03-23 23:47:11,663 - spam_application - INFO - - creating an instance of auxiliary_module.Auxiliary - 2005-03-23 23:47:11,665 - spam_application.auxiliary.Auxiliary - INFO - - creating an instance of Auxiliary - 2005-03-23 23:47:11,665 - spam_application - INFO - - created an instance of auxiliary_module.Auxiliary - 2005-03-23 23:47:11,668 - spam_application - INFO - - calling auxiliary_module.Auxiliary.do_something - 2005-03-23 23:47:11,668 - spam_application.auxiliary.Auxiliary - INFO - - doing something - 2005-03-23 23:47:11,669 - spam_application.auxiliary.Auxiliary - INFO - - done doing something - 2005-03-23 23:47:11,670 - spam_application - INFO - - finished auxiliary_module.Auxiliary.do_something - 2005-03-23 23:47:11,671 - spam_application - INFO - - calling auxiliary_module.some_function() - 2005-03-23 23:47:11,672 - spam_application.auxiliary - INFO - - received a call to "some_function" - 2005-03-23 23:47:11,673 - spam_application - INFO - - done with auxiliary_module.some_function() + FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' + logging.basicConfig(format=FORMAT) + d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} + logging.warning('Protocol problem: %s', 'connection reset', extra=d) + + would print something like:: + + 2006-02-08 22:20:02,165 192.168.0.1 fbloggs Protocol problem: connection reset + + The keys in the dictionary passed in *extra* should not clash with the keys used + by the logging system. (See the :class:`Formatter` documentation for more + information on which keys are used by the logging system.) + + If you choose to use these attributes in logged messages, you need to exercise + some care. In the above example, for instance, the :class:`Formatter` has been + set up with a format string which expects 'clientip' and 'user' in the attribute + dictionary of the LogRecord. If these are missing, the message will not be + logged because a string formatting exception will occur. So in this case, you + always need to pass the *extra* dictionary with these keys. + + While this might be annoying, this feature is intended for use in specialized + circumstances, such as multi-threaded servers where the same code executes in + many contexts, and interesting conditions which arise are dependent on this + context (such as remote client IP address and authenticated user name, in the + above example). In such circumstances, it is likely that specialized + :class:`Formatter`\ s would be used with particular :class:`Handler`\ s. + + .. versionadded:: 3.2 + The *stack_info* parameter was added. + +.. function:: info(msg, *args, **kwargs) + + Logs a message with level :const:`INFO` on the root logger. The arguments are + interpreted as for :func:`debug`. + + +.. function:: warning(msg, *args, **kwargs) + + Logs a message with level :const:`WARNING` on the root logger. The arguments are + interpreted as for :func:`debug`. + + +.. function:: error(msg, *args, **kwargs) + + Logs a message with level :const:`ERROR` on the root logger. The arguments are + interpreted as for :func:`debug`. + + +.. function:: critical(msg, *args, **kwargs) + + Logs a message with level :const:`CRITICAL` on the root logger. The arguments + are interpreted as for :func:`debug`. + + +.. function:: exception(msg, *args) + + Logs a message with level :const:`ERROR` on the root logger. The arguments are + interpreted as for :func:`debug`. Exception info is added to the logging + message. This function should only be called from an exception handler. + +.. function:: log(level, msg, *args, **kwargs) + + Logs a message with level *level* on the root logger. The other arguments are + interpreted as for :func:`debug`. + + PLEASE NOTE: The above module-level functions which delegate to the root + logger should *not* be used in threads, in versions of Python earlier than + 2.7.1 and 3.2, unless at least one handler has been added to the root + logger *before* the threads are started. These convenience functions call + :func:`basicConfig` to ensure that at least one handler is available; in + earlier versions of Python, this can (under rare circumstances) lead to + handlers being added multiple times to the root logger, which can in turn + lead to multiple messages for the same event. + +.. function:: disable(lvl) + + Provides an overriding level *lvl* for all loggers which takes precedence over + the logger's own level. When the need arises to temporarily throttle logging + output down across the whole application, this function can be useful. Its + effect is to disable all logging calls of severity *lvl* and below, so that + if you call it with a value of INFO, then all INFO and DEBUG events would be + discarded, whereas those of severity WARNING and above would be processed + according to the logger's effective level. + + +.. function:: addLevelName(lvl, levelName) + + Associates level *lvl* with text *levelName* in an internal dictionary, which is + used to map numeric levels to a textual representation, for example when a + :class:`Formatter` formats a message. This function can also be used to define + your own levels. The only constraints are that all levels used must be + registered using this function, levels should be positive integers and they + should increase in increasing order of severity. + + NOTE: If you are thinking of defining your own levels, please see the section + on :ref:`custom-levels`. + +.. function:: getLevelName(lvl) + + Returns the textual representation of logging level *lvl*. If the level is one + of the predefined levels :const:`CRITICAL`, :const:`ERROR`, :const:`WARNING`, + :const:`INFO` or :const:`DEBUG` then you get the corresponding string. If you + have associated levels with names using :func:`addLevelName` then the name you + have associated with *lvl* is returned. If a numeric value corresponding to one + of the defined levels is passed in, the corresponding string representation is + returned. Otherwise, the string 'Level %s' % lvl is returned. + + +.. function:: makeLogRecord(attrdict) + + Creates and returns a new :class:`LogRecord` instance whose attributes are + defined by *attrdict*. This function is useful for taking a pickled + :class:`LogRecord` attribute dictionary, sent over a socket, and reconstituting + it as a :class:`LogRecord` instance at the receiving end. + + +.. function:: basicConfig(**kwargs) + + Does basic configuration for the logging system by creating a + :class:`StreamHandler` with a default :class:`Formatter` and adding it to the + root logger. The functions :func:`debug`, :func:`info`, :func:`warning`, + :func:`error` and :func:`critical` will call :func:`basicConfig` automatically + if no handlers are defined for the root logger. + + This function does nothing if the root logger already has handlers + configured for it. + + PLEASE NOTE: This function should be called from the main thread + before other threads are started. In versions of Python prior to + 2.7.1 and 3.2, if this function is called from multiple threads, + it is possible (in rare circumstances) that a handler will be added + to the root logger more than once, leading to unexpected results + such as messages being duplicated in the log. + + The following keyword arguments are supported. + + +--------------+---------------------------------------------+ + | Format | Description | + +==============+=============================================+ + | ``filename`` | Specifies that a FileHandler be created, | + | | using the specified filename, rather than a | + | | StreamHandler. | + +--------------+---------------------------------------------+ + | ``filemode`` | Specifies the mode to open the file, if | + | | filename is specified (if filemode is | + | | unspecified, it defaults to 'a'). | + +--------------+---------------------------------------------+ + | ``format`` | Use the specified format string for the | + | | handler. | + +--------------+---------------------------------------------+ + | ``datefmt`` | Use the specified date/time format. | + +--------------+---------------------------------------------+ + | ``style`` | If ``format`` is specified, use this style | + | | for the format string. One of '%', '{' or | + | | '$' for %-formatting, :meth:`str.format` or | + | | :class:`string.Template` respectively, and | + | | defaulting to '%' if not specified. | + +--------------+---------------------------------------------+ + | ``level`` | Set the root logger level to the specified | + | | level. | + +--------------+---------------------------------------------+ + | ``stream`` | Use the specified stream to initialize the | + | | StreamHandler. Note that this argument is | + | | incompatible with 'filename' - if both are | + | | present, 'stream' is ignored. | + +--------------+---------------------------------------------+ + + .. versionchanged:: 3.2 + The ``style`` argument was added. + + +.. function:: shutdown() + + Informs the logging system to perform an orderly shutdown by flushing and + closing all handlers. This should be called at application exit and no + further use of the logging system should be made after this call. + + +.. function:: setLoggerClass(klass) + + Tells the logging system to use the class *klass* when instantiating a logger. + The class should define :meth:`__init__` such that only a name argument is + required, and the :meth:`__init__` should call :meth:`Logger.__init__`. This + function is typically called before any loggers are instantiated by applications + which need to use custom logger behavior. + + +.. function:: setLogRecordFactory(factory) + + Set a callable which is used to create a :class:`LogRecord`. + + :param factory: The factory callable to be used to instantiate a log record. + + .. versionadded:: 3.2 + This function has been provided, along with :func:`getLogRecordFactory`, to + allow developers more control over how the :class:`LogRecord` representing + a logging event is constructed. + + The factory has the following signature: + + ``factory(name, level, fn, lno, msg, args, exc_info, func=None, sinfo=None, **kwargs)`` + + :name: The logger name. + :level: The logging level (numeric). + :fn: The full pathname of the file where the logging call was made. + :lno: The line number in the file where the logging call was made. + :msg: The logging message. + :args: The arguments for the logging message. + :exc_info: An exception tuple, or None. + :func: The name of the function or method which invoked the logging + call. + :sinfo: A stack traceback such as is provided by + :func:`traceback.print_stack`, showing the call hierarchy. + :kwargs: Additional keyword arguments. + + +Integration with the warnings module +------------------------------------ + +The :func:`captureWarnings` function can be used to integrate :mod:`logging` +with the :mod:`warnings` module. + +.. function:: captureWarnings(capture) + + This function is used to turn the capture of warnings by logging on and + off. + + If *capture* is ``True``, warnings issued by the :mod:`warnings` module will + be redirected to the logging system. Specifically, a warning will be + formatted using :func:`warnings.formatwarning` and the resulting string + logged to a logger named 'py.warnings' with a severity of `WARNING`. + + If *capture* is ``False``, the redirection of warnings to the logging system + will stop, and warnings will be redirected to their original destinations + (i.e. those in effect before `captureWarnings(True)` was called). + + +.. seealso:: + + Module :mod:`logging.config` + Configuration API for the logging module. + + Module :mod:`logging.handlers` + Useful handlers included with the logging module. + + :pep:`282` - A Logging System + The proposal which described this feature for inclusion in the Python standard + library. + + `Original Python logging package <http://www.red-dove.com/python_logging.html>`_ + This is the original source for the :mod:`logging` package. The version of the + package available from this site is suitable for use with Python 1.5.2, 2.1.x + and 2.2.x, which do not include the :mod:`logging` package in the standard + library. diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst index 7409af5..ff8cfea 100644 --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -81,13 +81,16 @@ Maildir, mbox, MH, Babyl, and MMDF. it. Parameter *message* may be a :class:`Message` instance, an - :class:`email.Message.Message` instance, a string, or a file-like object - (which should be open in text mode). If *message* is an instance of the + :class:`email.Message.Message` instance, a string, a byte string, or a + file-like object (which should be open in binary mode). If *message* is + an instance of the appropriate format-specific :class:`Message` subclass (e.g., if it's an :class:`mboxMessage` instance and this is an :class:`mbox` instance), its format-specific information is used. Otherwise, reasonable defaults for format-specific information are used. + .. versionchanged:: 3.2 support for binary input + .. method:: remove(key) __delitem__(key) @@ -108,8 +111,9 @@ Maildir, mbox, MH, Babyl, and MMDF. :exc:`KeyError` exception if no message already corresponds to *key*. As with :meth:`add`, parameter *message* may be a :class:`Message` - instance, an :class:`email.Message.Message` instance, a string, or a - file-like object (which should be open in text mode). If *message* is an + instance, an :class:`email.Message.Message` instance, a string, a byte + string, or a file-like object (which should be open in binary mode). If + *message* is an instance of the appropriate format-specific :class:`Message` subclass (e.g., if it's an :class:`mboxMessage` instance and this is an :class:`mbox` instance), its format-specific information is @@ -171,24 +175,40 @@ Maildir, mbox, MH, Babyl, and MMDF. raise a :exc:`KeyError` exception if no such message exists. + .. method:: get_bytes(key) + + Return a byte representation of the message corresponding to *key*, or + raise a :exc:`KeyError` exception if no such message exists. + + .. versionadded:: 3.2 + + .. method:: get_string(key) Return a string representation of the message corresponding to *key*, or - raise a :exc:`KeyError` exception if no such message exists. + raise a :exc:`KeyError` exception if no such message exists. The + message is processed through :class:`email.message.Message` to + convert it to a 7bit clean representation. .. method:: get_file(key) Return a file-like representation of the message corresponding to *key*, - or raise a :exc:`KeyError` exception if no such message exists. The - file-like object behaves as if open in binary mode. This file should be + or raise a :exc:`KeyError` exception if no such message exists. The + file-like object behaves as if open in binary mode. This file should be closed once it is no longer needed. + .. versionchanged:: 3.2 + The file object really is a binary file; previously it was incorrectly + returned in text mode. Also, the file-like object now supports the + context manager protocol: you can use a :keyword:`with` statement to + automatically close it. + .. note:: Unlike other representations of messages, file-like representations are not necessarily independent of the :class:`Mailbox` instance that - created them or of the underlying mailbox. More specific documentation + created them or of the underlying mailbox. More specific documentation is provided by each subclass. @@ -452,7 +472,7 @@ Maildir, mbox, MH, Babyl, and MMDF. unlock() Three locking mechanisms are used---dot locking and, if available, the - :cfunc:`flock` and :cfunc:`lockf` system calls. + :c:func:`flock` and :c:func:`lockf` system calls. .. seealso:: @@ -566,7 +586,7 @@ Maildir, mbox, MH, Babyl, and MMDF. unlock() Three locking mechanisms are used---dot locking and, if available, the - :cfunc:`flock` and :cfunc:`lockf` system calls. For MH mailboxes, locking + :c:func:`flock` and :c:func:`lockf` system calls. For MH mailboxes, locking the mailbox means locking the :file:`.mh_sequences` file and, only for the duration of any operations that affect them, locking individual message files. @@ -664,7 +684,7 @@ Maildir, mbox, MH, Babyl, and MMDF. unlock() Three locking mechanisms are used---dot locking and, if available, the - :cfunc:`flock` and :cfunc:`lockf` system calls. + :c:func:`flock` and :c:func:`lockf` system calls. .. seealso:: @@ -715,7 +735,7 @@ Maildir, mbox, MH, Babyl, and MMDF. unlock() Three locking mechanisms are used---dot locking and, if available, the - :cfunc:`flock` and :cfunc:`lockf` system calls. + :c:func:`flock` and :c:func:`lockf` system calls. .. seealso:: @@ -742,9 +762,11 @@ Maildir, mbox, MH, Babyl, and MMDF. If *message* is omitted, the new instance is created in a default, empty state. If *message* is an :class:`email.Message.Message` instance, its contents are copied; furthermore, any format-specific information is converted insofar as - possible if *message* is a :class:`Message` instance. If *message* is a string + possible if *message* is a :class:`Message` instance. If *message* is a string, + a byte string, or a file, it should contain an :rfc:`2822`\ -compliant message, which is read - and parsed. + and parsed. Files should be open in binary mode, but text mode files + are accepted for backward compatibility. The format-specific state and behaviors offered by subclasses vary, but in general it is only the properties that are not specific to a particular diff --git a/Doc/library/mailcap.rst b/Doc/library/mailcap.rst index 0a0a790..4bb31bf 100644 --- a/Doc/library/mailcap.rst +++ b/Doc/library/mailcap.rst @@ -4,6 +4,9 @@ .. module:: mailcap :synopsis: Mailcap file handling. +**Source code:** :source:`Lib/mailcap.py` + +-------------- Mailcap files are used to configure how MIME-aware applications such as mail readers and Web browsers react to files with different MIME types. (The name diff --git a/Doc/library/markup.rst b/Doc/library/markup.rst index ae97b69..49794ef 100644 --- a/Doc/library/markup.rst +++ b/Doc/library/markup.rst @@ -20,6 +20,7 @@ definition of the Python bindings for the DOM and SAX interfaces. .. toctree:: + html.rst html.parser.rst html.entities.rst pyexpat.rst diff --git a/Doc/library/math.rst b/Doc/library/math.rst index c760701..98c5b33 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -97,15 +97,23 @@ Number-theoretic and representation functions <http://code.activestate.com/recipes/393090/>`_\. +.. function:: isfinite(x) + + Return ``True`` if *x* is neither an infinity nor a NaN, and + ``False`` otherwise. (Note that ``0.0`` *is* considered finite.) + + .. versionadded:: 3.2 + + .. function:: isinf(x) - Check if the float *x* is positive or negative infinity. + Return ``True`` if *x* is a positive or negative infinity, and + ``False`` otherwise. .. function:: isnan(x) - Check if the float *x* is a NaN (not a number). For more information - on NaNs, see the IEEE 754 standards. + Return ``True`` if *x* is a NaN (not a number), and ``False`` otherwise. .. function:: ldexp(x, i) @@ -146,6 +154,22 @@ Power and logarithmic functions Return ``e**x``. +.. function:: expm1(x) + + Return ``e**x - 1``. For small floats *x*, the subtraction in ``exp(x) - 1`` + can result in a `significant loss of precision + <http://en.wikipedia.org/wiki/Loss_of_significance>`_\; the :func:`expm1` + function provides a way to compute this quantity to full precision:: + + >>> from math import exp, expm1 + >>> exp(1e-5) - 1 # gives result accurate to 11 places + 1.0000050000069649e-05 + >>> expm1(1e-5) # result accurate to full precision + 1.0000050000166668e-05 + + .. versionadded:: 3.2 + + .. function:: log(x[, base]) With one argument, return the natural logarithm of *x* (to base *e*). @@ -245,6 +269,9 @@ Angular conversion Hyperbolic functions -------------------- +`Hyperbolic functions <http://en.wikipedia.org/wiki/Hyperbolic_function>`_ +are analogs of trigonometric functions that are based on hyperbolas +instead of circles. .. function:: acosh(x) @@ -276,6 +303,52 @@ Hyperbolic functions Return the hyperbolic tangent of *x*. +Special functions +----------------- + +.. function:: erf(x) + + Return the `error function <http://en.wikipedia.org/wiki/Error_function>`_ at + *x*. + + The :func:`erf` function can be used to compute traditional statistical + functions such as the `cumulative standard normal distribution + <http://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution_function>`_:: + + def phi(x): + 'Cumulative distribution function for the standard normal distribution' + return (1.0 + erf(x / sqrt(2.0))) / 2.0 + + .. versionadded:: 3.2 + + +.. function:: erfc(x) + + Return the complementary error function at *x*. The `complementary error + function <http://en.wikipedia.org/wiki/Error_function>`_ is defined as + ``1.0 - erf(x)``. It is used for large values of *x* where a subtraction + from one would cause a `loss of significance + <http://en.wikipedia.org/wiki/Loss_of_significance>`_\. + + .. versionadded:: 3.2 + + +.. function:: gamma(x) + + Return the `Gamma function <http://en.wikipedia.org/wiki/Gamma_function>`_ at + *x*. + + .. versionadded:: 3.2 + + +.. function:: lgamma(x) + + Return the natural logarithm of the absolute value of the Gamma + function at *x*. + + .. versionadded:: 3.2 + + Constants --------- diff --git a/Doc/library/mimetypes.rst b/Doc/library/mimetypes.rst index fe1437a..17b0c07 100644 --- a/Doc/library/mimetypes.rst +++ b/Doc/library/mimetypes.rst @@ -8,6 +8,10 @@ .. index:: pair: MIME; content type +**Source code:** :source:`Lib/mimetypes.py` + +-------------- + The :mod:`mimetypes` module converts between a filename or URL and the MIME type associated with the filename extension. Conversions are provided from filename to MIME type and from MIME type to filename extension; encodings are not @@ -76,9 +80,13 @@ behavior of the module. Initialize the internal data structures. If given, *files* must be a sequence of file names which should be used to augment the default type map. If omitted, - the file names to use are taken from :const:`knownfiles`. Each file named in - *files* or :const:`knownfiles` takes precedence over those named before it. - Calling :func:`init` repeatedly is allowed. + the file names to use are taken from :const:`knownfiles`; on Windows, the + current registry settings are loaded. Each file named in *files* or + :const:`knownfiles` takes precedence over those named before it. Calling + :func:`init` repeatedly is allowed. + + .. versionchanged:: 3.2 + Previously, Windows registry settings were ignored. .. function:: read_mime_types(filename) @@ -228,3 +236,9 @@ MimeTypes Objects Load MIME type information from an open file. The file must have the format of the standard :file:`mime.types` files. + +.. method:: MimeTypes.read_windows_registry() + + Load MIME type information from the Windows registry. Availability: Windows. + + .. versionadded:: 3.2 diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index 5001d4f..128bc90 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -111,6 +111,18 @@ To map anonymous memory, -1 should be passed as the fileno along with the length map.close() + :class:`mmap` can also be used as a context manager in a :keyword:`with` + statement.:: + + import mmap + + with mmap.mmap(-1, 13) as map: + map.write("Hello world!") + + .. versionadded:: 3.2 + Context manager support. + + The next example demonstrates how to create an anonymous map and exchange data between the parent and child processes:: @@ -131,13 +143,19 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Memory-mapped file objects support the following methods: - .. method:: close() Close the file. Subsequent calls to other methods of the object will result in an exception being raised. + .. attribute:: closed + + True if the file is closed. + + .. versionadded:: 3.2 + + .. method:: find(sub[, start[, end]]) Returns the lowest index in the object where the subsequence *sub* is diff --git a/Doc/library/modulefinder.rst b/Doc/library/modulefinder.rst index 41b387b..d42c6ab 100644 --- a/Doc/library/modulefinder.rst +++ b/Doc/library/modulefinder.rst @@ -7,6 +7,9 @@ .. module:: modulefinder :synopsis: Find modules used by a script. +**Source code:** :source:`Lib/modulefinder.py` + +-------------- This module provides a :class:`ModuleFinder` class that can be used to determine the set of modules imported by a script. ``modulefinder.py`` can also be run as diff --git a/Doc/library/msilib.rst b/Doc/library/msilib.rst index f27c3d8..270f4ff 100644 --- a/Doc/library/msilib.rst +++ b/Doc/library/msilib.rst @@ -42,7 +42,7 @@ structures. .. function:: UuidCreate() Return the string representation of a new unique identifier. This wraps the - Windows API functions :cfunc:`UuidCreate` and :cfunc:`UuidToString`. + Windows API functions :c:func:`UuidCreate` and :c:func:`UuidToString`. .. function:: OpenDatabase(path, persist) @@ -58,7 +58,7 @@ structures. .. function:: CreateRecord(count) - Return a new record object by calling :cfunc:`MSICreateRecord`. *count* is the + Return a new record object by calling :c:func:`MSICreateRecord`. *count* is the number of fields of the record. @@ -133,20 +133,20 @@ Database Objects .. method:: Database.OpenView(sql) - Return a view object, by calling :cfunc:`MSIDatabaseOpenView`. *sql* is the SQL + Return a view object, by calling :c:func:`MSIDatabaseOpenView`. *sql* is the SQL statement to execute. .. method:: Database.Commit() Commit the changes pending in the current transaction, by calling - :cfunc:`MSIDatabaseCommit`. + :c:func:`MSIDatabaseCommit`. .. method:: Database.GetSummaryInformation(count) Return a new summary information object, by calling - :cfunc:`MsiGetSummaryInformation`. *count* is the maximum number of updated + :c:func:`MsiGetSummaryInformation`. *count* is the maximum number of updated values. @@ -164,7 +164,7 @@ View Objects .. method:: View.Execute(params) - Execute the SQL query of the view, through :cfunc:`MSIViewExecute`. If + Execute the SQL query of the view, through :c:func:`MSIViewExecute`. If *params* is not ``None``, it is a record describing actual values of the parameter tokens in the query. @@ -172,18 +172,18 @@ View Objects .. method:: View.GetColumnInfo(kind) Return a record describing the columns of the view, through calling - :cfunc:`MsiViewGetColumnInfo`. *kind* can be either ``MSICOLINFO_NAMES`` or + :c:func:`MsiViewGetColumnInfo`. *kind* can be either ``MSICOLINFO_NAMES`` or ``MSICOLINFO_TYPES``. .. method:: View.Fetch() - Return a result record of the query, through calling :cfunc:`MsiViewFetch`. + Return a result record of the query, through calling :c:func:`MsiViewFetch`. .. method:: View.Modify(kind, data) - Modify the view, by calling :cfunc:`MsiViewModify`. *kind* can be one of + Modify the view, by calling :c:func:`MsiViewModify`. *kind* can be one of ``MSIMODIFY_SEEK``, ``MSIMODIFY_REFRESH``, ``MSIMODIFY_INSERT``, ``MSIMODIFY_UPDATE``, ``MSIMODIFY_ASSIGN``, ``MSIMODIFY_REPLACE``, ``MSIMODIFY_MERGE``, ``MSIMODIFY_DELETE``, ``MSIMODIFY_INSERT_TEMPORARY``, @@ -195,7 +195,7 @@ View Objects .. method:: View.Close() - Close the view, through :cfunc:`MsiViewClose`. + Close the view, through :c:func:`MsiViewClose`. .. seealso:: @@ -214,7 +214,7 @@ Summary Information Objects .. method:: SummaryInformation.GetProperty(field) - Return a property of the summary, through :cfunc:`MsiSummaryInfoGetProperty`. + Return a property of the summary, through :c:func:`MsiSummaryInfoGetProperty`. *field* is the name of the property, and can be one of the constants ``PID_CODEPAGE``, ``PID_TITLE``, ``PID_SUBJECT``, ``PID_AUTHOR``, ``PID_KEYWORDS``, ``PID_COMMENTS``, ``PID_TEMPLATE``, ``PID_LASTAUTHOR``, @@ -226,12 +226,12 @@ Summary Information Objects .. method:: SummaryInformation.GetPropertyCount() Return the number of summary properties, through - :cfunc:`MsiSummaryInfoGetPropertyCount`. + :c:func:`MsiSummaryInfoGetPropertyCount`. .. method:: SummaryInformation.SetProperty(field, value) - Set a property through :cfunc:`MsiSummaryInfoSetProperty`. *field* can have the + Set a property through :c:func:`MsiSummaryInfoSetProperty`. *field* can have the same values as in :meth:`GetProperty`, *value* is the new value of the property. Possible value types are integer and string. @@ -239,7 +239,7 @@ Summary Information Objects .. method:: SummaryInformation.Persist() Write the modified properties to the summary information stream, using - :cfunc:`MsiSummaryInfoPersist`. + :c:func:`MsiSummaryInfoPersist`. .. seealso:: @@ -258,7 +258,7 @@ Record Objects .. method:: Record.GetFieldCount() Return the number of fields of the record, through - :cfunc:`MsiRecordGetFieldCount`. + :c:func:`MsiRecordGetFieldCount`. .. method:: Record.GetInteger(field) @@ -275,25 +275,25 @@ Record Objects .. method:: Record.SetString(field, value) - Set *field* to *value* through :cfunc:`MsiRecordSetString`. *field* must be an + Set *field* to *value* through :c:func:`MsiRecordSetString`. *field* must be an integer; *value* a string. .. method:: Record.SetStream(field, value) Set *field* to the contents of the file named *value*, through - :cfunc:`MsiRecordSetStream`. *field* must be an integer; *value* a string. + :c:func:`MsiRecordSetStream`. *field* must be an integer; *value* a string. .. method:: Record.SetInteger(field, value) - Set *field* to *value* through :cfunc:`MsiRecordSetInteger`. Both *field* and + Set *field* to *value* through :c:func:`MsiRecordSetInteger`. Both *field* and *value* must be an integer. .. method:: Record.ClearData() - Set all fields of the record to 0, through :cfunc:`MsiRecordClearData`. + Set all fields of the record to 0, through :c:func:`MsiRecordClearData`. .. seealso:: diff --git a/Doc/library/msvcrt.rst b/Doc/library/msvcrt.rst index d968833..889a0c5 100644 --- a/Doc/library/msvcrt.rst +++ b/Doc/library/msvcrt.rst @@ -90,12 +90,12 @@ Console I/O .. function:: getch() - Read a keypress and return the resulting character. Nothing is echoed to the - console. This call will block if a keypress is not already available, but will - not wait for :kbd:`Enter` to be pressed. If the pressed key was a special - function key, this will return ``'\000'`` or ``'\xe0'``; the next call will - return the keycode. The :kbd:`Control-C` keypress cannot be read with this - function. + Read a keypress and return the resulting character as a byte string. + Nothing is echoed to the console. This call will block if a keypress + is not already available, but will not wait for :kbd:`Enter` to be + pressed. If the pressed key was a special function key, this will + return ``'\000'`` or ``'\xe0'``; the next call will return the keycode. + The :kbd:`Control-C` keypress cannot be read with this function. .. function:: getwch() @@ -116,7 +116,7 @@ Console I/O .. function:: putch(char) - Print the character *char* to the console without buffering. + Print the byte string *char* to the console without buffering. .. function:: putwch(unicode_char) @@ -126,8 +126,8 @@ Console I/O .. function:: ungetch(char) - Cause the character *char* to be "pushed back" into the console buffer; it will - be the next character read by :func:`getch` or :func:`getche`. + Cause the byte string *char* to be "pushed back" into the console buffer; + it will be the next character read by :func:`getch` or :func:`getche`. .. function:: ungetwch(unicode_char) @@ -143,5 +143,5 @@ Other Functions .. function:: heapmin() - Force the :cfunc:`malloc` heap to clean itself up and return unused blocks to + Force the :c:func:`malloc` heap to clean itself up and return unused blocks to the operating system. On failure, this raises :exc:`IOError`. diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index cb20a5c..92d5272 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -16,7 +16,7 @@ to this, the :mod:`multiprocessing` module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows. -.. warning:: +.. note:: Some of this package's functionality requires a functioning shared semaphore implementation on the host operating system. Without one, the @@ -120,7 +120,9 @@ processes: print(q.get()) # prints "[42, None, 'hello']" p.join() - Queues are thread and process safe. + Queues are thread and process safe, but note that they must never + be instantiated as a side effect of importing a module: this can lead + to a deadlock! (see :ref:`threaded-imports`) **Pipes** @@ -406,7 +408,7 @@ The :mod:`multiprocessing` package mostly replicates the API of the .. method:: terminate() Terminate the process. On Unix this is done using the ``SIGTERM`` signal; - on Windows :cfunc:`TerminateProcess` is used. Note that exit handlers and + on Windows :c:func:`TerminateProcess` is used. Note that exit handlers and finally clauses, etc., will not be executed. Note that descendant processes of the process will *not* be terminated -- @@ -420,9 +422,9 @@ The :mod:`multiprocessing` package mostly replicates the API of the acquired a lock or semaphore etc. then terminating it is liable to cause other processes to deadlock. - Note that the :meth:`start`, :meth:`join`, :meth:`is_alive` and - :attr:`exit_code` methods should only be called by the process that created - the process object. + Note that the :meth:`start`, :meth:`join`, :meth:`is_alive`, + :meth:`terminate` and :attr:`exit_code` methods should only be called by + the process that created the process object. Example usage of some of the methods of :class:`Process`: @@ -1130,7 +1132,7 @@ their parent process exits. The manager classes are defined in the Create a BaseManager object. - Once created one should call :meth:`start` or :meth:`serve_forever` to ensure + Once created one should call :meth:`start` or ``get_server().serve_forever()`` to ensure that the manager object refers to a started manager process. *address* is the address on which the manager process listens for new @@ -1146,10 +1148,6 @@ their parent process exits. The manager classes are defined in the Start a subprocess to start the manager. If *initializer* is not ``None`` then the subprocess will call ``initializer(*initargs)`` when it starts. - .. method:: serve_forever() - - Run the server in the current process. - .. method:: get_server() Returns a :class:`Server` object which represents the actual server under @@ -1552,7 +1550,7 @@ Process Pools One can create a pool of processes which will carry out tasks submitted to it with the :class:`Pool` class. -.. class:: multiprocessing.Pool([processes[, initializer[, initargs]]]) +.. class:: multiprocessing.Pool([processes[, initializer[, initargs[, maxtasksperchild]]]]) A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and @@ -1563,6 +1561,22 @@ with the :class:`Pool` class. *initializer* is not ``None`` then each worker process will call ``initializer(*initargs)`` when it starts. + .. versionadded:: 3.2 + *maxtasksperchild* is the number of tasks a worker process can complete + before it will exit and be replaced with a fresh worker process, to enable + unused resources to be freed. The default *maxtasksperchild* is None, which + means worker processes will live as long as the pool. + + .. note:: + + Worker processes within a :class:`Pool` typically live for the complete + duration of the Pool's work queue. A frequent pattern found in other + systems (such as Apache, mod_wsgi, etc) to free resources held by + workers is to allow a worker within a pool to complete only a set + amount of work before being exiting, being cleaned up and a new + process spawned to replace the old one. The *maxtasksperchild* + argument to the :class:`Pool` exposes this ability to the end user. + .. method:: apply(func[, args[, kwds]]) Call *func* with arguments *args* and keyword arguments *kwds*. It blocks @@ -1570,14 +1584,21 @@ with the :class:`Pool` class. suited for performing work in parallel. Additionally, the passed in function is only executed in one of the workers of the pool. - .. method:: apply_async(func[, args[, kwds[, callback]]]) + .. method:: apply_async(func[, args[, kwds[, callback[, error_callback]]]]) A variant of the :meth:`apply` method which returns a result object. If *callback* is specified then it should be a callable which accepts a single argument. When the result becomes ready *callback* is applied to - it (unless the call failed). *callback* should complete immediately since - otherwise the thread which handles the results will get blocked. + it, that is unless the call failed, in which case the *error_callback* + is applied instead + + If *error_callback* is specified then it should be a callable which + accepts a single argument. If the target function fails, then + the *error_callback* is called with the exception instance. + + Callbacks should complete immediately since otherwise the thread which + handles the results will get blocked. .. method:: map(func, iterable[, chunksize]) @@ -1594,8 +1615,15 @@ with the :class:`Pool` class. If *callback* is specified then it should be a callable which accepts a single argument. When the result becomes ready *callback* is applied to - it (unless the call failed). *callback* should complete immediately since - otherwise the thread which handles the results will get blocked. + it, that is unless the call failed, in which case the *error_callback* + is applied instead + + If *error_callback* is specified then it should be a callable which + accepts a single argument. If the target function fails, then + the *error_callback* is called with the exception instance. + + Callbacks should complete immediately since otherwise the thread which + handles the results will get blocked. .. method:: imap(func, iterable[, chunksize]) diff --git a/Doc/library/netrc.rst b/Doc/library/netrc.rst index 91990df..3f38cbc 100644 --- a/Doc/library/netrc.rst +++ b/Doc/library/netrc.rst @@ -7,6 +7,9 @@ .. moduleauthor:: Eric S. Raymond <esr@snark.thyrsus.com> .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> +**Source code:** :source:`Lib/netrc.py` + +-------------- The :class:`netrc` class parses and encapsulates the netrc file format used by the Unix :program:`ftp` program and other FTP clients. diff --git a/Doc/library/nntplib.rst b/Doc/library/nntplib.rst index c3cbd2b..164f149 100644 --- a/Doc/library/nntplib.rst +++ b/Doc/library/nntplib.rst @@ -10,101 +10,122 @@ pair: NNTP; protocol single: Network News Transfer Protocol +**Source code:** :source:`Lib/nntplib.py` + +-------------- + This module defines the class :class:`NNTP` which implements the client side of -the NNTP protocol. It can be used to implement a news reader or poster, or -automated news processors. For more information on NNTP (Network News Transfer -Protocol), see Internet :rfc:`977`. +the Network News Transfer Protocol. It can be used to implement a news reader +or poster, or automated news processors. It is compatible with :rfc:`3977` +as well as the older :rfc:`977` and :rfc:`2980`. Here are two small examples of how it can be used. To list some statistics about a newsgroup and print the subjects of the last 10 articles:: - >>> s = NNTP('news.gmane.org') + >>> s = nntplib.NNTP('news.gmane.org') >>> resp, count, first, last, name = s.group('gmane.comp.python.committers') >>> print('Group', name, 'has', count, 'articles, range', first, 'to', last) - Group gmane.comp.python.committers has 1071 articles, range 1 to 1071 - >>> resp, subs = s.xhdr('subject', first + '-' + last) - >>> for id, sub in subs[-10:]: print(id, sub) + Group gmane.comp.python.committers has 1096 articles, range 1 to 1096 + >>> resp, overviews = s.over((last - 9, last)) + >>> for id, over in overviews: + ... print(id, nntplib.decode_header(over['subject'])) ... - 1062 Re: Mercurial Status? - 1063 Re: [python-committers] (Windows) buildbots on 3.x - 1064 Re: Mercurial Status? - 1065 Re: Mercurial Status? - 1066 Python 2.6.6 status - 1067 Commit Privileges for Ask Solem - 1068 Re: Commit Privileges for Ask Solem - 1069 Re: Commit Privileges for Ask Solem - 1070 Re: Commit Privileges for Ask Solem - 1071 2.6.6 rc 2 + 1087 Re: Commit privileges for Łukasz Langa + 1088 Re: 3.2 alpha 2 freeze + 1089 Re: 3.2 alpha 2 freeze + 1090 Re: Commit privileges for Łukasz Langa + 1091 Re: Commit privileges for Łukasz Langa + 1092 Updated ssh key + 1093 Re: Updated ssh key + 1094 Re: Updated ssh key + 1095 Hello fellow committers! + 1096 Re: Hello fellow committers! >>> s.quit() '205 Bye!' -To post an article from a file (this assumes that the article has valid +To post an article from a binary file (this assumes that the article has valid headers, and that you have right to post on the particular newsgroup):: - >>> s = NNTP('news.gmane.org') - >>> f = open('/tmp/article') + >>> s = nntplib.NNTP('news.gmane.org') + >>> f = open('/tmp/article.txt', 'rb') >>> s.post(f) '240 Article posted successfully.' >>> s.quit() '205 Bye!' -The module itself defines the following items: +The module itself defines the following classes: -.. class:: NNTP(host[, port [, user[, password [, readermode][, usenetrc]]]]) +.. class:: NNTP(host, port=119, user=None, password=None, readermode=None, usenetrc=False, [timeout]) - Return a new instance of the :class:`NNTP` class, representing a connection - to the NNTP server running on host *host*, listening at port *port*. The - default *port* is 119. If the optional *user* and *password* are provided, - or if suitable credentials are present in :file:`/.netrc` and the optional - flag *usenetrc* is true (the default), the ``AUTHINFO USER`` and ``AUTHINFO - PASS`` commands are used to identify and authenticate the user to the server. - If the optional flag *readermode* is true, then a ``mode reader`` command is - sent before authentication is performed. Reader mode is sometimes necessary - if you are connecting to an NNTP server on the local machine and intend to - call reader-specific commands, such as ``group``. If you get unexpected + Return a new :class:`NNTP` object, representing a connection + to the NNTP server running on host *host*, listening at port *port*. + An optional *timeout* can be specified for the socket connection. + If the optional *user* and *password* are provided, or if suitable + credentials are present in :file:`/.netrc` and the optional flag *usenetrc* + is true, the ``AUTHINFO USER`` and ``AUTHINFO PASS`` commands are used + to identify and authenticate the user to the server. If the optional + flag *readermode* is true, then a ``mode reader`` command is sent before + authentication is performed. Reader mode is sometimes necessary if you are + connecting to an NNTP server on the local machine and intend to call + reader-specific commands, such as ``group``. If you get unexpected :exc:`NNTPPermanentError`\ s, you might need to set *readermode*. - *readermode* defaults to ``None``. *usenetrc* defaults to ``True``. + + .. versionchanged:: 3.2 + *usenetrc* is now False by default. + + +.. class:: NNTP_SSL(host, port=563, user=None, password=None, ssl_context=None, readermode=None, usenetrc=False, [timeout]) + + Return a new :class:`NNTP_SSL` object, representing an encrypted + connection to the NNTP server running on host *host*, listening at + port *port*. :class:`NNTP_SSL` objects have the same methods as + :class:`NNTP` objects. If *port* is omitted, port 563 (NNTPS) is used. + *ssl_context* is also optional, and is a :class:`~ssl.SSLContext` object. + All other parameters behave the same as for :class:`NNTP`. + + Note that SSL-on-563 is discouraged per :rfc:`4642`, in favor of + STARTTLS as described below. However, some servers only support the + former. + + .. versionadded:: 3.2 .. exception:: NNTPError - Derived from the standard exception :exc:`Exception`, this is the base class for - all exceptions raised by the :mod:`nntplib` module. + Derived from the standard exception :exc:`Exception`, this is the base + class for all exceptions raised by the :mod:`nntplib` module. Instances + of this class have the following attribute: + + .. attribute:: response + + The response of the server if available, as a :class:`str` object. .. exception:: NNTPReplyError - Exception raised when an unexpected reply is received from the server. For - backwards compatibility, the exception ``error_reply`` is equivalent to this - class. + Exception raised when an unexpected reply is received from the server. .. exception:: NNTPTemporaryError - Exception raised when an error code in the range 400--499 is received. For - backwards compatibility, the exception ``error_temp`` is equivalent to this - class. + Exception raised when a response code in the range 400--499 is received. .. exception:: NNTPPermanentError - Exception raised when an error code in the range 500--599 is received. For - backwards compatibility, the exception ``error_perm`` is equivalent to this - class. + Exception raised when a response code in the range 500--599 is received. .. exception:: NNTPProtocolError Exception raised when a reply is received from the server that does not begin - with a digit in the range 1--5. For backwards compatibility, the exception - ``error_proto`` is equivalent to this class. + with a digit in the range 1--5. .. exception:: NNTPDataError - Exception raised when there is some error in the response data. For backwards - compatibility, the exception ``error_data`` is equivalent to this class. + Exception raised when there is some error in the response data. .. _nntp-objects: @@ -112,10 +133,51 @@ The module itself defines the following items: NNTP Objects ------------ -NNTP instances have the following methods. The *response* that is returned as -the first item in the return tuple of almost all methods is the server's -response: a string beginning with a three-digit code. If the server's response -indicates an error, the method raises one of the above exceptions. +When connected, :class:`NNTP` and :class:`NNTP_SSL` objects support the +following methods and attributes. + +Attributes +^^^^^^^^^^ + +.. attribute:: NNTP.nntp_version + + An integer representing the version of the NNTP protocol supported by the + server. In practice, this should be ``2`` for servers advertising + :rfc:`3977` compliance and ``1`` for others. + + .. versionadded:: 3.2 + +.. attribute:: NNTP.nntp_implementation + + A string describing the software name and version of the NNTP server, + or :const:`None` if not advertised by the server. + + .. versionadded:: 3.2 + +Methods +^^^^^^^ + +The *response* that is returned as the first item in the return tuple of almost +all methods is the server's response: a string beginning with a three-digit +code. If the server's response indicates an error, the method raises one of +the above exceptions. + +Many of the following methods take an optional keyword-only argument *file*. +When the *file* argument is supplied, it must be either a :term:`file object` +opened for binary writing, or the name of an on-disk file to be written to. +The method will then write any data returned by the server (except for the +response line and the terminating dot) to the file; any list of lines, +tuples or objects that the method normally returns will be empty. + +.. versionchanged:: 3.2 + Many of the following methods have been reworked and fixed, which makes + them incompatible with their 3.1 counterparts. + + +.. method:: NNTP.quit() + + Send a ``QUIT`` command and close the connection. Once this method has been + called, no other methods of the NNTP object should be called. .. method:: NNTP.getwelcome() @@ -125,62 +187,114 @@ indicates an error, the method raises one of the above exceptions. that may be relevant to the user.) -.. method:: NNTP.set_debuglevel(level) +.. method:: NNTP.getcapabilities() - Set the instance's debugging level. This controls the amount of debugging - output printed. The default, ``0``, produces no debugging output. A value of - ``1`` produces a moderate amount of debugging output, generally a single line - per request or response. A value of ``2`` or higher produces the maximum amount - of debugging output, logging each line sent and received on the connection - (including message text). + Return the :rfc:`3977` capabilities advertised by the server, as a + :class:`dict` instance mapping capability names to (possibly empty) lists + of values. On legacy servers which don't understand the ``CAPABILITIES`` + command, an empty dictionary is returned instead. + + >>> s = NNTP('news.gmane.org') + >>> 'POST' in s.getcapabilities() + True + + .. versionadded:: 3.2 + + +.. method:: NNTP.login(user=None, password=None, usenetrc=True) + + Send ``AUTHINFO`` commands with the user name and password. If *user* + and *password* are None and *usenetrc* is True, credentials from + ``~/.netrc`` will be used if possible. + Unless intentionally delayed, login is normally performed during the + :class:`NNTP` object initialization and separately calling this function + is unnecessary. To force authentication to be delayed, you must not set + *user* or *password* when creating the object, and must set *usenetrc* to + False. -.. method:: NNTP.newgroups(date, time, [file]) + .. versionadded:: 3.2 - Send a ``NEWGROUPS`` command. The *date* argument should be a string of the - form ``'yymmdd'`` indicating the date, and *time* should be a string of the form - ``'hhmmss'`` indicating the time. Return a pair ``(response, groups)`` where - *groups* is a list of group names that are new since the given date and time. If - the *file* parameter is supplied, then the output of the ``NEWGROUPS`` command - is stored in a file. If *file* is a string, then the method will open a file - object with that name, write to it then close it. If *file* is a :term:`file - object`, then it will start calling :meth:`write` on it to store the lines of - the command output. If *file* is supplied, then the returned *list* is an empty list. +.. method:: NNTP.starttls(ssl_context=None) -.. method:: NNTP.newnews(group, date, time, [file]) + Send a ``STARTTLS`` command. The *ssl_context* argument is optional + and should be a :class:`ssl.SSLContext` object. This will enable + encryption on the NNTP connection. + + Note that this may not be done after authentication information has + been transmitted, and authentication occurs by default if possible during a + :class:`NNTP` object initialization. See :meth:`NNTP.login` for information + on suppressing this behavior. + + .. versionadded:: 3.2 + + +.. method:: NNTP.newgroups(date, *, file=None) + + Send a ``NEWGROUPS`` command. The *date* argument should be a + :class:`datetime.date` or :class:`datetime.datetime` object. + Return a pair ``(response, groups)`` where *groups* is a list representing + the groups that are new since the given *date*. If *file* is supplied, + though, then *groups* will be empty. + + >>> from datetime import date, timedelta + >>> resp, groups = s.newgroups(date.today() - timedelta(days=3)) + >>> len(groups) + 85 + >>> groups[0] + GroupInfo(group='gmane.network.tor.devel', last='4', first='1', flag='m') + + +.. method:: NNTP.newnews(group, date, *, file=None) Send a ``NEWNEWS`` command. Here, *group* is a group name or ``'*'``, and - *date* and *time* have the same meaning as for :meth:`newgroups`. Return a pair - ``(response, articles)`` where *articles* is a list of message ids. If the - *file* parameter is supplied, then the output of the ``NEWNEWS`` command is - stored in a file. If *file* is a string, then the method will open a file - object with that name, write to it then close it. If *file* is a :term:`file - object`, then it will start calling :meth:`write` on it to store the lines of the - command output. If *file* is supplied, then the returned *list* is an empty list. - - -.. method:: NNTP.list([file]) - - Send a ``LIST`` command. Return a pair ``(response, list)`` where *list* is a - list of tuples. Each tuple has the form ``(group, last, first, flag)``, where - *group* is a group name, *last* and *first* are the last and first article - numbers (as strings), and *flag* is ``'y'`` if posting is allowed, ``'n'`` if - not, and ``'m'`` if the newsgroup is moderated. (Note the ordering: *last*, - *first*.) If the *file* parameter is supplied, then the output of the ``LIST`` - command is stored in a file. If *file* is a string, then the method will open - a file with that name, write to it then close it. If *file* is a :term:`file - object`, then it will start calling :meth:`write` on it to store the lines of - the command output. If *file* is supplied, then the returned *list* is an empty - list. + *date* has the same meaning as for :meth:`newgroups`. Return a pair + ``(response, articles)`` where *articles* is a list of message ids. + + This command is frequently disabled by NNTP server administrators. + + +.. method:: NNTP.list(group_pattern=None, *, file=None) + + Send a ``LIST`` or ``LIST ACTIVE`` command. Return a pair + ``(response, list)`` where *list* is a list of tuples representing all + the groups available from this NNTP server, optionally matching the + pattern string *group_pattern*. Each tuple has the form + ``(group, last, first, flag)``, where *group* is a group name, *last* + and *first* are the last and first article numbers, and *flag* usually + takes one of these values: + + * ``y``: Local postings and articles from peers are allowed. + * ``m``: The group is moderated and all postings must be approved. + * ``n``: No local postings are allowed, only articles from peers. + * ``j``: Articles from peers are filed in the junk group instead. + * ``x``: No local postings, and articles from peers are ignored. + * ``=foo.bar``: Articles are filed in the ``foo.bar`` group instead. + + If *flag* has another value, then the status of the newsgroup should be + considered unknown. + + This command can return very large results, especially if *group_pattern* + is not specified. It is best to cache the results offline unless you + really need to refresh them. + + .. versionchanged:: 3.2 + *group_pattern* was added. .. method:: NNTP.descriptions(grouppattern) Send a ``LIST NEWSGROUPS`` command, where *grouppattern* is a wildmat string as - specified in RFC2980 (it's essentially the same as DOS or UNIX shell wildcard - strings). Return a pair ``(response, list)``, where *list* is a list of tuples - containing ``(name, title)``. + specified in :rfc:`3977` (it's essentially the same as DOS or UNIX shell wildcard + strings). Return a pair ``(response, descriptions)``, where *descriptions* + is a dictionary mapping group names to textual descriptions. + + >>> resp, descs = s.descriptions('gmane.comp.python.*') + >>> len(descs) + 295 + >>> descs.popitem() + ('gmane.comp.python.bio.general', 'BioPython discussion list (Moderated)') .. method:: NNTP.description(group) @@ -195,30 +309,75 @@ indicates an error, the method raises one of the above exceptions. .. method:: NNTP.group(name) - Send a ``GROUP`` command, where *name* is the group name. Return a tuple - ``(response, count, first, last, name)`` where *count* is the (estimated) number - of articles in the group, *first* is the first article number in the group, - *last* is the last article number in the group, and *name* is the group name. - The numbers are returned as strings. + Send a ``GROUP`` command, where *name* is the group name. The group is + selected as the current group, if it exists. Return a tuple + ``(response, count, first, last, name)`` where *count* is the (estimated) + number of articles in the group, *first* is the first article number in + the group, *last* is the last article number in the group, and *name* + is the group name. + + +.. method:: NNTP.over(message_spec, *, file=None) + + Send a ``OVER`` command, or a ``XOVER`` command on legacy servers. + *message_spec* can be either a string representing a message id, or + a ``(first, last)`` tuple of numbers indicating a range of articles in + the current group, or a ``(first, None)`` tuple indicating a range of + articles starting from *first* to the last article in the current group, + or :const:`None` to select the current article in the current group. + + Return a pair ``(response, overviews)``. *overviews* is a list of + ``(article_number, overview)`` tuples, one for each article selected + by *message_spec*. Each *overview* is a dictionary with the same number + of items, but this number depends on the server. These items are either + message headers (the key is then the lower-cased header name) or metadata + items (the key is then the metadata name prepended with ``":"``). The + following items are guaranteed to be present by the NNTP specification: + + * the ``subject``, ``from``, ``date``, ``message-id`` and ``references`` + headers + * the ``:bytes`` metadata: the number of bytes in the entire raw article + (including headers and body) + * the ``:lines`` metadata: the number of lines in the article body + + The value of each item is either a string, or :const:`None` if not present. + + It is advisable to use the :func:`decode_header` function on header + values when they may contain non-ASCII characters:: + + >>> _, _, first, last, _ = s.group('gmane.comp.python.devel') + >>> resp, overviews = s.over((last, last)) + >>> art_num, over = overviews[0] + >>> art_num + 117216 + >>> list(over.keys()) + ['xref', 'from', ':lines', ':bytes', 'references', 'date', 'message-id', 'subject'] + >>> over['from'] + '=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= <martin@v.loewis.de>' + >>> nntplib.decode_header(over['from']) + '"Martin v. Löwis" <martin@v.loewis.de>' + + .. versionadded:: 3.2 -.. method:: NNTP.help([file]) +.. method:: NNTP.help(*, file=None) Send a ``HELP`` command. Return a pair ``(response, list)`` where *list* is a - list of help strings. If the *file* parameter is supplied, then the output of - the ``HELP`` command is stored in a file. If *file* is a string, then the - method will open a file with that name, write to it then close it. If *file* - is a :term:`file object`, then it will start calling :meth:`write` on it to store - the lines of the command output. If *file* is supplied, then the returned *list* - is an empty list. + list of help strings. -.. method:: NNTP.stat(id) +.. method:: NNTP.stat(message_spec=None) - Send a ``STAT`` command, where *id* is the message id (enclosed in ``'<'`` and - ``'>'``) or an article number (as a string). Return a triple ``(response, - number, id)`` where *number* is the article number (as a string) and *id* is the - message id (enclosed in ``'<'`` and ``'>'``). + Send a ``STAT`` command, where *message_spec* is either a message id + (enclosed in ``'<'`` and ``'>'``) or an article number in the current group. + If *message_spec* is omitted or :const:`None`, the current article in the + current group is considered. Return a triple ``(response, number, id)`` + where *number* is the article number and *id* is the message id. + + >>> _, _, first, last, _ = s.group('gmane.comp.python.devel') + >>> resp, number, message_id = s.stat(first) + >>> number, message_id + (9099, '<20030112190404.GE29873@epoch.metaslash.com>') .. method:: NNTP.next() @@ -231,28 +390,69 @@ indicates an error, the method raises one of the above exceptions. Send a ``LAST`` command. Return as for :meth:`stat`. -.. method:: NNTP.head(id) +.. method:: NNTP.article(message_spec=None, *, file=None) + + Send an ``ARTICLE`` command, where *message_spec* has the same meaning as + for :meth:`stat`. Return a tuple ``(response, info)`` where *info* + is a :class:`~collections.namedtuple` with three members *number*, + *message_id* and *lines* (in that order). *number* is the article number + in the group (or 0 if the information is not available), *message_id* the + message id as a string, and *lines* a list of lines (without terminating + newlines) comprising the raw message including headers and body. + + >>> resp, info = s.article('<20030112190404.GE29873@epoch.metaslash.com>') + >>> info.number + 0 + >>> info.message_id + '<20030112190404.GE29873@epoch.metaslash.com>' + >>> len(info.lines) + 65 + >>> info.lines[0] + b'Path: main.gmane.org!not-for-mail' + >>> info.lines[1] + b'From: Neal Norwitz <neal@metaslash.com>' + >>> info.lines[-3:] + [b'There is a patch for 2.3 as well as 2.2.', b'', b'Neal'] + + +.. method:: NNTP.head(message_spec=None, *, file=None) + + Same as :meth:`article()`, but sends a ``HEAD`` command. The *lines* + returned (or written to *file*) will only contain the message headers, not + the body. + - Send a ``HEAD`` command, where *id* has the same meaning as for :meth:`stat`. - Return a tuple ``(response, number, id, list)`` where the first three are the - same as for :meth:`stat`, and *list* is a list of the article's headers (an - uninterpreted list of lines, without trailing newlines). +.. method:: NNTP.body(message_spec=None, *, file=None) + Same as :meth:`article()`, but sends a ``BODY`` command. The *lines* + returned (or written to *file*) will only contain the message body, not the + headers. -.. method:: NNTP.body(id,[file]) - Send a ``BODY`` command, where *id* has the same meaning as for :meth:`stat`. - If the *file* parameter is supplied, then the body is stored in a file. If - *file* is a string, then the method will open a file with that name, write - to it then close it. If *file* is a :term:`file object`, then it will start - calling :meth:`write` on it to store the lines of the body. Return as for - :meth:`head`. If *file* is supplied, then the returned *list* is an empty list. +.. method:: NNTP.post(data) + Post an article using the ``POST`` command. The *data* argument is either + a :term:`file object` opened for binary reading, or any iterable of bytes + objects (representing raw lines of the article to be posted). It should + represent a well-formed news article, including the required headers. The + :meth:`post` method automatically escapes lines beginning with ``.`` and + appends the termination line. -.. method:: NNTP.article(id) + If the method succeeds, the server's response is returned. If the server + refuses posting, a :class:`NNTPReplyError` is raised. + + +.. method:: NNTP.ihave(message_id, data) + + Send an ``IHAVE`` command. *message_id* is the id of the message to send + to the server (enclosed in ``'<'`` and ``'>'``). The *data* parameter + and the return value are the same as for :meth:`post()`. + + +.. method:: NNTP.date() - Send an ``ARTICLE`` command, where *id* has the same meaning as for - :meth:`stat`. Return as for :meth:`head`. + Return a pair ``(response, date)``. *date* is a :class:`~datetime.datetime` + object containing the current date and time of the server. .. method:: NNTP.slave() @@ -260,10 +460,23 @@ indicates an error, the method raises one of the above exceptions. Send a ``SLAVE`` command. Return the server's *response*. -.. method:: NNTP.xhdr(header, string, [file]) +.. method:: NNTP.set_debuglevel(level) - Send an ``XHDR`` command. This command is not defined in the RFC but is a - common extension. The *header* argument is a header keyword, e.g. + Set the instance's debugging level. This controls the amount of debugging + output printed. The default, ``0``, produces no debugging output. A value of + ``1`` produces a moderate amount of debugging output, generally a single line + per request or response. A value of ``2`` or higher produces the maximum amount + of debugging output, logging each line sent and received on the connection + (including message text). + + +The following are optional NNTP extensions defined in :rfc:`2980`. Some of +them have been superseded by newer commands in :rfc:`3977`. + + +.. method:: NNTP.xhdr(header, string, *, file=None) + + Send an ``XHDR`` command. The *header* argument is a header keyword, e.g. ``'subject'``. The *string* argument should have the form ``'first-last'`` where *first* and *last* are the first and last article numbers to search. Return a pair ``(response, list)``, where *list* is a list of pairs ``(id, @@ -276,66 +489,55 @@ indicates an error, the method raises one of the above exceptions. returned *list* is an empty list. -.. method:: NNTP.post(file) - - Post an article using the ``POST`` command. The *file* argument is an open file - object which is read until EOF using its :meth:`readline` method. It should be - a well-formed news article, including the required headers. The :meth:`post` - method automatically escapes lines beginning with ``.``. +.. method:: NNTP.xover(start, end, *, file=None) + Send an ``XOVER`` command. *start* and *end* are article numbers + delimiting the range of articles to select. The return value is the + same of for :meth:`over()`. It is recommended to use :meth:`over()` + instead, since it will automatically use the newer ``OVER`` command + if available. -.. method:: NNTP.ihave(id, file) - - Send an ``IHAVE`` command. *id* is a message id (enclosed in ``'<'`` and - ``'>'``). If the response is not an error, treat *file* exactly as for the - :meth:`post` method. - - -.. method:: NNTP.date() - - Return a triple ``(response, date, time)``, containing the current date and time - in a form suitable for the :meth:`newnews` and :meth:`newgroups` methods. This - is an optional NNTP extension, and may not be supported by all servers. +.. method:: NNTP.xpath(id) -.. method:: NNTP.xgtitle(name, [file]) + Return a pair ``(resp, path)``, where *path* is the directory path to the + article with message ID *id*. Most of the time, this extension is not + enabled by NNTP server administrators. - Process an ``XGTITLE`` command, returning a pair ``(response, list)``, where - *list* is a list of tuples containing ``(name, title)``. If the *file* parameter - is supplied, then the output of the ``XGTITLE`` command is stored in a file. - If *file* is a string, then the method will open a file with that name, write - to it then close it. If *file* is a :term:`file object`, then it will start - calling :meth:`write` on it to store the lines of the command output. If *file* - is supplied, then the returned *list* is an empty list. This is an optional NNTP - extension, and may not be supported by all servers. - RFC2980 says "It is suggested that this extension be deprecated". Use - :meth:`descriptions` or :meth:`description` instead. +.. XXX deprecated: + .. method:: NNTP.xgtitle(name, *, file=None) -.. method:: NNTP.xover(start, end, [file]) + Process an ``XGTITLE`` command, returning a pair ``(response, list)``, where + *list* is a list of tuples containing ``(name, title)``. If the *file* parameter + is supplied, then the output of the ``XGTITLE`` command is stored in a file. + If *file* is a string, then the method will open a file with that name, write + to it then close it. If *file* is a :term:`file object`, then it will start + calling :meth:`write` on it to store the lines of the command output. If *file* + is supplied, then the returned *list* is an empty list. This is an optional NNTP + extension, and may not be supported by all servers. - Return a pair ``(resp, list)``. *list* is a list of tuples, one for each - article in the range delimited by the *start* and *end* article numbers. Each - tuple is of the form ``(article number, subject, poster, date, id, references, - size, lines)``. If the *file* parameter is supplied, then the output of the - ``XOVER`` command is stored in a file. If *file* is a string, then the method - will open a file with that name, write to it then close it. If *file* is a - :term:`file object`, then it will start calling :meth:`write` on it to store the - lines of the command output. If *file* is supplied, then the returned *list* is - an empty list. This is an optional NNTP extension, and may not be supported by - all servers. + RFC2980 says "It is suggested that this extension be deprecated". Use + :meth:`descriptions` or :meth:`description` instead. -.. method:: NNTP.xpath(id) +Utility functions +----------------- - Return a pair ``(resp, path)``, where *path* is the directory path to the - article with message ID *id*. This is an optional NNTP extension, and may not - be supported by all servers. +The module also defines the following utility function: -.. method:: NNTP.quit() +.. function:: decode_header(header_str) - Send a ``QUIT`` command and close the connection. Once this method has been - called, no other methods of the NNTP object should be called. + Decode a header value, un-escaping any escaped non-ASCII characters. + *header_str* must be a :class:`str` object. The unescaped value is + returned. Using this function is recommended to display some headers + in a human readable form:: + >>> decode_header("Some subject") + 'Some subject' + >>> decode_header("=?ISO-8859-15?Q?D=E9buter_en_Python?=") + 'Débuter en Python' + >>> decode_header("Re: =?UTF-8?B?cHJvYmzDqG1lIGRlIG1hdHJpY2U=?=") + 'Re: problème de matrice' diff --git a/Doc/library/numeric.rst b/Doc/library/numeric.rst index 641f588..ba22cb6 100644 --- a/Doc/library/numeric.rst +++ b/Doc/library/numeric.rst @@ -23,6 +23,3 @@ The following modules are documented in this chapter: decimal.rst fractions.rst random.rst - itertools.rst - functools.rst - operator.rst diff --git a/Doc/library/objects.rst b/Doc/library/objects.rst deleted file mode 100644 index 1b75161..0000000 --- a/Doc/library/objects.rst +++ /dev/null @@ -1,27 +0,0 @@ - -.. _builtin: - -**************** -Built-in Objects -**************** - -.. index:: - pair: built-in; types - pair: built-in; exceptions - pair: built-in; functions - pair: built-in; constants - single: symbol table - -Names for built-in exceptions and functions and a number of constants are found -in a separate symbol table. This table is searched last when the interpreter -looks up the meaning of a name, so local and global user-defined names can -override built-in names. Built-in types are described together here for easy -reference. - -The tables in this chapter document the priorities of operators by listing them -in order of ascending priority (within a table) and grouping operators that have -the same priority in the same box. Binary operators of the same priority group -from left to right. (Unary operators group from right to left, but there you -have no real choice.) See :ref:`operator-summary` for the complete picture on -operator priorities. - diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst index 43bdeef..1dc0e14 100644 --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -9,7 +9,7 @@ .. testsetup:: import operator - from operator import itemgetter + from operator import itemgetter, iadd The :mod:`operator` module exports a set of functions implemented in C @@ -225,91 +225,6 @@ Operations which work with sequences (some of them with mappings too) include: Set the value of *a* at index *b* to *c*. - -Many operations have an "in-place" version. The following functions provide a -more primitive access to in-place operators than the usual syntax does; for -example, the :term:`statement` ``x += y`` is equivalent to -``x = operator.iadd(x, y)``. Another way to put it is to say that -``z = operator.iadd(x, y)`` is equivalent to the compound statement -``z = x; z += y``. - -.. function:: iadd(a, b) - __iadd__(a, b) - - ``a = iadd(a, b)`` is equivalent to ``a += b``. - - -.. function:: iand(a, b) - __iand__(a, b) - - ``a = iand(a, b)`` is equivalent to ``a &= b``. - - -.. function:: iconcat(a, b) - __iconcat__(a, b) - - ``a = iconcat(a, b)`` is equivalent to ``a += b`` for *a* and *b* sequences. - - -.. function:: ifloordiv(a, b) - __ifloordiv__(a, b) - - ``a = ifloordiv(a, b)`` is equivalent to ``a //= b``. - - -.. function:: ilshift(a, b) - __ilshift__(a, b) - - ``a = ilshift(a, b)`` is equivalent to ``a <<= b``. - - -.. function:: imod(a, b) - __imod__(a, b) - - ``a = imod(a, b)`` is equivalent to ``a %= b``. - - -.. function:: imul(a, b) - __imul__(a, b) - - ``a = imul(a, b)`` is equivalent to ``a *= b``. - - -.. function:: ior(a, b) - __ior__(a, b) - - ``a = ior(a, b)`` is equivalent to ``a |= b``. - - -.. function:: ipow(a, b) - __ipow__(a, b) - - ``a = ipow(a, b)`` is equivalent to ``a **= b``. - - -.. function:: irshift(a, b) - __irshift__(a, b) - - ``a = irshift(a, b)`` is equivalent to ``a >>= b``. - - -.. function:: isub(a, b) - __isub__(a, b) - - ``a = isub(a, b)`` is equivalent to ``a -= b``. - - -.. function:: itruediv(a, b) - __itruediv__(a, b) - - ``a = itruediv(a, b)`` is equivalent to ``a /= b``. - - -.. function:: ixor(a, b) - __ixor__(a, b) - - ``a = ixor(a, b)`` is equivalent to ``a ^= b``. - Example: Build a dictionary that maps the ordinals from ``0`` to ``255`` to their character equivalents. @@ -335,6 +250,8 @@ expect a function argument. b.date)``. Equivalent to:: def attrgetter(*items): + if any(not isinstance(item, str) for item in items): + raise TypeError('attribute name must be a string') if len(items) == 1: attr = items[0] def g(obj): @@ -488,3 +405,112 @@ Python syntax and the functions in the :mod:`operator` module. | Ordering | ``a > b`` | ``gt(a, b)`` | +-----------------------+-------------------------+---------------------------------------+ +Inplace Operators +================= + +Many operations have an "in-place" version. Listed below are functions +providing a more primitive access to in-place operators than the usual syntax +does; for example, the :term:`statement` ``x += y`` is equivalent to +``x = operator.iadd(x, y)``. Another way to put it is to say that +``z = operator.iadd(x, y)`` is equivalent to the compound statement +``z = x; z += y``. + +In those examples, note that when an in-place method is called, the computation +and assignment are performed in two separate steps. The in-place functions +listed below only do the first step, calling the in-place method. The second +step, assignment, is not handled. + +For immutable targets such as strings, numbers, and tuples, the updated +value is computed, but not assigned back to the input variable: + +>>> a = 'hello' +>>> iadd(a, ' world') +'hello world' +>>> a +'hello' + +For mutable targets such as lists and dictionaries, the inplace method +will perform the update, so no subsequent assignment is necessary: + +>>> s = ['h', 'e', 'l', 'l', 'o'] +>>> iadd(s, [' ', 'w', 'o', 'r', 'l', 'd']) +['h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd'] +>>> s +['h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd'] + +.. function:: iadd(a, b) + __iadd__(a, b) + + ``a = iadd(a, b)`` is equivalent to ``a += b``. + + +.. function:: iand(a, b) + __iand__(a, b) + + ``a = iand(a, b)`` is equivalent to ``a &= b``. + + +.. function:: iconcat(a, b) + __iconcat__(a, b) + + ``a = iconcat(a, b)`` is equivalent to ``a += b`` for *a* and *b* sequences. + + +.. function:: ifloordiv(a, b) + __ifloordiv__(a, b) + + ``a = ifloordiv(a, b)`` is equivalent to ``a //= b``. + + +.. function:: ilshift(a, b) + __ilshift__(a, b) + + ``a = ilshift(a, b)`` is equivalent to ``a <<= b``. + + +.. function:: imod(a, b) + __imod__(a, b) + + ``a = imod(a, b)`` is equivalent to ``a %= b``. + + +.. function:: imul(a, b) + __imul__(a, b) + + ``a = imul(a, b)`` is equivalent to ``a *= b``. + + +.. function:: ior(a, b) + __ior__(a, b) + + ``a = ior(a, b)`` is equivalent to ``a |= b``. + + +.. function:: ipow(a, b) + __ipow__(a, b) + + ``a = ipow(a, b)`` is equivalent to ``a **= b``. + + +.. function:: irshift(a, b) + __irshift__(a, b) + + ``a = irshift(a, b)`` is equivalent to ``a >>= b``. + + +.. function:: isub(a, b) + __isub__(a, b) + + ``a = isub(a, b)`` is equivalent to ``a -= b``. + + +.. function:: itruediv(a, b) + __itruediv__(a, b) + + ``a = itruediv(a, b)`` is equivalent to ``a /= b``. + + +.. function:: ixor(a, b) + __ixor__(a, b) + + ``a = ixor(a, b)`` is equivalent to ``a ^= b``. diff --git a/Doc/library/optparse.rst b/Doc/library/optparse.rst index 87ec4de..58fb7cd 100644 --- a/Doc/library/optparse.rst +++ b/Doc/library/optparse.rst @@ -1,11 +1,19 @@ -:mod:`optparse` --- More powerful command line option parser -============================================================ +:mod:`optparse` --- Parser for command line options +=================================================== .. module:: optparse - :synopsis: More convenient, flexible, and powerful command-line parsing library. + :synopsis: Command-line option parsing library. + :deprecated: .. moduleauthor:: Greg Ward <gward@python.net> .. sectionauthor:: Greg Ward <gward@python.net> +**Source code:** :source:`Lib/optparse.py` + +-------------- + +.. deprecated:: 2.7 + The :mod:`optparse` module is deprecated and will not be developed further; + development will continue with the :mod:`argparse` module. :mod:`optparse` is a more convenient, flexible, and powerful library for parsing command-line options than the old :mod:`getopt` module. :mod:`optparse` uses a diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index 28a7aff..522b94c 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -201,6 +201,7 @@ applications should use string objects to access all files. Normalize the case of a pathname. On Unix and Mac OS X, this returns the path unchanged; on case-insensitive filesystems, it converts the path to lowercase. On Windows, it also converts forward slashes to backward slashes. + Raise a TypeError if the type of *path* is not ``str`` or ``bytes``. .. function:: normpath(path) @@ -220,7 +221,7 @@ applications should use string objects to access all files. links encountered in the path (if they are supported by the operating system). -.. function:: relpath(path[, start]) +.. function:: relpath(path, start=None) Return a relative filepath to *path* either from the current directory or from an optional *start* point. @@ -232,18 +233,27 @@ applications should use string objects to access all files. .. function:: samefile(path1, path2) - Return ``True`` if both pathname arguments refer to the same file or directory - (as indicated by device number and i-node number). Raise an exception if a - :func:`os.stat` call on either pathname fails. + Return ``True`` if both pathname arguments refer to the same file or directory. + On Unix, this is determined by the device number and i-node number and raises an + exception if a :func:`os.stat` call on either pathname fails. - Availability: Unix. + On Windows, two files are the same if they resolve to the same final path + name using the Windows API call GetFinalPathNameByHandle. This function + raises an exception if handles cannot be obtained to either file. + + Availability: Unix, Windows. + + .. versionchanged:: 3.2 + Added Windows support. .. function:: sameopenfile(fp1, fp2) Return ``True`` if the file descriptors *fp1* and *fp2* refer to the same file. - Availability: Unix. + Availability: Unix, Windows. + + .. versionchanged:: 3.2 Added Windows support. .. function:: samestat(stat1, stat2) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 3f6d69d..fe785fc 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -109,6 +109,10 @@ process and user. to modify the environment as well as query the environment. :func:`putenv` will be called automatically when the mapping is modified. + On Unix, keys and values use :func:`sys.getfilesystemencoding` and + ``'surrogateescape'`` error handler. Use :data:`environb` if you would like + to use a different encoding. + .. note:: Calling :func:`putenv` directly does not change ``os.environ``, so it's better @@ -118,7 +122,7 @@ process and user. On some platforms, including FreeBSD and Mac OS X, setting ``environ`` may cause memory leaks. Refer to the system documentation for - :cfunc:`putenv`. + :c:func:`putenv`. If :func:`putenv` is not provided, a modified copy of this mapping may be passed to the appropriate process-creation functions to cause child processes @@ -130,6 +134,19 @@ process and user. one of the :meth:`pop` or :meth:`clear` methods is called. +.. data:: environb + + Bytes version of :data:`environ`: a mapping object representing the + environment as byte strings. :data:`environ` and :data:`environb` are + synchronized (modify :data:`environb` updates :data:`environ`, and vice + versa). + + :data:`environb` is only available if :data:`supports_bytes_environ` is + True. + + .. versionadded:: 3.2 + + .. function:: chdir(path) fchdir(fd) getcwd() @@ -138,6 +155,37 @@ process and user. These functions are described in :ref:`os-file-dir`. +.. function:: fsencode(filename) + + Encode *filename* to the filesystem encoding with ``'surrogateescape'`` + error handler, or ``'strict'`` on Windows; return :class:`bytes` unchanged. + + :func:`fsdecode` is the reverse function. + + .. versionadded:: 3.2 + + +.. function:: fsdecode(filename) + + Decode *filename* from the filesystem encoding with ``'surrogateescape'`` + error handler, or ``'strict'`` on Windows; return :class:`str` unchanged. + + :func:`fsencode` is the reverse function. + + .. versionadded:: 3.2 + + +.. function:: get_exec_path(env=None) + + Returns the list of directories that will be searched for a named + executable, similar to a shell, when launching a process. + *env*, when specified, should be an environment variable dictionary + to lookup the PATH in. + By default, when *env* is None, :data:`environ` is used. + + .. versionadded:: 3.2 + + .. function:: ctermid() Return the filename corresponding to the controlling terminal of the process. @@ -178,15 +226,26 @@ process and user. Availability: Unix. +.. function:: initgroups(username, gid) + + Call the system initgroups() to initialize the group access list with all of + the groups of which the specified username is a member, plus the specified + group id. + + Availability: Unix. + + .. versionadded:: 3.2 + + .. function:: getlogin() Return the name of the user logged in on the controlling terminal of the - process. For most purposes, it is more useful to use the environment variable - :envvar:`LOGNAME` to find out who the user is, or + process. For most purposes, it is more useful to use the environment variables + :envvar:`LOGNAME` or :envvar:`USERNAME` to find out who the user is, or ``pwd.getpwuid(os.getuid())[0]`` to get the login name of the currently effective user id. - Availability: Unix. + Availability: Unix, Windows. .. function:: getpgid(pid) @@ -218,10 +277,34 @@ process and user. .. index:: single: process; id of parent - Return the parent's process id. + Return the parent's process id. When the parent process has exited, on Unix + the id returned is the one of the init process (1), on Windows it is still + the same id, which may be already reused by another process. + + Availability: Unix, Windows + + .. versionchanged:: 3.2 + Added support for Windows. + +.. function:: getresuid() + + Return a tuple (ruid, euid, suid) denoting the current process's + real, effective, and saved user ids. Availability: Unix. + .. versionadded:: 3.2 + + +.. function:: getresgid() + + Return a tuple (rgid, egid, sgid) denoting the current process's + real, effective, and saved group ids. + + Availability: Unix. + + .. versionadded:: 3.2 + .. function:: getuid() @@ -232,19 +315,33 @@ process and user. Availability: Unix. -.. function:: getenv(varname[, value]) +.. function:: getenv(key, default=None) + + Return the value of the environment variable *key* if it exists, or + *default* if it doesn't. *key*, *default* and the result are str. - Return the value of the environment variable *varname* if it exists, or *value* - if it doesn't. *value* defaults to ``None``. + On Unix, keys and values are decoded with :func:`sys.getfilesystemencoding` + and ``'surrogateescape'`` error handler. Use :func:`os.getenvb` if you + would like to use a different encoding. Availability: most flavors of Unix, Windows. -.. function:: putenv(varname, value) +.. function:: getenvb(key, default=None) + + Return the value of the environment variable *key* if it exists, or + *default* if it doesn't. *key*, *default* and the result are bytes. + + Availability: most flavors of Unix. + + .. versionadded:: 3.2 + + +.. function:: putenv(key, value) .. index:: single: environment variables; setting - Set the environment variable named *varname* to the string *value*. Such + Set the environment variable named *key* to the string *value*. Such changes to the environment affect subprocesses started with :func:`os.system`, :func:`popen` or :func:`fork` and :func:`execv`. @@ -293,7 +390,7 @@ process and user. .. function:: setpgrp() - Call the system call :cfunc:`setpgrp` or :cfunc:`setpgrp(0, 0)` depending on + Call the system call :c:func:`setpgrp` or :c:func:`setpgrp(0, 0)` depending on which version is implemented (if any). See the Unix manual for the semantics. Availability: Unix. @@ -301,37 +398,55 @@ process and user. .. function:: setpgid(pid, pgrp) - Call the system call :cfunc:`setpgid` to set the process group id of the + Call the system call :c:func:`setpgid` to set the process group id of the process with id *pid* to the process group with id *pgrp*. See the Unix manual for the semantics. Availability: Unix. -.. function:: setreuid(ruid, euid) +.. function:: setregid(rgid, egid) - Set the current process's real and effective user ids. + Set the current process's real and effective group ids. Availability: Unix. -.. function:: setregid(rgid, egid) +.. function:: setresgid(rgid, egid, sgid) - Set the current process's real and effective group ids. + Set the current process's real, effective, and saved group ids. + + Availability: Unix. + + .. versionadded:: 3.2 + + +.. function:: setresuid(ruid, euid, suid) + + Set the current process's real, effective, and saved user ids. + + Availability: Unix. + + .. versionadded:: 3.2 + + +.. function:: setreuid(ruid, euid) + + Set the current process's real and effective user ids. Availability: Unix. .. function:: getsid(pid) - Call the system call :cfunc:`getsid`. See the Unix manual for the semantics. + Call the system call :c:func:`getsid`. See the Unix manual for the semantics. Availability: Unix. .. function:: setsid() - Call the system call :cfunc:`setsid`. See the Unix manual for the semantics. + Call the system call :c:func:`setsid`. See the Unix manual for the semantics. Availability: Unix. @@ -349,12 +464,20 @@ process and user. .. function:: strerror(code) Return the error message corresponding to the error code in *code*. - On platforms where :cfunc:`strerror` returns ``NULL`` when given an unknown + On platforms where :c:func:`strerror` returns ``NULL`` when given an unknown error number, :exc:`ValueError` is raised. Availability: Unix, Windows. +.. data:: supports_bytes_environ + + True if the native OS type of the environment is bytes (eg. False on + Windows). + + .. versionadded:: 3.2 + + .. function:: umask(mask) Set the current numeric umask and return the previous umask. @@ -378,11 +501,11 @@ process and user. Availability: recent flavors of Unix. -.. function:: unsetenv(varname) +.. function:: unsetenv(key) .. index:: single: environment variables; deleting - Unset (delete) the environment variable named *varname*. Such changes to the + Unset (delete) the environment variable named *key*. Such changes to the environment affect subprocesses started with :func:`os.system`, :func:`popen` or :func:`fork` and :func:`execv`. @@ -414,7 +537,7 @@ These functions create new :term:`file objects <file object>`. (See also :func:` ``'r'``, ``'w'``, or ``'a'``, otherwise a :exc:`ValueError` is raised. On Unix, when the *mode* argument starts with ``'a'``, the *O_APPEND* flag is - set on the file descriptor (which the :cfunc:`fdopen` implementation already + set on the file descriptor (which the :c:func:`fdopen` implementation already does on most platforms). Availability: Unix, Windows. @@ -550,7 +673,7 @@ as internal buffering of data. .. function:: fsync(fd) Force write of file with filedescriptor *fd* to disk. On Unix, this calls the - native :cfunc:`fsync` function; on Windows, the MS :cfunc:`_commit` function. + native :c:func:`fsync` function; on Windows, the MS :c:func:`_commit` function. If you're starting with a buffered Python :term:`file object` *f*, first do ``f.flush()``, and then do ``os.fsync(f.fileno())``, to ensure that all internal @@ -612,7 +735,7 @@ as internal buffering of data. This function is intended for low-level I/O. For normal usage, use the built-in function :func:`open`, which returns a :term:`file object` with - :meth:`~file.read` and :meth:`~file.wprite` methods (and many more). To + :meth:`~file.read` and :meth:`~file.write` methods (and many more). To wrap a file descriptor in a file object, use :func:`fdopen`. @@ -935,13 +1058,16 @@ Files and Directories Create a hard link pointing to *source* named *link_name*. - Availability: Unix. + Availability: Unix, Windows. + + .. versionchanged:: 3.2 + Added Windows support. -.. function:: listdir(path) +.. function:: listdir(path='.') Return a list containing the names of the entries in the directory given by - *path*. The list is in arbitrary order. It does not include the special + *path* (default: ``'.'``). The list is in arbitrary order. It does not include the special entries ``'.'`` and ``'..'`` even if they are present in the directory. This function can be called with a bytes or string argument, and returns @@ -949,14 +1075,19 @@ Files and Directories Availability: Unix, Windows. + .. versionchanged:: 3.2 + The *path* parameter became optional. .. function:: lstat(path) - Perform the equivalent of an :cfunc:`lstat` system call on the given path. + Perform the equivalent of an :c:func:`lstat` system call on the given path. Similar to :func:`~os.stat`, but does not follow symbolic links. On platforms that do not support symbolic links, this is an alias for :func:`~os.stat`. + .. versionchanged:: 3.2 + Added support for Windows 6.0 (Vista) symbolic links. + .. function:: mkfifo(path[, mode]) @@ -973,28 +1104,27 @@ Files and Directories Availability: Unix. -.. function:: mknod(filename[, mode=0o600, device]) +.. function:: mknod(filename[, mode=0o600[, device]]) Create a filesystem node (file, device special file or named pipe) named - *filename*. *mode* specifies both the permissions to use and the type of node to - be created, being combined (bitwise OR) with one of ``stat.S_IFREG``, - ``stat.S_IFCHR``, ``stat.S_IFBLK``, - and ``stat.S_IFIFO`` (those constants are available in :mod:`stat`). - For ``stat.S_IFCHR`` and - ``stat.S_IFBLK``, *device* defines the newly created device special file (probably using + *filename*. *mode* specifies both the permissions to use and the type of node + to be created, being combined (bitwise OR) with one of ``stat.S_IFREG``, + ``stat.S_IFCHR``, ``stat.S_IFBLK``, and ``stat.S_IFIFO`` (those constants are + available in :mod:`stat`). For ``stat.S_IFCHR`` and ``stat.S_IFBLK``, + *device* defines the newly created device special file (probably using :func:`os.makedev`), otherwise it is ignored. .. function:: major(device) Extract the device major number from a raw device number (usually the - :attr:`st_dev` or :attr:`st_rdev` field from :ctype:`stat`). + :attr:`st_dev` or :attr:`st_rdev` field from :c:type:`stat`). .. function:: minor(device) Extract the device minor number from a raw device number (usually the - :attr:`st_dev` or :attr:`st_rdev` field from :ctype:`stat`). + :attr:`st_dev` or :attr:`st_rdev` field from :c:type:`stat`). .. function:: makedev(major, minor) @@ -1015,17 +1145,20 @@ Files and Directories Availability: Unix, Windows. -.. function:: makedirs(path, mode=0o777) +.. function:: makedirs(path, mode=0o777, exist_ok=False) .. index:: single: directory; creating single: UNC paths; and os.makedirs() Recursive directory creation function. Like :func:`mkdir`, but makes all - intermediate-level directories needed to contain the leaf directory. Raises - an :exc:`error` exception if the leaf directory already exists or cannot be - created. The default *mode* is ``0o777`` (octal). On some systems, *mode* - is ignored. Where it is used, the current umask value is first masked out. + intermediate-level directories needed to contain the leaf directory. If + the target directory with the same mode as specified already exists, + raises an :exc:`OSError` exception if *exist_ok* is False, otherwise no + exception is raised. If the directory cannot be created in other cases, + raises an :exc:`OSError` exception. The default *mode* is ``0o777`` (octal). + On some systems, *mode* is ignored. Where it is used, the current umask + value is first masked out. .. note:: @@ -1034,6 +1167,9 @@ Files and Directories This function handles UNC paths correctly. + .. versionadded:: 3.2 + The *exist_ok* parameter. + .. function:: pathconf(path, name) @@ -1072,7 +1208,10 @@ Files and Directories and the call may raise an UnicodeDecodeError. If the *path* is a bytes object, the result will be a bytes object. - Availability: Unix. + Availability: Unix, Windows + + .. versionchanged:: 3.2 + Added support for Windows 6.0 (Vista) symbolic links. .. function:: remove(path) @@ -1139,11 +1278,11 @@ Files and Directories .. function:: stat(path) - Perform the equivalent of a :cfunc:`stat` system call on the given path. + Perform the equivalent of a :c:func:`stat` system call on the given path. (This function follows symlinks; to stat a symlink use :func:`lstat`.) The return value is an object whose attributes correspond to the members - of the :ctype:`stat` structure, namely: + of the :c:type:`stat` structure, namely: * :attr:`st_mode` - protection bits, * :attr:`st_ino` - inode number, @@ -1187,7 +1326,7 @@ Files and Directories For backward compatibility, the return value of :func:`~os.stat` is also accessible as a tuple of at least 10 integers giving the most important (and portable) - members of the :ctype:`stat` structure, in the order :attr:`st_mode`, + members of the :c:type:`stat` structure, in the order :attr:`st_mode`, :attr:`st_ino`, :attr:`st_dev`, :attr:`st_nlink`, :attr:`st_uid`, :attr:`st_gid`, :attr:`st_size`, :attr:`st_atime`, :attr:`st_mtime`, :attr:`st_ctime`. More items may be added at the end by some implementations. @@ -1195,7 +1334,7 @@ Files and Directories .. index:: module: stat The standard module :mod:`stat` defines functions and constants that are useful - for extracting information from a :ctype:`stat` structure. (On Windows, some + for extracting information from a :c:type:`stat` structure. (On Windows, some items are filled with dummy values.) Example:: @@ -1203,9 +1342,11 @@ Files and Directories >>> import os >>> statinfo = os.stat('somefile.txt') >>> statinfo - (33188, 422511, 769, 1, 1032, 100, 926, 1105022698,1105022732, 1105022732) + posix.stat_result(st_mode=33188, st_ino=7876932, st_dev=234881026, + st_nlink=1, st_uid=501, st_gid=501, st_size=264, st_atime=1297230295, + st_mtime=1297230027, st_ctime=1297230027) >>> statinfo.st_size - 926 + 264 Availability: Unix, Windows. @@ -1237,21 +1378,58 @@ Files and Directories .. function:: statvfs(path) - Perform a :cfunc:`statvfs` system call on the given path. The return value is + Perform a :c:func:`statvfs` system call on the given path. The return value is an object whose attributes describe the filesystem on the given path, and - correspond to the members of the :ctype:`statvfs` structure, namely: + correspond to the members of the :c:type:`statvfs` structure, namely: :attr:`f_bsize`, :attr:`f_frsize`, :attr:`f_blocks`, :attr:`f_bfree`, :attr:`f_bavail`, :attr:`f_files`, :attr:`f_ffree`, :attr:`f_favail`, :attr:`f_flag`, :attr:`f_namemax`. + Two module-level constants are defined for the :attr:`f_flag` attribute's + bit-flags: if :const:`ST_RDONLY` is set, the filesystem is mounted + read-only, and if :const:`ST_NOSUID` is set, the semantics of + setuid/setgid bits are disabled or not supported. + + .. versionchanged:: 3.2 + The :const:`ST_RDONLY` and :const:`ST_NOSUID` constants were added. + Availability: Unix. .. function:: symlink(source, link_name) + symlink(source, link_name, target_is_directory=False) Create a symbolic link pointing to *source* named *link_name*. - Availability: Unix. + On Windows, symlink version takes an additional optional parameter, + *target_is_directory*, which defaults to ``False``. + + On Windows, a symlink represents a file or a directory, and does not morph to + the target dynamically. For this reason, when creating a symlink on Windows, + if the target is not already present, the symlink will default to being a + file symlink. If *target_is_directory* is set to ``True``, the symlink will + be created as a directory symlink. This parameter is ignored if the target + exists (and the symlink is created with the same type as the target). + + Symbolic link support was introduced in Windows 6.0 (Vista). :func:`symlink` + will raise a :exc:`NotImplementedError` on Windows versions earlier than 6.0. + + .. note:: + + The *SeCreateSymbolicLinkPrivilege* is required in order to successfully + create symlinks. This privilege is not typically granted to regular + users but is available to accounts which can escalate privileges to the + administrator level. Either obtaining the privilege or running your + application as an administrator are ways to successfully create symlinks. + + + :exc:`OSError` is raised when the function is called by an unprivileged + user. + + Availability: Unix, Windows. + + .. versionchanged:: 3.2 + Added support for Windows 6.0 (Vista) symbolic links. .. function:: unlink(path) @@ -1279,7 +1457,7 @@ Files and Directories Availability: Unix, Windows. -.. function:: walk(top[, topdown=True [, onerror=None[, followlinks=False]]]) +.. function:: walk(top, topdown=True, onerror=None, followlinks=False) .. index:: single: directory; walking @@ -1373,7 +1551,7 @@ The various :func:`exec\*` functions take a list of arguments for the new program loaded into the process. In each case, the first of these arguments is passed to the new program as its own name rather than as an argument a user may have typed on a command line. For the C programmer, this is the ``argv[0]`` -passed to a program's :cfunc:`main`. For example, ``os.execv('/bin/echo', +passed to a program's :c:func:`main`. For example, ``os.execv('/bin/echo', ['foo', 'bar'])`` will only print ``bar`` on standard output; ``foo`` will seem to be ignored. @@ -1616,7 +1794,17 @@ written in Python, such as a mail server's external command delivery program. Send signal *sig* to the process *pid*. Constants for the specific signals available on the host platform are defined in the :mod:`signal` module. - Availability: Unix. + + Windows: The :data:`signal.CTRL_C_EVENT` and + :data:`signal.CTRL_BREAK_EVENT` signals are special signals which can + only be sent to console processes which share a common console window, + e.g., some subprocesses. Any other value for *sig* will cause the process + to be unconditionally killed by the TerminateProcess API, and the exit code + will be set to *sig*. The Windows version of :func:`kill` additionally takes + process handles to be killed. + + .. versionadded:: 3.2 + Windows support. .. function:: killpg(pgid, sig) @@ -1767,7 +1955,7 @@ written in Python, such as a mail server's external command delivery program. There is no option to wait for the application to close, and no way to retrieve the application's exit status. The *path* parameter is relative to the current directory. If you want to use an absolute path, make sure the first character - is not a slash (``'/'``); the underlying Win32 :cfunc:`ShellExecute` function + is not a slash (``'/'``); the underlying Win32 :c:func:`ShellExecute` function doesn't work if it is. Use the :func:`os.path.normpath` function to ensure that the path is properly encoded for Win32. @@ -1777,14 +1965,14 @@ written in Python, such as a mail server's external command delivery program. .. function:: system(command) Execute the command (a string) in a subshell. This is implemented by calling - the Standard C function :cfunc:`system`, and has the same limitations. + the Standard C function :c:func:`system`, and has the same limitations. Changes to :data:`sys.stdin`, etc. are not reflected in the environment of the executed command. If *command* generates any output, it will be sent to the interpreter standard output stream. On Unix, the return value is the exit status of the process encoded in the format specified for :func:`wait`. Note that POSIX does not specify the - meaning of the return value of the C :cfunc:`system` function, so the return + meaning of the return value of the C :c:func:`system` function, so the return value of the Python function is system-dependent. On Windows, the return value is that returned by the system shell after diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst index 3972f14..3b5a7e4 100644 --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -1,4 +1,3 @@ - :mod:`ossaudiodev` --- Access to OSS-compatible audio devices ============================================================= @@ -57,7 +56,7 @@ the standard audio interface for Linux and recent versions of FreeBSD. what went wrong. (If :mod:`ossaudiodev` receives an error from a system call such as - :cfunc:`open`, :cfunc:`write`, or :cfunc:`ioctl`, it raises :exc:`IOError`. + :c:func:`open`, :c:func:`write`, or :c:func:`ioctl`, it raises :exc:`IOError`. Errors detected directly by :mod:`ossaudiodev` result in :exc:`OSSAudioError`.) (For backwards compatibility, the exception class is also available as @@ -160,6 +159,11 @@ and (read-only) attributes: is only useful in non-blocking mode. Has no return value, since the amount of data written is always equal to the amount of data supplied. +.. versionchanged:: 3.2 + Audio device objects also support the context manager protocol, i.e. they can + be used in a :keyword:`with` statement. + + The following methods each map to exactly one :func:`ioctl` system call. The correspondence is obvious: for example, :meth:`setfmt` corresponds to the ``SNDCTL_DSP_SETFMT`` ioctl, and :meth:`sync` to ``SNDCTL_DSP_SYNC`` (this can @@ -347,6 +351,10 @@ The mixer object provides two file-like methods: Returns the file handle number of the open mixer device file. +.. versionchanged:: 3.2 + Mixer objects also support the context manager protocol. + + The remaining methods are specific to audio mixing: diff --git a/Doc/library/parser.rst b/Doc/library/parser.rst index 50b94c1..3e1e31b 100644 --- a/Doc/library/parser.rst +++ b/Doc/library/parser.rst @@ -1,4 +1,3 @@ - :mod:`parser` --- Access Python parse trees =========================================== @@ -165,7 +164,7 @@ executable code objects. Parse trees may be extracted with or without line numbering information. -.. function:: st2list(st[, line_info]) +.. function:: st2list(st, line_info=False, col_info=False) This function accepts an ST object from the caller in *st* and returns a Python list representing the equivalent parse tree. The resulting list @@ -183,7 +182,7 @@ numbering information. This information is omitted if the flag is false or omitted. -.. function:: st2tuple(st[, line_info]) +.. function:: st2tuple(st, line_info=False, col_info=False) This function accepts an ST object from the caller in *st* and returns a Python tuple representing the equivalent parse tree. Other than returning a @@ -194,7 +193,7 @@ numbering information. information is omitted if the flag is false or omitted. -.. function:: compilest(st[, filename='<syntax-tree>']) +.. function:: compilest(st, filename='<syntax-tree>') .. index:: builtin: exec @@ -293,7 +292,7 @@ ST objects (using the :mod:`pickle` module) is also supported. ST objects have the following methods: -.. method:: ST.compile([filename]) +.. method:: ST.compile(filename='<syntax-tree>') Same as ``compilest(st, filename)``. @@ -308,14 +307,14 @@ ST objects have the following methods: Same as ``issuite(st)``. -.. method:: ST.tolist([line_info]) +.. method:: ST.tolist(line_info=False, col_info=False) - Same as ``st2list(st, line_info)``. + Same as ``st2list(st, line_info, col_info)``. -.. method:: ST.totuple([line_info]) +.. method:: ST.totuple(line_info=False, col_info=False) - Same as ``st2tuple(st, line_info)``. + Same as ``st2tuple(st, line_info, col_info)``. Example: Emulation of :func:`compile` diff --git a/Doc/library/pdb.rst b/Doc/library/pdb.rst index ee63ebb..1e9de63 100644 --- a/Doc/library/pdb.rst +++ b/Doc/library/pdb.rst @@ -20,9 +20,9 @@ supports post-mortem debugging and can be called under program control. module: bdb module: cmd -The debugger is extensible --- it is actually defined as the class :class:`Pdb`. +The debugger is extensible -- it is actually defined as the class :class:`Pdb`. This is currently undocumented but easily understood by reading the source. The -extension interface uses the modules :mod:`bdb` (undocumented) and :mod:`cmd`. +extension interface uses the modules :mod:`bdb` and :mod:`cmd`. The debugger's prompt is ``(Pdb)``. Typical usage to run a program under control of the debugger is:: @@ -44,19 +44,23 @@ example:: python3 -m pdb myscript.py When invoked as a script, pdb will automatically enter post-mortem debugging if -the program being debugged exits abnormally. After post-mortem debugging (or -after normal exit of the program), pdb will restart the program. Automatic +the program being debugged exits abnormally. After post-mortem debugging (or +after normal exit of the program), pdb will restart the program. Automatic restarting preserves pdb's state (such as breakpoints) and in most cases is more useful than quitting the debugger upon program's exit. +.. versionadded:: 3.2 + :file:`pdb.py` now accepts a ``-c`` option that executes commands as if given + in a :file:`.pdbrc` file, see :ref:`debugger-commands`. + The typical usage to break into the debugger from a running program is to insert :: import pdb; pdb.set_trace() at the location you want to break into the debugger. You can then step through -the code following this statement, and continue running without the debugger using -the ``c`` command. +the code following this statement, and continue running without the debugger +using the :pdbcmd:`continue` command. The typical usage to inspect a crashed program is:: @@ -79,29 +83,31 @@ The typical usage to inspect a crashed program is:: The module defines the following functions; each enters the debugger in a slightly different way: -.. function:: run(statement[, globals[, locals]]) +.. function:: run(statement, globals=None, locals=None) - Execute the *statement* (given as a string) under debugger control. The - debugger prompt appears before any code is executed; you can set breakpoints and - type ``continue``, or you can step through the statement using ``step`` or - ``next`` (all these commands are explained below). The optional *globals* and - *locals* arguments specify the environment in which the code is executed; by - default the dictionary of the module :mod:`__main__` is used. (See the - explanation of the built-in :func:`exec` or :func:`eval` functions.) + Execute the *statement* (given as a string or a code object) under debugger + control. The debugger prompt appears before any code is executed; you can + set breakpoints and type :pdbcmd:`continue`, or you can step through the + statement using :pdbcmd:`step` or :pdbcmd:`next` (all these commands are + explained below). The optional *globals* and *locals* arguments specify the + environment in which the code is executed; by default the dictionary of the + module :mod:`__main__` is used. (See the explanation of the built-in + :func:`exec` or :func:`eval` functions.) -.. function:: runeval(expression[, globals[, locals]]) +.. function:: runeval(expression, globals=None, locals=None) - Evaluate the *expression* (given as a string) under debugger control. When - :func:`runeval` returns, it returns the value of the expression. Otherwise this - function is similar to :func:`run`. + Evaluate the *expression* (given as a string or a code object) under debugger + control. When :func:`runeval` returns, it returns the value of the + expression. Otherwise this function is similar to :func:`run`. -.. function:: runcall(function[, argument, ...]) +.. function:: runcall(function, *args, **kwds) - Call the *function* (a function or method object, not a string) with the given - arguments. When :func:`runcall` returns, it returns whatever the function call - returned. The debugger prompt appears as soon as the function is entered. + Call the *function* (a function or method object, not a string) with the + given arguments. When :func:`runcall` returns, it returns whatever the + function call returned. The debugger prompt appears as soon as the function + is entered. .. function:: set_trace() @@ -111,7 +117,7 @@ slightly different way: being debugged (e.g. when an assertion fails). -.. function:: post_mortem([traceback]) +.. function:: post_mortem(traceback=None) Enter post-mortem debugging of the given *traceback* object. If no *traceback* is given, it uses the one of the exception that is currently @@ -129,7 +135,8 @@ The ``run*`` functions and :func:`set_trace` are aliases for instantiating the :class:`Pdb` class and calling the method of the same name. If you want to access further features, you have to do this yourself: -.. class:: Pdb(completekey='tab', stdin=None, stdout=None, skip=None) +.. class:: Pdb(completekey='tab', stdin=None, stdout=None, skip=None, \ + nosigint=False) :class:`Pdb` is the debugger class. @@ -140,6 +147,11 @@ access further features, you have to do this yourself: patterns. The debugger will not step into frames that originate in a module that matches one of these patterns. [1]_ + By default, Pdb sets a handler for the SIGINT signal (which is sent when the + user presses Ctrl-C on the console) when you give a ``continue`` command. + This allows you to break into the debugger again by pressing Ctrl-C. If you + want Pdb not to touch the SIGINT handler, set *nosigint* tot true. + Example call to enable tracing with *skip*:: import pdb; pdb.Pdb(skip=['django.*']).set_trace() @@ -147,9 +159,13 @@ access further features, you have to do this yourself: .. versionadded:: 3.1 The *skip* argument. - .. method:: run(statement[, globals[, locals]]) - runeval(expression[, globals[, locals]]) - runcall(function[, argument, ...]) + .. versionadded:: 3.2 + The *nosigint* argument. Previously, a SIGINT handler was never set by + Pdb. + + .. method:: run(statement, globals=None, locals=None) + runeval(expression, globals=None, locals=None) + runcall(function, *args, **kwds) set_trace() See the documentation for the functions explained above. @@ -160,16 +176,17 @@ access further features, you have to do this yourself: Debugger Commands ----------------- -The debugger recognizes the following commands. Most commands can be -abbreviated to one or two letters; e.g. ``h(elp)`` means that either ``h`` or -``help`` can be used to enter the help command (but not ``he`` or ``hel``, nor -``H`` or ``Help`` or ``HELP``). Arguments to commands must be separated by -whitespace (spaces or tabs). Optional arguments are enclosed in square brackets -(``[]``) in the command syntax; the square brackets must not be typed. -Alternatives in the command syntax are separated by a vertical bar (``|``). +The commands recognized by the debugger are listed below. Most commands can be +abbreviated to one or two letters as indicated; e.g. ``h(elp)`` means that +either ``h`` or ``help`` can be used to enter the help command (but not ``he`` +or ``hel``, nor ``H`` or ``Help`` or ``HELP``). Arguments to commands must be +separated by whitespace (spaces or tabs). Optional arguments are enclosed in +square brackets (``[]``) in the command syntax; the square brackets must not be +typed. Alternatives in the command syntax are separated by a vertical bar +(``|``). Entering a blank line repeats the last command entered. Exception: if the last -command was a ``list`` command, the next 11 lines are listed. +command was a :pdbcmd:`list` command, the next 11 lines are listed. Commands that the debugger doesn't recognize are assumed to be Python statements and are executed in the context of the program being debugged. Python @@ -179,92 +196,112 @@ change a variable or call a function. When an exception occurs in such a statement, the exception name is printed but the debugger's state is not changed. +The debugger supports :ref:`aliases <debugger-aliases>`. Aliases can have +parameters which allows one a certain level of adaptability to the context under +examination. + Multiple commands may be entered on a single line, separated by ``;;``. (A single ``;`` is not used as it is the separator for multiple commands in a line -that is passed to the Python parser.) No intelligence is applied to separating +that is passed to the Python parser.) No intelligence is applied to separating the commands; the input is split at the first ``;;`` pair, even if it is in the middle of a quoted string. -The debugger supports aliases. Aliases can have parameters which allows one a -certain level of adaptability to the context under examination. - .. index:: pair: .pdbrc; file triple: debugger; configuration; file -If a file :file:`.pdbrc` exists in the user's home directory or in the current +If a file :file:`.pdbrc` exists in the user's home directory or in the current directory, it is read in and executed as if it had been typed at the debugger -prompt. This is particularly useful for aliases. If both files exist, the one +prompt. This is particularly useful for aliases. If both files exist, the one in the home directory is read first and aliases defined there can be overridden by the local file. -h(elp) [*command*] +.. versionchanged:: 3.2 + :file:`.pdbrc` can now contain commands that continue debugging, such as + :pdbcmd:`continue` or :pdbcmd:`next`. Previously, these commands had no + effect. + + +.. pdbcommand:: h(elp) [command] + Without argument, print the list of available commands. With a *command* as argument, print help about that command. ``help pdb`` displays the full - documentation file; if the environment variable :envvar:`PAGER` is defined, the - file is piped through that command instead. Since the *command* argument must - be an identifier, ``help exec`` must be entered to get help on the ``!`` - command. + documentation (the docstring of the :mod:`pdb` module). Since the *command* + argument must be an identifier, ``help exec`` must be entered to get help on + the ``!`` command. + +.. pdbcommand:: w(here) -w(here) Print a stack trace, with the most recent frame at the bottom. An arrow indicates the current frame, which determines the context of most commands. -d(own) - Move the current frame one level down in the stack trace (to a newer frame). +.. pdbcommand:: d(own) [count] + + Move the current frame *count* (default one) levels down in the stack trace + (to a newer frame). + +.. pdbcommand:: u(p) [count] + + Move the current frame *count* (default one) levels up in the stack trace (to + an older frame). -u(p) - Move the current frame one level up in the stack trace (to an older frame). +.. pdbcommand:: b(reak) [([filename:]lineno | function) [, condition]] -b(reak) [[*filename*:]\ *lineno* | *function*\ [, *condition*]] With a *lineno* argument, set a break there in the current file. With a - *function* argument, set a break at the first executable statement within that - function. The line number may be prefixed with a filename and a colon, to - specify a breakpoint in another file (probably one that hasn't been loaded yet). - The file is searched on ``sys.path``. Note that each breakpoint is assigned a - number to which all the other breakpoint commands refer. + *function* argument, set a break at the first executable statement within + that function. The line number may be prefixed with a filename and a colon, + to specify a breakpoint in another file (probably one that hasn't been loaded + yet). The file is searched on :data:`sys.path`. Note that each breakpoint + is assigned a number to which all the other breakpoint commands refer. - If a second argument is present, it is an expression which must evaluate to true - before the breakpoint is honored. + If a second argument is present, it is an expression which must evaluate to + true before the breakpoint is honored. - Without argument, list all breaks, including for each breakpoint, the number of - times that breakpoint has been hit, the current ignore count, and the associated - condition if any. + Without argument, list all breaks, including for each breakpoint, the number + of times that breakpoint has been hit, the current ignore count, and the + associated condition if any. -tbreak [[*filename*:]\ *lineno* | *function*\ [, *condition*]] - Temporary breakpoint, which is removed automatically when it is first hit. The - arguments are the same as break. +.. pdbcommand:: tbreak [([filename:]lineno | function) [, condition]] + + Temporary breakpoint, which is removed automatically when it is first hit. + The arguments are the same as for :pdbcmd:`break`. + +.. pdbcommand:: cl(ear) [filename:lineno | bpnumber [bpnumber ...]] -cl(ear) [*filename:lineno* | *bpnumber* [*bpnumber ...*]] With a *filename:lineno* argument, clear all the breakpoints at this line. With a space separated list of breakpoint numbers, clear those breakpoints. Without argument, clear all breaks (but first ask confirmation). -disable [*bpnumber* [*bpnumber ...*]] - Disables the breakpoints given as a space separated list of breakpoint numbers. - Disabling a breakpoint means it cannot cause the program to stop execution, but - unlike clearing a breakpoint, it remains in the list of breakpoints and can be - (re-)enabled. +.. pdbcommand:: disable [bpnumber [bpnumber ...]] + + Disable the breakpoints given as a space separated list of breakpoint + numbers. Disabling a breakpoint means it cannot cause the program to stop + execution, but unlike clearing a breakpoint, it remains in the list of + breakpoints and can be (re-)enabled. -enable [*bpnumber* [*bpnumber ...*]] - Enables the breakpoints specified. +.. pdbcommand:: enable [bpnumber [bpnumber ...]] -ignore *bpnumber* [*count*] - Sets the ignore count for the given breakpoint number. If count is omitted, the - ignore count is set to 0. A breakpoint becomes active when the ignore count is - zero. When non-zero, the count is decremented each time the breakpoint is - reached and the breakpoint is not disabled and any associated condition - evaluates to true. + Enable the breakpoints specified. -condition *bpnumber* [*condition*] - Condition is an expression which must evaluate to true before the breakpoint is - honored. If condition is absent, any existing condition is removed; i.e., the - breakpoint is made unconditional. +.. pdbcommand:: ignore bpnumber [count] + + Set the ignore count for the given breakpoint number. If count is omitted, + the ignore count is set to 0. A breakpoint becomes active when the ignore + count is zero. When non-zero, the count is decremented each time the + breakpoint is reached and the breakpoint is not disabled and any associated + condition evaluates to true. + +.. pdbcommand:: condition bpnumber [condition] + + Set a new *condition* for the breakpoint, an expression which must evaluate + to true before the breakpoint is honored. If *condition* is absent, any + existing condition is removed; i.e., the breakpoint is made unconditional. + +.. pdbcommand:: commands [bpnumber] -commands [*bpnumber*] Specify a list of commands for breakpoint number *bpnumber*. The commands - themselves appear on the following lines. Type a line containing just 'end' to - terminate the commands. An example:: + themselves appear on the following lines. Type a line containing just + ``end`` to terminate the commands. An example:: (Pdb) commands 1 (com) print some_variable @@ -272,12 +309,12 @@ commands [*bpnumber*] (Pdb) To remove all commands from a breakpoint, type commands and follow it - immediately with end; that is, give no commands. + immediately with ``end``; that is, give no commands. With no *bpnumber* argument, commands refers to the last breakpoint set. - You can use breakpoint commands to start your program up again. Simply use the - continue command, or step, or any other command that resumes execution. + You can use breakpoint commands to start your program up again. Simply use + the continue command, or step, or any other command that resumes execution. Specifying any command resuming execution (currently continue, step, next, return, jump, quit and their abbreviations) terminates the command list (as if @@ -291,91 +328,169 @@ commands [*bpnumber*] that are to print a specific message and then continue. If none of the other commands print anything, you see no sign that the breakpoint was reached. -s(tep) +.. pdbcommand:: s(tep) + Execute the current line, stop at the first possible occasion (either in a function that is called or on the next line in the current function). -n(ext) - Continue execution until the next line in the current function is reached or it - returns. (The difference between ``next`` and ``step`` is that ``step`` stops - inside a called function, while ``next`` executes called functions at (nearly) - full speed, only stopping at the next line in the current function.) +.. pdbcommand:: n(ext) + + Continue execution until the next line in the current function is reached or + it returns. (The difference between :pdbcmd:`next` and :pdbcmd:`step` is + that :pdbcmd:`step` stops inside a called function, while :pdbcmd:`next` + executes called functions at (nearly) full speed, only stopping at the next + line in the current function.) + +.. pdbcommand:: unt(il) [lineno] -unt(il) - Continue execution until the line with the line number greater than the - current one is reached or when returning from current frame. + Without argument, continue execution until the line with a number greater + than the current one is reached. + + With a line number, continue execution until a line with a number greater or + equal to that is reached. In both cases, also stop when the current frame + returns. + + .. versionchanged:: 3.2 + Allow giving an explicit line number. + +.. pdbcommand:: r(eturn) -r(eturn) Continue execution until the current function returns. -c(ont(inue)) +.. pdbcommand:: c(ont(inue)) + Continue execution, only stop when a breakpoint is encountered. -j(ump) *lineno* +.. pdbcommand:: j(ump) lineno + Set the next line that will be executed. Only available in the bottom-most - frame. This lets you jump back and execute code again, or jump forward to skip - code that you don't want to run. + frame. This lets you jump back and execute code again, or jump forward to + skip code that you don't want to run. - It should be noted that not all jumps are allowed --- for instance it is not + It should be noted that not all jumps are allowed -- for instance it is not possible to jump into the middle of a :keyword:`for` loop or out of a :keyword:`finally` clause. -l(ist) [*first*\ [, *last*]] - List source code for the current file. Without arguments, list 11 lines around - the current line or continue the previous listing. With one argument, list 11 - lines around at that line. With two arguments, list the given range; if the - second argument is less than the first, it is interpreted as a count. +.. pdbcommand:: l(ist) [first[, last]] + + List source code for the current file. Without arguments, list 11 lines + around the current line or continue the previous listing. With ``.`` as + argument, list 11 lines around the current line. With one argument, + list 11 lines around at that line. With two arguments, list the given range; + if the second argument is less than the first, it is interpreted as a count. + + The current line in the current frame is indicated by ``->``. If an + exception is being debugged, the line where the exception was originally + raised or propagated is indicated by ``>>``, if it differs from the current + line. + + .. versionadded:: 3.2 + The ``>>`` marker. + +.. pdbcommand:: ll | longlist + + List all source code for the current function or frame. Interesting lines + are marked as for :pdbcmd:`list`. + + .. versionadded:: 3.2 + +.. pdbcommand:: a(rgs) -a(rgs) Print the argument list of the current function. -p(rint) *expression* +.. pdbcommand:: p(rint) expression + Evaluate the *expression* in the current context and print its value. -pp *expression* - Like the ``p`` command, except the value of the expression is pretty-printed - using the :mod:`pprint` module. +.. pdbcommand:: pp expression + + Like the :pdbcmd:`print` command, except the value of the expression is + pretty-printed using the :mod:`pprint` module. + +.. pdbcommand:: whatis expression + + Print the type of the *expression*. + +.. pdbcommand:: source expression + + Try to get source code for the given object and display it. + + .. versionadded:: 3.2 + +.. pdbcommand:: display [expression] + + Display the value of the expression if it changed, each time execution stops + in the current frame. -alias [*name* [command]] - Creates an alias called *name* that executes *command*. The command must *not* - be enclosed in quotes. Replaceable parameters can be indicated by ``%1``, - ``%2``, and so on, while ``%*`` is replaced by all the parameters. If no - command is given, the current alias for *name* is shown. If no arguments are - given, all aliases are listed. + Without expression, list all display expressions for the current frame. - Aliases may be nested and can contain anything that can be legally typed at the - pdb prompt. Note that internal pdb commands *can* be overridden by aliases. - Such a command is then hidden until the alias is removed. Aliasing is - recursively applied to the first word of the command line; all other words in - the line are left alone. + .. versionadded:: 3.2 + +.. pdbcommand:: undisplay [expression] + + Do not display the expression any more in the current frame. Without + expression, clear all display expressions for the current frame. + + .. versionadded:: 3.2 + +.. pdbcommand:: interact + + Start an interative interpreter (using the :mod:`code` module) whose global + namespace contains all the (global and local) names found in the current + scope. + + .. versionadded:: 3.2 + +.. _debugger-aliases: + +.. pdbcommand:: alias [name [command]] + + Create an alias called *name* that executes *command*. The command must + *not* be enclosed in quotes. Replaceable parameters can be indicated by + ``%1``, ``%2``, and so on, while ``%*`` is replaced by all the parameters. + If no command is given, the current alias for *name* is shown. If no + arguments are given, all aliases are listed. + + Aliases may be nested and can contain anything that can be legally typed at + the pdb prompt. Note that internal pdb commands *can* be overridden by + aliases. Such a command is then hidden until the alias is removed. Aliasing + is recursively applied to the first word of the command line; all other words + in the line are left alone. As an example, here are two useful aliases (especially when placed in the :file:`.pdbrc` file):: - #Print instance variables (usage "pi classInst") + # Print instance variables (usage "pi classInst") alias pi for k in %1.__dict__.keys(): print("%1.",k,"=",%1.__dict__[k]) - #Print instance variables in self + # Print instance variables in self alias ps pi self -unalias *name* - Deletes the specified alias. +.. pdbcommand:: unalias name + + Delete the specified alias. + +.. pdbcommand:: ! statement -[!]\ *statement* Execute the (one-line) *statement* in the context of the current stack frame. The exclamation point can be omitted unless the first word of the statement - resembles a debugger command. To set a global variable, you can prefix the - assignment command with a ``global`` command on the same line, e.g.:: + resembles a debugger command. To set a global variable, you can prefix the + assignment command with a :keyword:`global` statement on the same line, + e.g.:: (Pdb) global list_options; list_options = ['-l'] (Pdb) -run [*args* ...] - Restart the debugged Python program. If an argument is supplied, it is split - with "shlex" and the result is used as the new sys.argv. History, breakpoints, - actions and debugger options are preserved. "restart" is an alias for "run". +.. pdbcommand:: run [args ...] + restart [args ...] + + Restart the debugged Python program. If an argument is supplied, it is split + with :mod:`shlex` and the result is used as the new :data:`sys.argv`. + History, breakpoints, actions and debugger options are preserved. + :pdbcmd:`restart` is an alias for :pdbcmd:`run`. + +.. pdbcommand:: q(uit) -q(uit) - Quit from the debugger. The program being executed is aborted. + Quit from the debugger. The program being executed is aborted. .. rubric:: Footnotes diff --git a/Doc/library/persistence.rst b/Doc/library/persistence.rst index b90b2e1..d5bb193 100644 --- a/Doc/library/persistence.rst +++ b/Doc/library/persistence.rst @@ -1,4 +1,3 @@ - .. _persistence: **************** diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index 1850b00..73f0611 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -141,7 +141,7 @@ an unpickler, then you call the unpickler's :meth:`load` method. The The :mod:`pickle` module provides the following functions to make the pickling process more convenient: -.. function:: dump(obj, file[, protocol, \*, fix_imports=True]) +.. function:: dump(obj, file, protocol=None, \*, fix_imports=True) Write a pickled representation of *obj* to the open :term:`file object` *file*. This is equivalent to ``Pickler(file, protocol).dump(obj)``. @@ -163,7 +163,7 @@ process more convenient: map the new Python 3.x names to the old module names used in Python 2.x, so that the pickle data stream is readable with Python 2.x. -.. function:: dumps(obj[, protocol, \*, fix_imports=True]) +.. function:: dumps(obj, protocol=None, \*, fix_imports=True) Return the pickled representation of the object as a :class:`bytes` object, instead of writing it to a file. @@ -180,7 +180,7 @@ process more convenient: map the new Python 3.x names to the old module names used in Python 2.x, so that the pickle data stream is readable with Python 2.x. -.. function:: load(file, [\*, fix_imports=True, encoding="ASCII", errors="strict"]) +.. function:: load(file, \*, fix_imports=True, encoding="ASCII", errors="strict") Read a pickled object representation from the open :term:`file object` *file* and return the reconstituted object hierarchy specified therein. This is @@ -203,7 +203,7 @@ process more convenient: *errors* tell pickle how to decode 8-bit string instances pickled by Python 2.x; these default to 'ASCII' and 'strict', respectively. -.. function:: loads(bytes_object, [\*, fix_imports=True, encoding="ASCII", errors="strict"]) +.. function:: loads(bytes_object, \*, fix_imports=True, encoding="ASCII", errors="strict") Read a pickled object hierarchy from a :class:`bytes` object and return the reconstituted object hierarchy specified therein @@ -248,7 +248,7 @@ The :mod:`pickle` module defines three exceptions: The :mod:`pickle` module exports two classes, :class:`Pickler` and :class:`Unpickler`: -.. class:: Pickler(file[, protocol, \*, fix_imports=True]) +.. class:: Pickler(file, protocol=None, \*, fix_imports=True) This takes a binary file for writing a pickle data stream. @@ -296,7 +296,7 @@ The :mod:`pickle` module exports two classes, :class:`Pickler` and Use :func:`pickletools.optimize` if you need more compact pickles. -.. class:: Unpickler(file, [\*, fix_imports=True, encoding="ASCII", errors="strict"]) +.. class:: Unpickler(file, \*, fix_imports=True, encoding="ASCII", errors="strict") This takes a binary file for reading a pickle data stream. diff --git a/Doc/library/pickletools.rst b/Doc/library/pickletools.rst index 88ecbab..4c0a148 100644 --- a/Doc/library/pickletools.rst +++ b/Doc/library/pickletools.rst @@ -2,7 +2,13 @@ ================================================== .. module:: pickletools - :synopsis: Contains extensive comments about the pickle protocols and pickle-machine opcodes, as well as some useful functions. + :synopsis: Contains extensive comments about the pickle protocols and + pickle-machine opcodes, as well as some useful functions. + +**Source code:** :source:`Lib/pickletools.py` + +-------------- + This module contains various constants relating to the intimate details of the :mod:`pickle` module, some lengthy comments about the implementation, and a @@ -11,15 +17,81 @@ are useful for Python core developers who are working on the :mod:`pickle`; ordinary users of the :mod:`pickle` module probably won't find the :mod:`pickletools` module relevant. +Command line usage +------------------ + +.. versionadded:: 3.2 + +When invoked from the command line, ``python -m pickletools`` will +disassemble the contents of one or more pickle files. Note that if +you want to see the Python object stored in the pickle rather than the +details of pickle format, you may want to use ``-m pickle`` instead. +However, when the pickle file that you want to examine comes from an +untrusted source, ``-m pickletools`` is a safer option because it does +not execute pickle bytecode. + +For example, with a tuple ``(1, 2)`` pickled in file ``x.pickle``:: + + $ python -m pickle x.pickle + (1, 2) + + $ python -m pickletools x.pickle + 0: \x80 PROTO 3 + 2: K BININT1 1 + 4: K BININT1 2 + 6: \x86 TUPLE2 + 7: q BINPUT 0 + 9: . STOP + highest protocol among opcodes = 2 + +Command line options +^^^^^^^^^^^^^^^^^^^^ + +.. program:: pickletools + +.. cmdoption:: -a, --annotate + + Annotate each line with a short opcode description. + +.. cmdoption:: -o, --output=<file> + + Name of a file where the output should be written. + +.. cmdoption:: -l, --indentlevel=<num> + + The number of blanks by which to indent a new MARK level. + +.. cmdoption:: -m, --memo + + When multiple objects are disassembled, preserve memo between + disassemblies. + +.. cmdoption:: -p, --preamble=<preamble> + + When more than one pickle file are specified, print given preamble + before each disassembly. + + + +Programmatic Interface +---------------------- + + +.. function:: dis(pickle, out=None, memo=None, indentlevel=4, annotate=0) -.. function:: dis(pickle[, out=None, memo=None, indentlevel=4]) + Outputs a symbolic disassembly of the pickle to the file-like + object *out*, defaulting to ``sys.stdout``. *pickle* can be a + string or a file-like object. *memo* can be a Python dictionary + that will be used as the pickle's memo; it can be used to perform + disassemblies across multiple pickles created by the same + pickler. Successive levels, indicated by ``MARK`` opcodes in the + stream, are indented by *indentlevel* spaces. If a nonzero value + is given to *annotate*, each opcode in the output is annotated with + a short description. The value of *annotate* is used as a hint for + the column where annotation should start. - Outputs a symbolic disassembly of the pickle to the file-like object *out*, - defaulting to ``sys.stdout``. *pickle* can be a string or a file-like object. - *memo* can be a Python dictionary that will be used as the pickle's memo; it can - be used to perform disassemblies across multiple pickles created by the same - pickler. Successive levels, indicated by ``MARK`` opcodes in the stream, are - indented by *indentlevel* spaces. + .. versionadded:: 3.2 + The *annotate* argument. .. function:: genops(pickle) diff --git a/Doc/library/pipes.rst b/Doc/library/pipes.rst index 1f2b2ff..016a720 100644 --- a/Doc/library/pipes.rst +++ b/Doc/library/pipes.rst @@ -1,4 +1,3 @@ - :mod:`pipes` --- Interface to shell pipelines ============================================= @@ -7,6 +6,9 @@ :synopsis: A Python interface to Unix shell pipelines. .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/pipes.py` + +-------------- The :mod:`pipes` module defines a class to abstract the concept of a *pipeline* --- a sequence of converters from one file to another. diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 5a59f28..3118ff2 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -1,10 +1,13 @@ - :mod:`pkgutil` --- Package extension utility ============================================ .. module:: pkgutil :synopsis: Utilities for the import system. +**Source code:** :source:`Lib/pkgutil.py` + +-------------- + This module provides utilities for the import system, in particular package support. @@ -168,10 +171,10 @@ support. Get a resource from a package. - This is a wrapper for the PEP 302 loader :func:`get_data` API. The package - argument should be the name of a package, in standard module format - (foo.bar). The resource argument should be in the form of a relative - filename, using ``/`` as the path separator. The parent directory name + This is a wrapper for the :pep:`302` loader :func:`get_data` API. The + *package* argument should be the name of a package, in standard module format + (``foo.bar``). The *resource* argument should be in the form of a relative + filename, using ``/`` as the path separator. The parent directory name ``..`` is not allowed, and nor is a rooted name (starting with a ``/``). The function returns a binary string that is the contents of the specified @@ -183,5 +186,5 @@ support. d = os.path.dirname(sys.modules[package].__file__) data = open(os.path.join(d, resource), 'rb').read() - If the package cannot be located or loaded, or it uses a PEP 302 loader + If the package cannot be located or loaded, or it uses a :pep:`302` loader which does not support :func:`get_data`, then ``None`` is returned. diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index a71d533..b0431d9 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Marc-Andre Lemburg <mal@egenix.com> .. sectionauthor:: Bjorn Pettersen <bpettersen@corp.fairisaac.com> +**Source code:** :source:`Lib/platform.py` + +-------------- .. note:: @@ -27,8 +30,8 @@ Cross Platform returned as strings. Values that cannot be determined are returned as given by the parameter presets. - If bits is given as ``''``, the :cfunc:`sizeof(pointer)` (or - :cfunc:`sizeof(long)` on Python version < 1.5.2) is used as indicator for the + If bits is given as ``''``, the :c:func:`sizeof(pointer)` (or + :c:func:`sizeof(long)` on Python version < 1.5.2) is used as indicator for the supported pointer size. The function relies on the system's :file:`file` command to do the actual work. @@ -225,7 +228,7 @@ Mac OS Platform Entries which cannot be determined are set to ``''``. All tuple entries are strings. - Documentation for the underlying :cfunc:`gestalt` API is available online at + Documentation for the underlying :c:func:`gestalt` API is available online at http://www.rgaros.nl/gestalt/. diff --git a/Doc/library/plistlib.rst b/Doc/library/plistlib.rst index 36c9eb3..ae5e94d 100644 --- a/Doc/library/plistlib.rst +++ b/Doc/library/plistlib.rst @@ -11,6 +11,10 @@ pair: plist; file single: property list +**Source code:** :source:`Lib/plistlib.py` + +-------------- + This module provides an interface for reading and writing the "property list" XML files used mainly by Mac OS X. diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst index 0b2c033..d11d937 100644 --- a/Doc/library/poplib.rst +++ b/Doc/library/poplib.rst @@ -1,4 +1,3 @@ - :mod:`poplib` --- POP3 protocol client ====================================== @@ -9,6 +8,10 @@ .. index:: pair: POP3; protocol +**Source code:** :source:`Lib/poplib.py` + +-------------- + This module defines a class, :class:`POP3`, which encapsulates a connection to a POP3 server and implements the protocol as defined in :rfc:`1725`. The :class:`POP3` class supports both the minimal and optional command sets. @@ -24,7 +27,7 @@ mailserver supports IMAP, you would be better off using the A single class is provided by the :mod:`poplib` module: -.. class:: POP3(host[, port[, timeout]]) +.. class:: POP3(host, port=POP3_PORT[, timeout]) This class implements the actual POP3 protocol. The connection is created when the instance is initialized. If *port* is omitted, the standard POP3 port (110) @@ -33,12 +36,19 @@ A single class is provided by the :mod:`poplib` module: be used). -.. class:: POP3_SSL(host[, port[, keyfile[, certfile]]]) +.. class:: POP3_SSL(host, port=POP3_SSL_PORT, keyfile=None, certfile=None, timeout=None, context=None) This is a subclass of :class:`POP3` that connects to the server over an SSL encrypted socket. If *port* is not specified, 995, the standard POP3-over-SSL port is used. *keyfile* and *certfile* are also optional - they can contain a PEM formatted private key and certificate chain file for the SSL connection. + *timeout* works as in the :class:`POP3` constructor. *context* parameter is a + :class:`ssl.SSLContext` object which allows bundling SSL configuration + options, certificates and private keys into a single (potentially long-lived) + structure. + + .. versionchanged:: 3.2 + *context* parameter added. One exception is defined as an attribute of the :mod:`poplib` module: @@ -160,7 +170,7 @@ An :class:`POP3` instance has the following methods: POP3 servers you will use before trusting it. -.. method:: POP3.uidl([which]) +.. method:: POP3.uidl(which=None) Return message digest (unique id) list. If *which* is specified, result contains the unique id for that message in the form ``'response mesgnum uid``, otherwise diff --git a/Doc/library/posix.rst b/Doc/library/posix.rst index c33d9e5..07db2b2 100644 --- a/Doc/library/posix.rst +++ b/Doc/library/posix.rst @@ -38,13 +38,13 @@ Large File Support Several operating systems (including AIX, HP-UX, Irix and Solaris) provide support for files that are larger than 2 GB from a C programming model where -:ctype:`int` and :ctype:`long` are 32-bit values. This is typically accomplished +:c:type:`int` and :c:type:`long` are 32-bit values. This is typically accomplished by defining the relevant size and offset types as 64-bit values. Such files are sometimes referred to as :dfn:`large files`. -Large file support is enabled in Python when the size of an :ctype:`off_t` is -larger than a :ctype:`long` and the :ctype:`long long` type is available and is -at least as large as an :ctype:`off_t`. +Large file support is enabled in Python when the size of an :c:type:`off_t` is +larger than a :c:type:`long` and the :c:type:`long long` type is available and is +at least as large as an :c:type:`off_t`. It may be necessary to configure and compile Python with certain compiler flags to enable this mode. For example, it is enabled by default with recent versions of Irix, but with Solaris 2.6 and 2.7 you need to do something like:: @@ -69,17 +69,22 @@ In addition to many functions described in the :mod:`os` module documentation, .. data:: environ A dictionary representing the string environment at the time the interpreter - was started. For example, ``environ['HOME']`` is the pathname of your home - directory, equivalent to ``getenv("HOME")`` in C. + was started. Keys and values are bytes on Unix and str on Windows. For + example, ``environ[b'HOME']`` (``environ['HOME']`` on Windows) is the + pathname of your home directory, equivalent to ``getenv("HOME")`` in C. Modifying this dictionary does not affect the string environment passed on by :func:`execv`, :func:`popen` or :func:`system`; if you need to change the environment, pass ``environ`` to :func:`execve` or add variable assignments and export statements to the command string for :func:`system` or :func:`popen`. + .. versionchanged:: 3.2 + On Unix, keys and values are bytes. + .. note:: - The :mod:`os` module provides an alternate implementation of ``environ`` which - updates the environment on modification. Note also that updating ``os.environ`` - will render this dictionary obsolete. Use of the :mod:`os` module version of - this is recommended over direct access to the :mod:`posix` module. + The :mod:`os` module provides an alternate implementation of ``environ`` + which updates the environment on modification. Note also that updating + :data:`os.environ` will render this dictionary obsolete. Use of the + :mod:`os` module version of this is recommended over direct access to the + :mod:`posix` module. diff --git a/Doc/library/pprint.rst b/Doc/library/pprint.rst index d1d1bae..9ab12ee 100644 --- a/Doc/library/pprint.rst +++ b/Doc/library/pprint.rst @@ -1,4 +1,3 @@ - :mod:`pprint` --- Data pretty printer ===================================== @@ -7,6 +6,9 @@ .. moduleauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/pprint.py` + +-------------- The :mod:`pprint` module provides a capability to "pretty-print" arbitrary Python data structures in a form which can be used as input to the interpreter. @@ -27,7 +29,7 @@ The :mod:`pprint` module defines one class: .. First the implementation class: -.. class:: PrettyPrinter(...) +.. class:: PrettyPrinter(indent=1, width=80, depth=None, stream=None) Construct a :class:`PrettyPrinter` instance. This constructor understands several keyword parameters. An output stream may be set using the *stream* @@ -62,21 +64,20 @@ The :mod:`pprint` module defines one class: >>> pp.pprint(tup) ('spam', ('eggs', ('lumberjack', ('knights', ('ni', ('dead', (...))))))) -The :class:`PrettyPrinter` class supports several derivative functions: -.. Now the derivative functions: +The :class:`PrettyPrinter` class supports several derivative functions: -.. function:: pformat(object[, indent[, width[, depth]]]) +.. function:: pformat(object, indent=1, width=80, depth=None) Return the formatted representation of *object* as a string. *indent*, *width* and *depth* will be passed to the :class:`PrettyPrinter` constructor as formatting parameters. -.. function:: pprint(object[, stream[, indent[, width[, depth]]]]) +.. function:: pprint(object, stream=None, indent=1, width=80, depth=None) Prints the formatted representation of *object* on *stream*, followed by a - newline. If *stream* is omitted, ``sys.stdout`` is used. This may be used + newline. If *stream* is ``None``, ``sys.stdout`` is used. This may be used in the interactive interpreter instead of the :func:`print` function for inspecting values (you can even reassign ``print = pprint.pprint`` for use within a scope). *indent*, *width* and *depth* will be passed to the @@ -191,7 +192,8 @@ are converted to strings. The default implementation uses the internals of the pprint Example -------------- -This example demonstrates several uses of the :func:`pprint` function and its parameters. +This example demonstrates several uses of the :func:`pprint` function and its +parameters. >>> import pprint >>> tup = ('spam', ('eggs', ('lumberjack', ('knights', ('ni', ('dead', diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst index 82cc2eb..3931836 100644 --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -1,4 +1,3 @@ - .. _profile: ******************** @@ -10,6 +9,9 @@ The Python Profilers .. module:: profile :synopsis: Python source profiler. +**Source code:** :source:`Lib/profile.py` and :source:`Lib/pstats.py` + +-------------- .. _profiler-introduction: @@ -219,7 +221,7 @@ discussion of how to derive "better" profilers from the classes presented, or reading the source code for these modules. -.. function:: run(command[, filename]) +.. function:: run(command, filename=None, sort=-1) This function takes a single argument that can be passed to the :func:`exec` function, and an optional file name. In all cases this routine attempts to @@ -248,8 +250,8 @@ reading the source code for these modules. for the number of calls, tottime - for the total time spent in the given function (and excluding time made in calls - to sub-functions), + for the total time spent in the given function (and excluding time made in + calls to sub-functions), percall is the quotient of ``tottime`` divided by ``ncalls`` @@ -269,24 +271,25 @@ reading the source code for these modules. calls. Note that when the function does not recurse, these two values are the same, and only the single figure is printed. + If *sort* is given, it can be one of ``'stdname'`` (sort by filename:lineno), + ``'calls'`` (sort by number of calls), ``'time'`` (sort by total time) or + ``'cumulative'`` (sort by cumulative time). The default is ``'stdname'``. + -.. function:: runctx(command, globals, locals[, filename]) +.. function:: runctx(command, globals, locals, filename=None) This function is similar to :func:`run`, with added arguments to supply the globals and locals dictionaries for the *command* string. -Analysis of the profiler data is done using the :class:`Stats` class. - -.. note:: - The :class:`Stats` class is defined in the :mod:`pstats` module. +Analysis of the profiler data is done using the :class:`pstats.Stats` class. .. module:: pstats :synopsis: Statistics object for use with the profiler. -.. class:: Stats(filename[, stream=sys.stdout[, ...]]) +.. class:: Stats(*filenames, stream=sys.stdout) This class constructor creates an instance of a "statistics object" from a *filename* (or set of filenames). :class:`Stats` objects are manipulated by @@ -326,7 +329,7 @@ The :class:`Stats` Class accumulated into a single entry. -.. method:: Stats.add(filename[, ...]) +.. method:: Stats.add(*filenames) This method of the :class:`Stats` class accumulates additional profiling information into the current profiling object. Its arguments should refer to @@ -343,7 +346,7 @@ The :class:`Stats` Class :class:`profile.Profile` and :class:`cProfile.Profile` classes. -.. method:: Stats.sort_stats(key[, ...]) +.. method:: Stats.sort_stats(*keys) This method modifies the :class:`Stats` object by sorting it according to the supplied criteria. The argument is typically a string identifying the basis of @@ -410,7 +413,7 @@ The :class:`Stats` Class .. This method is provided primarily for compatibility with the old profiler. -.. method:: Stats.print_stats([restriction, ...]) +.. method:: Stats.print_stats(*restrictions) This method for the :class:`Stats` class prints out a report as described in the :func:`profile.run` definition. @@ -439,7 +442,7 @@ The :class:`Stats` Class then proceed to only print the first 10% of them. -.. method:: Stats.print_callers([restriction, ...]) +.. method:: Stats.print_callers(*restrictions) This method for the :class:`Stats` class prints a list of all functions that called each function in the profiled database. The ordering is identical to @@ -457,7 +460,7 @@ The :class:`Stats` Class the current function while it was invoked by this specific caller. -.. method:: Stats.print_callees([restriction, ...]) +.. method:: Stats.print_callees(*restrictions) This method for the :class:`Stats` class prints a list of all function that were called by the indicated function. Aside from this reversal of direction of diff --git a/Doc/library/pty.rst b/Doc/library/pty.rst index be879f2..2b9385b 100644 --- a/Doc/library/pty.rst +++ b/Doc/library/pty.rst @@ -1,4 +1,3 @@ - :mod:`pty` --- Pseudo-terminal utilities ======================================== @@ -46,3 +45,50 @@ The :mod:`pty` module defines the following functions: a file descriptor. The defaults try to read 1024 bytes each time they are called. + +Example +------- + +.. sectionauthor:: Steen Lumholt + +The following program acts like the Unix command :manpage:`script(1)`, using a +pseudo-terminal to record all input and output of a terminal session in a +"typescript". :: + + import sys, os, time, getopt + import pty + + mode = 'wb' + shell = 'sh' + filename = 'typescript' + if 'SHELL' in os.environ: + shell = os.environ['SHELL'] + + try: + opts, args = getopt.getopt(sys.argv[1:], 'ap') + except getopt.error as msg: + print('%s: %s' % (sys.argv[0], msg)) + sys.exit(2) + + for opt, arg in opts: + # option -a: append to typescript file + if opt == '-a': + mode = 'ab' + # option -p: use a Python shell as the terminal command + elif opt == '-p': + shell = sys.executable + if args: + filename = args[0] + + script = open(filename, mode) + + def read(fd): + data = os.read(fd, 1024) + script.write(data) + return data + + sys.stdout.write('Script started, file is %s\n' % filename) + script.write(('Script started on %s\n' % time.asctime()).encode()) + pty.spawn(shell, read) + script.write(('Script done on %s\n' % time.asctime()).encode()) + sys.stdout.write('Script done, file is %s\n' % filename) diff --git a/Doc/library/pwd.rst b/Doc/library/pwd.rst index 562afd9..2c17d9e 100644 --- a/Doc/library/pwd.rst +++ b/Doc/library/pwd.rst @@ -1,4 +1,3 @@ - :mod:`pwd` --- The password database ==================================== diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst index 0891862..07ddc25 100644 --- a/Doc/library/py_compile.rst +++ b/Doc/library/py_compile.rst @@ -8,6 +8,10 @@ .. index:: pair: file; byte-code +**Source code:** :source:`Lib/py_compile.py` + +-------------- + The :mod:`py_compile` module provides a function to generate a byte-code file from a source file, and another function used when the module source file is invoked as a script. @@ -22,24 +26,42 @@ byte-code cache files in the directory containing the source code. Exception raised when an error occurs while attempting to compile the file. -.. function:: compile(file[, cfile[, dfile[, doraise]]]) +.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1) + + Compile a source file to byte-code and write out the byte-code cache file. + The source code is loaded from the file name *file*. The byte-code is + written to *cfile*, which defaults to the :PEP:`3147` path, ending in + ``.pyc`` (``.pyo`` if optimization is enabled in the current interpreter). + For example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to + ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. If *dfile* is + specified, it is used as the name of the source file in error messages when + instead of *file*. If *doraise* is true, a :exc:`PyCompileError` is raised + when an error is encountered while compiling *file*. If *doraise* is false + (the default), an error string is written to ``sys.stderr``, but no exception + is raised. This function returns the path to byte-compiled file, i.e. + whatever *cfile* value was used. - Compile a source file to byte-code and write out the byte-code cache file. The - source code is loaded from the file name *file*. The byte-code is written to - *cfile*, which defaults to *file* ``+`` ``'c'`` (``'o'`` if optimization is - enabled in the current interpreter). If *dfile* is specified, it is used as the - name of the source file in error messages instead of *file*. If *doraise* is - true, a :exc:`PyCompileError` is raised when an error is encountered while - compiling *file*. If *doraise* is false (the default), an error string is - written to ``sys.stderr``, but no exception is raised. + *optimize* controls the optimization level and is passed to the built-in + :func:`compile` function. The default of ``-1`` selects the optimization + level of the current interpreter. + .. versionchanged:: 3.2 + Changed default value of *cfile* to be :PEP:`3147`-compliant. Previous + default was *file* + ``'c'`` (``'o'`` if optimization was enabled). + Also added the *optimize* parameter. -.. function:: main([args]) + +.. function:: main(args=None) Compile several source files. The files named in *args* (or on the command - line, if *args* is not specified) are compiled and the resulting bytecode is + line, if *args* is ``None``) are compiled and the resulting bytecode is cached in the normal manner. This function does not search a directory structure to locate source files; it only compiles files named explicitly. + If ``'-'`` is the only parameter in args, the list of files is taken from + standard input. + + .. versionchanged:: 3.2 + Added support for ``'-'``. When this module is run as a script, the :func:`main` is used to compile all the files named on the command line. The exit status is nonzero if one of the files diff --git a/Doc/library/pyclbr.rst b/Doc/library/pyclbr.rst index 36b46f4..d4a76a6 100644 --- a/Doc/library/pyclbr.rst +++ b/Doc/library/pyclbr.rst @@ -1,4 +1,3 @@ - :mod:`pyclbr` --- Python class browser support ============================================== @@ -6,6 +5,9 @@ :synopsis: Supports information extraction for a Python class browser. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/pyclbr.py` + +-------------- The :mod:`pyclbr` module can be used to determine some limited information about the classes, methods and top-level functions defined in a module. The @@ -17,7 +19,7 @@ not implemented in Python, including all standard and optional extension modules. -.. function:: readmodule(module[, path=None]) +.. function:: readmodule(module, path=None) Read a module and return a dictionary mapping class names to class descriptor objects. The parameter *module* should be the name of a @@ -26,7 +28,7 @@ modules. of ``sys.path``, which is used to locate module source code. -.. function:: readmodule_ex(module[, path=None]) +.. function:: readmodule_ex(module, path=None) Like :func:`readmodule`, but the returned dictionary, in addition to mapping class names to class descriptor objects, also maps top-level diff --git a/Doc/library/pydoc.rst b/Doc/library/pydoc.rst index 01f48b1..e100865 100644 --- a/Doc/library/pydoc.rst +++ b/Doc/library/pydoc.rst @@ -1,4 +1,3 @@ - :mod:`pydoc` --- Documentation generator and online help system =============================================================== @@ -13,6 +12,10 @@ single: documentation; online single: help; online +**Source code:** :source:`Lib/pydoc.py` + +-------------- + The :mod:`pydoc` module automatically generates documentation from Python modules. The documentation can be presented as pages of text on the console, served to a Web browser, or saved to HTML files. @@ -51,12 +54,21 @@ manner similar to the Unix :program:`man` command. The synopsis line of a module is the first line of its documentation string. You can also use :program:`pydoc` to start an HTTP server on the local machine -that will serve documentation to visiting Web browsers. :program:`pydoc -p 1234` -will start a HTTP server on port 1234, allowing you to browse -the documentation at ``http://localhost:1234/`` in your preferred Web browser. +that will serve documentation to visiting Web browsers. :program:`pydoc -p 1234` +will start a HTTP server on port 1234, allowing you to browse the +documentation at ``http://localhost:1234/`` in your preferred Web browser. +Specifying ``0`` as the port number will select an arbitrary unused port. + :program:`pydoc -g` will start the server and additionally bring up a small :mod:`tkinter`\ -based graphical interface to help you search for -documentation pages. +documentation pages. The ``-g`` option is deprecated, since the server can +now be controlled directly from HTTP clients. + +:program:`pydoc -b` will start the server and additionally open a web +browser to a module index page. Each served page has a navigation bar at the +top where you can *Get* help on an individual item, *Search* all modules with a +keyword in their synopsis line, and go to the *Module index*, *Topics* and +*Keywords* pages. When :program:`pydoc` generates documentation, it uses the current environment and path to locate modules. Thus, invoking :program:`pydoc spam` @@ -70,3 +82,5 @@ be overridden by setting the :envvar:`PYTHONDOCS` environment variable to a different URL or to a local directory containing the Library Reference Manual pages. +.. versionchanged:: 3.2 + Added the ``-b`` option, deprecated the ``-g`` option. diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst index 1eb9db1..a648cfa 100644 --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -1,4 +1,3 @@ - :mod:`xml.parsers.expat` --- Fast XML parsing using Expat ========================================================= @@ -56,7 +55,7 @@ The :mod:`xml.parsers.expat` module contains two functions: Returns an explanatory string for a given error number *errno*. -.. function:: ParserCreate([encoding[, namespace_separator]]) +.. function:: ParserCreate(encoding=None, namespace_separator=None) Creates and returns a new :class:`xmlparser` object. *encoding*, if specified, must be a string naming the encoding used by the XML data. Expat doesn't @@ -177,7 +176,7 @@ XMLParser Objects This method can only be called before the :meth:`Parse` or :meth:`ParseFile` methods are called; calling it after either of those have been called causes :exc:`ExpatError` to be raised with the :attr:`code` attribute set to - :const:`errors.XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING`. + ``errors.codes[errors.XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING]``. :class:`xmlparser` objects have the following attributes: @@ -475,8 +474,21 @@ ExpatError Exceptions .. attribute:: ExpatError.code - Expat's internal error number for the specific error. This will match one of - the constants defined in the ``errors`` object from this module. + Expat's internal error number for the specific error. The + :data:`errors.messages` dictionary maps these error numbers to Expat's error + messages. For example:: + + from xml.parsers.expat import ParserCreate, ExpatError, errors + + p = ParserCreate() + try: + p.Parse(some_xml_document) + except ExpatError as err: + print("Error:", errors.messages[err.code]) + + The :mod:`errors` module also provides error message constants and a + dictionary :data:`~errors.codes` mapping these messages back to the error + codes, see below. .. attribute:: ExpatError.lineno @@ -538,15 +550,16 @@ The output from this program is:: Content Model Descriptions -------------------------- -.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +.. module:: xml.parsers.expat.model +.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> Content modules are described using nested tuples. Each tuple contains four values: the type, the quantifier, the name, and a tuple of children. Children are simply additional content module descriptions. -The values of the first two fields are constants defined in the ``model`` object -of the :mod:`xml.parsers.expat` module. These constants can be collected in two +The values of the first two fields are constants defined in the +:mod:`xml.parsers.expat.model` module. These constants can be collected in two groups: the model type group and the quantifier group. The constants in the model type group are: @@ -618,143 +631,139 @@ The constants in the quantifier group are: Expat error constants --------------------- -The following constants are provided in the ``errors`` object of the -:mod:`xml.parsers.expat` module. These constants are useful in interpreting -some of the attributes of the :exc:`ExpatError` exception objects raised when an -error has occurred. +.. module:: xml.parsers.expat.errors + +The following constants are provided in the :mod:`xml.parsers.expat.errors` +module. These constants are useful in interpreting some of the attributes of +the :exc:`ExpatError` exception objects raised when an error has occurred. +Since for backwards compatibility reasons, the constants' value is the error +*message* and not the numeric error *code*, you do this by comparing its +:attr:`code` attribute with +:samp:`errors.codes[errors.XML_ERROR_{CONSTANT_NAME}]`. + +The ``errors`` module has the following attributes: + +.. data:: codes + + A dictionary mapping numeric error codes to their string descriptions. + + .. versionadded:: 3.2 + + +.. data:: messages -The ``errors`` object has the following attributes: + A dictionary mapping string descriptions to their error codes. + + .. versionadded:: 3.2 .. data:: XML_ERROR_ASYNC_ENTITY - :noindex: .. data:: XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF - :noindex: An entity reference in an attribute value referred to an external entity instead of an internal entity. .. data:: XML_ERROR_BAD_CHAR_REF - :noindex: A character reference referred to a character which is illegal in XML (for example, character ``0``, or '``�``'). .. data:: XML_ERROR_BINARY_ENTITY_REF - :noindex: An entity reference referred to an entity which was declared with a notation, so cannot be parsed. .. data:: XML_ERROR_DUPLICATE_ATTRIBUTE - :noindex: An attribute was used more than once in a start tag. .. data:: XML_ERROR_INCORRECT_ENCODING - :noindex: .. data:: XML_ERROR_INVALID_TOKEN - :noindex: Raised when an input byte could not properly be assigned to a character; for example, a NUL byte (value ``0``) in a UTF-8 input stream. .. data:: XML_ERROR_JUNK_AFTER_DOC_ELEMENT - :noindex: Something other than whitespace occurred after the document element. .. data:: XML_ERROR_MISPLACED_XML_PI - :noindex: An XML declaration was found somewhere other than the start of the input data. .. data:: XML_ERROR_NO_ELEMENTS - :noindex: The document contains no elements (XML requires all documents to contain exactly one top-level element).. .. data:: XML_ERROR_NO_MEMORY - :noindex: Expat was not able to allocate memory internally. .. data:: XML_ERROR_PARAM_ENTITY_REF - :noindex: A parameter entity reference was found where it was not allowed. .. data:: XML_ERROR_PARTIAL_CHAR - :noindex: An incomplete character was found in the input. .. data:: XML_ERROR_RECURSIVE_ENTITY_REF - :noindex: An entity reference contained another reference to the same entity; possibly via a different name, and possibly indirectly. .. data:: XML_ERROR_SYNTAX - :noindex: Some unspecified syntax error was encountered. .. data:: XML_ERROR_TAG_MISMATCH - :noindex: An end tag did not match the innermost open start tag. .. data:: XML_ERROR_UNCLOSED_TOKEN - :noindex: Some token (such as a start tag) was not closed before the end of the stream or the next token was encountered. .. data:: XML_ERROR_UNDEFINED_ENTITY - :noindex: A reference was made to a entity which was not defined. .. data:: XML_ERROR_UNKNOWN_ENCODING - :noindex: The document encoding is not supported by Expat. .. data:: XML_ERROR_UNCLOSED_CDATA_SECTION - :noindex: A CDATA marked section was not closed. .. data:: XML_ERROR_EXTERNAL_ENTITY_HANDLING - :noindex: .. data:: XML_ERROR_NOT_STANDALONE - :noindex: The parser determined that the document was not "standalone" though it declared itself to be in the XML declaration, and the :attr:`NotStandaloneHandler` was @@ -762,15 +771,12 @@ The ``errors`` object has the following attributes: .. data:: XML_ERROR_UNEXPECTED_STATE - :noindex: .. data:: XML_ERROR_ENTITY_DECLARED_IN_PE - :noindex: .. data:: XML_ERROR_FEATURE_REQUIRES_XML_DTD - :noindex: An operation was requested that requires DTD support to be compiled in, but Expat was configured without DTD support. This should never be reported by a @@ -778,7 +784,6 @@ The ``errors`` object has the following attributes: .. data:: XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING - :noindex: A behavioral change was requested after parsing started that can only be changed before parsing has started. This is (currently) only raised by @@ -786,63 +791,53 @@ The ``errors`` object has the following attributes: .. data:: XML_ERROR_UNBOUND_PREFIX - :noindex: An undeclared prefix was found when namespace processing was enabled. .. data:: XML_ERROR_UNDECLARING_PREFIX - :noindex: The document attempted to remove the namespace declaration associated with a prefix. .. data:: XML_ERROR_INCOMPLETE_PE - :noindex: A parameter entity contained incomplete markup. .. data:: XML_ERROR_XML_DECL - :noindex: The document contained no document element at all. .. data:: XML_ERROR_TEXT_DECL - :noindex: There was an error parsing a text declaration in an external entity. .. data:: XML_ERROR_PUBLICID - :noindex: Characters were found in the public id that are not allowed. .. data:: XML_ERROR_SUSPENDED - :noindex: The requested operation was made on a suspended parser, but isn't allowed. This includes attempts to provide additional input or to stop the parser. .. data:: XML_ERROR_NOT_SUSPENDED - :noindex: An attempt to resume the parser was made when the parser had not been suspended. .. data:: XML_ERROR_ABORTED - :noindex: This should not be reported to Python applications. .. data:: XML_ERROR_FINISHED - :noindex: The requested operation was made on a parser which was finished parsing input, but isn't allowed. This includes attempts to provide additional input or to @@ -850,7 +845,6 @@ The ``errors`` object has the following attributes: .. data:: XML_ERROR_SUSPEND_PE - :noindex: .. rubric:: Footnotes diff --git a/Doc/library/python.rst b/Doc/library/python.rst index 7d4d827..b67fbfc 100644 --- a/Doc/library/python.rst +++ b/Doc/library/python.rst @@ -1,4 +1,3 @@ - .. _python: *********************** @@ -13,6 +12,7 @@ overview: .. toctree:: sys.rst + sysconfig.rst builtins.rst __main__.rst warnings.rst @@ -25,3 +25,4 @@ overview: inspect.rst site.rst fpectl.rst + distutils.rst diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index e16d6e9..26336ef 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -1,10 +1,12 @@ - :mod:`queue` --- A synchronized queue class =========================================== .. module:: queue :synopsis: A synchronized queue class. +**Source code:** :source:`Lib/queue.py` + +-------------- The :mod:`queue` module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be @@ -20,17 +22,17 @@ the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the :mod:`heapq` module) and the lowest valued entry is retrieved first. + The :mod:`queue` module defines the following classes and exceptions: -.. class:: Queue(maxsize) +.. class:: Queue(maxsize=0) Constructor for a FIFO queue. *maxsize* is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If *maxsize* is less than or equal to zero, the queue size is infinite. - -.. class:: LifoQueue(maxsize) +.. class:: LifoQueue(maxsize=0) Constructor for a LIFO queue. *maxsize* is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will @@ -38,7 +40,7 @@ The :mod:`queue` module defines the following classes and exceptions: *maxsize* is less than or equal to zero, the queue size is infinite. -.. class:: PriorityQueue(maxsize) +.. class:: PriorityQueue(maxsize=0) Constructor for a priority queue. *maxsize* is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will @@ -61,12 +63,6 @@ The :mod:`queue` module defines the following classes and exceptions: Exception raised when non-blocking :meth:`put` (or :meth:`put_nowait`) is called on a :class:`Queue` object which is full. -.. seealso:: - - :class:`collections.deque` is an alternative implementation of unbounded - queues with fast atomic :func:`append` and :func:`popleft` operations that - do not require locking. - .. _queueobjects: @@ -100,7 +96,7 @@ provide the public methods described below. guarantee that a subsequent call to put() will not block. -.. method:: Queue.put(item[, block[, timeout]]) +.. method:: Queue.put(item, block=True, timeout=None) Put *item* into the queue. If optional args *block* is true and *timeout* is None (the default), block if necessary until a free slot is available. If @@ -116,7 +112,7 @@ provide the public methods described below. Equivalent to ``put(item, False)``. -.. method:: Queue.get([block[, timeout]]) +.. method:: Queue.get(block=True, timeout=None) Remove and return an item from the queue. If optional args *block* is true and *timeout* is None (the default), block if necessary until an item is available. @@ -177,3 +173,14 @@ Example of how to wait for enqueued tasks to be completed:: q.join() # block until all tasks are done + +.. seealso:: + + Class :class:`multiprocessing.Queue` + A queue class for use in a multi-processing (rather than multi-threading) + context. + + :class:`collections.deque` is an alternative implementation of unbounded + queues with fast atomic :func:`append` and :func:`popleft` operations that + do not require locking. + diff --git a/Doc/library/quopri.rst b/Doc/library/quopri.rst index a64337e..755811a 100644 --- a/Doc/library/quopri.rst +++ b/Doc/library/quopri.rst @@ -1,4 +1,3 @@ - :mod:`quopri` --- Encode and decode MIME quoted-printable data ============================================================== @@ -10,6 +9,10 @@ pair: quoted-printable; encoding single: MIME; quoted-printable encoding +**Source code:** :source:`Lib/quopri.py` + +-------------- + This module performs quoted-printable transport encoding and decoding, as defined in :rfc:`1521`: "MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies". @@ -18,8 +21,7 @@ few nonprintable characters; the base64 encoding scheme available via the :mod:`base64` module is more compact if there are many such characters, as when sending a graphics file. - -.. function:: decode(input, output[,header]) +.. function:: decode(input, output, header=False) Decode the contents of the *input* file and write the resulting decoded binary data to the *output* file. *input* and *output* must be :term:`file objects @@ -30,7 +32,7 @@ sending a graphics file. Part Two: Message Header Extensions for Non-ASCII Text". -.. function:: encode(input, output, quotetabs) +.. function:: encode(input, output, quotetabs, header=False) Encode the contents of the *input* file and write the resulting quoted-printable data to the *output* file. *input* and *output* must be :term:`file objects @@ -38,24 +40,24 @@ sending a graphics file. empty string. *quotetabs* is a flag which controls whether to encode embedded spaces and tabs; when true it encodes such embedded whitespace, and when false it leaves them unencoded. Note that spaces and tabs appearing at the - end of lines are always encoded, as per :rfc:`1521`. + end of lines are always encoded, as per :rfc:`1521`. *header* is a flag + which controls if spaces are encoded as underscores as per :rfc:`1522`. -.. function:: decodestring(s[,header]) +.. function:: decodestring(s, header=False) Like :func:`decode`, except that it accepts a source string and returns the corresponding decoded string. -.. function:: encodestring(s[, quotetabs]) +.. function:: encodestring(s, quotetabs=False, header=False) Like :func:`encode`, except that it accepts a source string and returns the - corresponding encoded string. *quotetabs* is optional (defaulting to 0), and is - passed straight through to :func:`encode`. + corresponding encoded string. *quotetabs* and *header* are optional + (defaulting to ``False``), and are passed straight through to :func:`encode`. .. seealso:: Module :mod:`base64` Encode and decode MIME base64 data - diff --git a/Doc/library/random.rst b/Doc/library/random.rst index 2976f5e..f0c4add 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -1,17 +1,20 @@ - :mod:`random` --- Generate pseudo-random numbers ================================================ .. module:: random :synopsis: Generate pseudo-random numbers with various common distributions. +**Source code:** :source:`Lib/random.py` + +-------------- This module implements pseudo-random number generators for various distributions. -For integers, uniform selection from a range. For sequences, uniform selection -of a random element, a function to generate a random permutation of a list -in-place, and a function for random sampling without replacement. +For integers, there is uniform selection from a range. For sequences, there is +uniform selection of a random element, a function to generate a random +permutation of a list in-place, and a function for random sampling without +replacement. On the real line, there are functions to compute uniform, normal (Gaussian), lognormal, negative exponential, gamma, and beta distributions. For generating @@ -36,22 +39,30 @@ basic generator of your own devising: in that case, override the :meth:`random`, Optionally, a new generator can supply a :meth:`getrandbits` method --- this allows :meth:`randrange` to produce selections over an arbitrarily large range. +The :mod:`random` module also provides the :class:`SystemRandom` class which +uses the system function :func:`os.urandom` to generate random numbers +from sources provided by the operating system. + Bookkeeping functions: +.. function:: seed([x], version=2) -.. function:: seed([x]) + Initialize the random number generator. - Initialize the basic random number generator. Optional argument *x* can be any - :term:`hashable` object. If *x* is omitted or ``None``, current system time is used; - current system time is also used to initialize the generator when the module is - first imported. If randomness sources are provided by the operating system, - they are used instead of the system time (see the :func:`os.urandom` function - for details on availability). + If *x* is omitted or ``None``, the current system time is used. If + randomness sources are provided by the operating system, they are used + instead of the system time (see the :func:`os.urandom` function for details + on availability). - If *x* is not ``None`` or an int, ``hash(x)`` is used instead. If *x* is an - int, *x* is used directly. + If *x* is an int, it is used directly. + With version 2 (the default), a :class:`str`, :class:`bytes`, or :class:`bytearray` + object gets converted to an :class:`int` and all of its bits are used. With version 1, + the :func:`hash` of *x* is used instead. + + .. versionchanged:: 3.2 + Moved to the version 2 scheme which uses all of the bits in a string seed. .. function:: getstate() @@ -82,6 +93,13 @@ Functions for integers: equivalent to ``choice(range(start, stop, step))``, but doesn't actually build a range object. + The positional argument pattern matches that of :func:`range`. Keyword arguments + should not be used because the function may use them in unexpected ways. + + .. versionchanged:: 3.2 + :meth:`randrange` is more sophisticated about producing equally distributed + values. Formerly it used a style like ``int(random()*n)`` which could produce + slightly uneven distributions. .. function:: randint(a, b) @@ -213,29 +231,67 @@ be found in any statistics text. parameter. -Alternative Generators: +Alternative Generator: .. class:: SystemRandom([seed]) Class that uses the :func:`os.urandom` function for generating random numbers from sources provided by the operating system. Not available on all systems. - Does not rely on software state and sequences are not reproducible. Accordingly, + Does not rely on software state, and sequences are not reproducible. Accordingly, the :meth:`seed` method has no effect and is ignored. The :meth:`getstate` and :meth:`setstate` methods raise :exc:`NotImplementedError` if called. -Examples of basic usage:: +.. seealso:: + + M. Matsumoto and T. Nishimura, "Mersenne Twister: A 623-dimensionally + equidistributed uniform pseudorandom number generator", ACM Transactions on + Modeling and Computer Simulation Vol. 8, No. 1, January pp.3-30 1998. + + + `Complementary-Multiply-with-Carry recipe + <http://code.activestate.com/recipes/576707/>`_ for a compatible alternative + random number generator with a long period and comparatively simple update + operations. + + +Notes on Reproducibility +------------------------ + +Sometimes it is useful to be able to reproduce the sequences given by a pseudo +random number generator. By re-using a seed value, the same sequence should be +reproducible from run to run as long as multiple threads are not running. - >>> random.random() # Random float x, 0.0 <= x < 1.0 +Most of the random module's algorithms and seeding functions are subject to +change across Python versions, but two aspects are guaranteed not to change: + +* If a new seeding method is added, then a backward compatible seeder will be + offered. + +* The generator's :meth:`random` method will continue to produce the same + sequence when the compatible seeder is given the same seed. + +.. _random-examples: + +Examples and Recipes +-------------------- + +Basic usage:: + + >>> random.random() # Random float x, 0.0 <= x < 1.0 0.37444887175646646 - >>> random.uniform(1, 10) # Random float x, 1.0 <= x < 10.0 + + >>> random.uniform(1, 10) # Random float x, 1.0 <= x < 10.0 1.1800146073117523 - >>> random.randint(1, 10) # Integer from 1 to 10, endpoints included + + >>> random.randrange(10) # Integer from 0 to 9 7 - >>> random.randrange(0, 101, 2) # Even integer from 0 to 100 + + >>> random.randrange(0, 101, 2) # Even integer from 0 to 100 26 - >>> random.choice('abcdefghij') # Choose a random element + + >>> random.choice('abcdefghij') # Single random element 'c' >>> items = [1, 2, 3, 4, 5, 6, 7] @@ -243,19 +299,25 @@ Examples of basic usage:: >>> items [7, 3, 2, 5, 6, 4, 1] - >>> random.sample([1, 2, 3, 4, 5], 3) # Choose 3 elements + >>> random.sample([1, 2, 3, 4, 5], 3) # Three samples without replacement [4, 1, 5] +A common task is to make a :func:`random.choice` with weighted probababilites. +If the weights are small integer ratios, a simple technique is to build a sample +population with repeats:: -.. seealso:: - - M. Matsumoto and T. Nishimura, "Mersenne Twister: A 623-dimensionally - equidistributed uniform pseudorandom number generator", ACM Transactions on - Modeling and Computer Simulation Vol. 8, No. 1, January pp.3-30 1998. + >>> weighted_choices = [('Red', 3), ('Blue', 2), ('Yellow', 1), ('Green', 4)] + >>> population = [val for val, cnt in weighted_choices for i in range(cnt)] + >>> random.choice(population) + 'Green' +A more general approach is to arrange the weights in a cumulative distribution +with :func:`itertools.accumulate`, and then locate the random value with +:func:`bisect.bisect`:: - `Complementary-Multiply-with-Carry recipe - <http://code.activestate.com/recipes/576707/>`_ for a compatible alternative - random number generator with a long period and comparatively simple update - operations. + >>> choices, weights = zip(*weighted_choices) + >>> cumdist = list(itertools.accumulate(weights)) + >>> x = random.random() * cumdist[-1] + >>> choices[bisect.bisect(cumdist, x)] + 'Blue' diff --git a/Doc/library/re.rst b/Doc/library/re.rst index 1d90f79..3e9cf02 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -1,4 +1,3 @@ - :mod:`re` --- Regular expression operations =========================================== @@ -34,8 +33,9 @@ newline. Usually patterns will be expressed in Python code using this raw string notation. It is important to note that most regular expression operations are available as -module-level functions and :class:`RegexObject` methods. The functions are -shortcuts that don't require you to compile a regex object first, but miss some +module-level functions and methods on +:ref:`compiled regular expressions <re-objects>`. The functions are shortcuts +that don't require you to compile a regex object first, but miss some fine-tuning parameters. .. seealso:: @@ -339,11 +339,12 @@ the second character. For example, ``\$`` matches the character ``'$'``. ``\d`` For Unicode (str) patterns: - Matches any Unicode digit (which includes ``[0-9]``, and also many - other digit characters). If the :const:`ASCII` flag is used only - ``[0-9]`` is matched (but the flag affects the entire regular - expression, so in such cases using an explicit ``[0-9]`` may be a - better choice). + Matches any Unicode decimal digit (that is, any character in + Unicode character category [Nd]). This includes ``[0-9]``, and + also many other digit characters. If the :const:`ASCII` flag is + used only ``[0-9]`` is matched (but the flag affects the entire + regular expression, so in such cases using an explicit ``[0-9]`` + may be a better choice). For 8-bit (bytes) patterns: Matches any decimal digit; this is equivalent to ``[0-9]``. @@ -446,7 +447,7 @@ regular expressions. Most non-trivial applications always use the compiled form. -.. function:: compile(pattern[, flags]) +.. function:: compile(pattern, flags=0) Compile a regular expression pattern into a regular expression object, which can be used for matching using its :func:`match` and :func:`search` methods, @@ -547,21 +548,21 @@ form. -.. function:: search(pattern, string[, flags]) +.. function:: search(pattern, string, flags=0) Scan through *string* looking for a location where the regular expression - *pattern* produces a match, and return a corresponding :class:`MatchObject` - instance. Return ``None`` if no position in the string matches the pattern; note - that this is different from finding a zero-length match at some point in the - string. + *pattern* produces a match, and return a corresponding :ref:`match object + <match-objects>`. Return ``None`` if no position in the string matches the + pattern; note that this is different from finding a zero-length match at some + point in the string. -.. function:: match(pattern, string[, flags]) +.. function:: match(pattern, string, flags=0) If zero or more characters at the beginning of *string* match the regular - expression *pattern*, return a corresponding :class:`MatchObject` instance. - Return ``None`` if the string does not match the pattern; note that this is - different from a zero-length match. + expression *pattern*, return a corresponding :ref:`match object + <match-objects>`. Return ``None`` if the string does not match the pattern; + note that this is different from a zero-length match. .. note:: @@ -569,7 +570,7 @@ form. instead. -.. function:: split(pattern, string[, maxsplit=0, flags=0]) +.. function:: split(pattern, string, maxsplit=0, flags=0) Split *string* by the occurrences of *pattern*. If capturing parentheses are used in *pattern*, then the text of all groups in the pattern are also returned @@ -609,7 +610,7 @@ form. Added the optional flags argument. -.. function:: findall(pattern, string[, flags]) +.. function:: findall(pattern, string, flags=0) Return all non-overlapping matches of *pattern* in *string*, as a list of strings. The *string* is scanned left-to-right, and matches are returned in @@ -619,16 +620,16 @@ form. beginning of another match. -.. function:: finditer(pattern, string[, flags]) +.. function:: finditer(pattern, string, flags=0) - Return an :term:`iterator` yielding :class:`MatchObject` instances over all - non-overlapping matches for the RE *pattern* in *string*. The *string* is - scanned left-to-right, and matches are returned in the order found. Empty + Return an :term:`iterator` yielding :ref:`match objects <match-objects>` over + all non-overlapping matches for the RE *pattern* in *string*. The *string* + is scanned left-to-right, and matches are returned in the order found. Empty matches are included in the result unless they touch the beginning of another match. -.. function:: sub(pattern, repl, string[, count, flags]) +.. function:: sub(pattern, repl, string, count=0, flags=0) Return the string obtained by replacing the leftmost non-overlapping occurrences of *pattern* in *string* by the replacement *repl*. If the pattern isn't found, @@ -677,7 +678,7 @@ form. Added the optional flags argument. -.. function:: subn(pattern, repl, string[, count, flags]) +.. function:: subn(pattern, repl, string, count=0, flags=0) Perform the same operation as :func:`sub`, but return a tuple ``(new_string, number_of_subs_made)``. @@ -711,107 +712,107 @@ form. Regular Expression Objects -------------------------- -.. class:: RegexObject - - The :class:`RegexObject` class supports the following methods and attributes: +Compiled regular expression objects support the following methods and +attributes. - .. method:: RegexObject.search(string[, pos[, endpos]]) +.. method:: regex.search(string[, pos[, endpos]]) - Scan through *string* looking for a location where this regular expression - produces a match, and return a corresponding :class:`MatchObject` instance. - Return ``None`` if no position in the string matches the pattern; note that this - is different from finding a zero-length match at some point in the string. + Scan through *string* looking for a location where this regular expression + produces a match, and return a corresponding :ref:`match object + <match-objects>`. Return ``None`` if no position in the string matches the + pattern; note that this is different from finding a zero-length match at some + point in the string. - The optional second parameter *pos* gives an index in the string where the - search is to start; it defaults to ``0``. This is not completely equivalent to - slicing the string; the ``'^'`` pattern character matches at the real beginning - of the string and at positions just after a newline, but not necessarily at the - index where the search is to start. + The optional second parameter *pos* gives an index in the string where the + search is to start; it defaults to ``0``. This is not completely equivalent to + slicing the string; the ``'^'`` pattern character matches at the real beginning + of the string and at positions just after a newline, but not necessarily at the + index where the search is to start. - The optional parameter *endpos* limits how far the string will be searched; it - will be as if the string is *endpos* characters long, so only the characters - from *pos* to ``endpos - 1`` will be searched for a match. If *endpos* is less - than *pos*, no match will be found, otherwise, if *rx* is a compiled regular - expression object, ``rx.search(string, 0, 50)`` is equivalent to - ``rx.search(string[:50], 0)``. + The optional parameter *endpos* limits how far the string will be searched; it + will be as if the string is *endpos* characters long, so only the characters + from *pos* to ``endpos - 1`` will be searched for a match. If *endpos* is less + than *pos*, no match will be found, otherwise, if *rx* is a compiled regular + expression object, ``rx.search(string, 0, 50)`` is equivalent to + ``rx.search(string[:50], 0)``. - >>> pattern = re.compile("d") - >>> pattern.search("dog") # Match at index 0 - <_sre.SRE_Match object at ...> - >>> pattern.search("dog", 1) # No match; search doesn't include the "d" + >>> pattern = re.compile("d") + >>> pattern.search("dog") # Match at index 0 + <_sre.SRE_Match object at ...> + >>> pattern.search("dog", 1) # No match; search doesn't include the "d" - .. method:: RegexObject.match(string[, pos[, endpos]]) +.. method:: regex.match(string[, pos[, endpos]]) - If zero or more characters at the *beginning* of *string* match this regular - expression, return a corresponding :class:`MatchObject` instance. Return - ``None`` if the string does not match the pattern; note that this is different - from a zero-length match. + If zero or more characters at the *beginning* of *string* match this regular + expression, return a corresponding :ref:`match object <match-objects>`. + Return ``None`` if the string does not match the pattern; note that this is + different from a zero-length match. - The optional *pos* and *endpos* parameters have the same meaning as for the - :meth:`~RegexObject.search` method. + The optional *pos* and *endpos* parameters have the same meaning as for the + :meth:`~regex.search` method. - .. note:: + .. note:: - If you want to locate a match anywhere in *string*, use - :meth:`~RegexObject.search` instead. + If you want to locate a match anywhere in *string*, use + :meth:`~regex.search` instead. - >>> pattern = re.compile("o") - >>> pattern.match("dog") # No match as "o" is not at the start of "dog". - >>> pattern.match("dog", 1) # Match as "o" is the 2nd character of "dog". - <_sre.SRE_Match object at ...> + >>> pattern = re.compile("o") + >>> pattern.match("dog") # No match as "o" is not at the start of "dog". + >>> pattern.match("dog", 1) # Match as "o" is the 2nd character of "dog". + <_sre.SRE_Match object at ...> - .. method:: RegexObject.split(string[, maxsplit=0]) +.. method:: regex.split(string, maxsplit=0) - Identical to the :func:`split` function, using the compiled pattern. + Identical to the :func:`split` function, using the compiled pattern. - .. method:: RegexObject.findall(string[, pos[, endpos]]) +.. method:: regex.findall(string[, pos[, endpos]]) - Similar to the :func:`findall` function, using the compiled pattern, but - also accepts optional *pos* and *endpos* parameters that limit the search - region like for :meth:`match`. + Similar to the :func:`findall` function, using the compiled pattern, but + also accepts optional *pos* and *endpos* parameters that limit the search + region like for :meth:`match`. - .. method:: RegexObject.finditer(string[, pos[, endpos]]) +.. method:: regex.finditer(string[, pos[, endpos]]) - Similar to the :func:`finditer` function, using the compiled pattern, but - also accepts optional *pos* and *endpos* parameters that limit the search - region like for :meth:`match`. + Similar to the :func:`finditer` function, using the compiled pattern, but + also accepts optional *pos* and *endpos* parameters that limit the search + region like for :meth:`match`. - .. method:: RegexObject.sub(repl, string[, count=0]) +.. method:: regex.sub(repl, string, count=0) - Identical to the :func:`sub` function, using the compiled pattern. + Identical to the :func:`sub` function, using the compiled pattern. - .. method:: RegexObject.subn(repl, string[, count=0]) +.. method:: regex.subn(repl, string, count=0) - Identical to the :func:`subn` function, using the compiled pattern. + Identical to the :func:`subn` function, using the compiled pattern. - .. attribute:: RegexObject.flags +.. attribute:: regex.flags - The flags argument used when the RE object was compiled, or ``0`` if no flags - were provided. + The flags argument used when the RE object was compiled, or ``0`` if no flags + were provided. - .. attribute:: RegexObject.groups +.. attribute:: regex.groups - The number of capturing groups in the pattern. + The number of capturing groups in the pattern. - .. attribute:: RegexObject.groupindex +.. attribute:: regex.groupindex - A dictionary mapping any symbolic group names defined by ``(?P<id>)`` to group - numbers. The dictionary is empty if no symbolic groups were used in the - pattern. + A dictionary mapping any symbolic group names defined by ``(?P<id>)`` to group + numbers. The dictionary is empty if no symbolic groups were used in the + pattern. - .. attribute:: RegexObject.pattern +.. attribute:: regex.pattern - The pattern string from which the RE object was compiled. + The pattern string from which the RE object was compiled. .. _match-objects: @@ -819,181 +820,182 @@ Regular Expression Objects Match Objects ------------- -.. class:: MatchObject +Match objects always have a boolean value of :const:`True`, so that you can test +whether e.g. :func:`match` resulted in a match with a simple if statement. They +support the following methods and attributes: - Match Objects always have a boolean value of :const:`True`, so that you can test - whether e.g. :func:`match` resulted in a match with a simple if statement. They - support the following methods and attributes: +.. method:: match.expand(template) - .. method:: MatchObject.expand(template) + Return the string obtained by doing backslash substitution on the template + string *template*, as done by the :meth:`~regex.sub` method. + Escapes such as ``\n`` are converted to the appropriate characters, + and numeric backreferences (``\1``, ``\2``) and named backreferences + (``\g<1>``, ``\g<name>``) are replaced by the contents of the + corresponding group. - Return the string obtained by doing backslash substitution on the template - string *template*, as done by the :meth:`~RegexObject.sub` method. Escapes - such as ``\n`` are converted to the appropriate characters, and numeric - backreferences (``\1``, ``\2``) and named backreferences (``\g<1>``, - ``\g<name>``) are replaced by the contents of the corresponding group. +.. method:: match.group([group1, ...]) - .. method:: MatchObject.group([group1, ...]) + Returns one or more subgroups of the match. If there is a single argument, the + result is a single string; if there are multiple arguments, the result is a + tuple with one item per argument. Without arguments, *group1* defaults to zero + (the whole match is returned). If a *groupN* argument is zero, the corresponding + return value is the entire matching string; if it is in the inclusive range + [1..99], it is the string matching the corresponding parenthesized group. If a + group number is negative or larger than the number of groups defined in the + pattern, an :exc:`IndexError` exception is raised. If a group is contained in a + part of the pattern that did not match, the corresponding result is ``None``. + If a group is contained in a part of the pattern that matched multiple times, + the last match is returned. - Returns one or more subgroups of the match. If there is a single argument, the - result is a single string; if there are multiple arguments, the result is a - tuple with one item per argument. Without arguments, *group1* defaults to zero - (the whole match is returned). If a *groupN* argument is zero, the corresponding - return value is the entire matching string; if it is in the inclusive range - [1..99], it is the string matching the corresponding parenthesized group. If a - group number is negative or larger than the number of groups defined in the - pattern, an :exc:`IndexError` exception is raised. If a group is contained in a - part of the pattern that did not match, the corresponding result is ``None``. - If a group is contained in a part of the pattern that matched multiple times, - the last match is returned. + >>> m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist") + >>> m.group(0) # The entire match + 'Isaac Newton' + >>> m.group(1) # The first parenthesized subgroup. + 'Isaac' + >>> m.group(2) # The second parenthesized subgroup. + 'Newton' + >>> m.group(1, 2) # Multiple arguments give us a tuple. + ('Isaac', 'Newton') - >>> m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist") - >>> m.group(0) # The entire match - 'Isaac Newton' - >>> m.group(1) # The first parenthesized subgroup. - 'Isaac' - >>> m.group(2) # The second parenthesized subgroup. - 'Newton' - >>> m.group(1, 2) # Multiple arguments give us a tuple. - ('Isaac', 'Newton') + If the regular expression uses the ``(?P<name>...)`` syntax, the *groupN* + arguments may also be strings identifying groups by their group name. If a + string argument is not used as a group name in the pattern, an :exc:`IndexError` + exception is raised. - If the regular expression uses the ``(?P<name>...)`` syntax, the *groupN* - arguments may also be strings identifying groups by their group name. If a - string argument is not used as a group name in the pattern, an :exc:`IndexError` - exception is raised. + A moderately complicated example: - A moderately complicated example: + >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds") + >>> m.group('first_name') + 'Malcolm' + >>> m.group('last_name') + 'Reynolds' - >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds") - >>> m.group('first_name') - 'Malcolm' - >>> m.group('last_name') - 'Reynolds' + Named groups can also be referred to by their index: - Named groups can also be referred to by their index: + >>> m.group(1) + 'Malcolm' + >>> m.group(2) + 'Reynolds' - >>> m.group(1) - 'Malcolm' - >>> m.group(2) - 'Reynolds' + If a group matches multiple times, only the last match is accessible: - If a group matches multiple times, only the last match is accessible: + >>> m = re.match(r"(..)+", "a1b2c3") # Matches 3 times. + >>> m.group(1) # Returns only the last match. + 'c3' - >>> m = re.match(r"(..)+", "a1b2c3") # Matches 3 times. - >>> m.group(1) # Returns only the last match. - 'c3' - .. method:: MatchObject.groups([default]) +.. method:: match.groups(default=None) - Return a tuple containing all the subgroups of the match, from 1 up to however - many groups are in the pattern. The *default* argument is used for groups that - did not participate in the match; it defaults to ``None``. + Return a tuple containing all the subgroups of the match, from 1 up to however + many groups are in the pattern. The *default* argument is used for groups that + did not participate in the match; it defaults to ``None``. - For example: + For example: - >>> m = re.match(r"(\d+)\.(\d+)", "24.1632") - >>> m.groups() - ('24', '1632') + >>> m = re.match(r"(\d+)\.(\d+)", "24.1632") + >>> m.groups() + ('24', '1632') - If we make the decimal place and everything after it optional, not all groups - might participate in the match. These groups will default to ``None`` unless - the *default* argument is given: + If we make the decimal place and everything after it optional, not all groups + might participate in the match. These groups will default to ``None`` unless + the *default* argument is given: - >>> m = re.match(r"(\d+)\.?(\d+)?", "24") - >>> m.groups() # Second group defaults to None. - ('24', None) - >>> m.groups('0') # Now, the second group defaults to '0'. - ('24', '0') + >>> m = re.match(r"(\d+)\.?(\d+)?", "24") + >>> m.groups() # Second group defaults to None. + ('24', None) + >>> m.groups('0') # Now, the second group defaults to '0'. + ('24', '0') - .. method:: MatchObject.groupdict([default]) +.. method:: match.groupdict(default=None) - Return a dictionary containing all the *named* subgroups of the match, keyed by - the subgroup name. The *default* argument is used for groups that did not - participate in the match; it defaults to ``None``. For example: + Return a dictionary containing all the *named* subgroups of the match, keyed by + the subgroup name. The *default* argument is used for groups that did not + participate in the match; it defaults to ``None``. For example: - >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds") - >>> m.groupdict() - {'first_name': 'Malcolm', 'last_name': 'Reynolds'} + >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds") + >>> m.groupdict() + {'first_name': 'Malcolm', 'last_name': 'Reynolds'} - .. method:: MatchObject.start([group]) - MatchObject.end([group]) +.. method:: match.start([group]) + match.end([group]) - Return the indices of the start and end of the substring matched by *group*; - *group* defaults to zero (meaning the whole matched substring). Return ``-1`` if - *group* exists but did not contribute to the match. For a match object *m*, and - a group *g* that did contribute to the match, the substring matched by group *g* - (equivalent to ``m.group(g)``) is :: + Return the indices of the start and end of the substring matched by *group*; + *group* defaults to zero (meaning the whole matched substring). Return ``-1`` if + *group* exists but did not contribute to the match. For a match object *m*, and + a group *g* that did contribute to the match, the substring matched by group *g* + (equivalent to ``m.group(g)``) is :: - m.string[m.start(g):m.end(g)] + m.string[m.start(g):m.end(g)] - Note that ``m.start(group)`` will equal ``m.end(group)`` if *group* matched a - null string. For example, after ``m = re.search('b(c?)', 'cba')``, - ``m.start(0)`` is 1, ``m.end(0)`` is 2, ``m.start(1)`` and ``m.end(1)`` are both - 2, and ``m.start(2)`` raises an :exc:`IndexError` exception. + Note that ``m.start(group)`` will equal ``m.end(group)`` if *group* matched a + null string. For example, after ``m = re.search('b(c?)', 'cba')``, + ``m.start(0)`` is 1, ``m.end(0)`` is 2, ``m.start(1)`` and ``m.end(1)`` are both + 2, and ``m.start(2)`` raises an :exc:`IndexError` exception. - An example that will remove *remove_this* from email addresses: + An example that will remove *remove_this* from email addresses: - >>> email = "tony@tiremove_thisger.net" - >>> m = re.search("remove_this", email) - >>> email[:m.start()] + email[m.end():] - 'tony@tiger.net' + >>> email = "tony@tiremove_thisger.net" + >>> m = re.search("remove_this", email) + >>> email[:m.start()] + email[m.end():] + 'tony@tiger.net' - .. method:: MatchObject.span([group]) +.. method:: match.span([group]) - For :class:`MatchObject` *m*, return the 2-tuple ``(m.start(group), - m.end(group))``. Note that if *group* did not contribute to the match, this is - ``(-1, -1)``. *group* defaults to zero, the entire match. + For a match *m*, return the 2-tuple ``(m.start(group), m.end(group))``. Note + that if *group* did not contribute to the match, this is ``(-1, -1)``. + *group* defaults to zero, the entire match. - .. attribute:: MatchObject.pos +.. attribute:: match.pos + + The value of *pos* which was passed to the :meth:`~regex.search` or + :meth:`~regex.match` method of a :ref:`match object <match-objects>`. This + is the index into the string at which the RE engine started looking for a + match. - The value of *pos* which was passed to the :meth:`~RegexObject.search` or - :meth:`~RegexObject.match` method of the :class:`RegexObject`. This is the - index into the string at which the RE engine started looking for a match. +.. attribute:: match.endpos - .. attribute:: MatchObject.endpos + The value of *endpos* which was passed to the :meth:`~regex.search` or + :meth:`~regex.match` method of a :ref:`match object <match-objects>`. This + is the index into the string beyond which the RE engine will not go. - The value of *endpos* which was passed to the :meth:`~RegexObject.search` or - :meth:`~RegexObject.match` method of the :class:`RegexObject`. This is the - index into the string beyond which the RE engine will not go. +.. attribute:: match.lastindex - .. attribute:: MatchObject.lastindex + The integer index of the last matched capturing group, or ``None`` if no group + was matched at all. For example, the expressions ``(a)b``, ``((a)(b))``, and + ``((ab))`` will have ``lastindex == 1`` if applied to the string ``'ab'``, while + the expression ``(a)(b)`` will have ``lastindex == 2``, if applied to the same + string. - The integer index of the last matched capturing group, or ``None`` if no group - was matched at all. For example, the expressions ``(a)b``, ``((a)(b))``, and - ``((ab))`` will have ``lastindex == 1`` if applied to the string ``'ab'``, while - the expression ``(a)(b)`` will have ``lastindex == 2``, if applied to the same - string. +.. attribute:: match.lastgroup - .. attribute:: MatchObject.lastgroup + The name of the last matched capturing group, or ``None`` if the group didn't + have a name, or if no group was matched at all. - The name of the last matched capturing group, or ``None`` if the group didn't - have a name, or if no group was matched at all. +.. attribute:: match.re - .. attribute:: MatchObject.re + The regular expression object whose :meth:`~regex.match` or + :meth:`~regex.search` method produced this match instance. - The regular expression object whose :meth:`~RegexObject.match` or - :meth:`~RegexObject.search` method produced this :class:`MatchObject` - instance. +.. attribute:: match.string - .. attribute:: MatchObject.string + The string passed to :meth:`~regex.match` or :meth:`~regex.search`. - The string passed to :meth:`~RegexObject.match` or - :meth:`~RegexObject.search`. +.. _re-examples: -Examples --------- +Regular Expression Examples +--------------------------- Checking For a Pair @@ -1035,8 +1037,7 @@ To match this with a regular expression, one could use backreferences as such: "<Match: '354aa', groups=('a',)>" To find out what card the pair consists of, one could use the -:meth:`~MatchObject.group` method of :class:`MatchObject` in the following -manner: +:meth:`~match.group` method of the match object in the following manner: .. doctest:: @@ -1059,14 +1060,14 @@ Simulating scanf() .. index:: single: scanf() -Python does not currently have an equivalent to :cfunc:`scanf`. Regular +Python does not currently have an equivalent to :c:func:`scanf`. Regular expressions are generally more powerful, though also more verbose, than -:cfunc:`scanf` format strings. The table below offers some more-or-less -equivalent mappings between :cfunc:`scanf` format tokens and regular +:c:func:`scanf` format strings. The table below offers some more-or-less +equivalent mappings between :c:func:`scanf` format tokens and regular expressions. +--------------------------------+---------------------------------------------+ -| :cfunc:`scanf` Token | Regular Expression | +| :c:func:`scanf` Token | Regular Expression | +================================+=============================================+ | ``%c`` | ``.`` | +--------------------------------+---------------------------------------------+ @@ -1091,7 +1092,7 @@ To extract the filename and numbers from a string like :: /usr/sbin/sendmail - 0 errors, 4 warnings -you would use a :cfunc:`scanf` format like :: +you would use a :c:func:`scanf` format like :: %s - %d errors, %d warnings @@ -1111,7 +1112,7 @@ recursion, you may encounter a :exc:`RuntimeError` exception with the message >>> re.match('Begin (\w| )*? end', s).end() Traceback (most recent call last): File "<stdin>", line 1, in ? - File "/usr/local/lib/python3.1/re.py", line 132, in match + File "/usr/local/lib/python3.2/re.py", line 132, in match return _compile(pattern, flags).match(string) RuntimeError: maximum recursion limit exceeded @@ -1250,10 +1251,10 @@ Finding all Adverbs and their Positions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If one wants more information about all matches of a pattern than the matched -text, :func:`finditer` is useful as it provides instances of -:class:`MatchObject` instead of strings. Continuing with the previous example, -if one was a writer who wanted to find all of the adverbs *and their positions* -in some text, he or she would use :func:`finditer` in the following manner: +text, :func:`finditer` is useful as it provides :ref:`match objects +<match-objects>` instead of strings. Continuing with the previous example, if +one was a writer who wanted to find all of the adverbs *and their positions* in +some text, he or she would use :func:`finditer` in the following manner: >>> text = "He was carefully disguised but captured quickly by police." >>> for m in re.finditer(r"\w+ly", text): @@ -1284,3 +1285,69 @@ functionally identical: <_sre.SRE_Match object at ...> >>> re.match("\\\\", r"\\") <_sre.SRE_Match object at ...> + + +Writing a Tokenizer +^^^^^^^^^^^^^^^^^^^ + +A `tokenizer or scanner <http://en.wikipedia.org/wiki/Lexical_analysis>`_ +analyzes a string to categorize groups of characters. This is a useful first +step in writing a compiler or interpreter. + +The text categories are specified with regular expressions. The technique is +to combine those into a single master regular expression and to loop over +successive matches:: + + Token = collections.namedtuple('Token', 'typ value line column') + + def tokenize(s): + keywords = {'IF', 'THEN', 'FOR', 'NEXT', 'GOSUB', 'RETURN'} + tok_spec = [ + ('NUMBER', r'\d+(\.\d*)?'), # Integer or decimal number + ('ASSIGN', r':='), # Assignment operator + ('END', ';'), # Statement terminator + ('ID', r'[A-Za-z]+'), # Identifiers + ('OP', r'[+*\/\-]'), # Arithmetic operators + ('NEWLINE', r'\n'), # Line endings + ('SKIP', r'[ \t]'), # Skip over spaces and tabs + ] + tok_re = '|'.join('(?P<%s>%s)' % pair for pair in tok_spec) + gettok = re.compile(tok_re).match + line = 1 + pos = line_start = 0 + mo = gettok(s) + while mo is not None: + typ = mo.lastgroup + if typ == 'NEWLINE': + line_start = pos + line += 1 + elif typ != 'SKIP': + if typ == 'ID' and val in keywords: + typ = val + yield Token(typ, mo.group(typ), line, mo.start()-line_start) + pos = mo.end() + mo = gettok(s, pos) + if pos != len(s): + raise RuntimeError('Unexpected character %r on line %d' %(s[pos], line)) + + >>> statements = '''\ + total := total + price * quantity; + tax := price * 0.05; + ''' + >>> for token in tokenize(statements): + ... print(token) + ... + Token(typ='ID', value='total', line=1, column=8) + Token(typ='ASSIGN', value=':=', line=1, column=14) + Token(typ='ID', value='total', line=1, column=17) + Token(typ='OP', value='+', line=1, column=23) + Token(typ='ID', value='price', line=1, column=25) + Token(typ='OP', value='*', line=1, column=31) + Token(typ='ID', value='quantity', line=1, column=33) + Token(typ='END', value=';', line=1, column=41) + Token(typ='ID', value='tax', line=2, column=9) + Token(typ='ASSIGN', value=':=', line=2, column=13) + Token(typ='ID', value='price', line=2, column=16) + Token(typ='OP', value='*', line=2, column=22) + Token(typ='NUMBER', value='0.05', line=2, column=24) + Token(typ='END', value=';', line=2, column=28) diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index ab7b4b6..ab55197 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -1,4 +1,3 @@ - :mod:`readline` --- GNU readline interface ========================================== @@ -25,6 +24,7 @@ function. you can check for the text "libedit" in :const:`readline.__doc__` to differentiate between GNU readline and libedit. + The :mod:`readline` module defines the following functions: @@ -177,7 +177,6 @@ The :mod:`readline` module defines the following functions: Append a line to the history buffer, as if it was the last line typed. - .. seealso:: Module :mod:`rlcompleter` diff --git a/Doc/library/reprlib.rst b/Doc/library/reprlib.rst index 958ead6..0e870da 100644 --- a/Doc/library/reprlib.rst +++ b/Doc/library/reprlib.rst @@ -1,11 +1,13 @@ :mod:`reprlib` --- Alternate :func:`repr` implementation ======================================================== - .. module:: reprlib :synopsis: Alternate repr() implementation with size limits. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/reprlib.py` + +-------------- The :mod:`reprlib` module provides a means for producing object representations with limits on the size of the resulting strings. This is used in the Python @@ -35,6 +37,29 @@ This module provides a class, an instance, and a function: similar to that returned by the built-in function of the same name, but with limits on most sizes. +In addition to size-limiting tools, the module also provides a decorator for +detecting recursive calls to :meth:`__repr__` and substituting a placeholder +string instead. + +.. decorator:: recursive_repr(fillvalue="...") + + Decorator for :meth:`__repr__` methods to detect recursive calls within the + same thread. If a recursive call is made, the *fillvalue* is returned, + otherwise, the usual :meth:`__repr__` call is made. For example: + + >>> class MyList(list): + ... @recursive_repr() + ... def __repr__(self): + ... return '<' + '|'.join(map(repr, self)) + '>' + ... + >>> m = MyList('abc') + >>> m.append(m) + >>> m.append('x') + >>> print(m) + <'a'|'b'|'c'|...|'x'> + + .. versionadded:: 3.2 + .. _repr-objects: diff --git a/Doc/library/resource.rst b/Doc/library/resource.rst index fbd7204..c16b013 100644 --- a/Doc/library/resource.rst +++ b/Doc/library/resource.rst @@ -1,4 +1,3 @@ - :mod:`resource` --- Resource usage information ============================================== @@ -218,14 +217,14 @@ function to specify which processes information should be provided for. .. data:: RUSAGE_SELF - :const:`RUSAGE_SELF` should be used to request information pertaining only to - the process itself. + Pass to :func:`getrusage` to request resources consumed by the calling + process, which is the sum of resources used by all threads in the process. .. data:: RUSAGE_CHILDREN - Pass to :func:`getrusage` to request resource information for child processes of - the calling process. + Pass to :func:`getrusage` to request resources consumed by child processes + of the calling process which have been terminated and waited for. .. data:: RUSAGE_BOTH @@ -233,3 +232,10 @@ function to specify which processes information should be provided for. Pass to :func:`getrusage` to request resources consumed by both the current process and child processes. May not be available on all systems. + +.. data:: RUSAGE_THREAD + + Pass to :func:`getrusage` to request resources consumed by the current + thread. May not be available on all systems. + + .. versionadded:: 3.2 diff --git a/Doc/library/rlcompleter.rst b/Doc/library/rlcompleter.rst index 6b3befc..633088d 100644 --- a/Doc/library/rlcompleter.rst +++ b/Doc/library/rlcompleter.rst @@ -1,4 +1,3 @@ - :mod:`rlcompleter` --- Completion function for GNU readline =========================================================== @@ -6,6 +5,9 @@ :synopsis: Python identifier completion, suitable for the GNU readline library. .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/rlcompleter.py` + +-------------- The :mod:`rlcompleter` module defines a completion function suitable for the :mod:`readline` module by completing valid Python identifiers and keywords. diff --git a/Doc/library/runpy.rst b/Doc/library/runpy.rst index 1b48167..4df622c 100644 --- a/Doc/library/runpy.rst +++ b/Doc/library/runpy.rst @@ -5,71 +5,132 @@ :synopsis: Locate and run Python modules without importing them first. .. moduleauthor:: Nick Coghlan <ncoghlan@gmail.com> +**Source code:** :source:`Lib/runpy.py` -The :mod:`runpy` module is used to locate and run Python modules without -importing them first. Its main use is to implement the :option:`-m` command line -switch that allows scripts to be located using the Python module namespace -rather than the filesystem. - -When executed as a script, the module effectively operates as follows:: +-------------- - del sys.argv[0] # Remove the runpy module from the arguments - run_module(sys.argv[0], run_name="__main__", alter_sys=True) +The :mod:`runpy` module is used to locate and run Python modules without +importing them first. Its main use is to implement the :option:`-m` command +line switch that allows scripts to be located using the Python module +namespace rather than the filesystem. -The :mod:`runpy` module provides a single function: +The :mod:`runpy` module provides two functions: -.. function:: run_module(mod_name[, init_globals] [, run_name][, alter_sys]) +.. function:: run_module(mod_name, init_globals=None, run_name=None, alter_sys=False) - Execute the code of the specified module and return the resulting module globals - dictionary. The module's code is first located using the standard import - mechanism (refer to PEP 302 for details) and then executed in a fresh module - namespace. + Execute the code of the specified module and return the resulting module + globals dictionary. The module's code is first located using the standard + import mechanism (refer to :pep:`302` for details) and then executed in a + fresh module namespace. - If the supplied module name refers to a package rather than a normal module, - then that package is imported and the ``__main__`` submodule within that - package is then executed and the resulting module globals dictionary returned. + If the supplied module name refers to a package rather than a normal + module, then that package is imported and the ``__main__`` submodule within + that package is then executed and the resulting module globals dictionary + returned. - The optional dictionary argument *init_globals* may be used to pre-populate the - globals dictionary before the code is executed. The supplied dictionary will not - be modified. If any of the special global variables below are defined in the - supplied dictionary, those definitions are overridden by the ``run_module`` - function. + The optional dictionary argument *init_globals* may be used to pre-populate + the module's globals dictionary before the code is executed. The supplied + dictionary will not be modified. If any of the special global variables + below are defined in the supplied dictionary, those definitions are + overridden by :func:`run_module`. - The special global variables ``__name__``, ``__file__``, ``__loader__``, - ``__builtins__`` and ``__package__`` are set in the globals dictionary before - the module code is executed. + The special global variables ``__name__``, ``__file__``, ``__cached__``, + ``__loader__`` + and ``__package__`` are set in the globals dictionary before the module + code is executed (Note that this is a minimal set of variables - other + variables may be set implicitly as an interpreter implementation detail). - ``__name__`` is set to *run_name* if this optional argument is supplied, to - ``mod_name + '.__main__'`` if the named module is a package and to the - *mod_name* argument otherwise. + ``__name__`` is set to *run_name* if this optional argument is not + :const:`None`, to ``mod_name + '.__main__'`` if the named module is a + package and to the *mod_name* argument otherwise. - ``__loader__`` is set to the PEP 302 module loader used to retrieve the code for - the module (This loader may be a wrapper around the standard import mechanism). + ``__file__`` is set to the name provided by the module loader. If the + loader does not make filename information available, this variable is set + to :const:`None`. - ``__file__`` is set to the name provided by the module loader. If the loader - does not make filename information available, this variable is set to ``None``. + ``__cached__`` will be set to ``None``. - ``__builtins__`` is automatically initialised with a reference to the top level - namespace of the :mod:`builtins` module. + ``__loader__`` is set to the :pep:`302` module loader used to retrieve the + code for the module (This loader may be a wrapper around the standard + import mechanism). - ``__package__`` is set to *mod_name* if the named module is a package and to - ``mod_name.rpartition('.')[0]`` otherwise. + ``__package__`` is set to *mod_name* if the named module is a package and + to ``mod_name.rpartition('.')[0]`` otherwise. - If the argument *alter_sys* is supplied and evaluates to ``True``, then - ``sys.argv[0]`` is updated with the value of ``__file__`` and + If the argument *alter_sys* is supplied and evaluates to :const:`True`, + then ``sys.argv[0]`` is updated with the value of ``__file__`` and ``sys.modules[__name__]`` is updated with a temporary module object for the module being executed. Both ``sys.argv[0]`` and ``sys.modules[__name__]`` are restored to their original values before the function returns. - Note that this manipulation of :mod:`sys` is not thread-safe. Other threads may - see the partially initialised module, as well as the altered list of arguments. - It is recommended that the :mod:`sys` module be left alone when invoking this - function from threaded code. + Note that this manipulation of :mod:`sys` is not thread-safe. Other threads + may see the partially initialised module, as well as the altered list of + arguments. It is recommended that the :mod:`sys` module be left alone when + invoking this function from threaded code. + .. versionchanged:: 3.1 Added ability to execute packages by looking for a ``__main__`` submodule. + .. versionchanged:: 3.2 + Added ``__cached__`` global variable (see :PEP:`3147`). + + +.. function:: run_path(file_path, init_globals=None, run_name=None) + + Execute the code at the named filesystem location and return the resulting + module globals dictionary. As with a script name supplied to the CPython + command line, the supplied path may refer to a Python source file, a + compiled bytecode file or a valid sys.path entry containing a ``__main__`` + module (e.g. a zipfile containing a top-level ``__main__.py`` file). + + For a simple script, the specified code is simply executed in a fresh + module namespace. For a valid sys.path entry (typically a zipfile or + directory), the entry is first added to the beginning of ``sys.path``. The + function then looks for and executes a :mod:`__main__` module using the + updated path. Note that there is no special protection against invoking + an existing :mod:`__main__` entry located elsewhere on ``sys.path`` if + there is no such module at the specified location. + + The optional dictionary argument *init_globals* may be used to pre-populate + the module's globals dictionary before the code is executed. The supplied + dictionary will not be modified. If any of the special global variables + below are defined in the supplied dictionary, those definitions are + overridden by :func:`run_path`. + + The special global variables ``__name__``, ``__file__``, ``__loader__`` + and ``__package__`` are set in the globals dictionary before the module + code is executed (Note that this is a minimal set of variables - other + variables may be set implicitly as an interpreter implementation detail). + + ``__name__`` is set to *run_name* if this optional argument is not + :const:`None` and to ``'<run_path>'`` otherwise. + + ``__file__`` is set to the name provided by the module loader. If the + loader does not make filename information available, this variable is set + to :const:`None`. For a simple script, this will be set to ``file_path``. + + ``__loader__`` is set to the :pep:`302` module loader used to retrieve the + code for the module (This loader may be a wrapper around the standard + import mechanism). For a simple script, this will be set to :const:`None`. + + ``__package__`` is set to ``__name__.rpartition('.')[0]``. + + A number of alterations are also made to the :mod:`sys` module. Firstly, + ``sys.path`` may be altered as described above. ``sys.argv[0]`` is updated + with the value of ``file_path`` and ``sys.modules[__name__]`` is updated + with a temporary module object for the module being executed. All + modifications to items in :mod:`sys` are reverted before the function + returns. + + Note that, unlike :func:`run_module`, the alterations made to :mod:`sys` + are not optional in this function as these adjustments are essential to + allowing the execution of sys.path entries. As the thread-safety + limitations still apply, use of this function in threaded code should be + either serialised with the import lock or delegated to a separate process. + + .. versionadded:: 3.2 .. seealso:: @@ -79,3 +140,4 @@ The :mod:`runpy` module provides a single function: :pep:`366` - Main module explicit relative imports PEP written and implemented by Nick Coghlan. + :ref:`using-on-general` - CPython command line details diff --git a/Doc/library/sched.rst b/Doc/library/sched.rst index 0290ec4..ab58237 100644 --- a/Doc/library/sched.rst +++ b/Doc/library/sched.rst @@ -7,10 +7,13 @@ .. index:: single: event scheduling +**Source code:** :source:`Lib/sched.py` + +-------------- + The :mod:`sched` module defines a class which implements a general purpose event scheduler: - .. class:: scheduler(timefunc, delayfunc) The :class:`scheduler` class defines a generic interface to scheduling events. diff --git a/Doc/library/select.rst b/Doc/library/select.rst index 70f7370..f1fd126 100644 --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -1,4 +1,3 @@ - :mod:`select` --- Waiting for I/O completion ============================================ @@ -6,9 +5,9 @@ :synopsis: Wait for I/O completion on multiple streams. -This module provides access to the :cfunc:`select` and :cfunc:`poll` functions -available in most operating systems, :cfunc:`epoll` available on Linux 2.5+ and -:cfunc:`kqueue` available on most BSD. +This module provides access to the :c:func:`select` and :c:func:`poll` functions +available in most operating systems, :c:func:`epoll` available on Linux 2.5+ and +:c:func:`kqueue` available on most BSD. Note that on Windows, it only works for sockets; on other operating systems, it also works for other file types (in particular, on Unix, it works on pipes). It cannot be used on regular files to determine whether a file has grown since @@ -20,11 +19,11 @@ The module defines the following: .. exception:: error The exception raised when an error occurs. The accompanying value is a pair - containing the numeric error code from :cdata:`errno` and the corresponding - string, as would be printed by the C function :cfunc:`perror`. + containing the numeric error code from :c:data:`errno` and the corresponding + string, as would be printed by the C function :c:func:`perror`. -.. function:: epoll([sizehint=-1]) +.. function:: epoll(sizehint=-1) (Only supported on Linux 2.5.44 and newer.) Returns an edge polling object, which can be used as Edge or Level Triggered interface for I/O events; see @@ -54,7 +53,7 @@ The module defines the following: .. function:: select(rlist, wlist, xlist[, timeout]) - This is a straightforward interface to the Unix :cfunc:`select` system call. + This is a straightforward interface to the Unix :c:func:`select` system call. The first three arguments are sequences of 'waitable objects': either integers representing file descriptors or objects with a parameterless method named :meth:`fileno` returning such an integer: @@ -91,10 +90,21 @@ The module defines the following: .. index:: single: WinSock File objects on Windows are not acceptable, but sockets are. On Windows, - the underlying :cfunc:`select` function is provided by the WinSock + the underlying :c:func:`select` function is provided by the WinSock library, and does not handle file descriptors that don't originate from WinSock. +.. attribute:: PIPE_BUF + + The minimum number of bytes which can be written without blocking to a pipe + when the pipe has been reported as ready for writing by :func:`select`, + :func:`poll` or another interface in this module. This doesn't apply + to other kind of file-like objects such as sockets. + + This value is guaranteed by POSIX to be at least 512. Availability: Unix. + + .. versionadded:: 3.2 + .. _epoll-objects: @@ -124,15 +134,15 @@ Edge and Level Trigger Polling (epoll) Objects | :const:`EPOLLONESHOT` | Set one-shot behavior. After one event is | | | pulled out, the fd is internally disabled | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLRDNORM` | ??? | + | :const:`EPOLLRDNORM` | Equivalent to :const:`EPOLLIN` | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLRDBAND` | ??? | + | :const:`EPOLLRDBAND` | Priority data band can be read. | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLWRNORM` | ??? | + | :const:`EPOLLWRNORM` | Equivalent to :const:`EPOLLOUT` | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLWRBAND` | ??? | + | :const:`EPOLLWRBAND` | Priority data may be written. | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLMSG` | ??? | + | :const:`EPOLLMSG` | Ignored. | +-----------------------+-----------------------------------------------+ @@ -181,13 +191,13 @@ Edge and Level Trigger Polling (epoll) Objects Polling Objects --------------- -The :cfunc:`poll` system call, supported on most Unix systems, provides better +The :c:func:`poll` system call, supported on most Unix systems, provides better scalability for network servers that service many, many clients at the same -time. :cfunc:`poll` scales better because the system call only requires listing -the file descriptors of interest, while :cfunc:`select` builds a bitmap, turns +time. :c:func:`poll` scales better because the system call only requires listing +the file descriptors of interest, while :c:func:`select` builds a bitmap, turns on bits for the fds of interest, and then afterward the whole bitmap has to be -linearly scanned again. :cfunc:`select` is O(highest file descriptor), while -:cfunc:`poll` is O(number of file descriptors). +linearly scanned again. :c:func:`select` is O(highest file descriptor), while +:c:func:`poll` is O(number of file descriptors). .. method:: poll.register(fd[, eventmask]) @@ -226,7 +236,7 @@ linearly scanned again. :cfunc:`select` is O(highest file descriptor), while .. method:: poll.modify(fd, eventmask) Modifies an already registered fd. This has the same effect as - :meth:`register(fd, eventmask)`. Attempting to modify a file descriptor + ``register(fd, eventmask)``. Attempting to modify a file descriptor that was never registered causes an :exc:`IOError` exception with errno :const:`ENOENT` to be raised. diff --git a/Doc/library/shelve.rst b/Doc/library/shelve.rst index b281814..9d7d504 100644 --- a/Doc/library/shelve.rst +++ b/Doc/library/shelve.rst @@ -7,6 +7,10 @@ .. index:: module: pickle +**Source code:** :source:`Lib/shelve.py` + +-------------- + A "shelf" is a persistent, dictionary-like object. The difference with "dbm" databases is that the values (not the keys!) in a shelf can be essentially arbitrary Python objects --- anything that the :mod:`pickle` module can handle. @@ -14,7 +18,7 @@ This includes most class instances, recursive data types, and objects containing lots of shared sub-objects. The keys are ordinary strings. -.. function:: open(filename[, flag='c'[, protocol=None[, writeback=False]]]) +.. function:: open(filename, flag='c', protocol=None, writeback=False) Open a persistent dictionary. The filename specified is the base filename for the underlying database. As a side-effect, an extension may be added to the @@ -97,7 +101,7 @@ Restrictions implementation used. -.. class:: Shelf(dict[, protocol=None[, writeback=False]]) +.. class:: Shelf(dict, protocol=None, writeback=False, keyencoding='utf-8') A subclass of :class:`collections.MutableMapping` which stores pickled values in the *dict* object. @@ -111,8 +115,15 @@ Restrictions This allows natural operations on mutable entries, but can consume much more memory and make sync and close take a long time. + The *keyencoding* parameter is the encoding used to encode keys before they + are used with the underlying dict. + + .. versionadded:: 3.2 + The *keyencoding* parameter; previously, keys were always encoded in + UTF-8. -.. class:: BsdDbShelf(dict[, protocol=None[, writeback=False]]) + +.. class:: BsdDbShelf(dict, protocol=None, writeback=False, keyencoding='utf-8') A subclass of :class:`Shelf` which exposes :meth:`first`, :meth:`!next`, :meth:`previous`, :meth:`last` and :meth:`set_location` which are available @@ -121,11 +132,11 @@ Restrictions modules. The *dict* object passed to the constructor must support those methods. This is generally accomplished by calling one of :func:`bsddb.hashopen`, :func:`bsddb.btopen` or :func:`bsddb.rnopen`. The - optional *protocol* and *writeback* parameters have the same interpretation - as for the :class:`Shelf` class. + optional *protocol*, *writeback*, and *keyencoding* parameters have the same + interpretation as for the :class:`Shelf` class. -.. class:: DbfilenameShelf(filename[, flag='c'[, protocol=None[, writeback=False]]]) +.. class:: DbfilenameShelf(filename, flag='c', protocol=None, writeback=False) A subclass of :class:`Shelf` which accepts a *filename* instead of a dict-like object. The underlying file will be opened using :func:`dbm.open`. By diff --git a/Doc/library/shlex.rst b/Doc/library/shlex.rst index ee241f4..03c9f98 100644 --- a/Doc/library/shlex.rst +++ b/Doc/library/shlex.rst @@ -1,4 +1,3 @@ - :mod:`shlex` --- Simple lexical analysis ======================================== @@ -9,6 +8,9 @@ .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> .. sectionauthor:: Gustavo Niemeyer <niemeyer@conectiva.com> +**Source code:** :source:`Lib/shlex.py` + +-------------- The :class:`shlex` class makes it easy to write lexical analyzers for simple syntaxes resembling that of the Unix shell. This will often be useful for @@ -18,7 +20,7 @@ applications) or for parsing quoted strings. The :mod:`shlex` module defines the following functions: -.. function:: split(s[, comments[, posix]]) +.. function:: split(s, comments=False, posix=True) Split the string *s* using shell-like syntax. If *comments* is :const:`False` (the default), the parsing of comments in the given string will be disabled @@ -28,13 +30,14 @@ The :mod:`shlex` module defines the following functions: .. note:: - Since the :func:`split` function instantiates a :class:`shlex` instance, passing - ``None`` for *s* will read the string to split from standard input. + Since the :func:`split` function instantiates a :class:`shlex` instance, + passing ``None`` for *s* will read the string to split from standard + input. The :mod:`shlex` module defines the following class: -.. class:: shlex([instream[, infile[, posix]]]) +.. class:: shlex(instream=None, infile=None, posix=False) A :class:`shlex` instance or subclass instance is a lexical analyzer object. The initialization argument, if present, specifies where to read characters @@ -111,7 +114,7 @@ A :class:`shlex` instance has the following methods: :meth:`pop_source` methods. -.. method:: shlex.push_source(stream[, filename]) +.. method:: shlex.push_source(newstream, newfile=None) Push an input source stream onto the input stack. If the filename argument is specified it will later be available for use in error messages. This is the @@ -124,7 +127,7 @@ A :class:`shlex` instance has the following methods: used internally when the lexer reaches EOF on a stacked input stream. -.. method:: shlex.error_leader([file[, line]]) +.. method:: shlex.error_leader(infile=None, lineno=None) This method generates an error message leader in the format of a Unix C compiler error label; the format is ``'"%s", line %d: '``, where the ``%s`` is replaced diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 7cf8550..1a878d5 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -1,4 +1,3 @@ - :mod:`shutil` --- High-level file operations ============================================ @@ -11,6 +10,10 @@ single: file; copying single: copying files +**Source code:** :source:`Lib/shutil.py` + +-------------- + The :mod:`shutil` module offers a number of high-level operations on files and collections of files. In particular, functions are provided which support file copying and removal. For operations on individual files, see also the @@ -19,7 +22,7 @@ copying and removal. For operations on individual files, see also the .. warning:: Even the higher-level file copying functions (:func:`copy`, :func:`copy2`) - can't copy all file metadata. + cannot copy all file metadata. On POSIX platforms, this means that file owner and group are lost as well as ACLs. On Mac OS, the resource fork and other metadata are not used. @@ -28,6 +31,9 @@ copying and removal. For operations on individual files, see also the are not copied. +Directory and files operations +------------------------------ + .. function:: copyfileobj(fsrc, fdst[, length]) Copy the contents of the file-like object *fsrc* to the file-like object *fdst*. @@ -86,7 +92,7 @@ copying and removal. For operations on individual files, see also the match one of the glob-style *patterns* provided. See the example below. -.. function:: copytree(src, dst[, symlinks=False[, ignore=None]]) +.. function:: copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, ignore_dangling_symlinks=False) Recursively copy an entire directory tree rooted at *src*. The destination directory, named by *dst*, must not already exist; it will be created as well @@ -98,6 +104,13 @@ copying and removal. For operations on individual files, see also the symbolic links in the new tree; if false or omitted, the contents of the linked files are copied to the new tree. + When *symlinks* is false, if the file pointed by the symlink doesn't + exist, a exception will be added in the list of errors raised in + a :exc:`Error` exception at the end of the copy process. + You can set the optional *ignore_dangling_symlinks* flag to true if you + want to silence this exception. Notice that this option has no effect + on platforms that don't support :func:`os.symlink`. + If *ignore* is given, it must be a callable that will receive as its arguments the directory being visited by :func:`copytree`, and a list of its contents, as returned by :func:`os.listdir`. Since :func:`copytree` is @@ -110,11 +123,21 @@ copying and removal. For operations on individual files, see also the If exception(s) occur, an :exc:`Error` is raised with a list of reasons. - The source code for this should be considered an example rather than the - ultimate tool. + If *copy_function* is given, it must be a callable that will be used + to copy each file. It will be called with the source path and the + destination path as arguments. By default, :func:`copy2` is used, but any + function that supports the same signature (like :func:`copy`) can be used. + .. versionchanged:: 3.2 + Added the *copy_function* argument to be able to provide a custom copy + function. -.. function:: rmtree(path[, ignore_errors[, onerror]]) + .. versionchanged:: 3.2 + Added the *ignore_dangling_symlinks* argument to silent dangling symlinks + errors when *symlinks* is false. + + +.. function:: rmtree(path, ignore_errors=False, onerror=None) .. index:: single: directory; deleting @@ -151,8 +174,8 @@ copying and removal. For operations on individual files, see also the .. _shutil-example: -Example -------- +copytree example +:::::::::::::::: This example is the implementation of the :func:`copytree` function, described above, with the docstring omitted. It demonstrates many of the other functions @@ -190,3 +213,182 @@ provided by this module. :: if errors: raise Error(errors) +Another example that uses the :func:`ignore_patterns` helper:: + + from shutil import copytree, ignore_patterns + + copytree(source, destination, ignore=ignore_patterns('*.pyc', 'tmp*')) + +This will copy everything except ``.pyc`` files and files or directories whose +name starts with ``tmp``. + +Another example that uses the *ignore* argument to add a logging call:: + + from shutil import copytree + import logging + + def _logpath(path, names): + logging.info('Working in %s' % path) + return [] # nothing will be ignored + + copytree(source, destination, ignore=_logpath) + + +.. _archiving-operations: + +Archiving operations +-------------------- + +.. function:: make_archive(base_name, format, [root_dir, [base_dir, [verbose, [dry_run, [owner, [group, [logger]]]]]]]) + + Create an archive file (such as zip or tar) and return its name. + + *base_name* is the name of the file to create, including the path, minus + any format-specific extension. *format* is the archive format: one of + "zip", "tar", "bztar" (if the :mod:`bz2` module is available) or "gztar". + + *root_dir* is a directory that will be the root directory of the + archive; for example, we typically chdir into *root_dir* before creating the + archive. + + *base_dir* is the directory where we start archiving from; + i.e. *base_dir* will be the common prefix of all files and + directories in the archive. + + *root_dir* and *base_dir* both default to the current directory. + + *owner* and *group* are used when creating a tar archive. By default, + uses the current owner and group. + + *logger* is an instance of :class:`logging.Logger`. + + .. versionadded:: 3.2 + + +.. function:: get_archive_formats() + + Returns a list of supported formats for archiving. + Each element of the returned sequence is a tuple ``(name, description)`` + + By default :mod:`shutil` provides these formats: + + - *gztar*: gzip'ed tar-file + - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *tar*: uncompressed tar file + - *zip*: ZIP file + + You can register new formats or provide your own archiver for any existing + formats, by using :func:`register_archive_format`. + + .. versionadded:: 3.2 + + +.. function:: register_archive_format(name, function, [extra_args, [description]]) + + Registers an archiver for the format *name*. *function* is a callable that + will be used to invoke the archiver. + + If given, *extra_args* is a sequence of ``(name, value)`` pairs that will be + used as extra keywords arguments when the archiver callable is used. + + *description* is used by :func:`get_archive_formats` which returns the + list of archivers. Defaults to an empty list. + + .. versionadded:: 3.2 + + +.. function:: unregister_archive_format(name) + + Remove the archive format *name* from the list of supported formats. + + .. versionadded:: 3.2 + + +.. function:: unpack_archive(filename[, extract_dir[, format]]) + + Unpack an archive. *filename* is the full path of the archive. + + *extract_dir* is the name of the target directory where the archive is + unpacked. If not provided, the current working directory is used. + + *format* is the archive format: one of "zip", "tar", or "gztar". Or any + other format registered with :func:`register_unpack_format`. If not + provided, :func:`unpack_archive` will use the archive file name extension + and see if an unpacker was registered for that extension. In case none is + found, a :exc:`ValueError` is raised. + + .. versionadded:: 3.2 + + +.. function:: register_unpack_format(name, extensions, function[, extra_args[, description]]) + + Registers an unpack format. *name* is the name of the format and + *extensions* is a list of extensions corresponding to the format, like + ``.zip`` for Zip files. + + *function* is the callable that will be used to unpack archives. The + callable will receive the path of the archive, followed by the directory + the archive must be extracted to. + + When provided, *extra_args* is a sequence of ``(name, value)`` tuples that + will be passed as keywords arguments to the callable. + + *description* can be provided to describe the format, and will be returned + by the :func:`get_unpack_formats` function. + + .. versionadded:: 3.2 + + +.. function:: unregister_unpack_format(name) + + Unregister an unpack format. *name* is the name of the format. + + .. versionadded:: 3.2 + + +.. function:: get_unpack_formats() + + Return a list of all registered formats for unpacking. + Each element of the returned sequence is a tuple + ``(name, extensions, description)``. + + By default :mod:`shutil` provides these formats: + + - *gztar*: gzip'ed tar-file + - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *tar*: uncompressed tar file + - *zip*: ZIP file + + You can register new formats or provide your own unpacker for any existing + formats, by using :func:`register_unpack_format`. + + .. versionadded:: 3.2 + + + +Archiving example +::::::::::::::::: + +In this example, we create a gzip'ed tar-file archive containing all files +found in the :file:`.ssh` directory of the user:: + + >>> from shutil import make_archive + >>> import os + >>> archive_name = os.path.expanduser(os.path.join('~', 'myarchive')) + >>> root_dir = os.path.expanduser(os.path.join('~', '.ssh')) + >>> make_archive(archive_name, 'gztar', root_dir) + '/Users/tarek/myarchive.tar.gz' + +The resulting archive contains:: + + $ tar -tzvf /Users/tarek/myarchive.tar.gz + drwx------ tarek/staff 0 2010-02-01 16:23:40 ./ + -rw-r--r-- tarek/staff 609 2008-06-09 13:26:54 ./authorized_keys + -rwxr-xr-x tarek/staff 65 2008-06-09 13:26:54 ./config + -rwx------ tarek/staff 668 2008-06-09 13:26:54 ./id_dsa + -rwxr-xr-x tarek/staff 609 2008-06-09 13:26:54 ./id_dsa.pub + -rw------- tarek/staff 1675 2008-06-09 13:26:54 ./id_rsa + -rw-r--r-- tarek/staff 397 2008-06-09 13:26:54 ./id_rsa.pub + -rw-r--r-- tarek/staff 37192 2010-02-06 18:23:10 ./known_hosts + + diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 309f71b..698b1e7 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -1,4 +1,3 @@ - :mod:`signal` --- Set handlers for asynchronous events ====================================================== @@ -69,12 +68,32 @@ The variables defined in the :mod:`signal` module are: All the signal numbers are defined symbolically. For example, the hangup signal is defined as :const:`signal.SIGHUP`; the variable names are identical to the names used in C programs, as found in ``<signal.h>``. The Unix man page for - ':cfunc:`signal`' lists the existing signals (on some systems this is + ':c:func:`signal`' lists the existing signals (on some systems this is :manpage:`signal(2)`, on others the list is in :manpage:`signal(7)`). Note that not all systems define the same set of signal names; only those names defined by the system are defined by this module. +.. data:: CTRL_C_EVENT + + The signal corresponding to the CTRL+C keystroke event. This signal can + only be used with :func:`os.kill`. + + Availability: Windows. + + .. versionadded:: 3.2 + + +.. data:: CTRL_BREAK_EVENT + + The signal corresponding to the CTRL+BREAK keystroke event. This signal can + only be used with :func:`os.kill`. + + Availability: Windows. + + .. versionadded:: 3.2 + + .. data:: NSIG One more than the number of the highest signal number. @@ -82,7 +101,8 @@ The variables defined in the :mod:`signal` module are: .. data:: ITIMER_REAL - Decrements interval timer in real time, and delivers :const:`SIGALRM` upon expiration. + Decrements interval timer in real time, and delivers :const:`SIGALRM` upon + expiration. .. data:: ITIMER_VIRTUAL @@ -190,7 +210,7 @@ The :mod:`signal` module defines the following functions: Note that installing a signal handler with :func:`signal` will reset the restart behaviour to interruptible by implicitly calling - :cfunc:`siginterrupt` with a true *flag* value for the given signal. + :c:func:`siginterrupt` with a true *flag* value for the given signal. .. function:: signal(signalnum, handler) diff --git a/Doc/library/site.rst b/Doc/library/site.rst index bf65d17..b77f3cf 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -1,10 +1,12 @@ - :mod:`site` --- Site-specific configuration hook ================================================ .. module:: site :synopsis: A standard way to reference site-specific modules. +**Source code:** :source:`Lib/site.py` + +-------------- **This module is automatically imported during initialization.** The automatic import can be suppressed using the interpreter's :option:`-S` option. @@ -116,6 +118,32 @@ empty, and the path manipulations are skipped; however the import of Adds a directory to sys.path and processes its pth files. +.. function:: getsitepackages() + + Returns a list containing all global site-packages directories + (and possibly site-python). + + .. versionadded:: 3.2 + +.. function:: getuserbase() + + Returns the "user base" directory path. + + The "user base" directory can be used to store data. If the global + variable ``USER_BASE`` is not initialized yet, this function will also set + it. + + .. versionadded:: 3.2 + +.. function:: getusersitepackages() + + Returns the user-specific site-packages directory path. + + If the global variable ``USER_SITE`` is not initialized yet, this + function will also set it. + + .. versionadded:: 3.2 .. XXX Update documentation .. XXX document python -m site --user-base --user-site + diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst index 2767516..c391f71 100644 --- a/Doc/library/smtpd.rst +++ b/Doc/library/smtpd.rst @@ -7,13 +7,18 @@ .. moduleauthor:: Barry Warsaw <barry@zope.com> .. sectionauthor:: Moshe Zadka <moshez@moshez.org> +**Source code:** :source:`Lib/smtpd.py` +-------------- +This module offers several classes to implement SMTP (email) servers. -This module offers several classes to implement SMTP servers. One is a generic +Several server implementations are present; one is a generic do-nothing implementation, which can be overridden, while the other two offer specific mail-sending strategies. +Additionally the SMTPChannel may be extended to implement very specific +interaction behaviour with SMTP clients. SMTPServer Objects ------------------ @@ -26,7 +31,6 @@ SMTPServer Objects inherits from :class:`asyncore.dispatcher`, and so will insert itself into :mod:`asyncore`'s event loop on instantiation. - .. method:: process_message(peer, mailfrom, rcpttos, data) Raise :exc:`NotImplementedError` exception. Override this in subclasses to @@ -37,6 +41,11 @@ SMTPServer Objects containing the contents of the e-mail (which should be in :rfc:`2822` format). + .. attribute:: channel_class + + Override this in subclasses to use a custom :class:`SMTPChannel` for + managing SMTP clients. + DebuggingServer Objects ----------------------- @@ -71,3 +80,91 @@ MailmanProxy Objects running this has a good chance to make you into an open relay, so please be careful. +SMTPChannel Objects +------------------- + +.. class:: SMTPChannel(server, conn, addr) + + Create a new :class:`SMTPChannel` object which manages the communication + between the server and a single SMTP client. + + To use a custom SMTPChannel implementation you need to override the + :attr:`SMTPServer.channel_class` of your :class:`SMTPServer`. + + The :class:`SMTPChannel` has the following instance variables: + + .. attribute:: smtp_server + + Holds the :class:`SMTPServer` that spawned this channel. + + .. attribute:: conn + + Holds the socket object connecting to the client. + + .. attribute:: addr + + Holds the address of the client, the second value returned by + socket.accept() + + .. attribute:: received_lines + + Holds a list of the line strings (decoded using UTF-8) received from + the client. The lines have their "\r\n" line ending translated to "\n". + + .. attribute:: smtp_state + + Holds the current state of the channel. This will be either + :attr:`COMMAND` initially and then :attr:`DATA` after the client sends + a "DATA" line. + + .. attribute:: seen_greeting + + Holds a string containing the greeting sent by the client in its "HELO". + + .. attribute:: mailfrom + + Holds a string containing the address identified in the "MAIL FROM:" line + from the client. + + .. attribute:: rcpttos + + Holds a list of strings containing the addresses identified in the + "RCPT TO:" lines from the client. + + .. attribute:: received_data + + Holds a string containing all of the data sent by the client during the + DATA state, up to but not including the terminating "\r\n.\r\n". + + .. attribute:: fqdn + + Holds the fully-qualified domain name of the server as returned by + ``socket.getfqdn()``. + + .. attribute:: peer + + Holds the name of the client peer as returned by ``conn.getpeername()`` + where ``conn`` is :attr:`conn`. + + The :class:`SMTPChannel` operates by invoking methods named ``smtp_<command>`` + upon reception of a command line from the client. Built into the base + :class:`SMTPChannel` class are methods for handling the following commands + (and responding to them appropriately): + + ======== =================================================================== + Command Action taken + ======== =================================================================== + HELO Accepts the greeting from the client and stores it in + :attr:`seen_greeting`. + NOOP Takes no action. + QUIT Closes the connection cleanly. + MAIL Accepts the "MAIL FROM:" syntax and stores the supplied address as + :attr:`mailfrom`. + RCPT Accepts the "RCPT TO:" syntax and stores the supplied addresses in + the :attr:`rcpttos` list. + RSET Resets the :attr:`mailfrom`, :attr:`rcpttos`, and + :attr:`received_data`, but not the greeting. + DATA Sets the internal state to :attr:`DATA` and stores remaining lines + from the client in :attr:`received_data` until the terminator + "\r\n.\r\n" is received. + ======== =================================================================== diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index 9303c3e..531a64d 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -1,4 +1,3 @@ - :mod:`smtplib` --- SMTP protocol client ======================================= @@ -11,13 +10,17 @@ pair: SMTP; protocol single: Simple Mail Transfer Protocol +**Source code:** :source:`Lib/smtplib.py` + +-------------- + The :mod:`smtplib` module defines an SMTP client session object that can be used to send mail to any Internet machine with an SMTP or ESMTP listener daemon. For details of SMTP and ESMTP operation, consult :rfc:`821` (Simple Mail Transfer Protocol) and :rfc:`1869` (SMTP Service Extensions). -.. class:: SMTP([host[, port[, local_hostname[, timeout]]]]) +.. class:: SMTP(host='', port=0, local_hostname=None[, timeout]) A :class:`SMTP` instance encapsulates an SMTP connection. It has methods that support a full repertoire of SMTP and ESMTP operations. If the optional @@ -32,13 +35,13 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). :meth:`sendmail`, and :meth:`quit` methods. An example is included below. -.. class:: SMTP_SSL([host[, port[, local_hostname[, keyfile[, certfile[, timeout]]]]]]) +.. class:: SMTP_SSL(host='', port=0, local_hostname=None, keyfile=None, certfile=None[, timeout]) A :class:`SMTP_SSL` instance behaves exactly the same as instances of :class:`SMTP`. :class:`SMTP_SSL` should be used for situations where SSL is required from the beginning of the connection and using :meth:`starttls` is not appropriate. If *host* is not specified, the local host is used. If - *port* is omitted, the standard SMTP-over-SSL port (465) is used. *keyfile* + *port* is zero, the standard SMTP-over-SSL port (465) is used. *keyfile* and *certfile* are also optional, and can contain a PEM formatted private key and certificate chain file for the SSL connection. The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the @@ -46,7 +49,7 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). will be used). -.. class:: LMTP([host[, port[, local_hostname]]]) +.. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None) The LMTP protocol, which is very similar to ESMTP, is heavily based on the standard SMTP client. It's common to use Unix sockets for LMTP, so our :meth:`connect` @@ -142,7 +145,7 @@ An :class:`SMTP` instance has the following methods: for connection and for all messages sent to and received from the server. -.. method:: SMTP.connect([host[, port]]) +.. method:: SMTP.connect(host='localhost', port=0) Connect to a host on a given port. The defaults are to connect to the local host at the standard SMTP port (25). If the hostname ends with a colon (``':'``) @@ -151,9 +154,9 @@ An :class:`SMTP` instance has the following methods: the constructor if a host is specified during instantiation. -.. method:: SMTP.docmd(cmd, [, argstring]) +.. method:: SMTP.docmd(cmd, args='') - Send a command *cmd* to the server. The optional argument *argstring* is simply + Send a command *cmd* to the server. The optional argument *args* is simply concatenated to the command, separated by a space. This returns a 2-tuple composed of a numeric response code and the actual @@ -167,7 +170,7 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPServerDisconnected` will be raised. -.. method:: SMTP.helo([hostname]) +.. method:: SMTP.helo(name='') Identify yourself to the SMTP server using ``HELO``. The hostname argument defaults to the fully qualified domain name of the local host. @@ -178,7 +181,7 @@ An :class:`SMTP` instance has the following methods: It will be implicitly called by the :meth:`sendmail` when necessary. -.. method:: SMTP.ehlo([hostname]) +.. method:: SMTP.ehlo(name='') Identify yourself to an ESMTP server using ``EHLO``. The hostname argument defaults to the fully qualified domain name of the local host. Examine the @@ -239,7 +242,7 @@ An :class:`SMTP` instance has the following methods: No suitable authentication method was found. -.. method:: SMTP.starttls([keyfile[, certfile]]) +.. method:: SMTP.starttls(keyfile=None, certfile=None) Put the SMTP connection in TLS (Transport Layer Security) mode. All SMTP commands that follow will be encrypted. You should then call :meth:`ehlo` @@ -261,7 +264,7 @@ An :class:`SMTP` instance has the following methods: SSL/TLS support is not available to your Python interpreter. -.. method:: SMTP.sendmail(from_addr, to_addrs, msg[, mail_options, rcpt_options]) +.. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]) Send mail. The required arguments are an :rfc:`822` from-address string, a list of :rfc:`822` to-address strings (a bare string will be treated as a list with 1 @@ -275,9 +278,14 @@ An :class:`SMTP` instance has the following methods: .. note:: The *from_addr* and *to_addrs* parameters are used to construct the message - envelope used by the transport agents. The :class:`SMTP` does not modify the + envelope used by the transport agents. ``sendmail`` does not modify the message headers in any way. + msg may be a string containing characters in the ASCII range, or a byte + string. A string is encoded to bytes using the ascii codec, and lone ``\r`` + and ``\n`` characters are converted to ``\r\n`` characters. A byte string + is not modified. + If there has been no previous ``EHLO`` or ``HELO`` command this session, this method tries ESMTP ``EHLO`` first. If the server does ESMTP, message size and each of the specified options will be passed to it (if the option is in the @@ -312,6 +320,27 @@ An :class:`SMTP` instance has the following methods: Unless otherwise noted, the connection will be open even after an exception is raised. + .. versionchanged:: 3.2 *msg* may be a byte string. + + +.. method:: SMTP.send_message(msg, from_addr=None, to_addrs=None, mail_options=[], rcpt_options=[]) + + This is a convenience method for calling :meth:`sendmail` with the message + represented by an :class:`email.message.Message` object. The arguments have + the same meaning as for :meth:`sendmail`, except that *msg* is a ``Message`` + object. + + If *from_addr* is ``None``, ``send_message`` sets its value to the value of + the :mailheader:`From` header from *msg*. If *to_addrs* is ``None``, + ``send_message`` combines the values (if any) of the :mailheader:`To`, + :mailheader:`CC`, and :mailheader:`Bcc` fields from *msg*. Regardless of + the values of *from_addr* and *to_addrs*, ``send_message`` deletes any Bcc + field from *msg*. It then serializes *msg* using + :class:`~email.generator.BytesGenerator` with ``\r\n`` as the *linesep*, and + calls :meth:`sendmail` to transmit the resulting message. + + .. versionadded:: 3.2 + .. method:: SMTP.quit() @@ -367,5 +396,5 @@ example doesn't do any processing of the :rfc:`822` headers. In particular, the .. note:: In general, you will want to use the :mod:`email` package's features to - construct an email message, which you can then convert to a string and send - via :meth:`sendmail`; see :ref:`email-examples`. + construct an email message, which you can then send + via :meth:`~smtplib.SMTP.send_message`; see :ref:`email-examples`. diff --git a/Doc/library/sndhdr.rst b/Doc/library/sndhdr.rst index 01a3917..f36df68 100644 --- a/Doc/library/sndhdr.rst +++ b/Doc/library/sndhdr.rst @@ -1,4 +1,3 @@ - :mod:`sndhdr` --- Determine type of sound file ============================================== @@ -11,6 +10,10 @@ single: A-LAW single: u-LAW +**Source code:** :source:`Lib/sndhdr.py` + +-------------- + The :mod:`sndhdr` provides utility functions which attempt to determine the type of sound data which is in a file. When these functions are able to determine what type of sound data is stored in a file, they return a tuple ``(type, diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index d61398c..bfb8ae9 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -1,4 +1,3 @@ - :mod:`socket` --- Low-level networking interface ================================================ @@ -118,39 +117,44 @@ The module :mod:`socket` exports the following constants and functions: .. index:: module: errno - This exception is raised for socket-related errors. The accompanying value is - either a string telling what went wrong or a pair ``(errno, string)`` - representing an error returned by a system call, similar to the value - accompanying :exc:`os.error`. See the module :mod:`errno`, which contains names - for the error codes defined by the underlying operating system. + A subclass of :exc:`IOError`, this exception is raised for socket-related + errors. It is recommended that you inspect its ``errno`` attribute to + discriminate between different kinds of errors. + .. seealso:: + The :mod:`errno` module contains symbolic names for the error codes + defined by the underlying operating system. -.. exception:: herror - This exception is raised for address-related errors, i.e. for functions that use - *h_errno* in the C API, including :func:`gethostbyname_ex` and - :func:`gethostbyaddr`. +.. exception:: herror - The accompanying value is a pair ``(h_errno, string)`` representing an error - returned by a library call. *string* represents the description of *h_errno*, as - returned by the :cfunc:`hstrerror` C function. + A subclass of :exc:`socket.error`, this exception is raised for + address-related errors, i.e. for functions that use *h_errno* in the POSIX + C API, including :func:`gethostbyname_ex` and :func:`gethostbyaddr`. + The accompanying value is a pair ``(h_errno, string)`` representing an + error returned by a library call. *h_errno* is a numeric value, while + *string* represents the description of *h_errno*, as returned by the + :c:func:`hstrerror` C function. .. exception:: gaierror - This exception is raised for address-related errors, for :func:`getaddrinfo` and - :func:`getnameinfo`. The accompanying value is a pair ``(error, string)`` - representing an error returned by a library call. *string* represents the - description of *error*, as returned by the :cfunc:`gai_strerror` C function. The - *error* value will match one of the :const:`EAI_\*` constants defined in this - module. + A subclass of :exc:`socket.error`, this exception is raised for + address-related errors by :func:`getaddrinfo` and :func:`getnameinfo`. + The accompanying value is a pair ``(error, string)`` representing an error + returned by a library call. *string* represents the description of + *error*, as returned by the :c:func:`gai_strerror` C function. The + numeric *error* value will match one of the :const:`EAI_\*` constants + defined in this module. .. exception:: timeout - This exception is raised when a timeout occurs on a socket which has had - timeouts enabled via a prior call to :meth:`~socket.settimeout`. The - accompanying value is a string whose value is currently always "timed out". + A subclass of :exc:`socket.error`, this exception is raised when a timeout + occurs on a socket which has had timeouts enabled via a prior call to + :meth:`~socket.settimeout` (or implicitly through + :func:`~socket.setdefaulttimeout`). The accompanying value is a string + whose value is currently always "timed out". .. data:: AF_UNIX @@ -174,6 +178,21 @@ The module :mod:`socket` exports the following constants and functions: (Only :const:`SOCK_STREAM` and :const:`SOCK_DGRAM` appear to be generally useful.) +.. data:: SOCK_CLOEXEC + SOCK_NONBLOCK + + These two constants, if defined, can be combined with the socket types and + allow you to set some flags atomically (thus avoiding possible race + conditions and the need for separate calls). + + .. seealso:: + + `Secure File Descriptor Handling <http://udrepper.livejournal.com/20407.html>`_ + for a more thorough explanation. + + Availability: Linux >= 2.6.27. + + .. versionadded:: 3.2 .. data:: SO_* SOMAXCONN @@ -215,7 +234,7 @@ The module :mod:`socket` exports the following constants and functions: this platform. -.. function:: create_connection(address[, timeout]) +.. function:: create_connection(address[, timeout[, source_address]]) Convenience function. Connect to *address* (a 2-tuple ``(host, port)``), and return the socket object. Passing the optional *timeout* parameter will @@ -223,8 +242,18 @@ The module :mod:`socket` exports the following constants and functions: *timeout* is supplied, the global default timeout setting returned by :func:`getdefaulttimeout` is used. + If supplied, *source_address* must be a 2-tuple ``(host, port)`` for the + socket to bind to as its source address before connecting. If host or port + are '' or 0 respectively the OS default behavior will be used. + + .. versionchanged:: 3.2 + *source_address* was added. -.. function:: getaddrinfo(host, port, family=0, socktype=0, proto=0, flags=0) + .. versionchanged:: 3.2 + support for the :keyword:`with` statement was added. + + +.. function:: getaddrinfo(host, port, family=0, type=0, proto=0, flags=0) Translate the *host*/*port* argument into a sequence of 5-tuples that contain all the necessary arguments for creating a socket connected to that service. @@ -233,7 +262,7 @@ The module :mod:`socket` exports the following constants and functions: port number or ``None``. By passing ``None`` as the value of *host* and *port*, you can pass ``NULL`` to the underlying C API. - The *family*, *socktype* and *proto* arguments can be optionally specified + The *family*, *type* and *proto* arguments can be optionally specified in order to narrow the list of addresses returned. Passing zero as a value for each of these arguments selects the full range of results. The *flags* argument can be one or several of the ``AI_*`` constants, @@ -243,9 +272,9 @@ The module :mod:`socket` exports the following constants and functions: The function returns a list of 5-tuples with the following structure: - ``(family, socktype, proto, canonname, sockaddr)`` + ``(family, type, proto, canonname, sockaddr)`` - In these tuples, *family*, *socktype*, *proto* are all integers and are + In these tuples, *family*, *type*, *proto* are all integers and are meant to be passed to the :func:`socket` function. *canonname* will be a string representing the canonical name of the *host* if :const:`AI_CANONNAME` is part of the *flags* argument; else *canonname* @@ -259,10 +288,13 @@ The module :mod:`socket` exports the following constants and functions: connection to ``www.python.org`` on port 80 (results may differ on your system if IPv6 isn't enabled):: - >>> socket.getaddrinfo("www.python.org", 80, 0, 0, socket.SOL_TCP) + >>> socket.getaddrinfo("www.python.org", 80, proto=socket.SOL_TCP) [(2, 1, 6, '', ('82.94.164.162', 80)), (10, 1, 6, '', ('2001:888:2000:d::a2', 80, 0, 0))] + .. versionchanged:: 3.2 + parameters can now be passed as single keyword arguments. + .. function:: getfqdn([name]) Return a fully qualified domain name for *name*. If *name* is omitted or empty, @@ -368,6 +400,10 @@ The module :mod:`socket` exports the following constants and functions: if defined on the platform; otherwise, the default is :const:`AF_INET`. Availability: Unix. + .. versionchanged:: 3.2 + The returned socket objects now support the whole socket API, rather + than a subset. + .. function:: fromfd(fd, family, type[, proto]) @@ -379,7 +415,6 @@ The module :mod:`socket` exports the following constants and functions: This function is rarely needed, but can be used to get or set socket options on a socket passed to a program as standard input or output (such as a server started by the Unix inet daemon). The socket is assumed to be in blocking mode. - Availability: Unix. .. function:: ntohl(x) @@ -415,7 +450,7 @@ The module :mod:`socket` exports the following constants and functions: Convert an IPv4 address from dotted-quad string format (for example, '123.45.67.89') to 32-bit packed binary format, as a bytes object four characters in length. This is useful when conversing with a program that uses the standard C - library and needs objects of type :ctype:`struct in_addr`, which is the C type + library and needs objects of type :c:type:`struct in_addr`, which is the C type for the 32-bit packed binary this function returns. :func:`inet_aton` also accepts strings with less than three dots; see the @@ -423,7 +458,7 @@ The module :mod:`socket` exports the following constants and functions: If the IPv4 address string passed to this function is invalid, :exc:`socket.error` will be raised. Note that exactly what is valid depends on - the underlying C implementation of :cfunc:`inet_aton`. + the underlying C implementation of :c:func:`inet_aton`. :func:`inet_aton` does not support IPv6, and :func:`inet_pton` should be used instead for IPv4/v6 dual stack support. @@ -434,7 +469,7 @@ The module :mod:`socket` exports the following constants and functions: Convert a 32-bit packed IPv4 address (a bytes object four characters in length) to its standard dotted-quad string representation (for example, '123.45.67.89'). This is useful when conversing with a program that uses the - standard C library and needs objects of type :ctype:`struct in_addr`, which + standard C library and needs objects of type :c:type:`struct in_addr`, which is the C type for the 32-bit packed binary data this function takes as an argument. @@ -448,14 +483,14 @@ The module :mod:`socket` exports the following constants and functions: Convert an IP address from its family-specific string format to a packed, binary format. :func:`inet_pton` is useful when a library or network protocol - calls for an object of type :ctype:`struct in_addr` (similar to - :func:`inet_aton`) or :ctype:`struct in6_addr`. + calls for an object of type :c:type:`struct in_addr` (similar to + :func:`inet_aton`) or :c:type:`struct in6_addr`. Supported values for *address_family* are currently :const:`AF_INET` and :const:`AF_INET6`. If the IP address string *ip_string* is invalid, :exc:`socket.error` will be raised. Note that exactly what is valid depends on both the value of *address_family* and the underlying implementation of - :cfunc:`inet_pton`. + :c:func:`inet_pton`. Availability: Unix (maybe not all platforms). @@ -465,8 +500,8 @@ The module :mod:`socket` exports the following constants and functions: Convert a packed IP address (a bytes object of some number of characters) to its standard, family-specific string representation (for example, ``'7.10.0.5'`` or ``'5aef:2b::8'``). :func:`inet_ntop` is useful when a library or network protocol - returns an object of type :ctype:`struct in_addr` (similar to :func:`inet_ntoa`) - or :ctype:`struct in6_addr`. + returns an object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) + or :c:type:`struct in6_addr`. Supported values for *address_family* are currently :const:`AF_INET` and :const:`AF_INET6`. If the string *packed_ip* is not the correct length for the @@ -542,13 +577,22 @@ correspond to Unix system calls applicable to sockets. .. method:: socket.connect_ex(address) Like ``connect(address)``, but return an error indicator instead of raising an - exception for errors returned by the C-level :cfunc:`connect` call (other + exception for errors returned by the C-level :c:func:`connect` call (other problems, such as "host not found," can still raise exceptions). The error indicator is ``0`` if the operation succeeded, otherwise the value of the - :cdata:`errno` variable. This is useful to support, for example, asynchronous + :c:data:`errno` variable. This is useful to support, for example, asynchronous connects. +.. method:: socket.detach() + + Put the socket object into closed state without actually closing the + underlying file descriptor. The file descriptor is returned, and can + be reused for other purposes. + + .. versionadded:: 3.2 + + .. method:: socket.fileno() Return the socket's file descriptor (a small integer). This is useful with @@ -612,7 +656,8 @@ correspond to Unix system calls applicable to sockets. is system-dependent (usually 5). -.. method:: socket.makefile(mode='r', buffering=None, *, encoding=None, errors=None, newline=None) +.. method:: socket.makefile(mode='r', buffering=None, *, encoding=None, \ + errors=None, newline=None) .. index:: single: I/O control; buffering @@ -668,9 +713,9 @@ correspond to Unix system calls applicable to sockets. Receive up to *nbytes* bytes from the socket, storing the data into a buffer rather than creating a new bytestring. If *nbytes* is not specified (or 0), - receive up to the size available in the given buffer. See the Unix manual page - :manpage:`recv(2)` for the meaning of the optional argument *flags*; it defaults - to zero. + receive up to the size available in the given buffer. Returns the number of + bytes received. See the Unix manual page :manpage:`recv(2)` for the meaning + of the optional argument *flags*; it defaults to zero. .. method:: socket.send(bytes[, flags]) @@ -807,6 +852,21 @@ before calling :meth:`~socket.connect` or pass a timeout parameter to return a connection timeout error of its own regardless of any Python socket timeout setting. +Timeouts and the ``accept`` method +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If :func:`getdefaulttimeout` is not :const:`None`, sockets returned by +the :meth:`~socket.accept` method inherit that timeout. Otherwise, the +behaviour depends on settings of the listening socket: + +* if the listening socket is in *blocking mode* or in *timeout mode*, + the socket returned by :meth:`~socket.accept` is in *blocking mode*; + +* if the listening socket is in *non-blocking mode*, whether the socket + returned by :meth:`~socket.accept` is in blocking or non-blocking mode + is operating system-dependent. If you want to ensure cross-platform + behaviour, it is recommended you manually override this setting. + .. _socket-example: diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index 82d1107..ed547f5e 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -1,10 +1,13 @@ - :mod:`socketserver` --- A framework for network servers ======================================================= .. module:: socketserver :synopsis: A framework for network servers. +**Source code:** :source:`Lib/socketserver.py` + +-------------- + The :mod:`socketserver` module simplifies the task of writing network servers. There are four basic server classes: :class:`TCPServer` uses the Internet TCP diff --git a/Doc/library/someos.rst b/Doc/library/someos.rst index cf3eb6b..d2009bb 100644 --- a/Doc/library/someos.rst +++ b/Doc/library/someos.rst @@ -15,6 +15,7 @@ some other systems as well (e.g. Windows). Here's an overview: select.rst threading.rst multiprocessing.rst + concurrent.futures.rst mmap.rst readline.rst rlcompleter.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 0603738..7367674 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -90,7 +90,7 @@ This example uses the iterator form:: .. seealso:: - http://www.pysqlite.org + http://code.google.com/p/pysqlite/ The pysqlite web page -- sqlite3 is developed externally under the name "pysqlite". @@ -227,6 +227,12 @@ Connection Objects one of "DEFERRED", "IMMEDIATE" or "EXCLUSIVE". See section :ref:`sqlite3-controlling-transactions` for a more detailed explanation. +.. attribute:: Connection.in_transaction + + :const:`True` if a transaction is active (there are uncommitted changes), + :const:`False` otherwise. Read-only attribute. + + .. versionadded:: 3.2 .. method:: Connection.cursor([cursorClass]) @@ -363,6 +369,29 @@ Connection Objects method with :const:`None` for *handler*. +.. method:: Connection.enable_load_extension(enabled) + + This routine allows/disallows the SQLite engine to load SQLite extensions + from shared libraries. SQLite extensions can define new functions, + aggregates or whole new virtual table implementations. One well-known + extension is the fulltext-search extension distributed with SQLite. + + .. versionadded:: 3.2 + + .. literalinclude:: ../includes/sqlite3/load_extension.py + + Loadable extensions are disabled by default. See [#f1]_. + +.. method:: Connection.load_extension(path) + + This routine loads a SQLite extension from a shared library. You have to + enable extension loading with :meth:`enable_load_extension` before you can + use this routine. + + .. versionadded:: 3.2 + + Loadable extensions are disabled by default. See [#f1]_. + .. attribute:: Connection.row_factory You can change this attribute to a callable that accepts the cursor and the @@ -785,7 +814,8 @@ So if you are within a transaction and issue a command like ``CREATE TABLE before executing that command. There are two reasons for doing that. The first is that some of these commands don't work within transactions. The other reason is that sqlite3 needs to keep track of the transaction state (if a transaction -is active or not). +is active or not). The current transaction state is exposed through the +:attr:`Connection.in_transaction` attribute of the connection object. You can control which kind of ``BEGIN`` statements sqlite3 implicitly executes (or none at all) via the *isolation_level* parameter to the :func:`connect` @@ -852,3 +882,10 @@ threads. If you still try to do so, you will get an exception at runtime. The only exception is calling the :meth:`~Connection.interrupt` method, which only makes sense to call from a different thread. + +.. rubric:: Footnotes + +.. [#f1] The sqlite3 module is not built with loadable extension support by + default, because some platforms (notably Mac OS X) have SQLite libraries which + are compiled without this feature. To get loadable extension support, you must + pass --enable-loadable-sqlite-extensions to configure. diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index ff7a016..011d378 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -12,6 +12,10 @@ .. index:: TLS, SSL, Transport Layer Security, Secure Sockets Layer +**Source code:** :source:`Lib/ssl.py` + +-------------- + This module provides access to Transport Layer Security (often known as "Secure Sockets Layer") encryption and peer authentication facilities for network sockets, both client-side and server-side. This module uses the OpenSSL @@ -31,23 +35,43 @@ the documents in the "See Also" section at the bottom. This module provides a class, :class:`ssl.SSLSocket`, which is derived from the :class:`socket.socket` type, and provides a socket-like wrapper that also encrypts and decrypts the data going over the socket with SSL. It supports -additional :meth:`read` and :meth:`write` methods, along with a method, -:meth:`getpeercert`, to retrieve the certificate of the other side of the -connection, and a method, :meth:`cipher`, to retrieve the cipher being used for -the secure connection. +additional methods such as :meth:`getpeercert`, which retrieves the +certificate of the other side of the connection, and :meth:`cipher`,which +retrieves the cipher being used for the secure connection. + +For more sophisticated applications, the :class:`ssl.SSLContext` class +helps manage settings and certificates, which can then be inherited +by SSL sockets created through the :meth:`SSLContext.wrap_socket` method. + Functions, Constants, and Exceptions ------------------------------------ .. exception:: SSLError - Raised to signal an error from the underlying SSL implementation. This - signifies some problem in the higher-level encryption and authentication - layer that's superimposed on the underlying network connection. This error + Raised to signal an error from the underlying SSL implementation + (currently provided by the OpenSSL library). This signifies some + problem in the higher-level encryption and authentication layer that's + superimposed on the underlying network connection. This error is a subtype of :exc:`socket.error`, which in turn is a subtype of - :exc:`IOError`. + :exc:`IOError`. The error code and message of :exc:`SSLError` instances + are provided by the OpenSSL library. + +.. exception:: CertificateError + + Raised to signal an error with a certificate (such as mismatching + hostname). Certificate errors detected by OpenSSL, though, raise + an :exc:`SSLError`. + + +Socket creation +^^^^^^^^^^^^^^^ -.. function:: wrap_socket(sock, keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ssl_version={see docs}, ca_certs=None, do_handshake_on_connect=True, suppress_ragged_eofs=True) +The following function allows for standalone socket creation. Starting from +Python 3.2, it can be more flexible to use :meth:`SSLContext.wrap_socket` +instead. + +.. function:: wrap_socket(sock, keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ssl_version={see docs}, ca_certs=None, do_handshake_on_connect=True, suppress_ragged_eofs=True, ciphers=None) Takes an instance ``sock`` of :class:`socket.socket`, and returns an instance of :class:`ssl.SSLSocket`, a subtype of :class:`socket.socket`, which wraps @@ -64,19 +88,6 @@ Functions, Constants, and Exceptions connection. See the discussion of :ref:`ssl-certificates` for more information on how the certificate is stored in the ``certfile``. - Often the private key is stored in the same file as the certificate; in this - case, only the ``certfile`` parameter need be passed. If the private key is - stored in a separate file, both parameters must be used. If the private key - is stored in the ``certfile``, it should come before the first certificate in - the certificate chain:: - - -----BEGIN RSA PRIVATE KEY----- - ... (private key in base64 encoding) ... - -----END RSA PRIVATE KEY----- - -----BEGIN CERTIFICATE----- - ... (certificate in base64 PEM encoding) ... - -----END CERTIFICATE----- - The parameter ``server_side`` is a boolean which identifies whether server-side or client-side behavior is desired from this socket. @@ -110,14 +121,26 @@ Functions, Constants, and Exceptions ======================== ========= ========= ========== ========= *client* / **server** **SSLv2** **SSLv3** **SSLv23** **TLSv1** ------------------------ --------- --------- ---------- --------- - *SSLv2* yes no yes* no + *SSLv2* yes no yes no *SSLv3* yes yes yes no *SSLv23* yes no yes no *TLSv1* no no yes yes ======================== ========= ========= ========== ========= - In some older versions of OpenSSL (for instance, 0.9.7l on OS X 10.4), an - SSLv2 client could not connect to an SSLv23 server. + .. note:: + + Which connections succeed will vary depending on the version of + OpenSSL. For instance, in some older versions of OpenSSL (such + as 0.9.7l on OS X 10.4), an SSLv2 client could not connect to an + SSLv23 server. Another example: beginning with OpenSSL 1.0.0, + an SSLv23 client will not actually attempt SSLv2 connections + unless you explicitly enable SSLv2 ciphers; for example, you + might specify ``"ALL"`` or ``"SSLv2"`` as the *ciphers* parameter + to enable them. + + The *ciphers* parameter sets the available ciphers for this SSL object. + It should be a string in the `OpenSSL cipher list format + <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`_. The parameter ``do_handshake_on_connect`` specifies whether to do the SSL handshake automatically after doing a :meth:`socket.connect`, or whether the @@ -127,10 +150,17 @@ Functions, Constants, and Exceptions blocking behavior of the socket I/O involved in the handshake. The parameter ``suppress_ragged_eofs`` specifies how the - :meth:`SSLSocket.read` method should signal unexpected EOF from the other end + :meth:`SSLSocket.recv` method should signal unexpected EOF from the other end of the connection. If specified as :const:`True` (the default), it returns a - normal EOF in response to unexpected EOF errors raised from the underlying - socket; if :const:`False`, it will raise the exceptions back to the caller. + normal EOF (an empty bytes object) in response to unexpected EOF errors + raised from the underlying socket; if :const:`False`, it will raise the + exceptions back to the caller. + + .. versionchanged:: 3.2 + New optional argument *ciphers*. + +Random generation +^^^^^^^^^^^^^^^^^ .. function:: RAND_status() @@ -157,6 +187,32 @@ Functions, Constants, and Exceptions string (so you can always use :const:`0.0`). See :rfc:`1750` for more information on sources of entropy. +Certificate handling +^^^^^^^^^^^^^^^^^^^^ + +.. function:: match_hostname(cert, hostname) + + Verify that *cert* (in decoded format as returned by + :meth:`SSLSocket.getpeercert`) matches the given *hostname*. The rules + applied are those for checking the identity of HTTPS servers as outlined + in :rfc:`2818`, except that IP addresses are not currently supported. + In addition to HTTPS, this function should be suitable for checking the + identity of servers in various SSL-based protocols such as FTPS, IMAPS, + POPS and others. + + :exc:`CertificateError` is raised on failure. On success, the function + returns nothing:: + + >>> cert = {'subject': ((('commonName', 'example.com'),),)} + >>> ssl.match_hostname(cert, "example.com") + >>> ssl.match_hostname(cert, "example.org") + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + File "/home/py3k/Lib/ssl.py", line 130, in match_hostname + ssl.CertificateError: hostname 'example.org' doesn't match 'example.com' + + .. versionadded:: 3.2 + .. function:: cert_time_to_seconds(timestring) Returns a floating-point value containing a normal seconds-after-the-epoch @@ -171,7 +227,6 @@ Functions, Constants, and Exceptions >>> import time >>> time.ctime(ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT")) 'Wed May 9 00:00:00 2007' - >>> .. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv3, ca_certs=None) @@ -194,26 +249,41 @@ Functions, Constants, and Exceptions Given a certificate as an ASCII PEM string, returns a DER-encoded sequence of bytes for that same certificate. +Constants +^^^^^^^^^ + .. data:: CERT_NONE - Value to pass to the ``cert_reqs`` parameter to :func:`sslobject` when no - certificates will be required or validated from the other side of the socket - connection. + Possible value for :attr:`SSLContext.verify_mode`, or the ``cert_reqs`` + parameter to :func:`wrap_socket`. In this mode (the default), no + certificates will be required from the other side of the socket connection. + If a certificate is received from the other end, no attempt to validate it + is made. + + See the discussion of :ref:`ssl-security` below. .. data:: CERT_OPTIONAL - Value to pass to the ``cert_reqs`` parameter to :func:`sslobject` when no - certificates will be required from the other side of the socket connection, - but if they are provided, will be validated. Note that use of this setting - requires a valid certificate validation file also be passed as a value of the - ``ca_certs`` parameter. + Possible value for :attr:`SSLContext.verify_mode`, or the ``cert_reqs`` + parameter to :func:`wrap_socket`. In this mode no certificates will be + required from the other side of the socket connection; but if they + are provided, validation will be attempted and an :class:`SSLError` + will be raised on failure. + + Use of this setting requires a valid set of CA certificates to + be passed, either to :meth:`SSLContext.load_verify_locations` or as a + value of the ``ca_certs`` parameter to :func:`wrap_socket`. .. data:: CERT_REQUIRED - Value to pass to the ``cert_reqs`` parameter to :func:`sslobject` when - certificates will be required from the other side of the socket connection. - Note that use of this setting requires a valid certificate validation file - also be passed as a value of the ``ca_certs`` parameter. + Possible value for :attr:`SSLContext.verify_mode`, or the ``cert_reqs`` + parameter to :func:`wrap_socket`. In this mode, certificates are + required from the other side of the socket connection; an :class:`SSLError` + will be raised if no certificate is provided, or if its validation fails. + + Use of this setting requires a valid set of CA certificates to + be passed, either to :meth:`SSLContext.load_verify_locations` or as a + value of the ``ca_certs`` parameter to :func:`wrap_socket`. .. data:: PROTOCOL_SSLv2 @@ -241,22 +311,101 @@ Functions, Constants, and Exceptions modern version, and probably the best choice for maximum protection, if both sides can speak it. +.. data:: OP_ALL + + Enables workarounds for various bugs present in other SSL implementations. + This option is set by default. -SSLSocket Objects ------------------ + .. versionadded:: 3.2 -.. method:: SSLSocket.read(nbytes=1024, buffer=None) +.. data:: OP_NO_SSLv2 - Reads up to ``nbytes`` bytes from the SSL-encrypted channel and returns them. - If the ``buffer`` is specified, it will attempt to read into the buffer the - minimum of the size of the buffer and ``nbytes``, if that is specified. If - no buffer is specified, an immutable buffer is allocated and returned with - the data read from the socket. + Prevents an SSLv2 connection. This option is only applicable in + conjunction with :const:`PROTOCOL_SSLv23`. It prevents the peers from + choosing SSLv2 as the protocol version. -.. method:: SSLSocket.write(data) + .. versionadded:: 3.2 - Writes the ``data`` to the other side of the connection, using the SSL - channel to encrypt. Returns the number of bytes written. +.. data:: OP_NO_SSLv3 + + Prevents an SSLv3 connection. This option is only applicable in + conjunction with :const:`PROTOCOL_SSLv23`. It prevents the peers from + choosing SSLv3 as the protocol version. + + .. versionadded:: 3.2 + +.. data:: OP_NO_TLSv1 + + Prevents a TLSv1 connection. This option is only applicable in + conjunction with :const:`PROTOCOL_SSLv23`. It prevents the peers from + choosing TLSv1 as the protocol version. + + .. versionadded:: 3.2 + +.. data:: HAS_SNI + + Whether the OpenSSL library has built-in support for the *Server Name + Indication* extension to the SSLv3 and TLSv1 protocols (as defined in + :rfc:`4366`). When true, you can use the *server_hostname* argument to + :meth:`SSLContext.wrap_socket`. + + .. versionadded:: 3.2 + +.. data:: OPENSSL_VERSION + + The version string of the OpenSSL library loaded by the interpreter:: + + >>> ssl.OPENSSL_VERSION + 'OpenSSL 0.9.8k 25 Mar 2009' + + .. versionadded:: 3.2 + +.. data:: OPENSSL_VERSION_INFO + + A tuple of five integers representing version information about the + OpenSSL library:: + + >>> ssl.OPENSSL_VERSION_INFO + (0, 9, 8, 11, 15) + + .. versionadded:: 3.2 + +.. data:: OPENSSL_VERSION_NUMBER + + The raw version number of the OpenSSL library, as a single integer:: + + >>> ssl.OPENSSL_VERSION_NUMBER + 9470143 + >>> hex(ssl.OPENSSL_VERSION_NUMBER) + '0x9080bf' + + .. versionadded:: 3.2 + + +SSL Sockets +----------- + +SSL sockets provide the following methods of :ref:`socket-objects`: + +- :meth:`~socket.socket.accept()` +- :meth:`~socket.socket.bind()` +- :meth:`~socket.socket.close()` +- :meth:`~socket.socket.connect()` +- :meth:`~socket.socket.detach()` +- :meth:`~socket.socket.fileno()` +- :meth:`~socket.socket.getpeername()`, :meth:`~socket.socket.getsockname()` +- :meth:`~socket.socket.getsockopt()`, :meth:`~socket.socket.setsockopt()` +- :meth:`~socket.socket.gettimeout()`, :meth:`~socket.socket.settimeout()`, + :meth:`~socket.socket.setblocking()` +- :meth:`~socket.socket.listen()` +- :meth:`~socket.socket.makefile()` +- :meth:`~socket.socket.recv()`, :meth:`~socket.socket.recv_into()` + (but passing a non-zero ``flags`` argument is not allowed) +- :meth:`~socket.socket.send()`, :meth:`~socket.socket.sendall()` (with + the same limitation) +- :meth:`~socket.socket.shutdown()` + +They also have the following additional methods and attributes: .. method:: SSLSocket.do_handshake() @@ -288,11 +437,9 @@ SSLSocket Objects certificate was not validated, the dict is empty. If the certificate was validated, it returns a dict with the keys ``subject`` (the principal for which the certificate was issued), and ``notAfter`` (the time after which the - certificate should not be trusted). The certificate was already validated, - so the ``notBefore`` and ``issuer`` fields are not returned. If a - certificate contains an instance of the *Subject Alternative Name* extension - (see :rfc:`3280`), there will also be a ``subjectAltName`` key in the - dictionary. + certificate should not be trusted). If a certificate contains an instance + of the *Subject Alternative Name* extension (see :rfc:`3280`), there will + also be a ``subjectAltName`` key in the dictionary. The "subject" field is a tuple containing the sequence of relative distinguished names (RDNs) given in the certificate's data structure for the @@ -314,6 +461,10 @@ SSLSocket Objects been validated, but if :const:`CERT_NONE` was used to establish the connection, the certificate, if present, will not have been validated. + .. versionchanged:: 3.2 + The returned dictionary includes additional items such as ``issuer`` + and ``notBefore``. + .. method:: SSLSocket.cipher() Returns a three-value tuple containing the name of the cipher being used, the @@ -329,6 +480,142 @@ SSLSocket Objects returned socket should always be used for further communication with the other side of the connection, rather than the original socket. + +.. attribute:: SSLSocket.context + + The :class:`SSLContext` object this SSL socket is tied to. If the SSL + socket was created using the top-level :func:`wrap_socket` function + (rather than :meth:`SSLContext.wrap_socket`), this is a custom context + object created for this SSL socket. + + .. versionadded:: 3.2 + + +SSL Contexts +------------ + +.. versionadded:: 3.2 + +An SSL context holds various data longer-lived than single SSL connections, +such as SSL configuration options, certificate(s) and private key(s). +It also manages a cache of SSL sessions for server-side sockets, in order +to speed up repeated connections from the same clients. + +.. class:: SSLContext(protocol) + + Create a new SSL context. You must pass *protocol* which must be one + of the ``PROTOCOL_*`` constants defined in this module. + :data:`PROTOCOL_SSLv23` is recommended for maximum interoperability. + + +:class:`SSLContext` objects have the following methods and attributes: + +.. method:: SSLContext.load_cert_chain(certfile, keyfile=None) + + Load a private key and the corresponding certificate. The *certfile* + string must be the path to a single file in PEM format containing the + certificate as well as any number of CA certificates needed to establish + the certificate's authenticity. The *keyfile* string, if present, must + point to a file containing the private key in. Otherwise the private + key will be taken from *certfile* as well. See the discussion of + :ref:`ssl-certificates` for more information on how the certificate + is stored in the *certfile*. + + An :class:`SSLError` is raised if the private key doesn't + match with the certificate. + +.. method:: SSLContext.load_verify_locations(cafile=None, capath=None) + + Load a set of "certification authority" (CA) certificates used to validate + other peers' certificates when :data:`verify_mode` is other than + :data:`CERT_NONE`. At least one of *cafile* or *capath* must be specified. + + The *cafile* string, if present, is the path to a file of concatenated + CA certificates in PEM format. See the discussion of + :ref:`ssl-certificates` for more information about how to arrange the + certificates in this file. + + The *capath* string, if present, is + the path to a directory containing several CA certificates in PEM format, + following an `OpenSSL specific layout + <http://www.openssl.org/docs/ssl/SSL_CTX_load_verify_locations.html>`_. + +.. method:: SSLContext.set_default_verify_paths() + + Load a set of default "certification authority" (CA) certificates from + a filesystem path defined when building the OpenSSL library. Unfortunately, + there's no easy way to know whether this method succeeds: no error is + returned if no certificates are to be found. When the OpenSSL library is + provided as part of the operating system, though, it is likely to be + configured properly. + +.. method:: SSLContext.set_ciphers(ciphers) + + Set the available ciphers for sockets created with this context. + It should be a string in the `OpenSSL cipher list format + <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`_. + If no cipher can be selected (because compile-time options or other + configuration forbids use of all the specified ciphers), an + :class:`SSLError` will be raised. + + .. note:: + when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will + give the currently selected cipher. + +.. method:: SSLContext.wrap_socket(sock, server_side=False, \ + do_handshake_on_connect=True, suppress_ragged_eofs=True, \ + server_hostname=None) + + Wrap an existing Python socket *sock* and return an :class:`SSLSocket` + object. The SSL socket is tied to the context, its settings and + certificates. The parameters *server_side*, *do_handshake_on_connect* + and *suppress_ragged_eofs* have the same meaning as in the top-level + :func:`wrap_socket` function. + + On client connections, the optional parameter *server_hostname* specifies + the hostname of the service which we are connecting to. This allows a + single server to host multiple SSL-based services with distinct certificates, + quite similarly to HTTP virtual hosts. Specifying *server_hostname* + will raise a :exc:`ValueError` if the OpenSSL library doesn't have support + for it (that is, if :data:`HAS_SNI` is :const:`False`). Specifying + *server_hostname* will also raise a :exc:`ValueError` if *server_side* + is true. + +.. method:: SSLContext.session_stats() + + Get statistics about the SSL sessions created or managed by this context. + A dictionary is returned which maps the names of each `piece of information + <http://www.openssl.org/docs/ssl/SSL_CTX_sess_number.html>`_ to their + numeric values. For example, here is the total number of hits and misses + in the session cache since the context was created:: + + >>> stats = context.session_stats() + >>> stats['hits'], stats['misses'] + (0, 0) + +.. attribute:: SSLContext.options + + An integer representing the set of SSL options enabled on this context. + The default value is :data:`OP_ALL`, but you can specify other options + such as :data:`OP_NO_SSLv2` by ORing them together. + + .. note:: + With versions of OpenSSL older than 0.9.8m, it is only possible + to set options, not to clear them. Attempting to clear an option + (by resetting the corresponding bits) will raise a ``ValueError``. + +.. attribute:: SSLContext.protocol + + The protocol version chosen when constructing the context. This attribute + is read-only. + +.. attribute:: SSLContext.verify_mode + + Whether to try to verify other peers' certificates and how to behave + if verification fails. This attribute must be one of + :data:`CERT_NONE`, :data:`CERT_OPTIONAL` or :data:`CERT_REQUIRED`. + + .. index:: single: certificates .. index:: single: X509 certificate @@ -374,6 +661,9 @@ and a footer line:: ... (certificate in base64 PEM encoding) ... -----END CERTIFICATE----- +Certificate chains +^^^^^^^^^^^^^^^^^^ + The Python files which contain certificates can contain a sequence of certificates, sometimes called a *certificate chain*. This chain should start with the specific certificate for the principal who "is" the client or server, @@ -397,6 +687,9 @@ certification authority's certificate:: ... (the root certificate for the CA's issuer)... -----END CERTIFICATE----- +CA certificates +^^^^^^^^^^^^^^^ + If you are going to require validation of the other side of the connection's certificate, you need to provide a "CA certs" file, filled with the certificate chains for each issuer you are willing to trust. Again, this file just contains @@ -416,6 +709,25 @@ peer is supposed to furnish the other certificates necessary to chain from its certificate to a root certificate. See :rfc:`4158` for more discussion of the way in which certification chains can be built. +Combined key and certificate +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Often the private key is stored in the same file as the certificate; in this +case, only the ``certfile`` parameter to :meth:`SSLContext.load_cert_chain` +and :func:`wrap_socket` needs to be passed. If the private key is stored +with the certificate, it should come before the first certificate in +the certificate chain:: + + -----BEGIN RSA PRIVATE KEY----- + ... (private key in base64 encoding) ... + -----END RSA PRIVATE KEY----- + -----BEGIN CERTIFICATE----- + ... (certificate in base64 PEM encoding) ... + -----END CERTIFICATE----- + +Self-signed certificates +^^^^^^^^^^^^^^^^^^^^^^^^ + If you are going to create a server that provides SSL-encrypted connection services, you will need to acquire a certificate for that service. There are many ways of acquiring appropriate certificates, such as buying one from a @@ -469,101 +781,180 @@ should use the following idiom:: Client-side operation ^^^^^^^^^^^^^^^^^^^^^ -This example connects to an SSL server, prints the server's address and -certificate, sends some bytes, and reads part of the response:: +This example connects to an SSL server and prints the server's certificate:: import socket, ssl, pprint s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - # require a certificate from the server ssl_sock = ssl.wrap_socket(s, ca_certs="/etc/ca_certs_file", cert_reqs=ssl.CERT_REQUIRED) - ssl_sock.connect(('www.verisign.com', 443)) - print(repr(ssl_sock.getpeername())) pprint.pprint(ssl_sock.getpeercert()) - print(pprint.pformat(ssl_sock.getpeercert())) - - # Set a simple HTTP request -- use http.client in actual code. - ssl_sock.write("""GET / HTTP/1.0\r - Host: www.verisign.com\r\n\r\n""") - - # Read a chunk of data. Will not necessarily - # read all the data returned by the server. - data = ssl_sock.read() - # note that closing the SSLSocket will also close the underlying socket ssl_sock.close() -As of September 6, 2007, the certificate printed by this program looked like +As of October 6, 2010, the certificate printed by this program looks like this:: - {'notAfter': 'May 8 23:59:59 2009 GMT', - 'subject': ((('serialNumber', '2497886'),), - (('1.3.6.1.4.1.311.60.2.1.3', 'US'),), - (('1.3.6.1.4.1.311.60.2.1.2', 'Delaware'),), - (('countryName', 'US'),), - (('postalCode', '94043'),), - (('stateOrProvinceName', 'California'),), - (('localityName', 'Mountain View'),), - (('streetAddress', '487 East Middlefield Road'),), - (('organizationName', 'VeriSign, Inc.'),), - (('organizationalUnitName', - 'Production Security Services'),), - (('organizationalUnitName', - 'Terms of use at www.verisign.com/rpa (c)06'),), - (('commonName', 'www.verisign.com'),))} - -which is a fairly poorly-formed ``subject`` field. + {'notAfter': 'May 25 23:59:59 2012 GMT', + 'subject': ((('1.3.6.1.4.1.311.60.2.1.3', 'US'),), + (('1.3.6.1.4.1.311.60.2.1.2', 'Delaware'),), + (('businessCategory', 'V1.0, Clause 5.(b)'),), + (('serialNumber', '2497886'),), + (('countryName', 'US'),), + (('postalCode', '94043'),), + (('stateOrProvinceName', 'California'),), + (('localityName', 'Mountain View'),), + (('streetAddress', '487 East Middlefield Road'),), + (('organizationName', 'VeriSign, Inc.'),), + (('organizationalUnitName', ' Production Security Services'),), + (('commonName', 'www.verisign.com'),))} + +This other example first creates an SSL context, instructs it to verify +certificates sent by peers, and feeds it a set of recognized certificate +authorities (CA):: + + >>> context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + >>> context.verify_mode = ssl.CERT_REQUIRED + >>> context.load_verify_locations("/etc/ssl/certs/ca-bundle.crt") + +(it is assumed your operating system places a bundle of all CA certificates +in ``/etc/ssl/certs/ca-bundle.crt``; if not, you'll get an error and have +to adjust the location) + +When you use the context to connect to a server, :const:`CERT_REQUIRED` +validates the server certificate: it ensures that the server certificate +was signed with one of the CA certificates, and checks the signature for +correctness:: + + >>> conn = context.wrap_socket(socket.socket(socket.AF_INET)) + >>> conn.connect(("linuxfr.org", 443)) + +You should then fetch the certificate and check its fields for conformity:: + + >>> cert = conn.getpeercert() + >>> ssl.match_hostname(cert, "linuxfr.org") + +Visual inspection shows that the certificate does identify the desired service +(that is, the HTTPS host ``linuxfr.org``):: + + >>> pprint.pprint(cert) + {'notAfter': 'Jun 26 21:41:46 2011 GMT', + 'subject': ((('commonName', 'linuxfr.org'),),), + 'subjectAltName': (('DNS', 'linuxfr.org'), ('othername', '<unsupported>'))} + +Now that you are assured of its authenticity, you can proceed to talk with +the server:: + + >>> conn.sendall(b"HEAD / HTTP/1.0\r\nHost: linuxfr.org\r\n\r\n") + >>> pprint.pprint(conn.recv(1024).split(b"\r\n")) + [b'HTTP/1.1 302 Found', + b'Date: Sun, 16 May 2010 13:43:28 GMT', + b'Server: Apache/2.2', + b'Location: https://linuxfr.org/pub/', + b'Vary: Accept-Encoding', + b'Connection: close', + b'Content-Type: text/html; charset=iso-8859-1', + b'', + b''] + +See the discussion of :ref:`ssl-security` below. + Server-side operation ^^^^^^^^^^^^^^^^^^^^^ -For server operation, typically you'd need to have a server certificate, and -private key, each in a file. You'd open a socket, bind it to a port, call -:meth:`listen` on it, then start waiting for clients to connect:: +For server operation, typically you'll need to have a server certificate, and +private key, each in a file. You'll first create a context holding the key +and the certificate, so that clients can check your authenticity. Then +you'll open a socket, bind it to a port, call :meth:`listen` on it, and start +waiting for clients to connect:: import socket, ssl + context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context.load_cert_chain(certfile="mycertfile", keyfile="mykeyfile") + bindsocket = socket.socket() bindsocket.bind(('myaddr.mydomain.com', 10023)) bindsocket.listen(5) -When one did, you'd call :meth:`accept` on the socket to get the new socket from -the other end, and use :func:`wrap_socket` to create a server-side SSL context -for it:: +When a client connects, you'll call :meth:`accept` on the socket to get the +new socket from the other end, and use the context's :meth:`SSLContext.wrap_socket` +method to create a server-side SSL socket for the connection:: while True: newsocket, fromaddr = bindsocket.accept() - connstream = ssl.wrap_socket(newsocket, - server_side=True, - certfile="mycertfile", - keyfile="mykeyfile", - ssl_version=ssl.PROTOCOL_TLSv1) + connstream = context.wrap_socket(newsocket, server_side=True) try: deal_with_client(connstream) finally: connstream.shutdown(socket.SHUT_RDWR) connstream.close() -Then you'd read data from the ``connstream`` and do something with it till you +Then you'll read data from the ``connstream`` and do something with it till you are finished with the client (or the client is finished with you):: def deal_with_client(connstream): - data = connstream.read() - # null data means the client is finished with us + data = connstream.recv(1024) + # empty data means the client is finished with us while data: if not do_something(connstream, data): # we'll assume do_something returns False # when we're finished with client break - data = connstream.read() + data = connstream.recv(1024) # finished with client -And go back to listening for new client connections. +And go back to listening for new client connections (of course, a real server +would probably handle each client connection in a separate thread, or put +the sockets in non-blocking mode and use an event loop). + + +.. _ssl-security: + +Security considerations +----------------------- + +Verifying certificates +^^^^^^^^^^^^^^^^^^^^^^ + +:const:`CERT_NONE` is the default. Since it does not authenticate the other +peer, it can be insecure, especially in client mode where most of time you +would like to ensure the authenticity of the server you're talking to. +Therefore, when in client mode, it is highly recommended to use +:const:`CERT_REQUIRED`. However, it is in itself not sufficient; you also +have to check that the server certificate, which can be obtained by calling +:meth:`SSLSocket.getpeercert`, matches the desired service. For many +protocols and applications, the service can be identified by the hostname; +in this case, the :func:`match_hostname` function can be used. + +In server mode, if you want to authenticate your clients using the SSL layer +(rather than using a higher-level authentication mechanism), you'll also have +to specify :const:`CERT_REQUIRED` and similarly check the client certificate. + + .. note:: + + In client mode, :const:`CERT_OPTIONAL` and :const:`CERT_REQUIRED` are + equivalent unless anonymous ciphers are enabled (they are disabled + by default). + +Protocol versions +^^^^^^^^^^^^^^^^^ + +SSL version 2 is considered insecure and is therefore dangerous to use. If +you want maximum compatibility between clients and servers, it is recommended +to use :const:`PROTOCOL_SSLv23` as the protocol version and then disable +SSLv2 explicitly using the :data:`SSLContext.options` attribute:: + + context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + context.options |= ssl.OP_NO_SSLv2 + +The SSL context created above will allow SSLv3 and TLSv1 connections, but +not SSLv2. .. seealso:: @@ -582,3 +973,6 @@ And go back to listening for new client connections. `RFC 3280: Internet X.509 Public Key Infrastructure Certificate and CRL Profile <http://www.ietf.org/rfc/rfc3280>`_ Housley et. al. + + `RFC 4366: Transport Layer Security (TLS) Extensions <http://www.ietf.org/rfc/rfc4366>`_ + Blake-Wilson et. al. diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index 3419820..cc879de 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -6,11 +6,14 @@ os.lstat() and os.fstat(). .. sectionauthor:: Skip Montanaro <skip@automatrix.com> +**Source code:** :source:`Lib/stat.py` + +-------------- The :mod:`stat` module defines constants and functions for interpreting the results of :func:`os.stat`, :func:`os.fstat` and :func:`os.lstat` (if they -exist). For complete details about the :cfunc:`stat`, :cfunc:`fstat` and -:cfunc:`lstat` calls, consult the documentation for your system. +exist). For complete details about the :c:func:`stat`, :c:func:`fstat` and +:c:func:`lstat` calls, consult the documentation for your system. The :mod:`stat` module defines the following functions to test for specific file types: @@ -68,7 +71,7 @@ mode: Normally, you would use the :func:`os.path.is\*` functions for testing the type of a file; the functions here are useful when you are doing multiple tests of -the same file and wish to avoid the overhead of the :cfunc:`stat` system call +the same file and wish to avoid the overhead of the :c:func:`stat` system call for each test. These are also useful when checking for information about a file that isn't handled by :mod:`os.path`, like the tests for block and character devices. diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index a1a3879..72f93be 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -168,8 +168,9 @@ Objects of different types, except different numeric types, never compare equal. Furthermore, some types (for example, function objects) support only a degenerate notion of comparison where any two objects of that type are unequal. The ``<``, ``<=``, ``>`` and ``>=`` operators will raise a :exc:`TypeError` exception when -any operand is a complex number, the objects are of different types that cannot -be compared, or other cases where there is no defined ordering. +comparing a complex number with another built-in numeric type, when the objects +are of different types that cannot be compared, or in other cases where there is +no defined ordering. .. index:: single: __eq__() (instance method) @@ -196,8 +197,8 @@ exception. operator: in operator: not in -Two more operations with the same syntactic priority, ``in`` and ``not in``, are -supported only by sequence types (below). +Two more operations with the same syntactic priority, :keyword:`in` and +:keyword:`not in`, are supported only by sequence types (below). .. _typesnumeric: @@ -216,7 +217,7 @@ Numeric Types --- :class:`int`, :class:`float`, :class:`complex` There are three distinct numeric types: :dfn:`integers`, :dfn:`floating point numbers`, and :dfn:`complex numbers`. In addition, Booleans are a subtype of integers. Integers have unlimited precision. Floating point -numbers are usually implemented using :ctype:`double` in C; information +numbers are usually implemented using :c:type:`double` in C; information about the precision and internal representation of floating point numbers for the machine on which your program is running is available in :data:`sys.float_info`. Complex numbers have a real and imaginary @@ -464,6 +465,69 @@ Additional Methods on Integer Types .. versionadded:: 3.1 +.. method:: int.to_bytes(length, byteorder, \*, signed=False) + + Return an array of bytes representing an integer. + + >>> (1024).to_bytes(2, byteorder='big') + b'\x04\x00' + >>> (1024).to_bytes(10, byteorder='big') + b'\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00' + >>> (-1024).to_bytes(10, byteorder='big', signed=True) + b'\xff\xff\xff\xff\xff\xff\xff\xff\xfc\x00' + >>> x = 1000 + >>> x.to_bytes((x.bit_length() // 8) + 1, byteorder='little') + b'\xe8\x03' + + The integer is represented using *length* bytes. An :exc:`OverflowError` + is raised if the integer is not representable with the given number of + bytes. + + The *byteorder* argument determines the byte order used to represent the + integer. If *byteorder* is ``"big"``, the most significant byte is at the + beginning of the byte array. If *byteorder* is ``"little"``, the most + significant byte is at the end of the byte array. To request the native + byte order of the host system, use :data:`sys.byteorder` as the byte order + value. + + The *signed* argument determines whether two's complement is used to + represent the integer. If *signed* is ``False`` and a negative integer is + given, an :exc:`OverflowError` is raised. The default value for *signed* + is ``False``. + + .. versionadded:: 3.2 + +.. classmethod:: int.from_bytes(bytes, byteorder, \*, signed=False) + + Return the integer represented by the given array of bytes. + + >>> int.from_bytes(b'\x00\x10', byteorder='big') + 16 + >>> int.from_bytes(b'\x00\x10', byteorder='little') + 4096 + >>> int.from_bytes(b'\xfc\x00', byteorder='big', signed=True) + -1024 + >>> int.from_bytes(b'\xfc\x00', byteorder='big', signed=False) + 64512 + >>> int.from_bytes([255, 0, 0], byteorder='big') + 16711680 + + The argument *bytes* must either support the buffer protocol or be an + iterable producing bytes. :class:`bytes` and :class:`bytearray` are + examples of built-in objects that support the buffer protocol. + + The *byteorder* argument determines the byte order used to represent the + integer. If *byteorder* is ``"big"``, the most significant byte is at the + beginning of the byte array. If *byteorder* is ``"little"``, the most + significant byte is at the end of the byte array. To request the native + byte order of the host system, use :data:`sys.byteorder` as the byte order + value. + + The *signed* argument indicates whether two's complement is used to + represent the integer. + + .. versionadded:: 3.2 + Additional Methods on Float --------------------------- @@ -548,6 +612,109 @@ hexadecimal string representing the same number:: '0x1.d380000000000p+11' +.. _numeric-hash: + +Hashing of numeric types +------------------------ + +For numbers ``x`` and ``y``, possibly of different types, it's a requirement +that ``hash(x) == hash(y)`` whenever ``x == y`` (see the :meth:`__hash__` +method documentation for more details). For ease of implementation and +efficiency across a variety of numeric types (including :class:`int`, +:class:`float`, :class:`decimal.Decimal` and :class:`fractions.Fraction`) +Python's hash for numeric types is based on a single mathematical function +that's defined for any rational number, and hence applies to all instances of +:class:`int` and :class:`fraction.Fraction`, and all finite instances of +:class:`float` and :class:`decimal.Decimal`. Essentially, this function is +given by reduction modulo ``P`` for a fixed prime ``P``. The value of ``P`` is +made available to Python as the :attr:`modulus` attribute of +:data:`sys.hash_info`. + +.. impl-detail:: + + Currently, the prime used is ``P = 2**31 - 1`` on machines with 32-bit C + longs and ``P = 2**61 - 1`` on machines with 64-bit C longs. + +Here are the rules in detail: + + - If ``x = m / n`` is a nonnegative rational number and ``n`` is not divisible + by ``P``, define ``hash(x)`` as ``m * invmod(n, P) % P``, where ``invmod(n, + P)`` gives the inverse of ``n`` modulo ``P``. + + - If ``x = m / n`` is a nonnegative rational number and ``n`` is + divisible by ``P`` (but ``m`` is not) then ``n`` has no inverse + modulo ``P`` and the rule above doesn't apply; in this case define + ``hash(x)`` to be the constant value ``sys.hash_info.inf``. + + - If ``x = m / n`` is a negative rational number define ``hash(x)`` + as ``-hash(-x)``. If the resulting hash is ``-1``, replace it with + ``-2``. + + - The particular values ``sys.hash_info.inf``, ``-sys.hash_info.inf`` + and ``sys.hash_info.nan`` are used as hash values for positive + infinity, negative infinity, or nans (respectively). (All hashable + nans have the same hash value.) + + - For a :class:`complex` number ``z``, the hash values of the real + and imaginary parts are combined by computing ``hash(z.real) + + sys.hash_info.imag * hash(z.imag)``, reduced modulo + ``2**sys.hash_info.width`` so that it lies in + ``range(-2**(sys.hash_info.width - 1), 2**(sys.hash_info.width - + 1))``. Again, if the result is ``-1``, it's replaced with ``-2``. + + +To clarify the above rules, here's some example Python code, +equivalent to the builtin hash, for computing the hash of a rational +number, :class:`float`, or :class:`complex`:: + + + import sys, math + + def hash_fraction(m, n): + """Compute the hash of a rational number m / n. + + Assumes m and n are integers, with n positive. + Equivalent to hash(fractions.Fraction(m, n)). + + """ + P = sys.hash_info.modulus + # Remove common factors of P. (Unnecessary if m and n already coprime.) + while m % P == n % P == 0: + m, n = m // P, n // P + + if n % P == 0: + hash_ = sys.hash_info.inf + else: + # Fermat's Little Theorem: pow(n, P-1, P) is 1, so + # pow(n, P-2, P) gives the inverse of n modulo P. + hash_ = (abs(m) % P) * pow(n, P - 2, P) % P + if m < 0: + hash_ = -hash_ + if hash_ == -1: + hash_ = -2 + return hash_ + + def hash_float(x): + """Compute the hash of a float x.""" + + if math.isnan(x): + return sys.hash_info.nan + elif math.isinf(x): + return sys.hash_info.inf if x > 0 else -sys.hash_info.inf + else: + return hash_fraction(*x.as_integer_ratio()) + + def hash_complex(z): + """Compute the hash of a complex number z.""" + + hash_ = hash_float(z.real) + sys.hash_info.imag * hash_float(z.imag) + # do a signed reduction modulo 2**sys.hash_info.width + M = 2**(sys.hash_info.width - 1) + hash_ = (hash_ & (M - 1)) - (hash & M) + if hash_ == -1: + hash_ == -2 + return hash_ + .. _typeiter: Iterator Types @@ -653,22 +820,20 @@ constructor, :func:`bytes`, and from literals; use a ``b`` prefix with normal string syntax: ``b'xyzzy'``. To construct byte arrays, use the :func:`bytearray` function. -.. warning:: - - While string objects are sequences of characters (represented by strings of - length 1), bytes and bytearray objects are sequences of *integers* (between 0 - and 255), representing the ASCII value of single bytes. That means that for - a bytes or bytearray object *b*, ``b[0]`` will be an integer, while - ``b[0:1]`` will be a bytes or bytearray object of length 1. The - representation of bytes objects uses the literal format (``b'...'``) since it - is generally more useful than e.g. ``bytes([50, 19, 100])``. You can always - convert a bytes object into a list of integers using ``list(b)``. - - Also, while in previous Python versions, byte strings and Unicode strings - could be exchanged for each other rather freely (barring encoding issues), - strings and bytes are now completely separate concepts. There's no implicit - en-/decoding if you pass an object of the wrong type. A string always - compares unequal to a bytes or bytearray object. +While string objects are sequences of characters (represented by strings of +length 1), bytes and bytearray objects are sequences of *integers* (between 0 +and 255), representing the ASCII value of single bytes. That means that for +a bytes or bytearray object *b*, ``b[0]`` will be an integer, while +``b[0:1]`` will be a bytes or bytearray object of length 1. The +representation of bytes objects uses the literal format (``b'...'``) since it +is generally more useful than e.g. ``bytes([50, 19, 100])``. You can always +convert a bytes object into a list of integers using ``list(b)``. + +Also, while in previous Python versions, byte strings and Unicode strings +could be exchanged for each other rather freely (barring encoding issues), +strings and bytes are now completely separate concepts. There's no implicit +en-/decoding if you pass an object of the wrong type. A string always +compares unequal to a bytes or bytearray object. Lists are constructed with square brackets, separating items with commas: ``[a, b, c]``. Tuples are constructed by the comma operator (not within square @@ -677,8 +842,8 @@ the enclosing parentheses, such as ``a, b, c`` or ``()``. A single item tuple must have a trailing comma, such as ``(d,)``. Objects of type range are created using the :func:`range` function. They don't -support slicing, concatenation or repetition, and using ``in``, ``not in``, -:func:`min` or :func:`max` on them is inefficient. +support concatenation or repetition, and using :func:`min` or :func:`max` on +them is inefficient. Most sequence types support the following operations. The ``in`` and ``not in`` operations have the same priorities as the comparison operations. The ``+`` and @@ -687,7 +852,7 @@ operations have the same priorities as the comparison operations. The ``+`` and This table lists the sequence operations sorted in ascending priority (operations in the same box have the same priority). In the table, *s* and *t* -are sequences of the same type; *n*, *i* and *j* are integers: +are sequences of the same type; *n*, *i*, *j* and *k* are integers. +------------------+--------------------------------+----------+ | Operation | Result | Notes | @@ -813,11 +978,10 @@ String Methods .. index:: pair: string; methods -String objects support the methods listed below. Note that none of these -methods take keyword arguments. +String objects support the methods listed below. -In addition, Python's strings support the sequence type methods described in -the :ref:`typesseq` section. To output formatted strings, see the +In addition, Python's strings support the sequence type methods described in the +:ref:`typesseq` section. To output formatted strings, see the :ref:`string-formatting` section. Also, see the :mod:`re` module for string functions based on regular expressions. @@ -840,12 +1004,12 @@ functions based on regular expressions. interpreted as in slice notation. -.. method:: str.encode([encoding[, errors]]) +.. method:: str.encode(encoding="utf-8", errors="strict") - Return an encoded version of the string as a bytes object. Default encoding - is the current default string encoding. *errors* may be given to set a - different error handling scheme. The default for *errors* is ``'strict'``, - meaning that encoding errors raise a :exc:`UnicodeError`. Other possible + Return an encoded version of the string as a bytes object. Default encoding + is ``'utf-8'``. *errors* may be given to set a different error handling scheme. + The default for *errors* is ``'strict'``, meaning that encoding errors raise + a :exc:`UnicodeError`. Other possible values are ``'ignore'``, ``'replace'``, ``'xmlcharrefreplace'``, ``'backslashreplace'`` and any other name registered via :func:`codecs.register_error`, see section :ref:`codec-base-classes`. For a @@ -882,12 +1046,12 @@ functions based on regular expressions. .. method:: str.format(*args, **kwargs) - Perform a string formatting operation. The *format_string* argument can - contain literal text or replacement fields delimited by braces ``{}``. Each - replacement field contains either the numeric index of a positional argument, - or the name of a keyword argument. Returns a copy of *format_string* where - each replacement field is replaced with the string value of the corresponding - argument. + Perform a string formatting operation. The string on which this method is + called can contain literal text or replacement fields delimited by braces + ``{}``. Each replacement field contains either the numeric index of a + positional argument, or the name of a keyword argument. Returns a copy of + the string where each replacement field is replaced with the string value of + the corresponding argument. >>> "The sum of 1 + 2 is {0}".format(1+2) 'The sum of 1 + 2 is 3' @@ -896,6 +1060,22 @@ functions based on regular expressions. that can be specified in format strings. +.. method:: str.format_map(mapping) + + Similar to ``str.format(**mapping)``, except that ``mapping`` is + used directly and not copied to a :class:`dict` . This is useful + if for example ``mapping`` is a dict subclass: + + >>> class Default(dict): + ... def __missing__(self, key): + ... return key + ... + >>> '{name} was born in {country}'.format_map(Default(name='Guido')) + 'Guido was born in country' + + .. versionadded:: 3.2 + + .. method:: str.index(sub[, start[, end]]) Like :meth:`find`, but raise :exc:`ValueError` when the substring is not found. @@ -1248,7 +1428,7 @@ String objects have one unique built-in operation: the ``%`` operator (modulo). This is also known as the string *formatting* or *interpolation* operator. Given ``format % values`` (where *format* is a string), ``%`` conversion specifications in *format* are replaced with zero or more elements of *values*. -The effect is similar to the using :cfunc:`sprintf` in the C language. +The effect is similar to the using :c:func:`sprintf` in the C language. If *format* requires a single argument, *values* may be a single non-tuple object. [#]_ Otherwise, *values* must be a tuple with exactly the number of @@ -1421,11 +1601,23 @@ Range Type The :class:`range` type is an immutable sequence which is commonly used for looping. The advantage of the :class:`range` type is that an :class:`range` object will always take the same amount of memory, no matter the size of the -range it represents. There are no consistent performance advantages. +range it represents. + +Range objects have relatively little behavior: they support indexing, contains, +iteration, the :func:`len` function, and the following methods: + +.. method:: range.count(x) + + Return the number of *i*'s for which ``s[i] == x``. + + .. versionadded:: 3.2 + +.. method:: range.index(x) -Range objects have very little behavior: they only support indexing, iteration, -and the :func:`len` function. + Return the smallest *i* such that ``s[i] == x``. Raises + :exc:`ValueError` when *x* is not in the range. + .. versionadded:: 3.2 .. _typesseq-mutable: @@ -1543,6 +1735,9 @@ Notes: *key* specifies a function of one argument that is used to extract a comparison key from each list element: ``key=str.lower``. The default value is ``None``. + Use :func:`functools.cmp_to_key` to convert an + old-style *cmp* function to a *key* function. + *reverse* is a boolean value. If set to ``True``, then the list elements are sorted as if each comparison were reversed. @@ -1594,17 +1789,20 @@ Wherever one of these methods needs to interpret the bytes as characters b = a.replace(b"a", b"f") -.. method:: bytes.decode([encoding[, errors]]) - bytearray.decode([encoding[, errors]]) +.. method:: bytes.decode(encoding="utf-8", errors="strict") + bytearray.decode(encoding="utf-8", errors="strict") - Return a string decoded from the given bytes. Default encoding is the - current default string encoding. *errors* may be given to set a different + Return a string decoded from the given bytes. Default encoding is + ``'utf-8'``. *errors* may be given to set a different error handling scheme. The default for *errors* is ``'strict'``, meaning that encoding errors raise a :exc:`UnicodeError`. Other possible values are ``'ignore'``, ``'replace'`` and any other name registered via :func:`codecs.register_error`, see section :ref:`codec-base-classes`. For a list of possible encodings, see section :ref:`standard-encodings`. + .. versionchanged:: 3.1 + Added support for keyword arguments. + The bytes and bytearray types have an additional class method: @@ -1925,8 +2123,20 @@ pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: returned or raised by the ``__missing__(key)`` call if the key is not present. No other operations or methods invoke :meth:`__missing__`. If :meth:`__missing__` is not defined, :exc:`KeyError` is raised. - :meth:`__missing__` must be a method; it cannot be an instance variable. For - an example, see :class:`collections.defaultdict`. + :meth:`__missing__` must be a method; it cannot be an instance variable:: + + >>> class Counter(dict): + ... def __missing__(self, key): + ... return 0 + >>> c = Counter() + >>> c['red'] + 0 + >>> c['red'] += 1 + >>> c['red'] + 1 + + See :class:`collections.Counter` for a complete implementation including + other methods helpful for accumulating and managing tallies. .. describe:: d[key] = value @@ -2063,7 +2273,6 @@ since the entries are generally not unique.) For set-like views, all of the operations defined for the abstract base class :class:`collections.Set` are available (for example, ``==``, ``<``, or ``^``). - An example of dictionary view usage:: >>> dishes = {'eggs': 2, 'sausage': 1, 'bacon': 1, 'spam': 500} @@ -2098,8 +2307,8 @@ An example of dictionary view usage:: .. _typememoryview: -memoryview Types -================ +memoryview type +=============== :class:`memoryview` objects allow Python code to access the internal data of an object that supports the :ref:`buffer protocol <bufferobjects>` without @@ -2155,7 +2364,7 @@ copying. Memory is generally interpreted as simple bytes. Notice how the size of the memoryview object cannot be changed. - :class:`memoryview` has two methods: + :class:`memoryview` has several methods: .. method:: tobytes() @@ -2175,6 +2384,39 @@ copying. Memory is generally interpreted as simple bytes. >>> memoryview(b'abc').tolist() [97, 98, 99] + .. method:: release() + + Release the underlying buffer exposed by the memoryview object. Many + objects take special actions when a view is held on them (for example, + a :class:`bytearray` would temporarily forbid resizing); therefore, + calling release() is handy to remove these restrictions (and free any + dangling resources) as soon as possible. + + After this method has been called, any further operation on the view + raises a :class:`ValueError` (except :meth:`release()` itself which can + be called multiple times):: + + >>> m = memoryview(b'abc') + >>> m.release() + >>> m[0] + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ValueError: operation forbidden on released memoryview object + + The context management protocol can be used for a similar effect, + using the ``with`` statement:: + + >>> with memoryview(b'abc') as m: + ... m[0] + ... + b'a' + >>> m[0] + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ValueError: operation forbidden on released memoryview object + + .. versionadded:: 3.2 + There are also several readonly attributes available: .. attribute:: format diff --git a/Doc/library/string.rst b/Doc/library/string.rst index b2d0bba..d45eb36 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -11,6 +11,10 @@ :ref:`string-methods` +**Source code:** :source:`Lib/string.py` + +-------------- + String constants ---------------- @@ -340,9 +344,18 @@ following: | | positive numbers, and a minus sign on negative numbers. | +---------+----------------------------------------------------------+ -The ``'#'`` option is only valid for integers, and only for binary, octal, or -hexadecimal output. If present, it specifies that the output will be prefixed -by ``'0b'``, ``'0o'``, or ``'0x'``, respectively. + +The ``'#'`` option causes the "alternate form" to be used for the +conversion. The alternate form is defined differently for different +types. This option is only valid for integer, float, complex and +Decimal types. For integers, when binary, octal, or hexadecimal output +is used, this option adds the prefix respective ``'0b'``, ``'0o'``, or +``'0x'`` to the output value. For floats, complex and Decimal the +alternate form causes the result of the conversion to always contain a +decimal-point character, even if no digits follow it. Normally, a +decimal-point character appears in the result of these conversions +only if a digit follows it. In addition, for ``'g'`` and ``'G'`` +conversions, trailing zeros are not removed from the result. The ``','`` option signals the use of a comma for a thousands separator. For a locale aware separator, use the ``'n'`` integer presentation type @@ -705,6 +718,14 @@ to parse template strings. To do this, you can override these class attributes: appropriate). The default value is the regular expression ``[_a-z][_a-z0-9]*``. +* *flags* -- The regular expression flags that will be applied when compiling + the regular expression used for recognizing substitutions. The default value + is ``re.IGNORECASE``. Note that ``re.VERBOSE`` will always be added to the + flags, so custom *idpattern*\ s must follow conventions for verbose regular + expressions. + + .. versionadded:: 3.2 + Alternatively, you can provide the entire regular expression pattern by overriding the class attribute *pattern*. If you do this, the value must be a regular expression object with four named capturing groups. The capturing @@ -727,7 +748,7 @@ rule: Helper functions ---------------- -.. function:: capwords(s[, sep]) +.. function:: capwords(s, sep=None) Split the argument into words using :meth:`str.split`, capitalize each word using :meth:`str.capitalize`, and join the capitalized words using @@ -736,12 +757,3 @@ Helper functions and leading and trailing whitespace are removed, otherwise *sep* is used to split and join the words. - -.. function:: maketrans(frm, to) - - Return a translation table suitable for passing to :meth:`bytes.translate`, - that will map each character in *from* into the character at the same - position in *to*; *from* and *to* must have the same length. - - .. deprecated:: 3.1 - Use the :meth:`bytes.maketrans` static method instead. diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst index 1834cfd..42bfc14 100644 --- a/Doc/library/struct.rst +++ b/Doc/library/struct.rst @@ -21,8 +21,8 @@ structs and the intended conversion to/from Python values. alignment is taken into account when unpacking. This behavior is chosen so that the bytes of a packed struct correspond exactly to the layout in memory of the corresponding C struct. To handle platform-independent data formats - or omit implicit pad bytes, use `standard` size and alignment instead of - `native` size and alignment: see :ref:`struct-alignment` for details. + or omit implicit pad bytes, use ``standard`` size and alignment instead of + ``native`` size and alignment: see :ref:`struct-alignment` for details. Functions and Exceptions ------------------------ @@ -157,63 +157,66 @@ is, when the format string starts with one of ``'<'``, ``'>'``, ``'!'`` or ``'='``. When using native size, the size of the packed value is platform-dependent. -+--------+-------------------------+--------------------+----------------+------------+ -| Format | C Type | Python type | Standard size | Notes | -+========+=========================+====================+================+============+ -| ``x`` | pad byte | no value | | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``c`` | :ctype:`char` | bytes of length 1 | 1 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``b`` | :ctype:`signed char` | integer | 1 | \(1) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``B`` | :ctype:`unsigned char` | integer | 1 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``?`` | :ctype:`_Bool` | bool | 1 | \(2) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``h`` | :ctype:`short` | integer | 2 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``H`` | :ctype:`unsigned short` | integer | 2 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``i`` | :ctype:`int` | integer | 4 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``I`` | :ctype:`unsigned int` | integer | 4 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``l`` | :ctype:`long` | integer | 4 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``L`` | :ctype:`unsigned long` | integer | 4 | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``q`` | :ctype:`long long` | integer | 8 | \(3) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``Q`` | :ctype:`unsigned long | integer | 8 | \(3) | -| | long` | | | | -+--------+-------------------------+--------------------+----------------+------------+ -| ``f`` | :ctype:`float` | float | 4 | \(4) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``d`` | :ctype:`double` | float | 8 | \(4) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``s`` | :ctype:`char[]` | bytes | | \(1) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``p`` | :ctype:`char[]` | bytes | | \(1) | -+--------+-------------------------+--------------------+----------------+------------+ -| ``P`` | :ctype:`void \*` | integer | | \(5) | -+--------+-------------------------+--------------------+----------------+------------+ ++--------+--------------------------+--------------------+----------------+------------+ +| Format | C Type | Python type | Standard size | Notes | ++========+==========================+====================+================+============+ +| ``x`` | pad byte | no value | | | ++--------+--------------------------+--------------------+----------------+------------+ +| ``c`` | :c:type:`char` | bytes of length 1 | 1 | | ++--------+--------------------------+--------------------+----------------+------------+ +| ``b`` | :c:type:`signed char` | integer | 1 | \(1),\(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``B`` | :c:type:`unsigned char` | integer | 1 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``?`` | :c:type:`_Bool` | bool | 1 | \(1) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``h`` | :c:type:`short` | integer | 2 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``H`` | :c:type:`unsigned short` | integer | 2 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``i`` | :c:type:`int` | integer | 4 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``I`` | :c:type:`unsigned int` | integer | 4 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``l`` | :c:type:`long` | integer | 4 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``L`` | :c:type:`unsigned long` | integer | 4 | \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``q`` | :c:type:`long long` | integer | 8 | \(2), \(3) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``Q`` | :c:type:`unsigned long | integer | 8 | \(2), \(3) | +| | long` | | | | ++--------+--------------------------+--------------------+----------------+------------+ +| ``f`` | :c:type:`float` | float | 4 | \(4) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``d`` | :c:type:`double` | float | 8 | \(4) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``s`` | :c:type:`char[]` | bytes | | | ++--------+--------------------------+--------------------+----------------+------------+ +| ``p`` | :c:type:`char[]` | bytes | | | ++--------+--------------------------+--------------------+----------------+------------+ +| ``P`` | :c:type:`void \*` | integer | | \(5) | ++--------+--------------------------+--------------------+----------------+------------+ Notes: (1) - The ``c``, ``s`` and ``p`` conversion codes operate on :class:`bytes` - objects, but packing with such codes also supports :class:`str` objects, - which are encoded using UTF-8. + The ``'?'`` conversion code corresponds to the :c:type:`_Bool` type defined by + C99. If this type is not available, it is simulated using a :c:type:`char`. In + standard mode, it is always represented by one byte. (2) - The ``'?'`` conversion code corresponds to the :ctype:`_Bool` type defined by - C99. If this type is not available, it is simulated using a :ctype:`char`. In - standard mode, it is always represented by one byte. + The ``'q'`` and ``'Q'`` conversion codes are available in native mode only if + the platform C compiler supports C :c:type:`long long`, or, on Windows, + :c:type:`__int64`. They are always available in standard modes. (3) - The ``'q'`` and ``'Q'`` conversion codes are available in native mode only if - the platform C compiler supports C :ctype:`long long`, or, on Windows, - :ctype:`__int64`. They are always available in standard modes. + When attempting to pack a non-integer using any of the integer conversion + codes, if the non-integer has a :meth:`__index__` method then that method is + called to convert the argument to an integer before packing. + + .. versionchanged:: 3.2 + Use of the :meth:`__index__` method for non-integers is new in 3.2. (4) For the ``'f'`` and ``'d'`` conversion codes, the packed representation uses @@ -302,9 +305,9 @@ the result in a named tuple:: The ordering of format characters may have an impact on size since the padding needed to satisfy alignment requirements is different:: - >>> pack('ci', '*', 0x12131415) + >>> pack('ci', b'*', 0x12131415) b'*\x00\x00\x00\x12\x13\x14\x15' - >>> pack('ic', 0x12131415, '*') + >>> pack('ic', 0x12131415, b'*') b'\x12\x13\x14\x15*' >>> calcsize('ci') 8 diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index f146683..5677738 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -28,7 +28,7 @@ Using the subprocess Module This module defines one class called :class:`Popen`: -.. class:: Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0) +.. class:: Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0, restore_signals=True, start_new_session=False, pass_fds=()) Arguments are: @@ -41,7 +41,8 @@ This module defines one class called :class:`Popen`: name for the executing program in utilities such as :program:`ps`. On Unix, with *shell=False* (default): In this case, the Popen class uses - :meth:`os.execvp` to execute the child program. *args* should normally be a + :meth:`os.execvp` like behavior to execute the child program. + *args* should normally be a sequence. If a string is specified for *args*, it will be used as the name or path of the program to execute; this will only work if the program is being given no arguments. @@ -130,25 +131,65 @@ This module defines one class called :class:`Popen`: applications should be captured into the same file handle as for stdout. If *preexec_fn* is set to a callable object, this object will be called in the - child process just before the child is executed. (Unix only) + child process just before the child is executed. + (Unix only) + + .. warning:: + + The *preexec_fn* parameter is not safe to use in the presence of threads + in your application. The child process could deadlock before exec is + called. + If you must use it, keep it trivial! Minimize the number of libraries + you call into. + + .. note:: + + If you need to modify the environment for the child use the *env* + parameter rather than doing it in a *preexec_fn*. + The *start_new_session* parameter can take the place of a previously + common use of *preexec_fn* to call os.setsid() in the child. If *close_fds* is true, all file descriptors except :const:`0`, :const:`1` and :const:`2` will be closed before the child process is executed. (Unix only). - Or, on Windows, if *close_fds* is true then no handles will be inherited by the + The default varies by platform: Always true on Unix. On Windows it is + true when *stdin*/*stdout*/*stderr* are :const:`None`, false otherwise. + On Windows, if *close_fds* is true then no handles will be inherited by the child process. Note that on Windows, you cannot set *close_fds* to true and also redirect the standard handles by setting *stdin*, *stdout* or *stderr*. - If *shell* is :const:`True`, the specified command will be executed through the - shell. + .. versionchanged:: 3.2 + The default for *close_fds* was changed from :const:`False` to + what is described above. + + *pass_fds* is an optional sequence of file descriptors to keep open + between the parent and child. Providing any *pass_fds* forces + *close_fds* to be :const:`True`. (Unix only) + + .. versionadded:: 3.2 + The *pass_fds* parameter was added. If *cwd* is not ``None``, the child's current directory will be changed to *cwd* before it is executed. Note that this directory is not considered when searching the executable, so you can't specify the program's path relative to *cwd*. + If *restore_signals* is True (the default) all signals that Python has set to + SIG_IGN are restored to SIG_DFL in the child process before the exec. + Currently this includes the SIGPIPE, SIGXFZ and SIGXFSZ signals. + (Unix only) + + .. versionchanged:: 3.2 + *restore_signals* was added. + + If *start_new_session* is True the setsid() system call will be made in the + child process prior to the execution of the subprocess. (Unix only) + + .. versionchanged:: 3.2 + *start_new_session* was added. + If *env* is not ``None``, it must be a mapping that defines the environment - variables for the new process; these are used instead of inheriting the current - process' environment, which is the default behavior. + variables for the new process; these are used instead of the default + behavior of inheriting the current process' environment. .. note:: @@ -175,6 +216,16 @@ This module defines one class called :class:`Popen`: underlying CreateProcess() function. They can specify things such as appearance of the main window and priority for the new process. (Windows only) + Popen objects are supported as context managers via the :keyword:`with` statement, + closing any open file descriptors on exit. + :: + + with Popen(["ifconfig"], stdout=PIPE) as proc: + log.write(proc.stdout.read()) + + .. versionchanged:: 3.2 + Added context manager support. + .. data:: PIPE @@ -201,15 +252,16 @@ This module also defines the following shortcut functions: Run command with arguments. Wait for command to complete, then return the :attr:`returncode` attribute. - The arguments are the same as for the Popen constructor. Example:: + The arguments are the same as for the :class:`Popen` constructor. Example:: >>> retcode = subprocess.call(["ls", "-l"]) .. warning:: - Like :meth:`Popen.wait`, this will deadlock if the child process - generates enough output to a stdout or stderr pipe such that it blocks - waiting for the OS pipe buffer to accept more data. + Like :meth:`Popen.wait`, this will deadlock when using + ``stdout=PIPE`` and/or ``stderr=PIPE`` and the child process + generates enough output to a pipe such that it blocks waiting + for the OS pipe buffer to accept more data. .. function:: check_call(*popenargs, **kwargs) @@ -219,7 +271,7 @@ This module also defines the following shortcut functions: :exc:`CalledProcessError` object will have the return code in the :attr:`returncode` attribute. - The arguments are the same as for the Popen constructor. Example:: + The arguments are the same as for the :class:`Popen` constructor. Example:: >>> subprocess.check_call(["ls", "-l"]) 0 @@ -262,7 +314,7 @@ This module also defines the following shortcut functions: ``(status, output)``. *cmd* is actually run as ``{ cmd ; } 2>&1``, so that the returned output will contain output or error messages. A trailing newline is stripped from the output. The exit status for the command can be interpreted - according to the rules for the C function :cfunc:`wait`. Example:: + according to the rules for the C function :c:func:`wait`. Example:: >>> subprocess.getstatusoutput('ls /bin/ls') (0, '/bin/ls') @@ -333,9 +385,10 @@ Instances of the :class:`Popen` class have the following methods: .. warning:: - This will deadlock if the child process generates enough output to a - stdout or stderr pipe such that it blocks waiting for the OS pipe buffer - to accept more data. Use :meth:`communicate` to avoid that. + This will deadlock when using ``stdout=PIPE`` and/or + ``stderr=PIPE`` and the child process generates enough output to + a pipe such that it blocks waiting for the OS pipe buffer to + accept more data. Use :meth:`communicate` to avoid that. .. method:: Popen.communicate(input=None) @@ -364,14 +417,15 @@ Instances of the :class:`Popen` class have the following methods: .. note:: - On Windows only SIGTERM is supported so far. It's an alias for - :meth:`terminate`. + On Windows, SIGTERM is an alias for :meth:`terminate`. CTRL_C_EVENT and + CTRL_BREAK_EVENT can be sent to processes started with a *creationflags* + parameter which includes `CREATE_NEW_PROCESS_GROUP`. .. method:: Popen.terminate() Stop the child. On Posix OSs the method sends SIGTERM to the - child. On Windows the Win32 API function :cfunc:`TerminateProcess` is called + child. On Windows the Win32 API function :c:func:`TerminateProcess` is called to stop the child. @@ -606,7 +660,8 @@ Replacing functions from the :mod:`popen2` module * ``stdin=PIPE`` and ``stdout=PIPE`` must be specified. * popen2 closes all file descriptors by default, but you have to specify - ``close_fds=True`` with :class:`Popen`. + ``close_fds=True`` with :class:`Popen` to guarantee this behavior on + all platforms or past Python versions. Notes ----- @@ -640,3 +695,4 @@ runtime): backslash escapes the next double quotation mark as described in rule 3. + diff --git a/Doc/library/sunau.rst b/Doc/library/sunau.rst index fc141e9..4bdb99b 100644 --- a/Doc/library/sunau.rst +++ b/Doc/library/sunau.rst @@ -5,6 +5,9 @@ :synopsis: Provide an interface to the Sun AU sound format. .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/sunau.py` + +-------------- The :mod:`sunau` module provides a convenient interface to the Sun AU sound format. Note that this module is interface-compatible with the modules diff --git a/Doc/library/symbol.rst b/Doc/library/symbol.rst index 5134d47..75a4792 100644 --- a/Doc/library/symbol.rst +++ b/Doc/library/symbol.rst @@ -5,6 +5,9 @@ :synopsis: Constants representing internal nodes of the parse tree. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/symbol.py` + +-------------- This module provides constants which represent the numeric values of internal nodes of the parse tree. Unlike most Python constants, these use lower-case diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index c61fedb..ecac1fe 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -10,6 +10,13 @@ interpreter and to functions that interact strongly with the interpreter. It is always available. +.. data:: abiflags + + On POSIX systems where Python is build with the standard ``configure`` + script, this contains the ABI flags as specified by :pep:`3149`. + + .. versionadded:: 3.2 + .. data:: argv The list of command line arguments passed to a Python script. ``argv[0]`` is the @@ -92,13 +99,39 @@ always available. .. function:: displayhook(value) - If *value* is not ``None``, this function prints it to ``sys.stdout``, and saves - it in ``builtins._``. + If *value* is not ``None``, this function prints ``repr(value)`` to + ``sys.stdout``, and saves *value* in ``builtins._``. If ``repr(value)`` is + not encodable to ``sys.stdout.encoding`` with ``sys.stdout.errors`` error + handler (which is probably ``'strict'``), encode it to + ``sys.stdout.encoding`` with ``'backslashreplace'`` error handler. ``sys.displayhook`` is called on the result of evaluating an :term:`expression` entered in an interactive Python session. The display of these values can be customized by assigning another one-argument function to ``sys.displayhook``. + Pseudo-code:: + + def displayhook(value): + if value is None: + return + # Set '_' to None to avoid recursion + builtins._ = None + text = repr(value) + try: + sys.stdout.write(text) + except UnicodeEncodeError: + bytes = text.encode(sys.stdout.encoding, 'backslashreplace') + if hasattr(sys.stdout, 'buffer'): + sys.stdout.buffer.write(bytes) + else: + text = bytes.decode(sys.stdout.encoding, 'strict') + sys.stdout.write(text) + sys.stdout.write("\n") + builtins._ = value + + .. versionchanged:: 3.2 + Use ``'backslashreplace'`` error handler on :exc:`UnicodeEncodeError`. + .. function:: excepthook(type, value, traceback) @@ -220,8 +253,12 @@ always available. :const:`ignore_environment` :option:`-E` :const:`verbose` :option:`-v` :const:`bytes_warning` :option:`-b` + :const:`quiet` :option:`-q` ============================= ============================= + .. versionchanged:: 3.2 + Added ``quiet`` attribute for the new :option:`-q` flag. + .. data:: float_info @@ -303,6 +340,9 @@ always available. Return the interpreter's "check interval"; see :func:`setcheckinterval`. + .. deprecated:: 3.2 + Use :func:`getswitchinterval` instead. + .. function:: getdefaultencoding() @@ -312,22 +352,20 @@ always available. .. function:: getdlopenflags() - Return the current value of the flags that are used for :cfunc:`dlopen` calls. + Return the current value of the flags that are used for :c:func:`dlopen` calls. The flag constants are defined in the :mod:`ctypes` and :mod:`DLFCN` modules. Availability: Unix. .. function:: getfilesystemencoding() - Return the name of the encoding used to convert Unicode filenames into system - file names, or ``None`` if the system default encoding is used. The result value - depends on the operating system: + Return the name of the encoding used to convert Unicode filenames into + system file names. The result value depends on the operating system: * On Mac OS X, the encoding is ``'utf-8'``. * On Unix, the encoding is the user's preference according to the result of - nl_langinfo(CODESET), or ``None`` if the ``nl_langinfo(CODESET)`` - failed. + nl_langinfo(CODESET), or ``'utf-8'`` if ``nl_langinfo(CODESET)`` failed. * On Windows NT+, file names are Unicode natively, so no conversion is performed. :func:`getfilesystemencoding` still returns ``'mbcs'``, as @@ -337,6 +375,10 @@ always available. * On Windows 9x, the encoding is ``'mbcs'``. + .. versionchanged:: 3.2 + On Unix, use ``'utf-8'`` instead of ``None`` if ``nl_langinfo(CODESET)`` + failed. :func:`getfilesystemencoding` result cannot be ``None``. + .. function:: getrefcount(object) @@ -367,6 +409,17 @@ always available. additional garbage collector overhead if the object is managed by the garbage collector. + See `recursive sizeof recipe <http://code.activestate.com/recipes/577504>`_ + for an example of using :func:`getsizeof` recursively to find the size of + containers and all their contents. + +.. function:: getswitchinterval() + + Return the interpreter's "thread switch interval"; see + :func:`setswitchinterval`. + + .. versionadded:: 3.2 + .. function:: _getframe([depth]) @@ -408,9 +461,15 @@ always available. .. function:: getwindowsversion() - Return a tuple containing five components, describing the Windows version - currently running. The elements are *major*, *minor*, *build*, *platform*, and - *text*. *text* contains a string while all other values are integers. + Return a named tuple describing the Windows version + currently running. The named elements are *major*, *minor*, + *build*, *platform*, *service_pack*, *service_pack_minor*, + *service_pack_major*, *suite_mask*, and *product_type*. + *service_pack* contains a string while all other values are + integers. The components can also be accessed by name, so + ``sys.getwindowsversion()[0]`` is equivalent to + ``sys.getwindowsversion().major``. For compatibility with prior + versions, only the first 5 elements are retrievable by indexing. *platform* may be one of the following values: @@ -426,11 +485,54 @@ always available. | :const:`3 (VER_PLATFORM_WIN32_CE)` | Windows CE | +-----------------------------------------+-------------------------+ - This function wraps the Win32 :cfunc:`GetVersionEx` function; see the Microsoft - documentation for more information about these fields. + *product_type* may be one of the following values: + + +---------------------------------------+---------------------------------+ + | Constant | Meaning | + +=======================================+=================================+ + | :const:`1 (VER_NT_WORKSTATION)` | The system is a workstation. | + +---------------------------------------+---------------------------------+ + | :const:`2 (VER_NT_DOMAIN_CONTROLLER)` | The system is a domain | + | | controller. | + +---------------------------------------+---------------------------------+ + | :const:`3 (VER_NT_SERVER)` | The system is a server, but not | + | | a domain controller. | + +---------------------------------------+---------------------------------+ + + + This function wraps the Win32 :c:func:`GetVersionEx` function; see the + Microsoft documentation on :c:func:`OSVERSIONINFOEX` for more information + about these fields. Availability: Windows. + .. versionchanged:: 3.2 + Changed to a named tuple and added *service_pack_minor*, + *service_pack_major*, *suite_mask*, and *product_type*. + + +.. data:: hash_info + + A structseq giving parameters of the numeric hash implementation. For + more details about hashing of numeric types, see :ref:`numeric-hash`. + + +---------------------+--------------------------------------------------+ + | attribute | explanation | + +=====================+==================================================+ + | :const:`width` | width in bits used for hash values | + +---------------------+--------------------------------------------------+ + | :const:`modulus` | prime modulus P used for numeric hash scheme | + +---------------------+--------------------------------------------------+ + | :const:`inf` | hash value returned for a positive infinity | + +---------------------+--------------------------------------------------+ + | :const:`nan` | hash value returned for a nan | + +---------------------+--------------------------------------------------+ + | :const:`imag` | multiplier used for the imaginary part of a | + | | complex number | + +---------------------+--------------------------------------------------+ + + .. versionadded:: 3.2 + .. data:: hexversion @@ -525,7 +627,7 @@ always available. .. data:: maxsize - An integer giving the maximum value a variable of type :ctype:`Py_ssize_t` can + An integer giving the maximum value a variable of type :c:type:`Py_ssize_t` can take. It's usually ``2**31 - 1`` on a 32-bit platform and ``2**63 - 1`` on a 64-bit platform. @@ -544,7 +646,7 @@ always available. imported. The :meth:`find_module` method is called at least with the absolute name of the module being imported. If the module to be imported is contained in package then the parent package's :attr:`__path__` attribute - is passed in as a second argument. The method returns :keyword:`None` if + is passed in as a second argument. The method returns ``None`` if the module cannot be found, else returns a :term:`loader`. :data:`sys.meta_path` is searched before any implicit default finders or @@ -597,7 +699,7 @@ always available. A dictionary acting as a cache for :term:`finder` objects. The keys are paths that have been passed to :data:`sys.path_hooks` and the values are the finders that are found. If a path is a valid file system path but no - explicit finder is found on :data:`sys.path_hooks` then :keyword:`None` is + explicit finder is found on :data:`sys.path_hooks` then ``None`` is stored to represent the implicit default finder should be used. If the path is not an existing path then :class:`imp.NullImporter` is set. @@ -622,7 +724,6 @@ always available. Mac OS X ``'darwin'`` OS/2 ``'os2'`` OS/2 EMX ``'os2emx'`` - AtheOS ``'atheos'`` ================ =========================== @@ -671,22 +772,15 @@ always available. performance for programs using threads. Setting it to a value ``<=`` 0 checks every virtual instruction, maximizing responsiveness as well as overhead. - -.. function:: setdefaultencoding(name) - - Set the current default string encoding used by the Unicode implementation. If - *name* does not match any available encoding, :exc:`LookupError` is raised. - This function is only intended to be used by the :mod:`site` module - implementation and, where needed, by :mod:`sitecustomize`. Once used by the - :mod:`site` module, it is removed from the :mod:`sys` module's namespace. - - .. Note that :mod:`site` is not imported if the :option:`-S` option is passed - to the interpreter, in which case this function will remain available. + .. deprecated:: 3.2 + This function doesn't have an effect anymore, as the internal logic for + thread switching and asynchronous tasks has been rewritten. Use + :func:`setswitchinterval` instead. .. function:: setdlopenflags(n) - Set the flags used by the interpreter for :cfunc:`dlopen` calls, such as when + Set the flags used by the interpreter for :c:func:`dlopen` calls, such as when the interpreter loads extension modules. Among other things, this will enable a lazy resolving of symbols when importing a module, if called as ``sys.setdlopenflags(0)``. To share symbols across extension modules, call as @@ -696,15 +790,6 @@ always available. :file:`/usr/include/dlfcn.h` using the :program:`h2py` script. Availability: Unix. -.. function:: setfilesystemencoding(enc) - - Set the encoding used when converting Python strings to file names to *enc*. - By default, Python tries to determine the encoding it should use automatically - on Unix; on Windows, it avoids such conversion completely. This function can - be used when Python's determination of the encoding needs to be overwritten, - e.g. when not all file names on disk can be decoded using the encoding that - Python had chosen. - .. function:: setprofile(profilefunc) .. index:: @@ -734,6 +819,19 @@ always available. limit can lead to a crash. +.. function:: setswitchinterval(interval) + + Set the interpreter's thread switch interval (in seconds). This floating-point + value determines the ideal duration of the "timeslices" allocated to + concurrently running Python threads. Please note that the actual value + can be higher, especially if long-running internal functions or methods + are used. Also, which thread becomes scheduled at the end of the interval + is the operating system's decision. The interpreter doesn't have its + own scheduler. + + .. versionadded:: 3.2 + + .. function:: settrace(tracefunc) .. index:: @@ -766,9 +864,11 @@ always available. specifies the local trace function. ``'line'`` - The interpreter is about to execute a new line of code (sometimes multiple - line events on one line exist). The local trace function is called; *arg* - is ``None``; the return value specifies the new local trace function. + The interpreter is about to execute a new line of code or re-execute the + condition of a loop. The local trace function is called; *arg* is + ``None``; the return value specifies the new local trace function. See + :file:`Objects/lnotab_notes.txt` for a detailed explanation of how this + works. ``'return'`` A function (or other code block) is about to return. The local trace @@ -811,6 +911,10 @@ always available. available only if Python was compiled with ``--with-tsc``. To understand the output of this dump, read :file:`Python/ceval.c` in the Python sources. + .. impl-detail:: + This function is intimately bound to CPython implementation details and + thus not likely to be implemented elsewhere. + .. data:: stdin stdout @@ -917,6 +1021,30 @@ always available. module for informational purposes; modifying this value has no effect on the registry keys used by Python. Availability: Windows. + +.. data:: _xoptions + + A dictionary of the various implementation-specific flags passed through + the :option:`-X` command-line option. Option names are either mapped to + their values, if given explicitly, or to :const:`True`. Example:: + + $ ./python -Xa=b -Xc + Python 3.2a3+ (py3k, Oct 16 2010, 20:14:50) + [GCC 4.4.3] on linux2 + Type "help", "copyright", "credits" or "license" for more information. + >>> import sys + >>> sys._xoptions + {'a': 'b', 'c': True} + + .. impl-detail:: + + This is a CPython-specific way of accessing options passed through + :option:`-X`. Other implementations may export them through other + means, or not at all. + + .. versionadded:: 3.2 + + .. rubric:: Citations .. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf . diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst new file mode 100644 index 0000000..1e89bd0 --- /dev/null +++ b/Doc/library/sysconfig.rst @@ -0,0 +1,259 @@ +:mod:`sysconfig` --- Provide access to Python's configuration information +========================================================================= + +.. module:: sysconfig + :synopsis: Python's configuration information +.. moduleauthor:: Tarek Ziade <tarek@ziade.org> +.. sectionauthor:: Tarek Ziade <tarek@ziade.org> +.. index:: + single: configuration information + +**Source code:** :source:`Lib/sysconfig.py` + +.. versionadded:: 3.2 + +-------------- + +The :mod:`sysconfig` module provides access to Python's configuration +information like the list of installation paths and the configuration variables +relevant for the current platform. + +Configuration variables +----------------------- + +A Python distribution contains a :file:`Makefile` and a :file:`pyconfig.h` +header file that are necessary to build both the Python binary itself and +third-party C extensions compiled using :mod:`distutils`. + +:mod:`sysconfig` puts all variables found in these files in a dictionary that +can be accessed using :func:`get_config_vars` or :func:`get_config_var`. + +Notice that on Windows, it's a much smaller set. + +.. function:: get_config_vars(\*args) + + With no arguments, return a dictionary of all configuration variables + relevant for the current platform. + + With arguments, return a list of values that result from looking up each + argument in the configuration variable dictionary. + + For each argument, if the value is not found, return ``None``. + + +.. function:: get_config_var(name) + + Return the value of a single variable *name*. Equivalent to + ``get_config_vars().get(name)``. + + If *name* is not found, return ``None``. + +Example of usage:: + + >>> import sysconfig + >>> sysconfig.get_config_var('Py_ENABLE_SHARED') + 0 + >>> sysconfig.get_config_var('LIBDIR') + '/usr/local/lib' + >>> sysconfig.get_config_vars('AR', 'CXX') + ['ar', 'g++'] + + +Installation paths +------------------ + +Python uses an installation scheme that differs depending on the platform and on +the installation options. These schemes are stored in :mod:`sysconfig` under +unique identifiers based on the value returned by :const:`os.name`. + +Every new component that is installed using :mod:`distutils` or a +Distutils-based system will follow the same scheme to copy its file in the right +places. + +Python currently supports seven schemes: + +- *posix_prefix*: scheme for Posix platforms like Linux or Mac OS X. This is + the default scheme used when Python or a component is installed. +- *posix_home*: scheme for Posix platforms used when a *home* option is used + upon installation. This scheme is used when a component is installed through + Distutils with a specific home prefix. +- *posix_user*: scheme for Posix platforms used when a component is installed + through Distutils and the *user* option is used. This scheme defines paths + located under the user home directory. +- *nt*: scheme for NT platforms like Windows. +- *nt_user*: scheme for NT platforms, when the *user* option is used. +- *os2*: scheme for OS/2 platforms. +- *os2_home*: scheme for OS/2 patforms, when the *user* option is used. + +Each scheme is itself composed of a series of paths and each path has a unique +identifier. Python currently uses eight paths: + +- *stdlib*: directory containing the standard Python library files that are not + platform-specific. +- *platstdlib*: directory containing the standard Python library files that are + platform-specific. +- *platlib*: directory for site-specific, platform-specific files. +- *purelib*: directory for site-specific, non-platform-specific files. +- *include*: directory for non-platform-specific header files. +- *platinclude*: directory for platform-specific header files. +- *scripts*: directory for script files. +- *data*: directory for data files. + +:mod:`sysconfig` provides some functions to determine these paths. + +.. function:: get_scheme_names() + + Return a tuple containing all schemes currently supported in + :mod:`sysconfig`. + + +.. function:: get_path_names() + + Return a tuple containing all path names currently supported in + :mod:`sysconfig`. + + +.. function:: get_path(name, [scheme, [vars, [expand]]]) + + Return an installation path corresponding to the path *name*, from the + install scheme named *scheme*. + + *name* has to be a value from the list returned by :func:`get_path_names`. + + :mod:`sysconfig` stores installation paths corresponding to each path name, + for each platform, with variables to be expanded. For instance the *stdlib* + path for the *nt* scheme is: ``{base}/Lib``. + + :func:`get_path` will use the variables returned by :func:`get_config_vars` + to expand the path. All variables have default values for each platform so + one may call this function and get the default value. + + If *scheme* is provided, it must be a value from the list returned by + :func:`get_path_names`. Otherwise, the default scheme for the current + platform is used. + + If *vars* is provided, it must be a dictionary of variables that will update + the dictionary return by :func:`get_config_vars`. + + If *expand* is set to ``False``, the path will not be expanded using the + variables. + + If *name* is not found, return ``None``. + + +.. function:: get_paths([scheme, [vars, [expand]]]) + + Return a dictionary containing all installation paths corresponding to an + installation scheme. See :func:`get_path` for more information. + + If *scheme* is not provided, will use the default scheme for the current + platform. + + If *vars* is provided, it must be a dictionary of variables that will + update the dictionary used to expand the paths. + + If *expand* is set to False, the paths will not be expanded. + + If *scheme* is not an existing scheme, :func:`get_paths` will raise a + :exc:`KeyError`. + + +Other functions +--------------- + +.. function:: get_python_version() + + Return the ``MAJOR.MINOR`` Python version number as a string. Similar to + ``sys.version[:3]``. + + +.. function:: get_platform() + + Return a string that identifies the current platform. + + This is used mainly to distinguish platform-specific build directories and + platform-specific built distributions. Typically includes the OS name and + version and the architecture (as supplied by :func:`os.uname`), although the + exact information included depends on the OS; e.g. for IRIX the architecture + isn't particularly important (IRIX only runs on SGI hardware), but for Linux + the kernel version isn't particularly important. + + Examples of returned values: + + - linux-i586 + - linux-alpha (?) + - solaris-2.6-sun4u + - irix-5.3 + - irix64-6.2 + + Windows will return one of: + + - win-amd64 (64bit Windows on AMD64 (aka x86_64, Intel64, EM64T, etc) + - win-ia64 (64bit Windows on Itanium) + - win32 (all others - specifically, sys.platform is returned) + + Mac OS X can return: + + - macosx-10.6-ppc + - macosx-10.4-ppc64 + - macosx-10.3-i386 + - macosx-10.4-fat + + For other non-POSIX platforms, currently just returns :data:`sys.platform`. + + +.. function:: is_python_build() + + Return ``True`` if the current Python installation was built from source. + + +.. function:: parse_config_h(fp[, vars]) + + Parse a :file:`config.h`\-style file. + + *fp* is a file-like object pointing to the :file:`config.h`\-like file. + + A dictionary containing name/value pairs is returned. If an optional + dictionary is passed in as the second argument, it is used instead of a new + dictionary, and updated with the values read in the file. + + +.. function:: get_config_h_filename() + + Return the path of :file:`pyconfig.h`. + +.. function:: get_makefile_filename() + + Return the path of :file:`Makefile`. + +Using :mod:`sysconfig` as a script +---------------------------------- + +You can use :mod:`sysconfig` as a script with Python's *-m* option:: + + $ python -m sysconfig + Platform: "macosx-10.4-i386" + Python version: "3.2" + Current installation scheme: "posix_prefix" + + Paths: + data = "/usr/local" + include = "/Users/tarek/Dev/svn.python.org/py3k/Include" + platinclude = "." + platlib = "/usr/local/lib/python3.2/site-packages" + platstdlib = "/usr/local/lib/python3.2" + purelib = "/usr/local/lib/python3.2/site-packages" + scripts = "/usr/local/bin" + stdlib = "/usr/local/lib/python3.2" + + Variables: + AC_APPLE_UNIVERSAL_BUILD = "0" + AIX_GENUINE_CPLUSPLUS = "0" + AR = "ar" + ARFLAGS = "rc" + ASDLGEN = "./Parser/asdl_c.py" + ... + +This call will print in the standard output the information returned by +:func:`get_platform`, :func:`get_python_version`, :func:`get_path` and +:func:`get_config_vars`. diff --git a/Doc/library/syslog.rst b/Doc/library/syslog.rst index 89dd38f..a3b374a 100644 --- a/Doc/library/syslog.rst +++ b/Doc/library/syslog.rst @@ -10,42 +10,63 @@ This module provides an interface to the Unix ``syslog`` library routines. Refer to the Unix manual pages for a detailed description of the ``syslog`` facility. +This module wraps the system ``syslog`` family of routines. A pure Python +library that can speak to a syslog server is available in the +:mod:`logging.handlers` module as :class:`SysLogHandler`. + The module defines the following functions: .. function:: syslog([priority,] message) - Send the string *message* to the system logger. A trailing newline is added if - necessary. Each message is tagged with a priority composed of a *facility* and - a *level*. The optional *priority* argument, which defaults to - :const:`LOG_INFO`, determines the message priority. If the facility is not - encoded in *priority* using logical-or (``LOG_INFO | LOG_USER``), the value - given in the :func:`openlog` call is used. + Send the string *message* to the system logger. A trailing newline is added + if necessary. Each message is tagged with a priority composed of a + *facility* and a *level*. The optional *priority* argument, which defaults + to :const:`LOG_INFO`, determines the message priority. If the facility is + not encoded in *priority* using logical-or (``LOG_INFO | LOG_USER``), the + value given in the :func:`openlog` call is used. + + If :func:`openlog` has not been called prior to the call to :func:`syslog`, + ``openlog()`` will be called with no arguments. + +.. function:: openlog([ident[, logopt[, facility]]]) -.. function:: openlog(ident[, logopt[, facility]]) + Logging options of subsequent :func:`syslog` calls can be set by calling + :func:`openlog`. :func:`syslog` will call :func:`openlog` with no arguments + if the log is not currently open. - Logging options other than the defaults can be set by explicitly opening the log - file with :func:`openlog` prior to calling :func:`syslog`. The defaults are - (usually) *ident* = ``'syslog'``, *logopt* = ``0``, *facility* = - :const:`LOG_USER`. The *ident* argument is a string which is prepended to every - message. The optional *logopt* argument is a bit field - see below for possible - values to combine. The optional *facility* argument sets the default facility - for messages which do not have a facility explicitly encoded. + The optional *ident* keyword argument is a string which is prepended to every + message, and defaults to ``sys.argv[0]`` with leading path components + stripped. The optional *logopt* keyword argument (default is 0) is a bit + field -- see below for possible values to combine. The optional *facility* + keyword argument (default is :const:`LOG_USER`) sets the default facility for + messages which do not have a facility explicitly encoded. + + .. versionchanged:: 3.2 + In previous versions, keyword arguments were not allowed, and *ident* was + required. The default for *ident* was dependent on the system libraries, + and often was ``python`` instead of the name of the python program file. .. function:: closelog() - Close the log file. + Reset the syslog module values and call the system library ``closelog()``. + + This causes the module to behave as it does when initially imported. For + example, :func:`openlog` will be called on the first :func:`syslog` call (if + :func:`openlog` hasn't already been called), and *ident* and other + :func:`openlog` parameters are reset to defaults. .. function:: setlogmask(maskpri) - Set the priority mask to *maskpri* and return the previous mask value. Calls to - :func:`syslog` with a priority level not set in *maskpri* are ignored. The - default is to log all priorities. The function ``LOG_MASK(pri)`` calculates the - mask for the individual priority *pri*. The function ``LOG_UPTO(pri)`` - calculates the mask for all priorities up to and including *pri*. + Set the priority mask to *maskpri* and return the previous mask value. Calls + to :func:`syslog` with a priority level not set in *maskpri* are ignored. + The default is to log all priorities. The function ``LOG_MASK(pri)`` + calculates the mask for the individual priority *pri*. The function + ``LOG_UPTO(pri)`` calculates the mask for all priorities up to and including + *pri*. The module defines the following constants: @@ -63,3 +84,24 @@ Log options: :const:`LOG_PID`, :const:`LOG_CONS`, :const:`LOG_NDELAY`, :const:`LOG_NOWAIT` and :const:`LOG_PERROR` if defined in ``<syslog.h>``. + +Examples +-------- + +Simple example +~~~~~~~~~~~~~~ + +A simple set of examples:: + + import syslog + + syslog.syslog('Processing started') + if error: + syslog.syslog(syslog.LOG_ERR, 'Processing started') + +An example of setting some log options, these would include the process ID in +logged messages, and write the messages to the destination facility used for +mail logging:: + + syslog.openlog(logopt=syslog.LOG_PID, facility=syslog.LOG_MAIL) + syslog.syslog('E-mail processing initiated...') diff --git a/Doc/library/tabnanny.rst b/Doc/library/tabnanny.rst index 549ce36..4f3e705 100644 --- a/Doc/library/tabnanny.rst +++ b/Doc/library/tabnanny.rst @@ -9,6 +9,10 @@ .. rudimentary documentation based on module comments +**Source code:** :source:`Lib/tabnanny.py` + +-------------- + For the time being this module is intended to be called as a script. However it is possible to import it into an IDE and use the function :func:`check` described below. diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index d5a511e..9b7071b 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -8,6 +8,9 @@ .. moduleauthor:: Lars Gustäbel <lars@gustaebel.de> .. sectionauthor:: Lars Gustäbel <lars@gustaebel.de> +**Source code:** :source:`Lib/tarfile.py` + +-------------- The :mod:`tarfile` module makes it possible to read and write tar archives, including those using gzip or bz2 compression. @@ -20,7 +23,8 @@ Some facts and figures: * read/write support for the POSIX.1-1988 (ustar) format. * read/write support for the GNU tar format including *longname* and *longlink* - extensions, read-only support for the *sparse* extension. + extensions, read-only support for all variants of the *sparse* extension + including restoration of sparse files. * read/write support for the POSIX.1-2001 (pax) format. @@ -185,8 +189,8 @@ The following variables are available on module level: .. data:: ENCODING - The default character encoding i.e. the value from either - :func:`sys.getfilesystemencoding` or :func:`sys.getdefaultencoding`. + The default character encoding: ``'utf-8'`` on Windows, + :func:`sys.getfilesystemencoding` otherwise. .. seealso:: @@ -209,8 +213,16 @@ a header block followed by data blocks. It is possible to store a file in a tar archive several times. Each archive member is represented by a :class:`TarInfo` object, see :ref:`tarinfo-objects` for details. +A :class:`TarFile` object can be used as a context manager in a :keyword:`with` +statement. It will automatically be closed when the block is completed. Please +note that in the event of an exception an archive opened for writing will not +be finalized; only the internally used file object will be closed. See the +:ref:`tar-examples` section for a use case. -.. class:: TarFile(name=None, mode='r', fileobj=None, format=DEFAULT_FORMAT, tarinfo=TarInfo, dereference=False, ignore_zeros=False, encoding=ENCODING, errors=None, pax_headers=None, debug=0, errorlevel=0) +.. versionadded:: 3.2 + Added support for the context manager protocol. + +.. class:: TarFile(name=None, mode='r', fileobj=None, format=DEFAULT_FORMAT, tarinfo=TarInfo, dereference=False, ignore_zeros=False, encoding=ENCODING, errors='surrogateescape', pax_headers=None, debug=0, errorlevel=0) All following arguments are optional and can be accessed as instance attributes as well. @@ -259,6 +271,9 @@ object, see :ref:`tarinfo-objects` for details. to be handled. The default settings will work for most users. See section :ref:`tar-unicode` for in-depth information. + .. versionchanged:: 3.2 + Use ``'surrogateescape'`` as the default for the *errors* argument. + The *pax_headers* argument is an optional dictionary of strings which will be added as a pax global header if *format* is :const:`PAX_FORMAT`. @@ -324,12 +339,13 @@ object, see :ref:`tarinfo-objects` for details. dots ``".."``. -.. method:: TarFile.extract(member, path="") +.. method:: TarFile.extract(member, path="", set_attrs=True) Extract a member from the archive to the current working directory, using its full name. Its file information is extracted as accurately as possible. *member* may be a filename or a :class:`TarInfo` object. You can specify a different - directory using *path*. + directory using *path*. File attributes (owner, mtime, mode) are set unless + *set_attrs* is False. .. note:: @@ -340,6 +356,8 @@ object, see :ref:`tarinfo-objects` for details. See the warning for :meth:`extractall`. + .. versionchanged:: 3.2 + Added the *set_attrs* parameter. .. method:: TarFile.extractfile(member) @@ -355,15 +373,27 @@ object, see :ref:`tarinfo-objects` for details. and :meth:`close`, and also supports iteration over its lines. -.. method:: TarFile.add(name, arcname=None, recursive=True, exclude=None) +.. method:: TarFile.add(name, arcname=None, recursive=True, exclude=None, *, filter=None) - Add the file *name* to the archive. *name* may be any type of file (directory, - fifo, symbolic link, etc.). If given, *arcname* specifies an alternative name - for the file in the archive. Directories are added recursively by default. This - can be avoided by setting *recursive* to :const:`False`. If *exclude* is given, - it must be a function that takes one filename argument and returns a boolean - value. Depending on this value the respective file is either excluded - (:const:`True`) or added (:const:`False`). + Add the file *name* to the archive. *name* may be any type of file + (directory, fifo, symbolic link, etc.). If given, *arcname* specifies an + alternative name for the file in the archive. Directories are added + recursively by default. This can be avoided by setting *recursive* to + :const:`False`. If *exclude* is given, it must be a function that takes one + filename argument and returns a boolean value. Depending on this value the + respective file is either excluded (:const:`True`) or added + (:const:`False`). If *filter* is specified it must be a keyword argument. It + should be a function that takes a :class:`TarInfo` object argument and + returns the changed :class:`TarInfo` object. If it instead returns + :const:`None` the :class:`TarInfo` object will be excluded from the + archive. See :ref:`tar-examples` for an example. + + .. versionchanged:: 3.2 + Added the *filter* parameter. + + .. deprecated:: 3.2 + The *exclude* parameter is deprecated, please use the *filter* parameter + instead. .. method:: TarFile.addfile(tarinfo, fileobj=None) @@ -430,11 +460,14 @@ It does *not* contain the file's data itself. a :class:`TarInfo` object. -.. method:: TarInfo.tobuf(format=DEFAULT_FORMAT, encoding=ENCODING, errors='strict') +.. method:: TarInfo.tobuf(format=DEFAULT_FORMAT, encoding=ENCODING, errors='surrogateescape') Create a string buffer from a :class:`TarInfo` object. For information on the arguments see the constructor of the :class:`TarFile` class. + .. versionchanged:: 3.2 + Use ``'surrogateescape'`` as the default for the *errors* argument. + A ``TarInfo`` object has the following public data attributes: @@ -582,6 +615,13 @@ How to create an uncompressed tar archive from a list of filenames:: tar.add(name) tar.close() +The same example using the :keyword:`with` statement:: + + import tarfile + with tarfile.open("sample.tar", "w") as tar: + for name in ["foo", "bar", "quux"]: + tar.add(name) + How to read a gzip compressed tar archive and display some member information:: import tarfile @@ -596,6 +636,18 @@ How to read a gzip compressed tar archive and display some member information:: print("something else.") tar.close() +How to create an archive and reset the user information using the *filter* +parameter in :meth:`TarFile.add`:: + + import tarfile + def reset(tarinfo): + tarinfo.uid = tarinfo.gid = 0 + tarinfo.uname = tarinfo.gname = "root" + return tarinfo + tar = tarfile.open("sample.tar.gz", "w:gz") + tar.add("foo", filter=reset) + tar.close() + .. _tar-formats: @@ -663,11 +715,12 @@ metadata must be either decoded or encoded. If *encoding* is not set appropriately, this conversion may fail. The *errors* argument defines how characters are treated that cannot be -converted. Possible values are listed in section :ref:`codec-base-classes`. In -read mode the default scheme is ``'replace'``. This avoids unexpected -:exc:`UnicodeError` exceptions and guarantees that an archive can always be -read. In write mode the default value for *errors* is ``'strict'``. This -ensures that name information is not altered unnoticed. - -In case of writing :const:`PAX_FORMAT` archives, *encoding* is ignored because -non-ASCII metadata is stored using *UTF-8*. +converted. Possible values are listed in section :ref:`codec-base-classes`. +The default scheme is ``'surrogateescape'`` which Python also uses for its +file system calls, see :ref:`os-filenames`. + +In case of :const:`PAX_FORMAT` archives, *encoding* is generally not needed +because all the metadata is stored using *UTF-8*. *encoding* is only used in +the rare cases when binary pax headers are decoded or when strings with +surrogate characters are stored. + diff --git a/Doc/library/telnetlib.rst b/Doc/library/telnetlib.rst index 6e3abde..646634d 100644 --- a/Doc/library/telnetlib.rst +++ b/Doc/library/telnetlib.rst @@ -8,6 +8,10 @@ .. index:: single: protocol; Telnet +**Source code:** :source:`Lib/telnetlib.py` + +-------------- + The :mod:`telnetlib` module provides a :class:`Telnet` class that implements the Telnet protocol. See :rfc:`854` for details about the protocol. In addition, it provides symbolic constants for the protocol characters (see below), and for the diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index a13df0d..01092fc 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -12,6 +12,10 @@ pair: temporary; file name pair: temporary; file +**Source code:** :source:`Lib/tempfile.py` + +-------------- + This module generates temporary files and directories. It works on all supported platforms. It provides three new functions, :func:`NamedTemporaryFile`, :func:`mkstemp`, and :func:`mkdtemp`, which should @@ -25,7 +29,7 @@ no longer necessary to use the global *tempdir* and *template* variables. To maintain backward compatibility, the argument order is somewhat odd; it is recommended to use keyword arguments for clarity. -The module defines the following user-callable functions: +The module defines the following user-callable items: .. function:: TemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None) @@ -83,6 +87,24 @@ The module defines the following user-callable functions: used in a :keyword:`with` statement, just like a normal file. +.. function:: TemporaryDirectory(suffix='', prefix='tmp', dir=None) + + This function creates a temporary directory using :func:`mkdtemp` + (the supplied arguments are passed directly to the underlying function). + The resulting object can be used as a context manager (see + :ref:`context-managers`). On completion of the context (or destruction + of the temporary directory object), the newly created temporary directory + and all its contents are removed from the filesystem. + + The directory name can be retrieved from the :attr:`name` member + of the returned object. + + The directory can be explicitly cleaned up by calling the + :func:`cleanup` method. + + .. versionadded:: 3.2 + + .. function:: mkstemp(suffix='', prefix='tmp', dir=None, text=False) Creates a temporary file in the most secure manner possible. There are @@ -210,3 +232,36 @@ the appropriate function arguments, instead. Return the filename prefix used to create temporary files. This does not contain the directory component. + +Examples +-------- + +Here are some examples of typical usage of the :mod:`tempfile` module:: + + >>> import tempfile + + # create a temporary file and write some data to it + >>> fp = tempfile.TemporaryFile() + >>> fp.write(b'Hello world!') + # read data from file + >>> fp.seek(0) + >>> fp.read() + b'Hello world!' + # close the file, it will be removed + >>> fp.close() + + # create a temporary file using a context manager + >>> with tempfile.TemporaryFile() as fp: + ... fp.write(b'Hello world!') + ... fp.seek(0) + ... fp.read() + b'Hello world!' + >>> + # file is now closed and removed + + # create a temporary directory using the context manager + >>> with tempfile.TemporaryDirectory() as tmpdirname: + ... print('created temporary directory', tmpdirname) + >>> + # directory and contents have been removed + diff --git a/Doc/library/test.rst b/Doc/library/test.rst index 12f4c13..9d02b0a 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -5,7 +5,7 @@ :synopsis: Regression tests package containing the testing suite for Python. .. sectionauthor:: Brett Cannon <brett@python.org> -.. warning:: +.. note:: The :mod:`test` package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python's standard library is discouraged as code @@ -96,17 +96,17 @@ The goal for regression testing is to try to break code. This leads to a few guidelines to be followed: * The testing suite should exercise all classes, functions, and constants. This - includes not just the external API that is to be presented to the outside world - but also "private" code. + includes not just the external API that is to be presented to the outside + world but also "private" code. * Whitebox testing (examining the code being tested when the tests are being written) is preferred. Blackbox testing (testing only the published user - interface) is not complete enough to make sure all boundary and edge cases are - tested. + interface) is not complete enough to make sure all boundary and edge cases + are tested. * Make sure all possible values are tested including invalid ones. This makes - sure that not only all valid values are acceptable but also that improper values - are handled correctly. + sure that not only all valid values are acceptable but also that improper + values are handled correctly. * Exhaust as many code paths as possible. Test where branching occurs and thus tailor input to make sure as many different paths through the code are taken. @@ -126,8 +126,8 @@ guidelines to be followed: behavior from side-effects of importing a module. * Try to maximize code reuse. On occasion, tests will vary by something as small - as what type of input is used. Minimize code duplication by subclassing a basic - test class with a class that specifies the input:: + as what type of input is used. Minimize code duplication by subclassing a + basic test class with a class that specifies the input:: class TestFuncAcceptsSequences(unittest.TestCase): @@ -137,13 +137,13 @@ guidelines to be followed: self.func(self.arg) class AcceptLists(TestFuncAcceptsSequences): - arg = [1,2,3] + arg = [1, 2, 3] class AcceptStrings(TestFuncAcceptsSequences): arg = 'abc' class AcceptTuples(TestFuncAcceptsSequences): - arg = (1,2,3) + arg = (1, 2, 3) .. seealso:: @@ -157,32 +157,34 @@ guidelines to be followed: Running tests using the command-line interface ---------------------------------------------- -The :mod:`test.regrtest` module can be run as a script to drive Python's regression -test suite, thanks to the :option:`-m` option: :program:`python -m test.regrtest`. +The :mod:`test` package can be run as a script to drive Python's regression +test suite, thanks to the :option:`-m` option: :program:`python -m test`. Under +the hood, it uses :mod:`test.regrtest`; the call :program:`python -m +test.regrtest` used in previous Python versions still works). Running the script by itself automatically starts running all regression tests in the :mod:`test` package. It does this by finding all modules in the package whose name starts with ``test_``, importing them, and executing the -function :func:`test_main` if present. The names of tests to execute may also be -passed to the script. Specifying a single regression test (:program:`python --m test.regrtest test_spam`) will minimize output and only print whether -the test passed or failed and thus minimize output. - -Running :mod:`test.regrtest` directly allows what resources are available for -tests to use to be set. You do this by using the :option:`-u` command-line -option. Run :program:`python -m test.regrtest -uall` to turn on all +function :func:`test_main` if present. The names of tests to execute may also +be passed to the script. Specifying a single regression test (:program:`python +-m test test_spam`) will minimize output and only print +whether the test passed or failed and thus minimize output. + +Running :mod:`test` directly allows what resources are available for +tests to use to be set. You do this by using the ``-u`` command-line +option. Run :program:`python -m test -uall` to turn on all resources; specifying ``all`` as an option for ``-u`` enables all possible resources. If all but one resource is desired (a more common case), a comma-separated list of resources that are not desired may be listed after -``all``. The command :program:`python -m test.regrtest -uall,-audio,-largefile` -will run :mod:`test.regrtest` with all resources except the ``audio`` and +``all``. The command :program:`python -m test -uall,-audio,-largefile` +will run :mod:`test` with all resources except the ``audio`` and ``largefile`` resources. For a list of all resources and more command-line -options, run :program:`python -m test.regrtest -h`. +options, run :program:`python -m test -h`. Some other ways to execute the regression tests depend on what platform the tests are being executed on. On Unix, you can run :program:`make test` at the -top-level directory where Python was built. On Windows, executing -:program:`rt.bat` from your :file:`PCBuild` directory will run all regression -tests. +top-level directory where Python was built. On Windows, +executing :program:`rt.bat` from your :file:`PCBuild` directory will run all +regression tests. :mod:`test.support` --- Utility functions for tests @@ -205,17 +207,11 @@ This module defines the following exceptions: methods. -.. exception:: TestSkipped - - Subclass of :exc:`TestFailed`. Raised when a test is skipped. This occurs when a - needed resource (such as a network connection) is not available at the time of - testing. - - .. exception:: ResourceDenied - Subclass of :exc:`TestSkipped`. Raised when a resource (such as a network - connection) is not available. Raised by the :func:`requires` function. + Subclass of :exc:`unittest.SkipTest`. Raised when a resource (such as a + network connection) is not available. Raised by the :func:`requires` + function. The :mod:`test.support` module defines the following constants: @@ -234,44 +230,45 @@ The :mod:`test.support` module defines the following constants: .. data:: TESTFN - Set to the path that a temporary file may be created at. Any temporary that is - created should be closed and unlinked (removed). + Set to a name that is safe to use as the name of a temporary file. Any + temporary file that is created should be closed and unlinked (removed). The :mod:`test.support` module defines the following functions: .. function:: forget(module_name) - Removes the module named *module_name* from ``sys.modules`` and deletes any + Remove the module named *module_name* from ``sys.modules`` and delete any byte-compiled files of the module. .. function:: is_resource_enabled(resource) - Returns :const:`True` if *resource* is enabled and available. The list of + Return :const:`True` if *resource* is enabled and available. The list of available resources is only set when :mod:`test.regrtest` is executing the tests. .. function:: requires(resource, msg=None) - Raises :exc:`ResourceDenied` if *resource* is not available. *msg* is the - argument to :exc:`ResourceDenied` if it is raised. Always returns true if called - by a function whose ``__name__`` is ``'__main__'``. Used when tests are executed - by :mod:`test.regrtest`. + Raise :exc:`ResourceDenied` if *resource* is not available. *msg* is the + argument to :exc:`ResourceDenied` if it is raised. Always returns + :const:`True` if called by a function whose ``__name__`` is ``'__main__'``. + Used when tests are executed by :mod:`test.regrtest`. .. function:: findfile(filename) - Return the path to the file named *filename*. If no match is found *filename* is - returned. This does not equal a failure since it could be the path to the file. + Return the path to the file named *filename*. If no match is found + *filename* is returned. This does not equal a failure since it could be the + path to the file. -.. function:: run_unittest(*classes) +.. function:: run_unittest(\*classes) Execute :class:`unittest.TestCase` subclasses passed to the function. The - function scans the classes for methods starting with the prefix ``test_`` and - executes the tests individually. + function scans the classes for methods starting with the prefix ``test_`` + and executes the tests individually. It is also legal to pass strings as parameters; these should be keys in ``sys.modules``. Each associated module will be scanned by @@ -284,37 +281,72 @@ The :mod:`test.support` module defines the following functions: This will run all tests defined in the named module. -.. function:: check_warnings() +.. function:: check_warnings(\*filters, quiet=True) + + A convenience wrapper for :func:`warnings.catch_warnings()` that makes it + easier to test that a warning was correctly raised. It is approximately + equivalent to calling ``warnings.catch_warnings(record=True)`` with + :meth:`warnings.simplefilter` set to ``always`` and with the option to + automatically validate the results that are recorded. + + ``check_warnings`` accepts 2-tuples of the form ``("message regexp", + WarningCategory)`` as positional arguments. If one or more *filters* are + provided, or if the optional keyword argument *quiet* is :const:`False`, + it checks to make sure the warnings are as expected: each specified filter + must match at least one of the warnings raised by the enclosed code or the + test fails, and if any warnings are raised that do not match any of the + specified filters the test fails. To disable the first of these checks, + set *quiet* to :const:`True`. + + If no arguments are specified, it defaults to:: + + check_warnings(("", Warning), quiet=True) - A convenience wrapper for ``warnings.catch_warnings()`` that makes - it easier to test that a warning was correctly raised with a single - assertion. It is approximately equivalent to calling - ``warnings.catch_warnings(record=True)``. + In this case all warnings are caught and no errors are raised. - The main difference is that on entry to the context manager, a - :class:`WarningRecorder` instance is returned instead of a simple list. - The underlying warnings list is available via the recorder object's - :attr:`warnings` attribute, while the attributes of the last raised - warning are also accessible directly on the object. If no warning has - been raised, then the latter attributes will all be :const:`None`. + On entry to the context manager, a :class:`WarningRecorder` instance is + returned. The underlying warnings list from + :func:`~warnings.catch_warnings` is available via the recorder object's + :attr:`warnings` attribute. As a convenience, the attributes of the object + representing the most recent warning can also be accessed directly through + the recorder object (see example below). If no warning has been raised, + then any of the attributes that would otherwise be expected on an object + representing a warning will return :const:`None`. - A :meth:`reset` method is also provided on the recorder object. This - method simply clears the warning list. + The recorder object also has a :meth:`reset` method, which clears the + warnings list. - The context manager is used like this:: + The context manager is designed to be used like this:: - with check_warnings() as w: - warnings.simplefilter("always") + with check_warnings(("assertion is always true", SyntaxWarning), + ("", UserWarning)): + exec('assert(False, "Hey!")') + warnings.warn(UserWarning("Hide me!")) + + In this case if either warning was not raised, or some other warning was + raised, :func:`check_warnings` would raise an error. + + When a test needs to look more deeply into the warnings, rather than + just checking whether or not they occurred, code like this can be used:: + + with check_warnings(quiet=True) as w: warnings.warn("foo") - assert str(w.message) == "foo" + assert str(w.args[0]) == "foo" warnings.warn("bar") - assert str(w.message) == "bar" - assert str(w.warnings[0].message) == "foo" - assert str(w.warnings[1].message) == "bar" + assert str(w.args[0]) == "bar" + assert str(w.warnings[0].args[0]) == "foo" + assert str(w.warnings[1].args[0]) == "bar" w.reset() assert len(w.warnings) == 0 + Here all warnings will be caught, and the test code tests the captured + warnings directly. + + .. versionchanged:: 3.2 + New optional arguments *filters* and *quiet*. + + .. function:: captured_stdout() This is a context manager that runs the :keyword:`with` statement body using @@ -389,18 +421,19 @@ The :mod:`test.support` module defines the following classes: .. class:: EnvironmentVarGuard() - Class used to temporarily set or unset environment variables. Instances can be - used as a context manager and have a complete dictionary interface for - querying/modifying the underlying ``os.environ``. After exit from the context - manager all changes to environment variables done through this instance will - be rolled back. + Class used to temporarily set or unset environment variables. Instances can + be used as a context manager and have a complete dictionary interface for + querying/modifying the underlying ``os.environ``. After exit from the + context manager all changes to environment variables done through this + instance will be rolled back. .. versionchanged:: 3.1 Added dictionary interface. .. method:: EnvironmentVarGuard.set(envvar, value) - Temporarily set the environment variable ``envvar`` to the value of ``value``. + Temporarily set the environment variable ``envvar`` to the value of + ``value``. .. method:: EnvironmentVarGuard.unset(envvar) @@ -412,4 +445,3 @@ The :mod:`test.support` module defines the following classes: Class used to record warnings for unit tests. See documentation of :func:`check_warnings` above for more details. - diff --git a/Doc/library/textwrap.rst b/Doc/library/textwrap.rst index 8357013..a814962 100644 --- a/Doc/library/textwrap.rst +++ b/Doc/library/textwrap.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Greg Ward <gward@python.net> .. sectionauthor:: Greg Ward <gward@python.net> +**Source code:** :source:`Lib/textwrap.py` + +-------------- The :mod:`textwrap` module provides two convenience functions, :func:`wrap` and :func:`fill`, as well as :class:`TextWrapper`, the class that does all the work, @@ -13,7 +16,6 @@ and a utility function :func:`dedent`. If you're just wrapping or filling one or two text strings, the convenience functions should be good enough; otherwise, you should use an instance of :class:`TextWrapper` for efficiency. - .. function:: wrap(text, width=70, **kwargs) Wraps the single paragraph in *text* (a string) so every line is at most diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index fb18809..5f1b9bf 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -4,6 +4,9 @@ .. module:: threading :synopsis: Thread-based parallelism. +**Source code:** :source:`Lib/threading.py` + +-------------- This module constructs higher-level threading interfaces on top of the lower level :mod:`_thread` module. See also the :mod:`queue` module. @@ -24,8 +27,9 @@ The :mod:`dummy_threading` module is provided for situations where libraries might overcome this limitation). If you want your application to make better of use of the computational resources of multi-core machines, you are advised to use - :mod:`multiprocessing`. However, threading is still an appropriate model - if you want to run multiple I/O-bound tasks simultaneously. + :mod:`multiprocessing` or :class:`concurrent.futures.ProcessPoolExecutor`. + However, threading is still an appropriate model if you want to run + multiple I/O-bound tasks simultaneously. This module defines the following functions and objects: @@ -182,6 +186,18 @@ This module defines the following functions and objects: Availability: Windows, systems with POSIX threads. +This module also defines the following constant: + +.. data:: TIMEOUT_MAX + + The maximum value allowed for the *timeout* parameter of blocking functions + (:meth:`Lock.acquire`, :meth:`RLock.acquire`, :meth:`Condition.wait`, etc.). + Specifying a timeout greater than this value will raise an + :exc:`OverflowError`. + + .. versionadded:: 3.2 + + Detailed interfaces for the objects are documented below. The design of this module is loosely based on Java's threading model. However, @@ -376,7 +392,7 @@ and may vary across implementations. All methods are executed atomically. -.. method:: Lock.acquire([blocking]) +.. method:: Lock.acquire(blocking=True, timeout=-1) Acquire a lock, blocking or non-blocking. @@ -390,6 +406,21 @@ All methods are executed atomically. without an argument would block, return false immediately; otherwise, do the same thing as when called without arguments, and return true. + When invoked with the floating-point *timeout* argument set to a positive + value, block for at most the number of seconds specified by *timeout* + and as long as the lock cannot be acquired. A negative *timeout* argument + specifies an unbounded wait. It is forbidden to specify a *timeout* + when *blocking* is false. + + The return value is ``True`` if the lock is acquired successfully, + ``False`` if not (for example if the *timeout* expired). + + .. versionchanged:: 3.2 + The *timeout* parameter is new. + + .. versionchanged:: 3.2 + Lock acquires can now be interrupted by signals on POSIX. + .. method:: Lock.release() @@ -423,7 +454,7 @@ pair) resets the lock to unlocked and allows another thread blocked in :meth:`acquire` to proceed. -.. method:: RLock.acquire(blocking=True) +.. method:: RLock.acquire(blocking=True, timeout=-1) Acquire a lock, blocking or non-blocking. @@ -442,6 +473,14 @@ pair) resets the lock to unlocked and allows another thread blocked in without an argument would block, return false immediately; otherwise, do the same thing as when called without arguments, and return true. + When invoked with the floating-point *timeout* argument set to a positive + value, block for at most the number of seconds specified by *timeout* + and as long as the lock cannot be acquired. Return true if the lock has + been acquired, false if the timeout has elapsed. + + .. versionchanged:: 3.2 + The *timeout* parameter is new. + .. method:: RLock.release() @@ -513,6 +552,13 @@ state change can be interesting for only one or several waiting threads. E.g. in a typical producer-consumer situation, adding one item to the buffer only needs to wake up one consumer thread. +Note: Condition variables can be, depending on the implementation, subject +to both spurious wakeups (when :meth:`wait` returns without a :meth:`notify` +call) and stolen wakeups (when another thread acquires the lock before the +awoken thread.) For this reason, it is always necessary to verify the state +the thread is waiting for when :meth:`wait` returns and optionally repeat +the call as often as necessary. + .. class:: Condition(lock=None) @@ -553,6 +599,41 @@ needs to wake up one consumer thread. interface is then used to restore the recursion level when the lock is reacquired. + The return value is ``True`` unless a given *timeout* expired, in which + case it is ``False``. + + .. versionchanged:: 3.2 + Previously, the method always returned ``None``. + + .. method:: wait_for(predicate, timeout=None) + + Wait until a condition evaluates to True. *predicate* should be a + callable which result will be interpreted as a boolean value. + A *timeout* may be provided giving the maximum time to wait. + + This utility method may call :meth:`wait` repeatedly until the predicate + is satisfied, or until a timeout occurs. The return value is + the last return value of the predicate and will evaluate to + ``False`` if the method timed out. + + Ignoring the timeout feature, calling this method is roughly equivalent to + writing:: + + while not predicate(): + cv.wait() + + Therefore, the same rules apply as with :meth:`wait`: The lock must be + held when called and is re-aquired on return. The predicate is evaluated + with the lock held. + + Using this method, the consumer example above can be written thus:: + + with cv: + cv.wait_for(an_item_is_available) + get_an_available_item() + + .. versionadded:: 3.2 + .. method:: notify() Wake up a thread waiting on this condition, if any. If the calling thread @@ -599,7 +680,7 @@ waiting until some other thread calls :meth:`release`. defaults to ``1``. If the *value* given is less than 0, :exc:`ValueError` is raised. - .. method:: acquire(blocking=True) + .. method:: acquire(blocking=True, timeout=None) Acquire a semaphore. @@ -610,14 +691,18 @@ waiting until some other thread calls :meth:`release`. interlocking so that if multiple :meth:`acquire` calls are blocked, :meth:`release` will wake exactly one of them up. The implementation may pick one at random, so the order in which blocked threads are awakened - should not be relied on. There is no return value in this case. - - When invoked with *blocking* set to true, do the same thing as when called - without arguments, and return true. + should not be relied on. Returns true (or blocks indefinitely). When invoked with *blocking* set to false, do not block. If a call - without an argument would block, return false immediately; otherwise, do - the same thing as when called without arguments, and return true. + without an argument would block, return false immediately; otherwise, + do the same thing as when called without arguments, and return true. + + When invoked with a *timeout* other than None, it will block for at + most *timeout* seconds. If acquire does not complete successfully in + that interval, return false. Return true otherwise. + + .. versionchanged:: 3.2 + The *timeout* parameter is new. .. method:: release() @@ -737,6 +822,108 @@ For example:: only work if the timer is still in its waiting stage. +Barrier Objects +--------------- + +.. versionadded:: 3.2 + +This class provides a simple synchronization primitive for use by a fixed number +of threads that need to wait for each other. Each of the threads tries to pass +the barrier by calling the :meth:`wait` method and will block until all of the +threads have made the call. At this points, the threads are released +simultanously. + +The barrier can be reused any number of times for the same number of threads. + +As an example, here is a simple way to synchronize a client and server thread:: + + b = Barrier(2, timeout=5) + + def server(): + start_server() + b.wait() + while True: + connection = accept_connection() + process_server_connection(connection) + + def client(): + b.wait() + while True: + connection = make_connection() + process_client_connection(connection) + + +.. class:: Barrier(parties, action=None, timeout=None) + + Create a barrier object for *parties* number of threads. An *action*, when + provided, is a callable to be called by one of the threads when they are + released. *timeout* is the default timeout value if none is specified for + the :meth:`wait` method. + + .. method:: wait(timeout=None) + + Pass the barrier. When all the threads party to the barrier have called + this function, they are all released simultaneously. If a *timeout* is + provided, is is used in preference to any that was supplied to the class + constructor. + + The return value is an integer in the range 0 to *parties* -- 1, different + for each thread. This can be used to select a thread to do some special + housekeeping, e.g.:: + + i = barrier.wait() + if i == 0: + # Only one thread needs to print this + print("passed the barrier") + + If an *action* was provided to the constructor, one of the threads will + have called it prior to being released. Should this call raise an error, + the barrier is put into the broken state. + + If the call times out, the barrier is put into the broken state. + + This method may raise a :class:`BrokenBarrierError` exception if the + barrier is broken or reset while a thread is waiting. + + .. method:: reset() + + Return the barrier to the default, empty state. Any threads waiting on it + will receive the :class:`BrokenBarrierError` exception. + + Note that using this function may can require some external + synchronization if there are other threads whose state is unknown. If a + barrier is broken it may be better to just leave it and create a new one. + + .. method:: abort() + + Put the barrier into a broken state. This causes any active or future + calls to :meth:`wait` to fail with the :class:`BrokenBarrierError`. Use + this for example if one of the needs to abort, to avoid deadlocking the + application. + + It may be preferable to simply create the barrier with a sensible + *timeout* value to automatically guard against one of the threads going + awry. + + .. attribute:: parties + + The number of threads required to pass the barrier. + + .. attribute:: n_waiting + + The number of threads currently waiting in the barrier. + + .. attribute:: broken + + A boolean that is ``True`` if the barrier is in the broken state. + + +.. exception:: BrokenBarrierError + + This exception, a subclass of :exc:`RuntimeError`, is raised when the + :class:`Barrier` object is reset or broken. + + .. _with-locks: Using locks, conditions, and semaphores in the :keyword:`with` statement diff --git a/Doc/library/time.rst b/Doc/library/time.rst index b91aa53..28e994c 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -24,9 +24,9 @@ An explanation of some terminology and conventions is in order. .. index:: single: Year 2038 -* The functions in this module do not handle dates and times before the epoch or +* The functions in this module may not handle dates and times before the epoch or far in the future. The cut-off point in the future is determined by the C - library; for Unix, it is typically in 2038. + library; for 32-bit systems, it is typically in 2038. .. index:: single: Year 2000 @@ -34,20 +34,31 @@ An explanation of some terminology and conventions is in order. .. _time-y2kissues: -* **Year 2000 (Y2K) issues**: Python depends on the platform's C library, which +* **Year 2000 (Y2K) issues**: Python depends on the platform's C library, which generally doesn't have year 2000 issues, since all dates and times are - represented internally as seconds since the epoch. Functions accepting a - :class:`struct_time` (see below) generally require a 4-digit year. For backward - compatibility, 2-digit years are supported if the module variable - ``accept2dyear`` is a non-zero integer; this variable is initialized to ``1`` - unless the environment variable :envvar:`PYTHONY2K` is set to a non-empty - string, in which case it is initialized to ``0``. Thus, you can set - :envvar:`PYTHONY2K` to a non-empty string in the environment to require 4-digit - years for all year input. When 2-digit years are accepted, they are converted - according to the POSIX or X/Open standard: values 69-99 are mapped to 1969-1999, - and values 0--68 are mapped to 2000--2068. Values 100--1899 are always illegal. - Note that this is new as of Python 1.5.2(a2); earlier versions, up to Python - 1.5.1 and 1.5.2a1, would add 1900 to year values below 1900. + represented internally as seconds since the epoch. Function :func:`strptime` + can parse 2-digit years when given ``%y`` format code. When 2-digit years are + parsed, they are converted according to the POSIX and ISO C standards: values + 69--99 are mapped to 1969--1999, and values 0--68 are mapped to 2000--2068. + + For backward compatibility, years with less than 4 digits are treated + specially by :func:`asctime`, :func:`mktime`, and :func:`strftime` functions + that operate on a 9-tuple or :class:`struct_time` values. If year (the first + value in the 9-tuple) is specified with less than 4 digits, its interpretation + depends on the value of ``accept2dyear`` variable. + + If ``accept2dyear`` is true (default), a backward compatibility behavior is + invoked as follows: + + - for 2-digit year, century is guessed according to POSIX rules for + ``%y`` strptime format. A deprecation warning is issued when century + information is guessed in this way. + + - for 3-digit or negative year, a :exc:`ValueError` exception is raised. + + If ``accept2dyear`` is false (set by the program or as a result of a + non-empty value assigned to ``PYTHONY2K`` environment variable) all year + values are interpreted as given. .. index:: single: UTC @@ -73,8 +84,8 @@ An explanation of some terminology and conventions is in order. * On the other hand, the precision of :func:`time` and :func:`sleep` is better than their Unix equivalents: times are expressed as floating point numbers, :func:`time` returns the most accurate time available (using Unix - :cfunc:`gettimeofday` where available), and :func:`sleep` will accept a time - with a nonzero fraction (Unix :cfunc:`select` is used to implement this, where + :c:func:`gettimeofday` where available), and :func:`sleep` will accept a time + with a nonzero fraction (Unix :c:func:`select` is used to implement this, where available). * The time value as returned by :func:`gmtime`, :func:`localtime`, and @@ -109,10 +120,19 @@ The module defines the following functions and data items: .. data:: accept2dyear - Boolean value indicating whether two-digit year values will be accepted. This - is true by default, but will be set to false if the environment variable - :envvar:`PYTHONY2K` has been set to a non-empty string. It may also be modified - at run time. + Boolean value indicating whether two-digit year values will be + mapped to 1969--2068 range by :func:`asctime`, :func:`mktime`, and + :func:`strftime` functions. This is true by default, but will be + set to false if the environment variable :envvar:`PYTHONY2K` has + been set to a non-empty string. It may also be modified at run + time. + + .. deprecated:: 3.2 + Mapping of 2-digit year values by :func:`asctime`, + :func:`mktime`, and :func:`strftime` functions to 1969--2068 + range is deprecated. Programs that need to process 2-digit + years should use ``%y`` code available in :func:`strptime` + function or convert 2-digit year values to 4-digit themselves. .. data:: altzone @@ -125,7 +145,7 @@ The module defines the following functions and data items: .. function:: asctime([t]) Convert a tuple or :class:`struct_time` representing a time as returned by - :func:`gmtime` or :func:`localtime` to a 24-character string of the following + :func:`gmtime` or :func:`localtime` to a string of the following form: ``'Sun Jun 20 23:21:05 1993'``. If *t* is not provided, the current time as returned by :func:`localtime` is used. Locale information is not used by :func:`asctime`. @@ -149,7 +169,7 @@ The module defines the following functions and data items: On Windows, this function returns wall-clock seconds elapsed since the first call to this function, as a floating point number, based on the Win32 function - :cfunc:`QueryPerformanceCounter`. The resolution is typically better than one + :c:func:`QueryPerformanceCounter`. The resolution is typically better than one microsecond. @@ -288,7 +308,7 @@ The module defines the following functions and data items: | ``%y`` | Year without century as a decimal number | | | | [00,99]. | | +-----------+------------------------------------------------+-------+ - | ``%Y`` | Year with century as a decimal number. | | + | ``%Y`` | Year with century as a decimal number. | \(4) | | | | | +-----------+------------------------------------------------+-------+ | ``%Z`` | Time zone name (no characters if no time zone | | @@ -304,13 +324,20 @@ The module defines the following functions and data items: the output hour field if the ``%I`` directive is used to parse the hour. (2) - The range really is ``0`` to ``61``; this accounts for leap seconds and the - (very rare) double leap seconds. + The range really is ``0`` to ``61``; value ``60`` is valid in + timestamps representing leap seconds and value ``61`` is supported + for historical reasons. (3) When used with the :func:`strptime` function, ``%U`` and ``%W`` are only used in calculations when the day of the week and the year are specified. + (4) + Produces different results depending on the value of + ``time.accept2dyear`` variable. See :ref:`Year 2000 (Y2K) + issues <time-y2kissues>` for details. + + Here is an example, a format for dates compatible with that specified in the :rfc:`2822` Internet email standard. [#]_ :: @@ -380,7 +407,7 @@ The module defines the following functions and data items: +-------+-------------------+---------------------------------+ | 4 | :attr:`tm_min` | range [0, 59] | +-------+-------------------+---------------------------------+ - | 5 | :attr:`tm_sec` | range [0, 61]; see **(1)** in | + | 5 | :attr:`tm_sec` | range [0, 61]; see **(2)** in | | | | :func:`strftime` description | +-------+-------------------+---------------------------------+ | 6 | :attr:`tm_wday` | range [0, 6], Monday is 0 | diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst index 4a0b9c2..a03e40e 100644 --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -9,6 +9,10 @@ single: Benchmarking single: Performance +**Source code:** :source:`Lib/timeit.py` + +-------------- + This module provides a simple way to time small bits of Python code. It has both command line as well as callable interfaces. It avoids a number of common traps for measuring execution times. See also Tim Peters' introduction to the diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst index 7bb54fd..ae5635f 100644 --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -9,7 +9,9 @@ The :mod:`tkinter` package ("Tk interface") is the standard Python interface to the Tk GUI toolkit. Both Tk and :mod:`tkinter` are available on most Unix platforms, as well as on Windows systems. (Tk itself is not part of Python; it -is maintained at ActiveState.) +is maintained at ActiveState.) You can check that :mod:`tkinter` is properly +installed on your system by running ``python -m tkinter`` from the command line; +this should open a window demonstrating a simple Tk interface. .. seealso:: @@ -657,9 +659,7 @@ relief scrollcommand This is almost always the :meth:`!set` method of some scrollbar widget, but can - be any widget method that takes a single argument. Refer to the file - :file:`Demo/tkinter/matt/canvas-with-scrollbars.py` in the Python source - distribution for an example. + be any widget method that takes a single argument. wrap: Must be one of: ``"none"``, ``"char"``, or ``"word"``. diff --git a/Doc/library/tkinter.tix.rst b/Doc/library/tkinter.tix.rst index beb91e6..289bffd 100644 --- a/Doc/library/tkinter.tix.rst +++ b/Doc/library/tkinter.tix.rst @@ -84,11 +84,7 @@ Tix Widgets ----------- `Tix <http://tix.sourceforge.net/dist/current/man/html/TixCmd/TixIntro.htm>`_ -introduces over 40 widget classes to the :mod:`tkinter` repertoire. There is a -demo of all the :mod:`tkinter.tix` widgets in the :file:`Demo/tix` directory of -the standard distribution. - -.. The Python sample code is still being added to Python, hence commented out +introduces over 40 widget classes to the :mod:`tkinter` repertoire. Basic Widgets diff --git a/Doc/library/token.rst b/Doc/library/token.rst index 991762f..4b98eac 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -5,6 +5,9 @@ :synopsis: Constants representing terminal nodes of the parse tree. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/token.py` + +-------------- This module provides constants which represent the numeric values of leaf nodes of the parse tree (terminal tokens). Refer to the file :file:`Grammar/Grammar` diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 7017045..577d7cc 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Ka Ping Yee .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/tokenize.py` + +-------------- The :mod:`tokenize` module provides a lexical scanner for Python source code, implemented in Python. The scanner in this module returns comments as tokens @@ -95,12 +98,25 @@ function it uses to do this is available: It detects the encoding from the presence of a UTF-8 BOM or an encoding cookie as specified in :pep:`263`. If both a BOM and a cookie are present, - but disagree, a SyntaxError will be raised. + but disagree, a SyntaxError will be raised. Note that if the BOM is found, + ``'utf-8-sig'`` will be returned as an encoding. + + If no encoding is specified, then the default of ``'utf-8'`` will be + returned. + + Use :func:`open` to open Python source files: it uses + :func:`detect_encoding` to detect the file encoding. + - If no encoding is specified, then the default of ``'utf-8'`` will be returned. +.. function:: open(filename) + Open a file in read only mode using the encoding detected by + :func:`detect_encoding`. -Example of a script re-writer that transforms float literals into Decimal + .. versionadded:: 3.2 + + +Example of a script rewriter that transforms float literals into Decimal objects:: from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP @@ -142,4 +158,3 @@ objects:: result.append((toknum, tokval)) return untokenize(result).decode('utf-8') - diff --git a/Doc/library/trace.rst b/Doc/library/trace.rst index 7b49c8f..c4ddc56 100644 --- a/Doc/library/trace.rst +++ b/Doc/library/trace.rst @@ -4,13 +4,15 @@ .. module:: trace :synopsis: Trace or track Python statement execution. +**Source code:** :source:`Lib/trace.py` + +-------------- The :mod:`trace` module allows you to trace program execution, generate annotated statement coverage listings, print caller/callee relationships and list functions executed during a program run. It can be used in another program or from the command line. - .. _trace-cli: Command-Line Usage diff --git a/Doc/library/turtle-star.pdf b/Doc/library/turtle-star.pdf Binary files differnew file mode 100644 index 0000000..e354073 --- /dev/null +++ b/Doc/library/turtle-star.pdf diff --git a/Doc/library/turtle-star.png b/Doc/library/turtle-star.png Binary files differnew file mode 100644 index 0000000..caf36a3 --- /dev/null +++ b/Doc/library/turtle-star.png diff --git a/Doc/library/turtle-star.ps b/Doc/library/turtle-star.ps new file mode 100644 index 0000000..46362cb --- /dev/null +++ b/Doc/library/turtle-star.ps @@ -0,0 +1,447 @@ +%!PS-Adobe-3.0 EPSF-3.0 +%%Creator: Tk Canvas Widget +%%For: Alexander Belopolsky +%%Title: Window .4315905424 +%%CreationDate: Tue Nov 9 12:54:06 2010 +%%XBoundingBox: -172 -52 785 845 +%%BoundingBox: 290 290 520 520 +%%Pages: 1 +%%DocumentData: Clean7Bit +%%Orientation: Portrait +%%EndComments + +%%BeginProlog +/CurrentEncoding [ +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/exclam/quotedbl/numbersign/dollar/percent/ampersand/quotesingle +/parenleft/parenright/asterisk/plus/comma/hyphen/period/slash +/zero/one/two/three/four/five/six/seven +/eight/nine/colon/semicolon/less/equal/greater/question +/at/A/B/C/D/E/F/G +/H/I/J/K/L/M/N/O +/P/Q/R/S/T/U/V/W +/X/Y/Z/bracketleft/backslash/bracketright/asciicircum/underscore +/grave/a/b/c/d/e/f/g +/h/i/j/k/l/m/n/o +/p/q/r/s/t/u/v/w +/x/y/z/braceleft/bar/braceright/asciitilde/space +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/space/space/space/space/space/space/space +/space/exclamdown/cent/sterling/currency/yen/brokenbar/section +/dieresis/copyright/ordfeminine/guillemotleft/logicalnot/hyphen/registered/macron +/degree/plusminus/twosuperior/threesuperior/acute/mu/paragraph/periodcentered +/cedilla/onesuperior/ordmasculine/guillemotright/onequarter/onehalf/threequarters/questiondown +/Agrave/Aacute/Acircumflex/Atilde/Adieresis/Aring/AE/Ccedilla +/Egrave/Eacute/Ecircumflex/Edieresis/Igrave/Iacute/Icircumflex/Idieresis +/Eth/Ntilde/Ograve/Oacute/Ocircumflex/Otilde/Odieresis/multiply +/Oslash/Ugrave/Uacute/Ucircumflex/Udieresis/Yacute/Thorn/germandbls +/agrave/aacute/acircumflex/atilde/adieresis/aring/ae/ccedilla +/egrave/eacute/ecircumflex/edieresis/igrave/iacute/icircumflex/idieresis +/eth/ntilde/ograve/oacute/ocircumflex/otilde/odieresis/divide +/oslash/ugrave/uacute/ucircumflex/udieresis/yacute/thorn/ydieresis +] def + +50 dict begin +% This is a standard prolog for Postscript generated by Tk's canvas +% widget. +% RCS: @(#) $Id$ + +% The definitions below just define all of the variables used in +% any of the procedures here. This is needed for obscure reasons +% explained on p. 716 of the Postscript manual (Section H.2.7, +% "Initializing Variables," in the section on Encapsulated Postscript). + +/baseline 0 def +/stipimage 0 def +/height 0 def +/justify 0 def +/lineLength 0 def +/spacing 0 def +/stipple 0 def +/strings 0 def +/xoffset 0 def +/yoffset 0 def +/tmpstip null def + + +/cstringshow { + { + dup type /stringtype eq + { show } { glyphshow } + ifelse + } + forall +} bind def + + + +/cstringwidth { + 0 exch 0 exch + { + dup type /stringtype eq + { stringwidth } { + currentfont /Encoding get exch 1 exch put (\001) stringwidth + } + ifelse + exch 3 1 roll add 3 1 roll add exch + } + forall +} bind def + +% font ISOEncode font +% This procedure changes the encoding of a font from the default +% Postscript encoding to current system encoding. It's typically invoked just +% before invoking "setfont". The body of this procedure comes from +% Section 5.6.1 of the Postscript book. + +/ISOEncode { + dup length dict begin + {1 index /FID ne {def} {pop pop} ifelse} forall + /Encoding CurrentEncoding def + currentdict + end + + % I'm not sure why it's necessary to use "definefont" on this new + % font, but it seems to be important; just use the name "Temporary" + % for the font. + + /Temporary exch definefont +} bind def + +% StrokeClip +% +% This procedure converts the current path into a clip area under +% the assumption of stroking. It's a bit tricky because some Postscript +% interpreters get errors during strokepath for dashed lines. If +% this happens then turn off dashes and try again. + +/StrokeClip { + {strokepath} stopped { + (This Postscript printer gets limitcheck overflows when) = + (stippling dashed lines; lines will be printed solid instead.) = + [] 0 setdash strokepath} if + clip +} bind def + +% desiredSize EvenPixels closestSize +% +% The procedure below is used for stippling. Given the optimal size +% of a dot in a stipple pattern in the current user coordinate system, +% compute the closest size that is an exact multiple of the device's +% pixel size. This allows stipple patterns to be displayed without +% aliasing effects. + +/EvenPixels { + % Compute exact number of device pixels per stipple dot. + dup 0 matrix currentmatrix dtransform + dup mul exch dup mul add sqrt + + % Round to an integer, make sure the number is at least 1, and compute + % user coord distance corresponding to this. + dup round dup 1 lt {pop 1} if + exch div mul +} bind def + +% width height string StippleFill -- +% +% Given a path already set up and a clipping region generated from +% it, this procedure will fill the clipping region with a stipple +% pattern. "String" contains a proper image description of the +% stipple pattern and "width" and "height" give its dimensions. Each +% stipple dot is assumed to be about one unit across in the current +% user coordinate system. This procedure trashes the graphics state. + +/StippleFill { + % The following code is needed to work around a NeWSprint bug. + + /tmpstip 1 index def + + % Change the scaling so that one user unit in user coordinates + % corresponds to the size of one stipple dot. + 1 EvenPixels dup scale + + % Compute the bounding box occupied by the path (which is now + % the clipping region), and round the lower coordinates down + % to the nearest starting point for the stipple pattern. Be + % careful about negative numbers, since the rounding works + % differently on them. + + pathbbox + 4 2 roll + 5 index div dup 0 lt {1 sub} if cvi 5 index mul 4 1 roll + 6 index div dup 0 lt {1 sub} if cvi 6 index mul 3 2 roll + + % Stack now: width height string y1 y2 x1 x2 + % Below is a doubly-nested for loop to iterate across this area + % in units of the stipple pattern size, going up columns then + % across rows, blasting out a stipple-pattern-sized rectangle at + % each position + + 6 index exch { + 2 index 5 index 3 index { + % Stack now: width height string y1 y2 x y + + gsave + 1 index exch translate + 5 index 5 index true matrix tmpstip imagemask + grestore + } for + pop + } for + pop pop pop pop pop +} bind def + +% -- AdjustColor -- +% Given a color value already set for output by the caller, adjusts +% that value to a grayscale or mono value if requested by the CL +% variable. + +/AdjustColor { + CL 2 lt { + currentgray + CL 0 eq { + .5 lt {0} {1} ifelse + } if + setgray + } if +} bind def + +% x y strings spacing xoffset yoffset justify stipple DrawText -- +% This procedure does all of the real work of drawing text. The +% color and font must already have been set by the caller, and the +% following arguments must be on the stack: +% +% x, y - Coordinates at which to draw text. +% strings - An array of strings, one for each line of the text item, +% in order from top to bottom. +% spacing - Spacing between lines. +% xoffset - Horizontal offset for text bbox relative to x and y: 0 for +% nw/w/sw anchor, -0.5 for n/center/s, and -1.0 for ne/e/se. +% yoffset - Vertical offset for text bbox relative to x and y: 0 for +% nw/n/ne anchor, +0.5 for w/center/e, and +1.0 for sw/s/se. +% justify - 0 for left justification, 0.5 for center, 1 for right justify. +% stipple - Boolean value indicating whether or not text is to be +% drawn in stippled fashion. If text is stippled, +% procedure StippleText must have been defined to call +% StippleFill in the right way. +% +% Also, when this procedure is invoked, the color and font must already +% have been set for the text. + +/DrawText { + /stipple exch def + /justify exch def + /yoffset exch def + /xoffset exch def + /spacing exch def + /strings exch def + + % First scan through all of the text to find the widest line. + + /lineLength 0 def + strings { + cstringwidth pop + dup lineLength gt {/lineLength exch def} {pop} ifelse + newpath + } forall + + % Compute the baseline offset and the actual font height. + + 0 0 moveto (TXygqPZ) false charpath + pathbbox dup /baseline exch def + exch pop exch sub /height exch def pop + newpath + + % Translate coordinates first so that the origin is at the upper-left + % corner of the text's bounding box. Remember that x and y for + % positioning are still on the stack. + + translate + lineLength xoffset mul + strings length 1 sub spacing mul height add yoffset mul translate + + % Now use the baseline and justification information to translate so + % that the origin is at the baseline and positioning point for the + % first line of text. + + justify lineLength mul baseline neg translate + + % Iterate over each of the lines to output it. For each line, + % compute its width again so it can be properly justified, then + % display it. + + strings { + dup cstringwidth pop + justify neg mul 0 moveto + stipple { + + + % The text is stippled, so turn it into a path and print + % by calling StippledText, which in turn calls StippleFill. + % Unfortunately, many Postscript interpreters will get + % overflow errors if we try to do the whole string at + % once, so do it a character at a time. + + gsave + /char (X) def + { + dup type /stringtype eq { + % This segment is a string. + { + char 0 3 -1 roll put + currentpoint + gsave + char true charpath clip StippleText + grestore + char stringwidth translate + moveto + } forall + } { + % This segment is glyph name + % Temporary override + currentfont /Encoding get exch 1 exch put + currentpoint + gsave (\001) true charpath clip StippleText + grestore + (\001) stringwidth translate + moveto + } ifelse + } forall + grestore + } {cstringshow} ifelse + 0 spacing neg translate + } forall +} bind def + +%%EndProlog +%%BeginSetup +/CL 2 def +%%EndSetup + +%%Page: 1 1 +save +306.0 396.0 translate +0.9995 0.9995 scale +4 -449 translate +-483 898 moveto 475 898 lineto 475 0 lineto -483 0 lineto closepath clip newpath +gsave +grestore +gsave +0 445 moveto +200 445 lineto +3.03844939755837 479.729635533386 lineto +190.97697355474 411.325606868252 lineto +17.7718927978523 511.325606868252 lineto +170.980781421648 382.768084930944 lineto +42.42325948434 535.97697355474 lineto +142.42325948434 362.771892797852 lineto +74.0192308192062 550.710416955034 lineto +108.748866352592 353.748866352592 lineto +108.748866352592 553.748866352592 lineto +74.0192308192064 356.787315750151 lineto +142.42325948434 544.725839907333 lineto +42.4232594843401 371.520759150445 lineto +170.980781421648 524.72964777424 lineto +17.7718927978524 396.172125836932 lineto +190.97697355474 496.172125836933 lineto +3.03844939755834 427.768097171799 lineto +200 462.497732705185 lineto +-1.13686837721616e-13 462.497732705185 lineto +196.961550602442 427.768097171799 lineto +9.02302644525972 496.172125836932 lineto +182.228107202148 396.172125836933 lineto +29.0192185783518 524.72964777424 lineto +157.57674051566 371.520759150445 lineto +57.5767405156596 544.725839907332 lineto +125.980769180794 356.787315750151 lineto +91.2511336474073 553.748866352592 lineto +91.2511336474079 353.748866352592 lineto +125.980769180793 550.710416955034 lineto +57.5767405156601 362.771892797852 lineto +157.57674051566 535.97697355474 lineto +29.0192185783522 382.768084930944 lineto +182.228107202148 511.325606868253 lineto +9.02302644525994 411.325606868252 lineto +196.961550602442 479.729635533386 lineto +-1.70530256582424e-13 445 lineto +0 445 lineto +1.000 1.000 0.000 setrgbcolor AdjustColor +eofill +grestore +gsave +0 445 moveto +200 445 lineto +3.03844939755837 479.729635533386 lineto +190.97697355474 411.325606868252 lineto +17.7718927978523 511.325606868252 lineto +170.980781421648 382.768084930944 lineto +42.42325948434 535.97697355474 lineto +142.42325948434 362.771892797852 lineto +74.0192308192062 550.710416955034 lineto +108.748866352592 353.748866352592 lineto +108.748866352592 553.748866352592 lineto +74.0192308192064 356.787315750151 lineto +142.42325948434 544.725839907333 lineto +42.4232594843401 371.520759150445 lineto +170.980781421648 524.72964777424 lineto +17.7718927978524 396.172125836932 lineto +190.97697355474 496.172125836933 lineto +3.03844939755834 427.768097171799 lineto +200 462.497732705185 lineto +-1.13686837721616e-13 462.497732705185 lineto +196.961550602442 427.768097171799 lineto +9.02302644525972 496.172125836932 lineto +182.228107202148 396.172125836933 lineto +29.0192185783518 524.72964777424 lineto +157.57674051566 371.520759150445 lineto +57.5767405156596 544.725839907332 lineto +125.980769180794 356.787315750151 lineto +91.2511336474073 553.748866352592 lineto +91.2511336474079 353.748866352592 lineto +125.980769180793 550.710416955034 lineto +57.5767405156601 362.771892797852 lineto +157.57674051566 535.97697355474 lineto +29.0192185783522 382.768084930944 lineto +182.228107202148 511.325606868253 lineto +9.02302644525994 411.325606868252 lineto +196.961550602442 479.729635533386 lineto +-1.70530256582424e-13 445 lineto +1 setlinecap +1 setlinejoin +1 setlinewidth +[] 0 setdash +1.000 0.000 0.000 setrgbcolor AdjustColor +stroke +grestore +gsave +grestore +gsave +-1.70530256582424e-13 445 moveto +-9.00000000000019 450 lineto +-7.00000000000017 445 lineto +-9.00000000000015 440 lineto +-1.70530256582424e-13 445 lineto +1.000 1.000 0.000 setrgbcolor AdjustColor +eofill +-1.70530256582424e-13 445 moveto +-9.00000000000019 450 lineto +-7.00000000000017 445 lineto +-9.00000000000015 440 lineto +-1.70530256582424e-13 445 lineto +1 setlinejoin 1 setlinecap +1 setlinewidth +[] 0 setdash +1.000 0.000 0.000 setrgbcolor AdjustColor +stroke +grestore +restore showpage + +%%Trailer +end +%%EOF + diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index 1fe9699..e995a7c 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -23,6 +23,16 @@ command ``turtle.forward(15)``, and it moves (on-screen!) 15 pixels in the direction it is facing, drawing a line as it moves. Give it the command ``turtle.left(25)``, and it rotates in-place 25 degrees clockwise. +.. sidebar:: Turtle star + + Turtle can draw intricate shapes using programs that repeat simple + moves. + + .. image:: turtle-star.* + :align: center + + .. literalinclude:: ../includes/turtle-star.py + By combining together these and similar commands, intricate shapes and pictures can easily be drawn. @@ -1868,7 +1878,7 @@ Settings and special methods >>> cv = screen.getcanvas() >>> cv - <turtle.ScrolledCanvas instance at 0x...> + <turtle.ScrolledCanvas object at ...> .. function:: getshapes() @@ -2258,7 +2268,7 @@ There can be a :file:`turtle.cfg` file in the directory where :mod:`turtle` is stored and an additional one in the current working directory. The latter will override the settings of the first one. -The :file:`Demo/turtle` directory contains a :file:`turtle.cfg` file. You can +The :file:`Lib/turtledemo` directory contains a :file:`turtle.cfg` file. You can study it as an example and see its effects when running the demos (preferably not from within the demo-viewer). @@ -2266,29 +2276,35 @@ not from within the demo-viewer). Demo scripts ============ -There is a set of demo scripts in the turtledemo directory located in the -:file:`Demo/turtle` directory in the source distribution. +There is a set of demo scripts in the :mod:`turtledemo` package. These +scripts can be run and viewed using the supplied demo viewer as follows:: + + python -m turtledemo + +Alternatively, you can run the demo scripts individually. For example, :: + + python -m turtledemo.bytedesign -It contains: +The :mod:`turtledemo` package directory contains: - a set of 15 demo scripts demonstrating different features of the new module - :mod:`turtle` -- a demo viewer :file:`turtleDemo.py` which can be used to view the sourcecode + :mod:`turtle`; +- a demo viewer :file:`__main__.py` which can be used to view the sourcecode of the scripts and run them at the same time. 14 of the examples can be accessed via the Examples menu; all of them can also be run standalone. -- The example :file:`turtledemo_two_canvases.py` demonstrates the simultaneous +- The example :mod:`turtledemo.two_canvases` demonstrates the simultaneous use of two canvases with the turtle module. Therefore it only can be run standalone. -- There is a :file:`turtle.cfg` file in this directory, which also serves as an +- There is a :file:`turtle.cfg` file in this directory, which serves as an example for how to write and use such files. -The demoscripts are: +The demo scripts are: +----------------+------------------------------+-----------------------+ | Name | Description | Features | +----------------+------------------------------+-----------------------+ | bytedesign | complex classical | :func:`tracer`, delay,| -| | turtlegraphics pattern | :func:`update` | +| | turtle graphics pattern | :func:`update` | +----------------+------------------------------+-----------------------+ | chaos | graphs Verhulst dynamics, | world coordinates | | | shows that computer's | | @@ -2386,8 +2402,7 @@ Changes since Python 3.0 strings and numbers respectively. - Two example scripts :file:`tdemo_nim.py` and :file:`tdemo_round_dance.py` - have been added to the Demo directory (source distribution only). As usual - they can be viewed and executed within the demo viewer :file:`turtleDemo.py`. + have been added to the :file:`Lib/turtledemo` directory. .. doctest:: diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 7caecaf..d4a76b6 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -4,6 +4,9 @@ .. module:: types :synopsis: Names for built-in types. +**Source code:** :source:`Lib/types.py` + +-------------- This module defines names for some object types that are used by the standard Python interpreter, but not exposed as builtins like :class:`int` or diff --git a/Doc/library/unicodedata.rst b/Doc/library/unicodedata.rst index e1e6dc1..bcb3da3 100644 --- a/Doc/library/unicodedata.rst +++ b/Doc/library/unicodedata.rst @@ -13,14 +13,15 @@ single: character pair: Unicode; database -This module provides access to the Unicode Character Database which defines -character properties for all Unicode characters. The data in this database is -based on the :file:`UnicodeData.txt` file version 5.1.0 which is publicly -available from ftp://ftp.unicode.org/. - -The module uses the same names and symbols as defined by the UnicodeData File -Format 5.1.0 (see http://www.unicode.org/Public/5.1.0/ucd/UCD.html). It defines -the following functions: +This module provides access to the Unicode Character Database (UCD) which +defines character properties for all Unicode characters. The data contained in +this database is compiled from the `UCD version 6.0.0 +<http://www.unicode.org/Public/6.0.0/ucd>`_. + +The module uses the same names and symbols as defined by Unicode +Standard Annex #44, `"Unicode Character Database" +<http://www.unicode.org/reports/tr44/tr44-6.html>`_. It defines the +following functions: .. function:: lookup(name) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 01a036d..beed4de 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -97,6 +97,13 @@ need to derive from a specific class. A special-interest-group for discussion of testing, and testing tools, in Python. + The script :file:`Tools/unittestgui/unittestgui.py` in the Python source distribution is + a GUI tool for test discovery and execution. This is intended largely for ease of use + for those new to unit testing. For production environments it is recommended that + tests be driven by a continuous integration system such as `Hudson <http://hudson-ci.org/>`_ + or `Buildbot <http://buildbot.net/trac>`_. + + .. _unittest-minimal-example: Basic example @@ -204,14 +211,141 @@ modules, classes or even individual test methods:: You can pass in a list with any combination of module names, and fully qualified class or method names. +Test modules can be specified by file path as well:: + + python -m unittest tests/test_something.py + +This allows you to use the shell filename completion to specify the test module. +The file specified must still be importable as a module. The path is converted +to a module name by removing the '.py' and converting path separators into '.'. +If you want to execute a test file that isn't importable as a module you should +execute the file directly instead. + You can run tests with more detail (higher verbosity) by passing in the -v flag:: python -m unittest -v test_module +When executed without arguments :ref:`unittest-test-discovery` is started:: + + python -m unittest + For a list of all the command-line options:: python -m unittest -h +.. versionchanged:: 3.2 + In earlier versions it was only possible to run individual test methods and + not modules or classes. + + +Command-line options +~~~~~~~~~~~~~~~~~~~~ + +:program:`unittest` supports these command-line options: + +.. program:: unittest + +.. cmdoption:: -b, --buffer + + The standard output and standard error streams are buffered during the test + run. Output during a passing test is discarded. Output is echoed normally + on test fail or error and is added to the failure messages. + +.. cmdoption:: -c, --catch + + Control-C during the test run waits for the current test to end and then + reports all the results so far. A second control-C raises the normal + :exc:`KeyboardInterrupt` exception. + + See `Signal Handling`_ for the functions that provide this functionality. + +.. cmdoption:: -f, --failfast + + Stop the test run on the first error or failure. + +.. versionadded:: 3.2 + The command-line options ``-b``, ``-c`` and ``-f`` were added. + +The command line can also be used for test discovery, for running all of the +tests in a project or just a subset. + + +.. _unittest-test-discovery: + +Test Discovery +-------------- + +.. versionadded:: 3.2 + +Unittest supports simple test discovery. In order to be compatible with test +discovery, all of the test files must be :ref:`modules <tut-modules>` or +:ref:`packages <tut-packages>` importable from the top-level directory of +the project (this means that their filenames must be valid +:ref:`identifiers <identifiers>`). + +Test discovery is implemented in :meth:`TestLoader.discover`, but can also be +used from the command line. The basic command-line usage is:: + + cd project_directory + python -m unittest discover + +.. note:: + + As a shortcut, ``python -m unittest`` is the equivalent of + ``python -m unittest discover``. If you want to pass arguments to test + discovery the `discover` sub-command must be used explicitly. + +The ``discover`` sub-command has the following options: + +.. program:: unittest discover + +.. cmdoption:: -v, --verbose + + Verbose output + +.. cmdoption:: -s directory + + Directory to start discovery ('.' default) + +.. cmdoption:: -p pattern + + Pattern to match test files ('test*.py' default) + +.. cmdoption:: -t directory + + Top level directory of project (defaults to start directory) + +The :option:`-s`, :option:`-p`, and :option:`-t` options can be passed in +as positional arguments in that order. The following two command lines +are equivalent:: + + python -m unittest discover -s project_directory -p '*_test.py' + python -m unittest discover project_directory '*_test.py' + +As well as being a path it is possible to pass a package name, for example +``myproject.subpackage.test``, as the start directory. The package name you +supply will then be imported and its location on the filesystem will be used +as the start directory. + +.. caution:: + + Test discovery loads tests by importing them. Once test discovery has found + all the test files from the start directory you specify it turns the paths + into package names to import. For example :file:`foo/bar/baz.py` will be + imported as ``foo.bar.baz``. + + If you have a package installed globally and attempt test discovery on + a different copy of the package then the import *could* happen from the + wrong place. If this happens test discovery will warn you and exit. + + If you supply the start directory as a package name rather than a + path to a directory then discover assumes that whichever location it + imports from is the location you intended, so you will not get the + warning. + +Test modules and packages can customize test loading and discovery by through +the `load_tests protocol`_. + .. _organizing-tests: @@ -220,9 +354,9 @@ Organizing test code The basic building blocks of unit testing are :dfn:`test cases` --- single scenarios that must be set up and checked for correctness. In :mod:`unittest`, -test cases are represented by instances of :mod:`unittest`'s :class:`TestCase` -class. To make your own test cases you must write subclasses of -:class:`TestCase`, or use :class:`FunctionTestCase`. +test cases are represented by :class:`unittest.TestCase` instances. +To make your own test cases you must write subclasses of +:class:`TestCase` or use :class:`FunctionTestCase`. An instance of a :class:`TestCase`\ -derived class is an object that can completely run a single test method, together with optional set-up and tidy-up @@ -532,24 +666,27 @@ the test unless the passed object has a certain attribute: :: The following decorators implement test skipping and expected failures: -.. function:: skip(reason) +.. decorator:: skip(reason) Unconditionally skip the decorated test. *reason* should describe why the test is being skipped. -.. function:: skipIf(condition, reason) +.. decorator:: skipIf(condition, reason) Skip the decorated test if *condition* is true. -.. function:: skipUnless(condition, reason) +.. decorator:: skipUnless(condition, reason) Skip the decorated test unless *condition* is true. -.. function:: expectedFailure +.. decorator:: expectedFailure Mark the test as an expected failure. If the test fails when run, the test is not counted as a failure. +Skipped tests will not have :meth:`setUp` or :meth:`tearDown` run around them. +Skipped classes will not have :meth:`setUpClass` or :meth:`tearDownClass` run. + .. _unittest-contents: @@ -586,6 +723,11 @@ Test cases Here, we create two instances of :class:`WidgetTestCase`, each of which runs a single test. + .. versionchanged:: + `TestCase` can be instantiated successfully without providing a method + name. This makes it easier to experiment with `TestCase` from the + interactive interpreter. + *methodName* defaults to :meth:`runTest`. :class:`TestCase` instances provide three groups of methods: one group used @@ -615,6 +757,36 @@ Test cases the outcome of the test method. The default implementation does nothing. + .. method:: setUpClass() + + A class method called before tests in an individual class run. + ``setUpClass`` is called with the class as the only argument + and must be decorated as a :func:`classmethod`:: + + @classmethod + def setUpClass(cls): + ... + + See `Class and Module Fixtures`_ for more details. + + .. versionadded:: 3.2 + + + .. method:: tearDownClass() + + A class method called after tests in an individual class have run. + ``tearDownClass`` is called with the class as the only argument + and must be decorated as a :meth:`classmethod`:: + + @classmethod + def tearDownClass(cls): + ... + + See `Class and Module Fixtures`_ for more details. + + .. versionadded:: 3.2 + + .. method:: run(result=None) Run the test, collecting the result into the test result object passed as @@ -678,16 +850,22 @@ Test cases | :meth:`assertNotIn(a, b) | ``a not in b`` | 3.1 | | <TestCase.assertNotIn>` | | | +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertIsInstance(a, b) | ``isinstance(a, b)`` | 3.2 | + | <TestCase.assertIsInstance>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertNotIsInstance(a, b) | ``not isinstance(a, b)`` | 3.2 | + | <TestCase.assertNotIsInstance>` | | | + +-----------------------------------------+-----------------------------+---------------+ All the assert methods (except :meth:`assertRaises`, - :meth:`assertRaisesRegexp`, :meth:`assertWarns`, :meth:`assertWarnsRegexp`) + :meth:`assertRaisesRegex`, :meth:`assertWarns`, :meth:`assertWarnsRegex`) accept a *msg* argument that, if specified, is used as the error message on failure (see also :data:`longMessage`). .. method:: assertEqual(first, second, msg=None) - Test that *first* and *second* are equal. If the values do not compare - equal, the test will fail. + Test that *first* and *second* are equal. If the values do not + compare equal, the test will fail. In addition, if *first* and *second* are the exact same type and one of list, tuple, dict, set, frozenset or str or any type that a subclass @@ -699,11 +877,15 @@ Test cases .. versionchanged:: 3.1 Added the automatic calling of type specific equality function. + .. versionchanged:: 3.2 + :meth:`assertMultiLineEqual` added as the default type equality + function for comparing strings. + .. method:: assertNotEqual(first, second, msg=None) - Test that *first* and *second* are not equal. If the values do compare - equal, the test will fail. + Test that *first* and *second* are not equal. If the values do + compare equal, the test will fail. .. method:: assertTrue(expr, msg=None) assertFalse(expr, msg=None) @@ -720,7 +902,8 @@ Test cases .. method:: assertIs(first, second, msg=None) assertIsNot(first, second, msg=None) - Test that *first* and *second* evaluate (or don't evaluate) to the same object. + Test that *first* and *second* evaluate (or don't evaluate) to the + same object. .. versionadded:: 3.1 @@ -741,6 +924,15 @@ Test cases .. versionadded:: 3.1 + .. method:: assertIsInstance(obj, cls, msg=None) + assertNotIsInstance(obj, cls, msg=None) + + Test that *obj* is (or is not) an instance of *cls* (which can be a + class or a tuple of classes, as supported by :func:`isinstance`). + + .. versionadded:: 3.2 + + It is also possible to check that exceptions and warnings are raised using the following methods: @@ -751,8 +943,14 @@ Test cases | :meth:`assertRaises(exc, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises `exc` | | | <TestCase.assertRaises>` | | | +---------------------------------------------------------+--------------------------------------+------------+ - | :meth:`assertRaisesRegexp(exc, re, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises `exc` | 3.1 | - | <TestCase.assertRaisesRegexp>` | and the message matches `re` | | + | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises `exc` | 3.1 | + | <TestCase.assertRaisesRegex>` | and the message matches `re` | | + +---------------------------------------------------------+--------------------------------------+------------+ + | :meth:`assertWarns(warn, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises `warn` | 3.2 | + | <TestCase.assertWarns>` | | | + +---------------------------------------------------------+--------------------------------------+------------+ + | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises `warn` | 3.2 | + | <TestCase.assertWarnsRegex>` | and the message matches `re` | | +---------------------------------------------------------+--------------------------------------+------------+ .. method:: assertRaises(exception, callable, *args, **kwds) @@ -771,27 +969,97 @@ Test cases with self.assertRaises(SomeException): do_something() + The context manager will store the caught exception object in its + :attr:`exception` attribute. This can be useful if the intention + is to perform additional checks on the exception raised:: + + with self.assertRaises(SomeException) as cm: + do_something() + + the_exception = cm.exception + self.assertEqual(the_exception.error_code, 3) + .. versionchanged:: 3.1 Added the ability to use :meth:`assertRaises` as a context manager. + .. versionchanged:: 3.2 + Added the :attr:`exception` attribute. - .. method:: assertRaisesRegexp(exception, regexp, callable, *args, **kwds) - assertRaisesRegexp(exception, regexp) - Like :meth:`assertRaises` but also tests that *regexp* matches - on the string representation of the raised exception. *regexp* may be + .. method:: assertRaisesRegex(exception, regex, callable, *args, **kwds) + assertRaisesRegex(exception, regex) + + Like :meth:`assertRaises` but also tests that *regex* matches + on the string representation of the raised exception. *regex* may be a regular expression object or a string containing a regular expression suitable for use by :func:`re.search`. Examples:: - self.assertRaisesRegexp(ValueError, 'invalid literal for.*XYZ$', - int, 'XYZ') + self.assertRaisesRegex(ValueError, 'invalid literal for.*XYZ$', + int, 'XYZ') or:: - with self.assertRaisesRegexp(ValueError, 'literal'): + with self.assertRaisesRegex(ValueError, 'literal'): int('XYZ') .. versionadded:: 3.1 + under the name ``assertRaisesRegexp``. + .. versionchanged:: 3.2 + Renamed to :meth:`assertRaisesRegex`. + + + .. method:: assertWarns(warning, callable, *args, **kwds) + assertWarns(warning) + + Test that a warning is triggered when *callable* is called with any + positional or keyword arguments that are also passed to + :meth:`assertWarns`. The test passes if *warning* is triggered and + fails if it isn't. Also, any unexpected exception is an error. + To catch any of a group of warnings, a tuple containing the warning + classes may be passed as *warnings*. + + If only the *warning* argument is given, returns a context manager so + that the code under test can be written inline rather than as a function:: + + with self.assertWarns(SomeWarning): + do_something() + + The context manager will store the caught warning object in its + :attr:`warning` attribute, and the source line which triggered the + warnings in the :attr:`filename` and :attr:`lineno` attributes. + This can be useful if the intention is to perform additional checks + on the exception raised:: + + with self.assertWarns(SomeWarning) as cm: + do_something() + + self.assertIn('myfile.py', cm.filename) + self.assertEqual(320, cm.lineno) + + This method works regardless of the warning filters in place when it + is called. + + .. versionadded:: 3.2 + + + .. method:: assertWarnsRegex(warning, regex, callable, *args, **kwds) + assertWarnsRegex(warning, regex) + + Like :meth:`assertWarns` but also tests that *regex* matches on the + message of the triggered warning. *regex* may be a regular expression + object or a string containing a regular expression suitable for use + by :func:`re.search`. Example:: + + self.assertWarnsRegex(DeprecationWarning, + r'legacy_function\(\) is deprecated', + legacy_function, 'XYZ') + + or:: + + with self.assertWarnsRegex(RuntimeWarning, 'unsafe frobnicating'): + frobnicate('/etc/passwd') + + .. versionadded:: 3.2 @@ -818,11 +1086,15 @@ Test cases | :meth:`assertLessEqual(a, b) | ``a <= b`` | 3.1 | | <TestCase.assertLessEqual>` | | | +---------------------------------------+--------------------------------+--------------+ - | :meth:`assertRegexpMatches(s, re) | ``regex.search(s)`` | 3.1 | - | <TestCase.assertRegexpMatches>` | | | + | :meth:`assertRegex(s, re) | ``regex.search(s)`` | 3.1 | + | <TestCase.assertRegex>` | | | + +---------------------------------------+--------------------------------+--------------+ + | :meth:`assertNotRegex(s, re) | ``not regex.search(s)`` | 3.2 | + | <TestCase.assertNotRegex>` | | | +---------------------------------------+--------------------------------+--------------+ - | :meth:`assertDictContainsSubset(a, b) | all the key/value pairs | 3.1 | - | <TestCase.assertDictContainsSubset>` | in `a` exist in `b` | | + | :meth:`assertCountEqual(a, b) | `a` and `b` have the same | 3.2 | + | <TestCase.assertCountEqual>` | elements in the same number, | | + | | regardless of their order | | +---------------------------------------+--------------------------------+--------------+ @@ -840,6 +1112,11 @@ Test cases Supplying both *delta* and *places* raises a ``TypeError``. + .. versionchanged:: 3.2 + :meth:`assertAlmostEqual` automatically considers almost equal objects + that compare equal. :meth:`assertNotAlmostEqual` automatically fails + if the objects compare equal. Added the *delta* keyword argument. + .. method:: assertGreater(first, second, msg=None) assertGreaterEqual(first, second, msg=None) @@ -855,38 +1132,64 @@ Test cases .. versionadded:: 3.1 - .. method:: assertRegexpMatches(text, regexp, msg=None) + .. method:: assertRegex(text, regex, msg=None) + assertNotRegex(text, regex, msg=None) - Test that a *regexp* search matches *text*. In case + Test that a *regex* search matches (or does not match) *text*. In case of failure, the error message will include the pattern and the *text* (or - the pattern and the part of *text* that unexpectedly matched). *regexp* + the pattern and the part of *text* that unexpectedly matched). *regex* may be a regular expression object or a string containing a regular expression suitable for use by :func:`re.search`. - .. versionadded:: 3.1 :meth:`~TestCase.assertRegexpMatches` + .. versionadded:: 3.1 + under the name ``assertRegexpMatches``. + .. versionchanged:: 3.2 + The method ``assertRegexpMatches()`` has been renamed to + :meth:`.assertRegex`. + .. versionadded:: 3.2 + :meth:`.assertNotRegex`. + + .. method:: assertDictContainsSubset(subset, dictionary, msg=None) - .. method:: assertDictContainsSubset(expected, actual, msg=None) + Tests whether the key/value pairs in *dictionary* are a superset of + those in *subset*. If not, an error message listing the missing keys + and mismatched values is generated. - Tests whether the key/value pairs in dictionary *actual* are a - superset of those in *expected*. If not, an error message listing - the missing keys and mismatched values is generated. + Note, the arguments are in the opposite order of what the method name + dictates. Instead, consider using the set-methods on :ref:`dictionary + views <dict-views>`, for example: ``d.keys() <= e.keys()`` or + ``d.items() <= d.items()``. .. versionadded:: 3.1 .. deprecated:: 3.2 - .. method:: assertSameElements(actual, expected, msg=None) + .. method:: assertCountEqual(first, second, msg=None) - Test that sequence *expected* contains the same elements as *actual*, + Test that sequence *first* contains the same elements as *second*, + regardless of their order. When they don't, an error message listing the + differences between the sequences will be generated. + + Duplicate elements are *not* ignored when comparing *first* and + *second*. It verifies whether each element has the same count in both + sequences. Equivalent to: + ``assertEqual(Counter(list(first)), Counter(list(second)))`` + but works with sequences of unhashable objects as well. + + .. versionadded:: 3.2 + + .. method:: assertSameElements(first, second, msg=None) + + Test that sequence *first* contains the same elements as *second*, regardless of their order. When they don't, an error message listing the differences between the sequences will be generated. - Duplicate elements are ignored when comparing *actual* and *expected*. - It is the equivalent of ``assertEqual(set(expected), set(actual))`` + Duplicate elements are ignored when comparing *first* and *second*. + It is the equivalent of ``assertEqual(set(first), set(second))`` but it works with sequences of unhashable objects as well. Because duplicates are ignored, this method has been deprecated in favour of - :meth:`assertItemsEqual`. + :meth:`assertCountEqual`. .. versionadded:: 3.1 .. deprecated:: 3.2 @@ -950,10 +1253,10 @@ Test cases .. versionadded:: 3.1 - .. method:: assertSequenceEqual(seq1, seq2, msg=None, seq_type=None) + .. method:: assertSequenceEqual(first, second, msg=None, seq_type=None) Tests that two sequences are equal. If a *seq_type* is supplied, both - *seq1* and *seq2* must be instances of *seq_type* or a failure will + *first* and *second* must be instances of *seq_type* or a failure will be raised. If the sequences are different an error message is constructed that shows the difference between the two. @@ -964,8 +1267,8 @@ Test cases .. versionadded:: 3.1 - .. method:: assertListEqual(list1, list2, msg=None) - assertTupleEqual(tuple1, tuple2, msg=None) + .. method:: assertListEqual(first, second, msg=None) + assertTupleEqual(first, second, msg=None) Tests that two lists or tuples are equal. If not an error message is constructed that shows only the differences between the two. An error @@ -976,19 +1279,19 @@ Test cases .. versionadded:: 3.1 - .. method:: assertSetEqual(set1, set2, msg=None) + .. method:: assertSetEqual(first, second, msg=None) Tests that two sets are equal. If not, an error message is constructed that lists the differences between the sets. This method is used by default when comparing sets or frozensets with :meth:`assertEqual`. - Fails if either of *set1* or *set2* does not have a :meth:`set.difference` + Fails if either of *first* or *second* does not have a :meth:`set.difference` method. .. versionadded:: 3.1 - .. method:: assertDictEqual(expected, actual, msg=None) + .. method:: assertDictEqual(first, second, msg=None) Test that two dictionaries are equal. If not, an error message is constructed that shows the differences in the dictionaries. This @@ -1029,8 +1332,8 @@ Test cases to ``True`` allows you to have a custom error message in addition to the normal one. - This attribute defaults to ``False``, meaning that a custom message passed - to an assert method will silence the normal message. + This attribute defaults to ``True``. If set to False then a custom message + passed to an assert method will silence the normal message. The class setting can be overridden in individual tests by assigning an instance attribute to ``True`` or ``False`` before calling the assert methods. @@ -1038,6 +1341,21 @@ Test cases .. versionadded:: 3.1 + .. attribute:: maxDiff + + This attribute controls the maximum length of diffs output by assert + methods that report diffs on failure. It defaults to 80*8 characters. + Assert methods affected by this attribute are + :meth:`assertSequenceEqual` (including all the sequence comparison + methods that delegate to it), :meth:`assertDictEqual` and + :meth:`assertMultiLineEqual`. + + Setting ``maxDiff`` to None means that there is no maximum length of + diffs. + + .. versionadded:: 3.2 + + Testing frameworks can use the following methods to collect information on the test: @@ -1070,13 +1388,13 @@ Test cases Returns a description of the test, or ``None`` if no description has been provided. The default implementation of this method returns the first line of the test method's docstring, if available, - along with the method name. + or ``None``. .. versionchanged:: 3.1 - In earlier versions this only returned the first line of the test - method's docstring, if available or the :const:`None`. That led to - undesirable behavior of not printing the test name when someone was - thoughtful enough to write a docstring. + In 3.1 this was changed to add the test name to the short description + even in the presence of a docstring. This caused compatibility issues + with unittest extensions and adding the test name was moved to the + :class:`TextTestResult` in Python 3.2. .. method:: addCleanup(function, *args, **kwargs) @@ -1118,6 +1436,8 @@ Test cases :mod:`unittest`-based test framework. +.. _deprecated-aliases: + Deprecated aliases ################## @@ -1125,21 +1445,27 @@ For historical reasons, some of the :class:`TestCase` methods had one or more aliases that are now deprecated. The following table lists the correct names along with their deprecated aliases: - ============================== =============================== - Method Name Deprecated alias(es) - ============================== =============================== - :meth:`.assertEqual` failUnlessEqual, assertEquals - :meth:`.assertNotEqual` failIfEqual - :meth:`.assertTrue` failUnless, assert\_ + ============================== ====================== ====================== + Method Name Deprecated alias Deprecated alias + ============================== ====================== ====================== + :meth:`.assertEqual` failUnlessEqual assertEquals + :meth:`.assertNotEqual` failIfEqual assertNotEquals + :meth:`.assertTrue` failUnless assert\_ :meth:`.assertFalse` failIf :meth:`.assertRaises` failUnlessRaises - :meth:`.assertAlmostEqual` failUnlessAlmostEqual - :meth:`.assertNotAlmostEqual` failIfAlmostEqual - ============================== =============================== + :meth:`.assertAlmostEqual` failUnlessAlmostEqual assertAlmostEquals + :meth:`.assertNotAlmostEqual` failIfAlmostEqual assertNotAlmostEquals + :meth:`.assertRegex` assertRegexpMatches + :meth:`.assertRaisesRegex` assertRaisesRegexp + ============================== ====================== ====================== .. deprecated:: 3.1 - the aliases listed in the second column - + the fail* aliases listed in the second column. + .. deprecated:: 3.2 + the assert* aliases listed in the third column. + .. deprecated:: 3.2 + ``assertRegexpMatches`` and ``assertRaisesRegexp`` have been renamed to + :meth:`.assertRegex` and :meth:`.assertRaisesRegex` .. _testsuite-objects: @@ -1209,6 +1535,11 @@ Grouping tests (for example when counting tests or comparing for equality) so the tests returned must be the same for repeated iterations. + .. versionchanged:: 3.2 + In earlier versions the :class:`TestSuite` accessed tests directly rather + than through iteration, so overriding :meth:`__iter__` wasn't sufficient + for providing tests. + In the typical usage of a :class:`TestSuite` object, the :meth:`run` method is invoked by a :class:`TestRunner` rather than by the end-user test harness. @@ -1248,6 +1579,13 @@ Loading and running tests directly does not play well with this method. Doing so, however, can be useful when the fixtures are different and defined in subclasses. + If a module provides a ``load_tests`` function it will be called to + load the tests. This allows modules to customize test loading. + This is the `load_tests protocol`_. + + .. versionchanged:: 3.2 + Support for ``load_tests`` added. + .. method:: loadTestsFromName(name, module=None) @@ -1287,6 +1625,39 @@ Loading and running tests this should be a subclass of :class:`TestCase`. + .. method:: discover(start_dir, pattern='test*.py', top_level_dir=None) + + Find and return all test modules from the specified start directory, + recursing into subdirectories to find them. Only test files that match + *pattern* will be loaded. (Using shell style pattern matching.) Only + module names that are importable (i.e. are valid Python identifiers) will + be loaded. + + All test modules must be importable from the top level of the project. If + the start directory is not the top level directory then the top level + directory must be specified separately. + + If importing a module fails, for example due to a syntax error, then this + will be recorded as a single error and discovery will continue. + + If a test package name (directory with :file:`__init__.py`) matches the + pattern then the package will be checked for a ``load_tests`` + function. If this exists then it will be called with *loader*, *tests*, + *pattern*. + + If load_tests exists then discovery does *not* recurse into the package, + ``load_tests`` is responsible for loading all tests in the package. + + The pattern is deliberately not stored as a loader attribute so that + packages can continue discovery themselves. *top_level_dir* is stored so + ``load_tests`` does not need to pass this argument in to + ``loader.discover()``. + + *start_dir* can be a dotted module name as well as a directory. + + .. versionadded:: 3.2 + + The following attributes of a :class:`TestLoader` can be configured either by subclassing or assignment on an instance: @@ -1375,6 +1746,24 @@ Loading and running tests The total number of tests run so far. + .. attribute:: buffer + + If set to true, ``sys.stdout`` and ``sys.stderr`` will be buffered in between + :meth:`startTest` and :meth:`stopTest` being called. Collected output will + only be echoed onto the real ``sys.stdout`` and ``sys.stderr`` if the test + fails or errors. Any output is also attached to the failure / error message. + + .. versionadded:: 3.2 + + + .. attribute:: failfast + + If set to true :meth:`stop` will be called on the first failure or error, + halting the test run. + + .. versionadded:: 3.2 + + .. method:: wasSuccessful() Return ``True`` if all tests run so far have passed, otherwise returns @@ -1403,18 +1792,11 @@ Loading and running tests Called when the test case *test* is about to be run. - The default implementation simply increments the instance's :attr:`testsRun` - counter. - - .. method:: stopTest(test) Called after the test case *test* has been executed, regardless of the outcome. - The default implementation does nothing. - - .. method:: startTestRun(test) Called once before any tests are executed. @@ -1485,6 +1867,16 @@ Loading and running tests :attr:`unexpectedSuccesses` attribute. +.. class:: TextTestResult(stream, descriptions, verbosity) + + A concrete implementation of :class:`TestResult` used by the + :class:`TextTestRunner`. + + .. versionadded:: 3.2 + This class was previously named ``_TextTestResult``. The old name still + exists as an alias but is deprecated. + + .. data:: defaultTestLoader Instance of the :class:`TestLoader` class intended to be shared. If no @@ -1492,20 +1884,46 @@ Loading and running tests instead of repeatedly creating new instances. -.. class:: TextTestRunner(stream=sys.stderr, descriptions=True, verbosity=1) +.. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, runnerclass=None, warnings=None) - A basic test runner implementation which prints results on standard error. It + A basic test runner implementation that outputs results to a stream. If *stream* + is `None`, the default, `sys.stderr` is used as the output stream. This class has a few configurable parameters, but is essentially very simple. Graphical applications which run test suites should provide alternate implementations. + By default this runner shows :exc:`DeprecationWarning`, + :exc:`PendingDeprecationWarning`, and :exc:`ImportWarning` even if they are + :ref:`ignored by default <warning-ignored>`. Deprecation warnings caused by + :ref:`deprecated unittest methods <deprecated-aliases>` are also + special-cased and, when the warning filters are ``'default'`` or ``'always'``, + they will appear only once per-module, in order to avoid too many warning + messages. This behavior can be overridden using the :option:`-Wd` or + :option:`-Wa` options and leaving *warnings* to ``None``. + + .. versionchanged:: 3.2 + Added the ``warnings`` argument. + + .. versionchanged:: 3.2 + The default stream is set to `sys.stderr` at instantiation time rather + than import time. + .. method:: _makeResult() This method returns the instance of ``TestResult`` used by :meth:`run`. It is not intended to be called directly, but can be overridden in subclasses to provide a custom ``TestResult``. + ``_makeResult()`` instantiates the class or callable passed in the + ``TextTestRunner`` constructor as the ``resultclass`` argument. It + defaults to :class:`TextTestResult` if no ``resultclass`` is provided. + The result class is instantiated with the following arguments:: -.. function:: main(module='__main__', defaultTest=None, argv=None, testRunner=TextTestRunner, testLoader=unittest.defaultTestLoader, exit=True) + stream, descriptions, verbosity + + +.. function:: main(module='__main__', defaultTest=None, argv=None, testRunner=None, \ + testLoader=unittest.defaultTestLoader, exit=True, verbosity=1, \ + failfast=None, catchbreak=None, buffer=None, warnings=None) A command-line program that runs a set of tests; this is primarily for making test modules conveniently executable. The simplest use for this function is to @@ -1514,6 +1932,11 @@ Loading and running tests if __name__ == '__main__': unittest.main() + You can run tests with more detailed information by passing in the verbosity + argument:: + + if __name__ == '__main__': + unittest.main(verbosity=2) The *testRunner* argument can either be a test runner class or an already created instance of it. By default ``main`` calls :func:`sys.exit` with @@ -1526,8 +1949,228 @@ Loading and running tests >>> from unittest import main >>> main(module='test_module', exit=False) + The ``failfast``, ``catchbreak`` and ``buffer`` parameters have the same + effect as the same-name `command-line options`_. + + The *warning* argument specifies the :ref:`warning filter <warning-filter>` + that should be used while running the tests. If it's not specified, it will + remain ``None`` if a :option:`-W` option is passed to :program:`python`, + otherwise it will be set to ``'default'``. + Calling ``main`` actually returns an instance of the ``TestProgram`` class. This stores the result of the tests run as the ``result`` attribute. .. versionchanged:: 3.1 The ``exit`` parameter was added. + + .. versionchanged:: 3.2 + The ``verbosity``, ``failfast``, ``catchbreak``, ``buffer`` + and ``warnings`` parameters were added. + + +load_tests Protocol +################### + +.. versionadded:: 3.2 + +Modules or packages can customize how tests are loaded from them during normal +test runs or test discovery by implementing a function called ``load_tests``. + +If a test module defines ``load_tests`` it will be called by +:meth:`TestLoader.loadTestsFromModule` with the following arguments:: + + load_tests(loader, standard_tests, None) + +It should return a :class:`TestSuite`. + +*loader* is the instance of :class:`TestLoader` doing the loading. +*standard_tests* are the tests that would be loaded by default from the +module. It is common for test modules to only want to add or remove tests +from the standard set of tests. +The third argument is used when loading packages as part of test discovery. + +A typical ``load_tests`` function that loads tests from a specific set of +:class:`TestCase` classes may look like:: + + test_cases = (TestCase1, TestCase2, TestCase3) + + def load_tests(loader, tests, pattern): + suite = TestSuite() + for test_class in test_cases: + tests = loader.loadTestsFromTestCase(test_class) + suite.addTests(tests) + return suite + +If discovery is started, either from the command line or by calling +:meth:`TestLoader.discover`, with a pattern that matches a package +name then the package :file:`__init__.py` will be checked for ``load_tests``. + +.. note:: + + The default pattern is 'test*.py'. This matches all Python files + that start with 'test' but *won't* match any test directories. + + A pattern like 'test*' will match test packages as well as + modules. + +If the package :file:`__init__.py` defines ``load_tests`` then it will be +called and discovery not continued into the package. ``load_tests`` +is called with the following arguments:: + + load_tests(loader, standard_tests, pattern) + +This should return a :class:`TestSuite` representing all the tests +from the package. (``standard_tests`` will only contain tests +collected from :file:`__init__.py`.) + +Because the pattern is passed into ``load_tests`` the package is free to +continue (and potentially modify) test discovery. A 'do nothing' +``load_tests`` function for a test package would look like:: + + def load_tests(loader, standard_tests, pattern): + # top level directory cached on loader instance + this_dir = os.path.dirname(__file__) + package_tests = loader.discover(start_dir=this_dir, pattern=pattern) + standard_tests.addTests(package_tests) + return standard_tests + + +Class and Module Fixtures +------------------------- + +Class and module level fixtures are implemented in :class:`TestSuite`. When +the test suite encounters a test from a new class then :meth:`tearDownClass` +from the previous class (if there is one) is called, followed by +:meth:`setUpClass` from the new class. + +Similarly if a test is from a different module from the previous test then +``tearDownModule`` from the previous module is run, followed by +``setUpModule`` from the new module. + +After all the tests have run the final ``tearDownClass`` and +``tearDownModule`` are run. + +Note that shared fixtures do not play well with [potential] features like test +parallelization and they break test isolation. They should be used with care. + +The default ordering of tests created by the unittest test loaders is to group +all tests from the same modules and classes together. This will lead to +``setUpClass`` / ``setUpModule`` (etc) being called exactly once per class and +module. If you randomize the order, so that tests from different modules and +classes are adjacent to each other, then these shared fixture functions may be +called multiple times in a single test run. + +Shared fixtures are not intended to work with suites with non-standard +ordering. A ``BaseTestSuite`` still exists for frameworks that don't want to +support shared fixtures. + +If there are any exceptions raised during one of the shared fixture functions +the test is reported as an error. Because there is no corresponding test +instance an ``_ErrorHolder`` object (that has the same interface as a +:class:`TestCase`) is created to represent the error. If you are just using +the standard unittest test runner then this detail doesn't matter, but if you +are a framework author it may be relevant. + + +setUpClass and tearDownClass +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These must be implemented as class methods:: + + import unittest + + class Test(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls._connection = createExpensiveConnectionObject() + + @classmethod + def tearDownClass(cls): + cls._connection.destroy() + +If you want the ``setUpClass`` and ``tearDownClass`` on base classes called +then you must call up to them yourself. The implementations in +:class:`TestCase` are empty. + +If an exception is raised during a ``setUpClass`` then the tests in the class +are not run and the ``tearDownClass`` is not run. Skipped classes will not +have ``setUpClass`` or ``tearDownClass`` run. If the exception is a +``SkipTest`` exception then the class will be reported as having been skipped +instead of as an error. + + +setUpModule and tearDownModule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These should be implemented as functions:: + + def setUpModule(): + createConnection() + + def tearDownModule(): + closeConnection() + +If an exception is raised in a ``setUpModule`` then none of the tests in the +module will be run and the ``tearDownModule`` will not be run. If the exception is a +``SkipTest`` exception then the module will be reported as having been skipped +instead of as an error. + + +Signal Handling +--------------- + +.. versionadded:: 3.2 + +The :option:`-c/--catch <unittest -c>` command-line option to unittest, +along with the ``catchbreak`` parameter to :func:`unittest.main()`, provide +more friendly handling of control-C during a test run. With catch break +behavior enabled control-C will allow the currently running test to complete, +and the test run will then end and report all the results so far. A second +control-c will raise a :exc:`KeyboardInterrupt` in the usual way. + +The control-c handling signal handler attempts to remain compatible with code or +tests that install their own :const:`signal.SIGINT` handler. If the ``unittest`` +handler is called but *isn't* the installed :const:`signal.SIGINT` handler, +i.e. it has been replaced by the system under test and delegated to, then it +calls the default handler. This will normally be the expected behavior by code +that replaces an installed handler and delegates to it. For individual tests +that need ``unittest`` control-c handling disabled the :func:`removeHandler` +decorator can be used. + +There are a few utility functions for framework authors to enable control-c +handling functionality within test frameworks. + +.. function:: installHandler() + + Install the control-c handler. When a :const:`signal.SIGINT` is received + (usually in response to the user pressing control-c) all registered results + have :meth:`~TestResult.stop` called. + + +.. function:: registerResult(result) + + Register a :class:`TestResult` object for control-c handling. Registering a + result stores a weak reference to it, so it doesn't prevent the result from + being garbage collected. + + Registering a :class:`TestResult` object has no side-effects if control-c + handling is not enabled, so test frameworks can unconditionally register + all results they create independently of whether or not handling is enabled. + + +.. function:: removeResult(result) + + Remove a registered result. Once a result has been removed then + :meth:`~TestResult.stop` will no longer be called on that result object in + response to a control-c. + + +.. function:: removeHandler(function=None) + + When called without arguments this function removes the control-c handler + if it has been installed. This function can also be used as a test decorator + to temporarily remove the handler whilst the test is being executed:: + + @unittest.removeHandler + def test_signal_handling(self): + ... diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index 0ed27ba4..a6d7267 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -24,7 +24,15 @@ following URL schemes: ``file``, ``ftp``, ``gopher``, ``hdl``, ``http``, ``rsync``, ``rtsp``, ``rtspu``, ``sftp``, ``shttp``, ``sip``, ``sips``, ``snews``, ``svn``, ``svn+ssh``, ``telnet``, ``wais``. -The :mod:`urllib.parse` module defines the following functions: +The :mod:`urllib.parse` module defines functions that fall into two broad +categories: URL parsing and URL quoting. These are covered in detail in +the following sections. + +URL Parsing +----------- + +The URL parsing functions focus on splitting a URL string into its components, +or on combining URL components into a URL string. .. function:: urlparse(urlstring, scheme='', allow_fragments=True) @@ -104,8 +112,11 @@ The :mod:`urllib.parse` module defines the following functions: See section :ref:`urlparse-result-object` for more information on the result object. + .. versionchanged:: 3.2 + Added IPv6 URL parsing capabilities. + -.. function:: parse_qs(qs, keep_blank_values=False, strict_parsing=False) +.. function:: parse_qs(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace') Parse a query string given as a string argument (data of type :mimetype:`application/x-www-form-urlencoded`). Data are returned as a @@ -122,11 +133,19 @@ The :mod:`urllib.parse` module defines the following functions: parsing errors. If false (the default), errors are silently ignored. If true, errors raise a :exc:`ValueError` exception. + The optional *encoding* and *errors* parameters specify how to decode + percent-encoded sequences into Unicode characters, as accepted by the + :meth:`bytes.decode` method. + Use the :func:`urllib.parse.urlencode` function to convert such dictionaries into query strings. -.. function:: parse_qsl(qs, keep_blank_values=False, strict_parsing=False) + .. versionchanged:: 3.2 + Add *encoding* and *errors* parameters. + + +.. function:: parse_qsl(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace') Parse a query string given as a string argument (data of type :mimetype:`application/x-www-form-urlencoded`). Data are returned as a list of @@ -142,9 +161,16 @@ The :mod:`urllib.parse` module defines the following functions: parsing errors. If false (the default), errors are silently ignored. If true, errors raise a :exc:`ValueError` exception. + The optional *encoding* and *errors* parameters specify how to decode + percent-encoded sequences into Unicode characters, as accepted by the + :meth:`bytes.decode` method. + Use the :func:`urllib.parse.urlencode` function to convert such lists of pairs into query strings. + .. versionchanged:: 3.2 + Add *encoding* and *errors* parameters. + .. function:: urlunparse(parts) @@ -239,6 +265,162 @@ The :mod:`urllib.parse` module defines the following functions: string. If there is no fragment identifier in *url*, return *url* unmodified and an empty string. + The return value is actually an instance of a subclass of :class:`tuple`. This + class has the following additional read-only convenience attributes: + + +------------------+-------+-------------------------+----------------------+ + | Attribute | Index | Value | Value if not present | + +==================+=======+=========================+======================+ + | :attr:`url` | 0 | URL with no fragment | empty string | + +------------------+-------+-------------------------+----------------------+ + | :attr:`fragment` | 1 | Fragment identifier | empty string | + +------------------+-------+-------------------------+----------------------+ + + See section :ref:`urlparse-result-object` for more information on the result + object. + + .. versionchanged:: 3.2 + Result is a structured object rather than a simple 2-tuple. + +.. _parsing-ascii-encoded-bytes: + +Parsing ASCII Encoded Bytes +--------------------------- + +The URL parsing functions were originally designed to operate on character +strings only. In practice, it is useful to be able to manipulate properly +quoted and encoded URLs as sequences of ASCII bytes. Accordingly, the +URL parsing functions in this module all operate on :class:`bytes` and +:class:`bytearray` objects in addition to :class:`str` objects. + +If :class:`str` data is passed in, the result will also contain only +:class:`str` data. If :class:`bytes` or :class:`bytearray` data is +passed in, the result will contain only :class:`bytes` data. + +Attempting to mix :class:`str` data with :class:`bytes` or +:class:`bytearray` in a single function call will result in a +:exc:`TypeError` being raised, while attempting to pass in non-ASCII +byte values will trigger :exc:`UnicodeDecodeError`. + +To support easier conversion of result objects between :class:`str` and +:class:`bytes`, all return values from URL parsing functions provide +either an :meth:`encode` method (when the result contains :class:`str` +data) or a :meth:`decode` method (when the result contains :class:`bytes` +data). The signatures of these methods match those of the corresponding +:class:`str` and :class:`bytes` methods (except that the default encoding +is ``'ascii'`` rather than ``'utf-8'``). Each produces a value of a +corresponding type that contains either :class:`bytes` data (for +:meth:`encode` methods) or :class:`str` data (for +:meth:`decode` methods). + +Applications that need to operate on potentially improperly quoted URLs +that may contain non-ASCII data will need to do their own decoding from +bytes to characters before invoking the URL parsing methods. + +The behaviour described in this section applies only to the URL parsing +functions. The URL quoting functions use their own rules when producing +or consuming byte sequences as detailed in the documentation of the +individual URL quoting functions. + +.. versionchanged:: 3.2 + URL parsing functions now accept ASCII encoded byte sequences + + +.. _urlparse-result-object: + +Structured Parse Results +------------------------ + +The result objects from the :func:`urlparse`, :func:`urlsplit` and +:func:`urldefrag` functions are subclasses of the :class:`tuple` type. +These subclasses add the attributes listed in the documentation for +those functions, the encoding and decoding support described in the +previous section, as well as an additional method: + +.. method:: urllib.parse.SplitResult.geturl() + + Return the re-combined version of the original URL as a string. This may + differ from the original URL in that the scheme may be normalized to lower + case and empty components may be dropped. Specifically, empty parameters, + queries, and fragment identifiers will be removed. + + For :func:`urldefrag` results, only empty fragment identifiers will be removed. + For :func:`urlsplit` and :func:`urlparse` results, all noted changes will be + made to the URL returned by this method. + + The result of this method remains unchanged if passed back through the original + parsing function: + + >>> from urllib.parse import urlsplit + >>> url = 'HTTP://www.Python.org/doc/#' + >>> r1 = urlsplit(url) + >>> r1.geturl() + 'http://www.Python.org/doc/' + >>> r2 = urlsplit(r1.geturl()) + >>> r2.geturl() + 'http://www.Python.org/doc/' + + +The following classes provide the implementations of the structured parse +results when operating on :class:`str` objects: + +.. class:: DefragResult(url, fragment) + + Concrete class for :func:`urldefrag` results containing :class:`str` + data. The :meth:`encode` method returns a :class:`DefragResultBytes` + instance. + + .. versionadded:: 3.2 + +.. class:: ParseResult(scheme, netloc, path, params, query, fragment) + + Concrete class for :func:`urlparse` results containing :class:`str` + data. The :meth:`encode` method returns a :class:`ParseResultBytes` + instance. + +.. class:: SplitResult(scheme, netloc, path, query, fragment) + + Concrete class for :func:`urlsplit` results containing :class:`str` + data. The :meth:`encode` method returns a :class:`SplitResultBytes` + instance. + + +The following classes provide the implementations of the parse results when +operating on :class:`bytes` or :class:`bytearray` objects: + +.. class:: DefragResultBytes(url, fragment) + + Concrete class for :func:`urldefrag` results containing :class:`bytes` + data. The :meth:`decode` method returns a :class:`DefragResult` + instance. + + .. versionadded:: 3.2 + +.. class:: ParseResultBytes(scheme, netloc, path, params, query, fragment) + + Concrete class for :func:`urlparse` results containing :class:`bytes` + data. The :meth:`decode` method returns a :class:`ParseResult` + instance. + + .. versionadded:: 3.2 + +.. class:: SplitResultBytes(scheme, netloc, path, query, fragment) + + Concrete class for :func:`urlsplit` results containing :class:`bytes` + data. The :meth:`decode` method returns a :class:`SplitResult` + instance. + + .. versionadded:: 3.2 + + +URL Quoting +----------- + +The URL quoting functions focus on taking program data and making it safe +for use as URL components by quoting special characters and appropriately +encoding non-ASCII text. They also support reversing these operations to +recreate the original data from the contents of a URL component if that +task isn't already covered by the URL parsing functions above. .. function:: quote(string, safe='/', encoding=None, errors=None) @@ -319,16 +501,16 @@ The :mod:`urllib.parse` module defines the following functions: If it is a :class:`str`, unescaped non-ASCII characters in *string* are encoded into UTF-8 bytes. - Example: ``unquote_to_bytes('a%26%EF')`` yields - ``b'a&\xef'``. + Example: ``unquote_to_bytes('a%26%EF')`` yields ``b'a&\xef'``. .. function:: urlencode(query, doseq=False, safe='', encoding=None, errors=None) Convert a mapping object or a sequence of two-element tuples, which may - either be a :class:`str` or a :class:`bytes`, to a "percent-encoded" string, - suitable to pass to :func:`urlopen` above as the optional *data* argument. - This is useful to pass a dictionary of form fields to a ``POST`` request. + either be a :class:`str` or a :class:`bytes`, to a "percent-encoded" + string. The resultant string must be converted to bytes using the + user-specified encoding before it is sent to :func:`urlopen` as the optional + *data* argument. The resulting string is a series of ``key=value`` pairs separated by ``'&'`` characters, where both *key* and *value* are quoted using :func:`quote_plus` above. When a sequence of two-element tuples is used as the *query* @@ -337,12 +519,16 @@ The :mod:`urllib.parse` module defines the following functions: the optional parameter *doseq* is evaluates to *True*, individual ``key=value`` pairs separated by ``'&'`` are generated for each element of the value sequence for the key. The order of parameters in the encoded - string will match the order of parameter tuples in the sequence. This module - provides the functions :func:`parse_qs` and :func:`parse_qsl` which are used - to parse query strings into Python data structures. + string will match the order of parameter tuples in the sequence. When *query* parameter is a :class:`str`, the *safe*, *encoding* and *error* - parameters are sent the :func:`quote_plus` for encoding. + parameters are passed down to :func:`quote_plus` for encoding. + + To reverse this encoding process, :func:`parse_qs` and :func:`parse_qsl` are + provided in this module to parse query strings into Python data structures. + + Refer to :ref:`urllib examples <urllib-examples>` to find out how urlencode + method can be used for generating query string for a URL or data for POST. .. versionchanged:: 3.2 Query parameter supports bytes and string objects. @@ -356,6 +542,9 @@ The :mod:`urllib.parse` module defines the following functions: mostly for backward compatibility purposes and for certain de-facto parsing requirements as commonly observed in major browsers. + :rfc:`2732` - Format for Literal IPv6 Addresses in URL's. + This specifies the parsing requirements of IPv6 URLs. + :rfc:`2396` - Uniform Resource Identifiers (URI): Generic Syntax Document describing the generic syntactic requirements for both Uniform Resource Names (URNs) and Uniform Resource Locators (URLs). @@ -370,57 +559,3 @@ The :mod:`urllib.parse` module defines the following functions: :rfc:`1738` - Uniform Resource Locators (URL) This specifies the formal syntax and semantics of absolute URLs. - - -.. _urlparse-result-object: - -Results of :func:`urlparse` and :func:`urlsplit` ------------------------------------------------- - -The result objects from the :func:`urlparse` and :func:`urlsplit` functions are -subclasses of the :class:`tuple` type. These subclasses add the attributes -described in those functions, as well as provide an additional method: - -.. method:: ParseResult.geturl() - - Return the re-combined version of the original URL as a string. This may differ - from the original URL in that the scheme will always be normalized to lower case - and empty components may be dropped. Specifically, empty parameters, queries, - and fragment identifiers will be removed. - - The result of this method is a fixpoint if passed back through the original - parsing function: - - >>> import urllib.parse - >>> url = 'HTTP://www.Python.org/doc/#' - - >>> r1 = urllib.parse.urlsplit(url) - >>> r1.geturl() - 'http://www.Python.org/doc/' - - >>> r2 = urllib.parse.urlsplit(r1.geturl()) - >>> r2.geturl() - 'http://www.Python.org/doc/' - - -The following classes provide the implementations of the parse results: - -.. class:: BaseResult - - Base class for the concrete result classes. This provides most of the - attribute definitions. It does not provide a :meth:`geturl` method. It is - derived from :class:`tuple`, but does not override the :meth:`__init__` or - :meth:`__new__` methods. - - -.. class:: ParseResult(scheme, netloc, path, params, query, fragment) - - Concrete class for :func:`urlparse` results. The :meth:`__new__` method is - overridden to support checking that the right number of arguments are passed. - - -.. class:: SplitResult(scheme, netloc, path, query, fragment) - - Concrete class for :func:`urlsplit` results. The :meth:`__new__` method is - overridden to support checking that the right number of arguments are passed. - diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 3ba2c15..3e24956 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -15,28 +15,37 @@ authentication, redirections, cookies and more. The :mod:`urllib.request` module defines the following functions: -.. function:: urlopen(url, data=None[, timeout]) +.. function:: urlopen(url, data=None[, timeout], *, cafile=None, capath=None) Open the URL *url*, which can be either a string or a :class:`Request` object. - .. warning:: - HTTPS requests do not do any verification of the server's certificate. - - *data* may be a string specifying additional data to send to the - server, or ``None`` if no such data is needed. Currently HTTP - requests are the only ones that use *data*; the HTTP request will - be a POST instead of a GET when the *data* parameter is provided. - *data* should be a buffer in the standard + *data* may be a bytes object specifying additional data to send to the + server, or ``None`` if no such data is needed. *data* may also be an + iterable object and in that case Content-Length value must be specified in + the headers. Currently HTTP requests are the only ones that use *data*; the + HTTP request will be a POST instead of a GET when the *data* parameter is + provided. *data* should be a buffer in the standard :mimetype:`application/x-www-form-urlencoded` format. The - :func:`urllib.parse.urlencode` function takes a mapping or sequence - of 2-tuples and returns a string in this format. + :func:`urllib.parse.urlencode` function takes a mapping or sequence of + 2-tuples and returns a string in this format. urllib.request module uses + HTTP/1.1 and includes ``Connection:close`` header in its HTTP requests. The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). This actually only works for HTTP, HTTPS and FTP connections. + The optional *cafile* and *capath* parameters specify a set of trusted + CA certificates for HTTPS requests. *cafile* should point to a single + file containing a bundle of CA certificates, whereas *capath* should + point to a directory of hashed certificate files. More information can + be found in :meth:`ssl.SSLContext.load_verify_locations`. + + .. warning:: + If neither *cafile* nor *capath* is specified, an HTTPS request + will not do any verification of the server's certificate. + This function returns a file-like object with two additional methods from the :mod:`urllib.response` module @@ -61,6 +70,16 @@ The :mod:`urllib.request` module defines the following functions: Proxy handling, which was done by passing a dictionary parameter to ``urllib.urlopen``, can be obtained by using :class:`ProxyHandler` objects. + .. versionchanged:: 3.2 + *cafile* and *capath* were added. + + .. versionchanged:: 3.2 + HTTPS virtual hosts are now supported if possible (that is, if + :data:`ssl.HAS_SNI` is true). + + .. versionadded:: 3.2 + *data* can be an iterable object. + .. function:: install_opener(opener) Install an :class:`OpenerDirector` instance as the default global opener. @@ -89,52 +108,6 @@ The :mod:`urllib.request` module defines the following functions: member variable to modify its position in the handlers list. -.. function:: urlretrieve(url, filename=None, reporthook=None, data=None) - - Copy a network object denoted by a URL to a local file, if necessary. If the URL - points to a local file, or a valid cached copy of the object exists, the object - is not copied. Return a tuple ``(filename, headers)`` where *filename* is the - local file name under which the object can be found, and *headers* is whatever - the :meth:`info` method of the object returned by :func:`urlopen` returned (for - a remote object, possibly cached). Exceptions are the same as for - :func:`urlopen`. - - The second argument, if present, specifies the file location to copy to (if - absent, the location will be a tempfile with a generated name). The third - argument, if present, is a hook function that will be called once on - establishment of the network connection and once after each block read - thereafter. The hook will be passed three arguments; a count of blocks - transferred so far, a block size in bytes, and the total size of the file. The - third argument may be ``-1`` on older FTP servers which do not return a file - size in response to a retrieval request. - - If the *url* uses the :file:`http:` scheme identifier, the optional *data* - argument may be given to specify a ``POST`` request (normally the request type - is ``GET``). The *data* argument must in standard - :mimetype:`application/x-www-form-urlencoded` format; see the :func:`urlencode` - function below. - - :func:`urlretrieve` will raise :exc:`ContentTooShortError` when it detects that - the amount of data available was less than the expected amount (which is the - size reported by a *Content-Length* header). This can occur, for example, when - the download is interrupted. - - The *Content-Length* is treated as a lower bound: if there's more data to read, - :func:`urlretrieve` reads more data, but if less data is available, it raises - the exception. - - You can still retrieve the downloaded data in this case, it is stored in the - :attr:`content` attribute of the exception instance. - - If no *Content-Length* header was supplied, :func:`urlretrieve` can not - check the size of the data it has downloaded, and just returns it. In - this case you just have to assume that the download was successful. - -.. function:: urlcleanup() - - Clear the cache that may have been built up by previous calls to - :func:`urlretrieve`. - .. function:: pathname2url(path) Convert the pathname *path* from the local syntax for a path to the form used in @@ -203,116 +176,6 @@ The following classes are provided: fetching of the image, this should be true. -.. class:: URLopener(proxies=None, **x509) - - Base class for opening and reading URLs. Unless you need to support opening - objects using schemes other than :file:`http:`, :file:`ftp:`, or :file:`file:`, - you probably want to use :class:`FancyURLopener`. - - By default, the :class:`URLopener` class sends a :mailheader:`User-Agent` header - of ``urllib/VVV``, where *VVV* is the :mod:`urllib` version number. - Applications can define their own :mailheader:`User-Agent` header by subclassing - :class:`URLopener` or :class:`FancyURLopener` and setting the class attribute - :attr:`version` to an appropriate string value in the subclass definition. - - The optional *proxies* parameter should be a dictionary mapping scheme names to - proxy URLs, where an empty dictionary turns proxies off completely. Its default - value is ``None``, in which case environmental proxy settings will be used if - present, as discussed in the definition of :func:`urlopen`, above. - - Additional keyword parameters, collected in *x509*, may be used for - authentication of the client when using the :file:`https:` scheme. The keywords - *key_file* and *cert_file* are supported to provide an SSL key and certificate; - both are needed to support client authentication. - - :class:`URLopener` objects will raise an :exc:`IOError` exception if the server - returns an error code. - - .. method:: open(fullurl, data=None) - - Open *fullurl* using the appropriate protocol. This method sets up cache and - proxy information, then calls the appropriate open method with its input - arguments. If the scheme is not recognized, :meth:`open_unknown` is called. - The *data* argument has the same meaning as the *data* argument of - :func:`urlopen`. - - - .. method:: open_unknown(fullurl, data=None) - - Overridable interface to open unknown URL types. - - - .. method:: retrieve(url, filename=None, reporthook=None, data=None) - - Retrieves the contents of *url* and places it in *filename*. The return value - is a tuple consisting of a local filename and either a - :class:`email.message.Message` object containing the response headers (for remote - URLs) or ``None`` (for local URLs). The caller must then open and read the - contents of *filename*. If *filename* is not given and the URL refers to a - local file, the input filename is returned. If the URL is non-local and - *filename* is not given, the filename is the output of :func:`tempfile.mktemp` - with a suffix that matches the suffix of the last path component of the input - URL. If *reporthook* is given, it must be a function accepting three numeric - parameters. It will be called after each chunk of data is read from the - network. *reporthook* is ignored for local URLs. - - If the *url* uses the :file:`http:` scheme identifier, the optional *data* - argument may be given to specify a ``POST`` request (normally the request type - is ``GET``). The *data* argument must in standard - :mimetype:`application/x-www-form-urlencoded` format; see the :func:`urlencode` - function below. - - - .. attribute:: version - - Variable that specifies the user agent of the opener object. To get - :mod:`urllib` to tell servers that it is a particular user agent, set this in a - subclass as a class variable or in the constructor before calling the base - constructor. - - -.. class:: FancyURLopener(...) - - :class:`FancyURLopener` subclasses :class:`URLopener` providing default handling - for the following HTTP response codes: 301, 302, 303, 307 and 401. For the 30x - response codes listed above, the :mailheader:`Location` header is used to fetch - the actual URL. For 401 response codes (authentication required), basic HTTP - authentication is performed. For the 30x response codes, recursion is bounded - by the value of the *maxtries* attribute, which defaults to 10. - - For all other response codes, the method :meth:`http_error_default` is called - which you can override in subclasses to handle the error appropriately. - - .. note:: - - According to the letter of :rfc:`2616`, 301 and 302 responses to POST requests - must not be automatically redirected without confirmation by the user. In - reality, browsers do allow automatic redirection of these responses, changing - the POST to a GET, and :mod:`urllib` reproduces this behaviour. - - The parameters to the constructor are the same as those for :class:`URLopener`. - - .. note:: - - When performing basic authentication, a :class:`FancyURLopener` instance calls - its :meth:`prompt_user_passwd` method. The default implementation asks the - users for the required information on the controlling terminal. A subclass may - override this method to support more appropriate behavior if needed. - - The :class:`FancyURLopener` class offers one additional method that should be - overloaded to provide the appropriate behavior: - - .. method:: prompt_user_passwd(host, realm) - - Return information needed to authenticate the user at the given host in the - specified security realm. The return value should be a tuple, ``(user, - password)``, which can be used for basic authentication. - - The implementation prompts for this information on the terminal; an application - should override this method to use an appropriate interaction model in the local - environment. - - .. class:: OpenerDirector() The :class:`OpenerDirector` class opens URLs via :class:`BaseHandler`\ s chained @@ -421,9 +284,13 @@ The following classes are provided: A class to handle opening of HTTP URLs. -.. class:: HTTPSHandler() +.. class:: HTTPSHandler(debuglevel=0, context=None, check_hostname=None) + + A class to handle opening of HTTPS URLs. *context* and *check_hostname* + have the same meaning as in :class:`http.client.HTTPSConnection`. - A class to handle opening of HTTPS URLs. + .. versionchanged:: 3.2 + *context* and *check_hostname* were added. .. class:: FileHandler() @@ -994,8 +861,12 @@ FileHandler Objects .. method:: FileHandler.file_open(req) Open the file locally, if there is no host name, or the host name is - ``'localhost'``. Change the protocol to ``ftp`` otherwise, and retry opening it - using :attr:`parent`. + ``'localhost'``. + + This method is applicable only for local hostnames. When a remote hostname + is given, an :exc:`URLError` is raised. + +.. versionchanged:: 3.2 .. _ftp-handler-objects: @@ -1100,7 +971,7 @@ when the Python installation supports SSL. :: >>> import urllib.request >>> req = urllib.request.Request(url='https://localhost/cgi-bin/test.cgi', - ... data='This data is passed to stdin of the CGI') + ... data=b'This data is passed to stdin of the CGI') >>> f = urllib.request.urlopen(req) >>> print(f.read().decode('utf-8')) Got Data: "This data is passed to stdin of the CGI" @@ -1176,11 +1047,13 @@ containing parameters:: >>> f = urllib.request.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params) >>> print(f.read().decode('utf-8')) -The following example uses the ``POST`` method instead:: +The following example uses the ``POST`` method instead. Note that params output +from urlencode is encoded to bytes before it is sent to urlopen as data:: >>> import urllib.request >>> import urllib.parse >>> params = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) + >>> params = params.encode('utf-8') >>> f = urllib.request.urlopen("http://www.musi-cal.com/cgi-bin/query", params) >>> print(f.read().decode('utf-8')) @@ -1201,6 +1074,170 @@ The following example uses no proxies at all, overriding environment settings:: >>> f.read().decode('utf-8') +Legacy interface +---------------- + +The following functions and classes are ported from the Python 2 module +``urllib`` (as opposed to ``urllib2``). They might become deprecated at +some point in the future. + + +.. function:: urlretrieve(url, filename=None, reporthook=None, data=None) + + Copy a network object denoted by a URL to a local file, if necessary. If the URL + points to a local file, or a valid cached copy of the object exists, the object + is not copied. Return a tuple ``(filename, headers)`` where *filename* is the + local file name under which the object can be found, and *headers* is whatever + the :meth:`info` method of the object returned by :func:`urlopen` returned (for + a remote object, possibly cached). Exceptions are the same as for + :func:`urlopen`. + + The second argument, if present, specifies the file location to copy to (if + absent, the location will be a tempfile with a generated name). The third + argument, if present, is a hook function that will be called once on + establishment of the network connection and once after each block read + thereafter. The hook will be passed three arguments; a count of blocks + transferred so far, a block size in bytes, and the total size of the file. The + third argument may be ``-1`` on older FTP servers which do not return a file + size in response to a retrieval request. + + If the *url* uses the :file:`http:` scheme identifier, the optional *data* + argument may be given to specify a ``POST`` request (normally the request type + is ``GET``). The *data* argument must in standard + :mimetype:`application/x-www-form-urlencoded` format; see the :func:`urlencode` + function below. + + :func:`urlretrieve` will raise :exc:`ContentTooShortError` when it detects that + the amount of data available was less than the expected amount (which is the + size reported by a *Content-Length* header). This can occur, for example, when + the download is interrupted. + + The *Content-Length* is treated as a lower bound: if there's more data to read, + :func:`urlretrieve` reads more data, but if less data is available, it raises + the exception. + + You can still retrieve the downloaded data in this case, it is stored in the + :attr:`content` attribute of the exception instance. + + If no *Content-Length* header was supplied, :func:`urlretrieve` can not check + the size of the data it has downloaded, and just returns it. In this case + you just have to assume that the download was successful. + +.. function:: urlcleanup() + + Clear the cache that may have been built up by previous calls to + :func:`urlretrieve`. + +.. class:: URLopener(proxies=None, **x509) + + Base class for opening and reading URLs. Unless you need to support opening + objects using schemes other than :file:`http:`, :file:`ftp:`, or :file:`file:`, + you probably want to use :class:`FancyURLopener`. + + By default, the :class:`URLopener` class sends a :mailheader:`User-Agent` header + of ``urllib/VVV``, where *VVV* is the :mod:`urllib` version number. + Applications can define their own :mailheader:`User-Agent` header by subclassing + :class:`URLopener` or :class:`FancyURLopener` and setting the class attribute + :attr:`version` to an appropriate string value in the subclass definition. + + The optional *proxies* parameter should be a dictionary mapping scheme names to + proxy URLs, where an empty dictionary turns proxies off completely. Its default + value is ``None``, in which case environmental proxy settings will be used if + present, as discussed in the definition of :func:`urlopen`, above. + + Additional keyword parameters, collected in *x509*, may be used for + authentication of the client when using the :file:`https:` scheme. The keywords + *key_file* and *cert_file* are supported to provide an SSL key and certificate; + both are needed to support client authentication. + + :class:`URLopener` objects will raise an :exc:`IOError` exception if the server + returns an error code. + + .. method:: open(fullurl, data=None) + + Open *fullurl* using the appropriate protocol. This method sets up cache and + proxy information, then calls the appropriate open method with its input + arguments. If the scheme is not recognized, :meth:`open_unknown` is called. + The *data* argument has the same meaning as the *data* argument of + :func:`urlopen`. + + + .. method:: open_unknown(fullurl, data=None) + + Overridable interface to open unknown URL types. + + + .. method:: retrieve(url, filename=None, reporthook=None, data=None) + + Retrieves the contents of *url* and places it in *filename*. The return value + is a tuple consisting of a local filename and either a + :class:`email.message.Message` object containing the response headers (for remote + URLs) or ``None`` (for local URLs). The caller must then open and read the + contents of *filename*. If *filename* is not given and the URL refers to a + local file, the input filename is returned. If the URL is non-local and + *filename* is not given, the filename is the output of :func:`tempfile.mktemp` + with a suffix that matches the suffix of the last path component of the input + URL. If *reporthook* is given, it must be a function accepting three numeric + parameters. It will be called after each chunk of data is read from the + network. *reporthook* is ignored for local URLs. + + If the *url* uses the :file:`http:` scheme identifier, the optional *data* + argument may be given to specify a ``POST`` request (normally the request type + is ``GET``). The *data* argument must in standard + :mimetype:`application/x-www-form-urlencoded` format; see the :func:`urlencode` + function below. + + + .. attribute:: version + + Variable that specifies the user agent of the opener object. To get + :mod:`urllib` to tell servers that it is a particular user agent, set this in a + subclass as a class variable or in the constructor before calling the base + constructor. + + +.. class:: FancyURLopener(...) + + :class:`FancyURLopener` subclasses :class:`URLopener` providing default handling + for the following HTTP response codes: 301, 302, 303, 307 and 401. For the 30x + response codes listed above, the :mailheader:`Location` header is used to fetch + the actual URL. For 401 response codes (authentication required), basic HTTP + authentication is performed. For the 30x response codes, recursion is bounded + by the value of the *maxtries* attribute, which defaults to 10. + + For all other response codes, the method :meth:`http_error_default` is called + which you can override in subclasses to handle the error appropriately. + + .. note:: + + According to the letter of :rfc:`2616`, 301 and 302 responses to POST requests + must not be automatically redirected without confirmation by the user. In + reality, browsers do allow automatic redirection of these responses, changing + the POST to a GET, and :mod:`urllib` reproduces this behaviour. + + The parameters to the constructor are the same as those for :class:`URLopener`. + + .. note:: + + When performing basic authentication, a :class:`FancyURLopener` instance calls + its :meth:`prompt_user_passwd` method. The default implementation asks the + users for the required information on the controlling terminal. A subclass may + override this method to support more appropriate behavior if needed. + + The :class:`FancyURLopener` class offers one additional method that should be + overloaded to provide the appropriate behavior: + + .. method:: prompt_user_passwd(host, realm) + + Return information needed to authenticate the user at the given host in the + specified security realm. The return value should be a tuple, ``(user, + password)``, which can be used for basic authentication. + + The implementation prompts for this information on the terminal; an application + should override this method to use an appropriate interaction model in the local + environment. + + :mod:`urllib.request` Restrictions ---------------------------------- diff --git a/Doc/library/uu.rst b/Doc/library/uu.rst index 7813e44..d61c178 100644 --- a/Doc/library/uu.rst +++ b/Doc/library/uu.rst @@ -5,6 +5,9 @@ :synopsis: Encode and decode files in uuencode format. .. moduleauthor:: Lance Ellinghouse +**Source code:** :source:`Lib/uu.py` + +-------------- This module encodes and decodes files in uuencode format, allowing arbitrary binary data to be transferred over ASCII-only connections. Wherever a file @@ -56,4 +59,3 @@ The :mod:`uu` module defines the following functions: Module :mod:`binascii` Support module containing ASCII-to-binary and binary-to-ASCII conversions. - diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst index e62be55..274840b 100644 --- a/Doc/library/warnings.rst +++ b/Doc/library/warnings.rst @@ -6,6 +6,9 @@ .. module:: warnings :synopsis: Issue warning messages and control their disposition. +**Source code:** :source:`Lib/warnings.py` + +-------------- Warning messages are typically issued in situations where it is useful to alert the user of some condition in a program, where that condition (normally) doesn't @@ -13,7 +16,7 @@ warrant raising an exception and terminating the program. For example, one might want to issue a warning when a program uses an obsolete module. Python programmers issue warnings by calling the :func:`warn` function defined -in this module. (C programmers use :cfunc:`PyErr_WarnEx`; see +in this module. (C programmers use :c:func:`PyErr_WarnEx`; see :ref:`exceptionhandling` for details). Warning messages are normally written to ``sys.stderr``, but their disposition @@ -57,7 +60,7 @@ following warnings category classes are currently defined: | :exc:`UserWarning` | The default category for :func:`warn`. | +----------------------------------+-----------------------------------------------+ | :exc:`DeprecationWarning` | Base category for warnings about deprecated | -| | features. | +| | features (ignored by default). | +----------------------------------+-----------------------------------------------+ | :exc:`SyntaxWarning` | Base category for warnings about dubious | | | syntactic features. | @@ -82,6 +85,9 @@ following warnings category classes are currently defined: | :exc:`BytesWarning` | Base category for warnings related to | | | :class:`bytes` and :class:`buffer`. | +----------------------------------+-----------------------------------------------+ +| :exc:`ResourceWarning` | Base category for warnings related to | +| | resource usage. | ++----------------------------------+-----------------------------------------------+ While these are technically built-in exceptions, they are documented here, @@ -150,14 +156,6 @@ interpreter command line. The interpreter saves the arguments for all :mod:`warnings` module parses these when it is first imported (invalid options are ignored, after printing a message to ``sys.stderr``). -The warnings that are ignored by default may be enabled by passing :option:`-Wd` -to the interpreter. This enables default handling for all warnings, including -those that are normally ignored by default. This is particular useful for -enabling ImportWarning when debugging problems importing a developed package. -ImportWarning can also be enabled explicitly in Python code using:: - - warnings.simplefilter('default', ImportWarning) - Default Warning Filters ~~~~~~~~~~~~~~~~~~~~~~~ @@ -166,12 +164,19 @@ By default, Python installs several warning filters, which can be overridden by the command-line options passed to :option:`-W` and calls to :func:`filterwarnings`. -* :exc:`PendingDeprecationWarning`, and :exc:`ImportWarning` are ignored. +* :exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning`, and + :exc:`ImportWarning` are ignored. * :exc:`BytesWarning` is ignored unless the :option:`-b` option is given once or twice; in this case this warning is either printed (``-b``) or turned into an exception (``-bb``). +* :exc:`ResourceWarning` is ignored unless Python was built in debug mode. + +.. versionchanged:: 3.2 + :exc:`DeprecationWarning` is now ignored by default in addition to + :exc:`PendingDeprecationWarning`. + .. _warning-suppress: @@ -194,7 +199,10 @@ the warning using the :class:`catch_warnings` context manager:: While within the context manager all warnings will simply be ignored. This allows you to use known-deprecated code without having to see the warning while not suppressing the warning for other code that might not be aware of its use -of deprecated code. +of deprecated code. Note: this can only be guaranteed in a single-threaded +application. If two or more threads use the :class:`catch_warnings` context +manager at the same time, the behavior is undefined. + .. _warning-testing: @@ -232,7 +240,9 @@ Once the context manager exits, the warnings filter is restored to its state when the context was entered. This prevents tests from changing the warnings filter in unexpected ways between tests and leading to indeterminate test results. The :func:`showwarning` function in the module is also restored to -its original value. +its original value. Note: this can only be guaranteed in a single-threaded +application. If two or more threads use the :class:`catch_warnings` context +manager at the same time, the behavior is undefined. When testing multiple operations that raise the same kind of warning, it is important to test them in a manner that confirms each operation is raising @@ -242,6 +252,42 @@ continues to increase after each operation, or else delete the previous entries from the warnings list before each new operation). +.. _warning-ignored: + +Updating Code For New Versions of Python +---------------------------------------- + +Warnings that are only of interest to the developer are ignored by default. As +such you should make sure to test your code with typically ignored warnings +made visible. You can do this from the command-line by passing :option:`-Wd` +to the interpreter (this is shorthand for :option:`-W default`). This enables +default handling for all warnings, including those that are ignored by default. +To change what action is taken for encountered warnings you simply change what +argument is passed to :option:`-W`, e.g. :option:`-W error`. See the +:option:`-W` flag for more details on what is possible. + +To programmatically do the same as :option:`-Wd`, use:: + + warnings.simplefilter('default') + +Make sure to execute this code as soon as possible. This prevents the +registering of what warnings have been raised from unexpectedly influencing how +future warnings are treated. + +Having certain warnings ignored by default is done to prevent a user from +seeing warnings that are only of interest to the developer. As you do not +necessarily have control over what interpreter a user uses to run their code, +it is possible that a new version of Python will be released between your +release cycles. The new interpreter release could trigger new warnings in your +code that were not there in an older interpreter, e.g. +:exc:`DeprecationWarning` for a module that you are using. While you as a +developer want to be notified that your code is using a deprecated module, to a +user this information is essentially noise and provides no benefit to them. + +The :mod:`unittest` module has been also updated to use the ``'default'`` +filter while running tests. + + .. _warning-functions: Available Functions @@ -351,3 +397,11 @@ Available Context Managers module returned when you import :mod:`warnings` whose filter will be protected. This argument exists primarily for testing the :mod:`warnings` module itself. + + .. note:: + + The :class:`catch_warnings` manager works by replacing and + then later restoring the module's + :func:`showwarning` function and internal list of filter + specifications. This means the context manager is modifying + global state and therefore is not thread-safe. diff --git a/Doc/library/wave.rst b/Doc/library/wave.rst index 6cfa1a4..afafb45 100644 --- a/Doc/library/wave.rst +++ b/Doc/library/wave.rst @@ -6,6 +6,10 @@ .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> .. Documentations stolen from comments in file. +**Source code:** :source:`Lib/wave.py` + +-------------- + The :mod:`wave` module provides a convenient interface to the WAV sound format. It does not support compression/decompression, but it does support mono/stereo. @@ -162,6 +166,10 @@ Wave_write objects, as returned by :func:`.open`, have the following methods: Set the frame rate to *n*. + .. versionchanged:: 3.2 + A non-integral input to this method is rounded to the nearest + integer. + .. method:: Wave_write.setnframes(n) diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index 10b69c9..63545ab 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -8,6 +8,9 @@ .. moduleauthor:: Martin von Löwis <martin@loewis.home.cs.tu-berlin.de> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/weakref.py` + +-------------- The :mod:`weakref` module allows the Python programmer to create :dfn:`weak references` to objects. @@ -59,8 +62,11 @@ is exposed by the :mod:`weakref` module for the benefit of advanced uses. Not all objects can be weakly referenced; those objects which can include class instances, functions written in Python (but not in C), instance methods, sets, frozensets, some :term:`file objects <file object>`, :term:`generator`\s, type -objects, sockets, arrays, deques and regular expression pattern objects. +objects, sockets, arrays, deques, regular expression pattern objects, and code +objects. +.. versionchanged:: 3.2 + Added support for thread.lock, threading.Lock, and code objects. Several built-in types such as :class:`list` and :class:`dict` do not directly support weak references but can add support through subclassing:: diff --git a/Doc/library/webbrowser.rst b/Doc/library/webbrowser.rst index 20c0913..23ba6c5 100644 --- a/Doc/library/webbrowser.rst +++ b/Doc/library/webbrowser.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/webbrowser.py` + +-------------- The :mod:`webbrowser` module provides a high-level interface to allow displaying Web-based documents to users. Under most circumstances, simply calling the diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst index 68f7e91..5cf30ee 100644 --- a/Doc/library/winreg.rst +++ b/Doc/library/winreg.rst @@ -60,6 +60,33 @@ This module offers the following functions: :exc:`WindowsError` exception is raised. +.. function:: CreateKeyEx(key, sub_key, reserved=0, access=KEY_ALL_ACCESS) + + Creates or opens the specified key, returning a + :ref:`handle object <handle-object>`. + + *key* is an already open key, or one of the predefined + :ref:`HKEY_* constants <hkey-constants>`. + + *sub_key* is a string that names the key this method opens or creates. + + *res* is a reserved integer, and must be zero. The default is zero. + + *sam* is an integer that specifies an access mask that describes the desired + security access for the key. Default is :const:`KEY_ALL_ACCESS`. See + :ref:`Access Rights <access-rights>` for other allowed values. + + If *key* is one of the predefined keys, *sub_key* may be ``None``. In that + case, the handle returned is the same key handle passed in to the function. + + If the key already exists, this function opens the existing key. + + The return value is the handle of the opened key. If the function fails, a + :exc:`WindowsError` exception is raised. + + .. versionadded:: 3.2 + + .. function:: DeleteKey(key, sub_key) Deletes the specified key. @@ -67,8 +94,8 @@ This module offers the following functions: *key* is an already open key, or one of the predefined :ref:`HKEY_* constants <hkey-constants>`. - *sub_key* is a string that must be a subkey of the key identified by the *key* - parameter. This value must not be ``None``, and the key may not have subkeys. + *sub_key* is a string that must be a subkey of the key identified by the *key* + parameter. This value must not be ``None``, and the key may not have subkeys. *This method can not delete keys with subkeys.* @@ -76,6 +103,39 @@ This module offers the following functions: If the method fails, a :exc:`WindowsError` exception is raised. +.. function:: DeleteKeyEx(key, sub_key, access=KEY_ALL_ACCESS, reserved=0) + + Deletes the specified key. + + .. note:: + The :func:`DeleteKeyEx` function is implemented with the RegDeleteKeyEx + Windows API function, which is specific to 64-bit versions of Windows. + See the `RegDeleteKeyEx documentation + <http://msdn.microsoft.com/en-us/library/ms724847%28VS.85%29.aspx>`__. + + *key* is an already open key, or one of the predefined + :ref:`HKEY_* constants <hkey-constants>`. + + *sub_key* is a string that must be a subkey of the key identified by the + *key* parameter. This value must not be ``None``, and the key may not have + subkeys. + + *res* is a reserved integer, and must be zero. The default is zero. + + *sam* is an integer that specifies an access mask that describes the desired + security access for the key. Default is :const:`KEY_ALL_ACCESS`. See + :ref:`Access Rights <access-rights>` for other allowed values. + + *This method can not delete keys with subkeys.* + + If the method succeeds, the entire key, including all of its values, is + removed. If the method fails, a :exc:`WindowsError` exception is raised. + + On unsupported Windows versions, :exc:`NotImplementedError` is raised. + + .. versionadded:: 3.2 + + .. function:: DeleteValue(key, value) Removes a named value from a registry key. @@ -183,7 +243,7 @@ This module offers the following functions: specified in *file_name* is relative to the remote computer. -.. function:: OpenKey(key, sub_key[, res[, sam]]) +.. function:: OpenKey(key, sub_key, reserved=0, access=KEY_ALL_ACCESS) Opens the specified key, returning a :ref:`handle object <handle-object>`. @@ -202,6 +262,8 @@ This module offers the following functions: If the function fails, :exc:`WindowsError` is raised. + .. versionchanged:: 3.2 Allow the use of named arguments. + .. function:: OpenKeyEx() diff --git a/Doc/library/winsound.rst b/Doc/library/winsound.rst index d54c999..8356062 100644 --- a/Doc/library/winsound.rst +++ b/Doc/library/winsound.rst @@ -22,7 +22,7 @@ provided by Windows platforms. It includes functions and several constants. .. function:: PlaySound(sound, flags) - Call the underlying :cfunc:`PlaySound` function from the Platform API. The + Call the underlying :c:func:`PlaySound` function from the Platform API. The *sound* parameter may be a filename, audio data as a string, or ``None``. Its interpretation depends on the value of *flags*, which can be a bitwise ORed combination of the constants described below. If the *sound* parameter is @@ -32,7 +32,7 @@ provided by Windows platforms. It includes functions and several constants. .. function:: MessageBeep(type=MB_OK) - Call the underlying :cfunc:`MessageBeep` function from the Platform API. This + Call the underlying :c:func:`MessageBeep` function from the Platform API. This plays a sound as specified in the registry. The *type* argument specifies which sound to play; possible values are ``-1``, ``MB_ICONASTERISK``, ``MB_ICONEXCLAMATION``, ``MB_ICONHAND``, ``MB_ICONQUESTION``, and ``MB_OK``, all diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index 231449c..3969ea4 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -22,7 +22,7 @@ be used to add WSGI support to a web server or framework. It provides utilities for manipulating WSGI environment variables and response headers, base classes for implementing WSGI servers, a demo HTTP server that serves WSGI applications, and a validation tool that checks WSGI servers and applications for conformance -to the WSGI specification (:pep:`333`). +to the WSGI specification (:pep:`3333`). See http://www.wsgi.org for more information about WSGI, and links to tutorials and other resources. @@ -39,9 +39,9 @@ and other resources. This module provides a variety of utility functions for working with WSGI environments. A WSGI environment is a dictionary containing HTTP request -variables as described in :pep:`333`. All of the functions taking an *environ* +variables as described in :pep:`3333`. All of the functions taking an *environ* parameter expect a WSGI-compliant dictionary to be supplied; please see -:pep:`333` for a detailed specification. +:pep:`3333` for a detailed specification. .. function:: guess_scheme(environ) @@ -60,7 +60,7 @@ parameter expect a WSGI-compliant dictionary to be supplied; please see .. function:: request_uri(environ, include_query=True) Return the full request URI, optionally including the query string, using the - algorithm found in the "URL Reconstruction" section of :pep:`333`. If + algorithm found in the "URL Reconstruction" section of :pep:`3333`. If *include_query* is false, the query string is not included in the resulting URI. @@ -104,7 +104,7 @@ parameter expect a WSGI-compliant dictionary to be supplied; please see This routine adds various parameters required for WSGI, including ``HTTP_HOST``, ``SERVER_NAME``, ``SERVER_PORT``, ``REQUEST_METHOD``, ``SCRIPT_NAME``, - ``PATH_INFO``, and all of the :pep:`333`\ -defined ``wsgi.*`` variables. It + ``PATH_INFO``, and all of the :pep:`3333`\ -defined ``wsgi.*`` variables. It only supplies default values, and does not replace any existing settings for these variables. @@ -152,8 +152,8 @@ also provides these miscellaneous utilities: support both :meth:`__getitem__` and :meth:`__iter__` iteration styles, for compatibility with Python 2.1 and Jython. As the object is iterated over, the optional *blksize* parameter will be repeatedly passed to the *filelike* - object's :meth:`read` method to obtain strings to yield. When :meth:`read` - returns an empty string, iteration is ended and is not resumable. + object's :meth:`read` method to obtain bytestrings to yield. When :meth:`read` + returns an empty bytestring, iteration is ended and is not resumable. If *filelike* has a :meth:`close` method, the returned object will also have a :meth:`close` method, and it will invoke the *filelike* object's :meth:`close` @@ -187,7 +187,7 @@ manipulation of WSGI response headers using a mapping-like interface. .. class:: Headers(headers) Create a mapping-like object wrapping *headers*, which must be a list of header - name/value tuples as described in :pep:`333`. + name/value tuples as described in :pep:`3333`. :class:`Headers` objects support typical mapping operations including :meth:`__getitem__`, :meth:`get`, :meth:`__setitem__`, :meth:`setdefault`, @@ -210,11 +210,11 @@ manipulation of WSGI response headers using a mapping-like interface. :meth:`items`, which is the same as the length of the wrapped header list. In fact, the :meth:`items` method just returns a copy of the wrapped header list. - Calling ``str()`` on a :class:`Headers` object returns a formatted string + Calling ``bytes()`` on a :class:`Headers` object returns a formatted bytestring suitable for transmission as HTTP response headers. Each header is placed on a line with its value, separated by a colon and a space. Each line is terminated - by a carriage return and line feed, and the string is terminated with a blank - line. + by a carriage return and line feed, and the bytestring is terminated with a + blank line. In addition to their mapping interface and formatting features, :class:`Headers` objects also have the following methods for querying and adding multi-valued @@ -272,7 +272,7 @@ request. (E.g., using the :func:`shift_path_info` function from Create a new WSGI server listening on *host* and *port*, accepting connections for *app*. The return value is an instance of the supplied *server_class*, and will process requests using the specified *handler_class*. *app* must be a WSGI - application object, as defined by :pep:`333`. + application object, as defined by :pep:`3333`. Example usage:: @@ -346,7 +346,7 @@ request. (E.g., using the :func:`shift_path_info` function from :attr:`base_environ` dictionary attribute and then adds various headers derived from the HTTP request. Each call to this method should return a new dictionary containing all of the relevant CGI environment variables as specified in - :pep:`333`. + :pep:`3333`. .. method:: WSGIRequestHandler.get_stderr() @@ -376,7 +376,7 @@ application objects that validate communications between a WSGI server or gateway and a WSGI application object, to check both sides for protocol conformance. -Note that this utility does not guarantee complete :pep:`333` compliance; an +Note that this utility does not guarantee complete :pep:`3333` compliance; an absence of errors from this module does not necessarily mean that errors do not exist. However, if this module does produce an error, then it is virtually certain that either the server or application is not 100% compliant. @@ -401,7 +401,7 @@ Paste" library. This wrapper may also generate output using the :mod:`warnings` module to indicate behaviors that are questionable but which may not actually be - prohibited by :pep:`333`. Unless they are suppressed using Python command-line + prohibited by :pep:`3333`. Unless they are suppressed using Python command-line options or the :mod:`warnings` API, any such warnings will be written to ``sys.stderr`` (*not* ``wsgi.errors``, unless they happen to be the same object). @@ -456,6 +456,32 @@ input, output, and error streams. environment. +.. class:: IISCGIHandler() + + A specialized alternative to :class:`CGIHandler`, for use when deploying on + Microsoft's IIS web server, without having set the config allowPathInfo + option (IIS>=7) or metabase allowPathInfoForScriptMappings (IIS<7). + + By default, IIS gives a ``PATH_INFO`` that duplicates the ``SCRIPT_NAME`` at + the front, causing problems for WSGI applications that wish to implement + routing. This handler strips any such duplicated path. + + IIS can be configured to pass the correct ``PATH_INFO``, but this causes + another bug where ``PATH_TRANSLATED`` is wrong. Luckily this variable is + rarely used and is not guaranteed by WSGI. On IIS<7, though, the + setting can only be made on a vhost level, affecting all other script + mappings, many of which break when exposed to the ``PATH_TRANSLATED`` bug. + For this reason IIS<7 is almost never deployed with the fix. (Even IIS7 + rarely uses it because there is still no UI for it.) + + There is no way for CGI code to tell whether the option was set, so a + separate handler class is provided. It is used in the same way as + :class:`CGIHandler`, i.e., by calling ``IISCGIHandler().run(app)``, where + ``app`` is the WSGI application object you wish to invoke. + + .. versionadded:: 3.2 + + .. class:: BaseCGIHandler(stdin, stdout, stderr, environ, multithread=True, multiprocess=False) Similar to :class:`CGIHandler`, but instead of using the :mod:`sys` and @@ -626,7 +652,7 @@ input, output, and error streams. This method can access the current error information using ``sys.exc_info()``, and should pass that information to *start_response* when calling it (as - described in the "Error Handling" section of :pep:`333`). + described in the "Error Handling" section of :pep:`3333`). The default implementation just uses the :attr:`error_status`, :attr:`error_headers`, and :attr:`error_body` attributes to generate an output @@ -641,23 +667,23 @@ input, output, and error streams. .. attribute:: BaseHandler.error_status The HTTP status used for error responses. This should be a status string as - defined in :pep:`333`; it defaults to a 500 code and message. + defined in :pep:`3333`; it defaults to a 500 code and message. .. attribute:: BaseHandler.error_headers The HTTP headers used for error responses. This should be a list of WSGI - response headers (``(name, value)`` tuples), as described in :pep:`333`. The + response headers (``(name, value)`` tuples), as described in :pep:`3333`. The default list just sets the content type to ``text/plain``. .. attribute:: BaseHandler.error_body - The error response body. This should be an HTTP response body string. It + The error response body. This should be an HTTP response body bytestring. It defaults to the plain text, "A server error occurred. Please contact the administrator." - Methods and attributes for :pep:`333`'s "Optional Platform-Specific File + Methods and attributes for :pep:`3333`'s "Optional Platform-Specific File Handling" feature: @@ -696,6 +722,24 @@ input, output, and error streams. version of the response set to the client. It defaults to ``"1.0"``. +.. function:: read_environ() + + Transcode CGI variables from ``os.environ`` to PEP 3333 "bytes in unicode" + strings, returning a new dictionary. This function is used by + :class:`CGIHandler` and :class:`IISCGIHandler` in place of directly using + ``os.environ``, which is not necessarily WSGI-compliant on all platforms + and web servers using Python 3 -- specifically, ones where the OS's + actual environment is Unicode (i.e. Windows), or ones where the environment + is bytes, but the system encoding used by Python to decode it is anything + other than ISO-8859-1 (e.g. Unix systems using UTF-8). + + If you are implementing a CGI-based handler of your own, you probably want + to use this routine instead of just copying values out of ``os.environ`` + directly. + + .. versionadded:: 3.2 + + Examples -------- diff --git a/Doc/library/xdrlib.rst b/Doc/library/xdrlib.rst index 4fcb9cd..1d3da0a 100644 --- a/Doc/library/xdrlib.rst +++ b/Doc/library/xdrlib.rst @@ -9,6 +9,10 @@ single: XDR single: External Data Representation +**Source code:** :source:`Lib/xdrlib.py` + +-------------- + The :mod:`xdrlib` module supports the External Data Representation Standard as described in :rfc:`1014`, written by Sun Microsystems, Inc. June 1987. It supports most of the data types described in the RFC. diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst index 8364f35..070967b 100644 --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -7,6 +7,9 @@ .. sectionauthor:: Paul Prescod <paul@prescod.net> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/xml/dom/minidom.py` + +-------------- :mod:`xml.dom.minidom` is a light-weight implementation of the Document Object Model interface. It is intended to be simpler than the full DOM and also @@ -82,22 +85,12 @@ document: the one that holds all others. Here is an example program:: dom3 = parseString("<myxml>Some data</myxml>") assert dom3.documentElement.tagName == "myxml" -When you are finished with a DOM, you should clean it up. This is necessary -because some versions of Python do not support garbage collection of objects -that refer to each other in a cycle. Until this restriction is removed from all -versions of Python, it is safest to write your code as if cycles would not be -cleaned up. - -The way to clean up a DOM is to call its :meth:`unlink` method:: - - dom1.unlink() - dom2.unlink() - dom3.unlink() - -:meth:`unlink` is a :mod:`xml.dom.minidom`\ -specific extension to the DOM API. -After calling :meth:`unlink` on a node, the node and its descendants are -essentially useless. - +When you are finished with a DOM tree, you may optionally call the +:meth:`unlink` method to encourage early cleanup of the now-unneeded +objects. :meth:`unlink` is a :mod:`xml.dom.minidom`\ -specific +extension to the DOM API that renders the node and its descendants are +essentially useless. Otherwise, Python's garbage collector will +eventually take care of the objects in the tree. .. seealso:: @@ -124,6 +117,13 @@ module documentation. This section lists the differences between the API and to be called on the :class:`Document` object, but may be called on child nodes to discard children of that node. + You can avoid calling this method explicitly by using the :keyword:`with` + statement. The following code will automatically unlink *dom* when the + :keyword:`with` block is exited:: + + with xml.dom.minidom.parse(datasource) as dom: + ... # Work with dom. + .. method:: Node.writexml(writer, indent="", addindent="", newl="") @@ -139,18 +139,20 @@ module documentation. This section lists the differences between the API and .. method:: Node.toxml(encoding=None) - Return the XML that the DOM represents as a string. - - With no argument, the XML header does not specify an encoding, and the result is - Unicode string if the default encoding cannot represent all characters in the - document. Encoding this string in an encoding other than UTF-8 is likely - incorrect, since UTF-8 is the default encoding of XML. + Return a string or byte string containing the XML represented by + the DOM node. - With an explicit *encoding* [1]_ argument, the result is a byte string in the - specified encoding. It is recommended that this argument is always specified. To - avoid :exc:`UnicodeError` exceptions in case of unrepresentable text data, the - encoding argument should be specified as "utf-8". + With an explicit *encoding* [1]_ argument, the result is a byte + string in the specified encoding. It is recommended that you + always specify an encoding; you may use any encoding you like, but + an argument of "utf-8" is the most common choice, avoiding + :exc:`UnicodeError` exceptions in case of unrepresentable text + data. + With no *encoding* argument, the result is a Unicode string, and the + XML declaration in the resulting string does not specify an + encoding. Encoding this string in an encoding other than UTF-8 is + likely incorrect, since UTF-8 is the default encoding of XML. .. method:: Node.toprettyxml(indent="", newl="", encoding="") @@ -158,7 +160,8 @@ module documentation. This section lists the differences between the API and indentation string and defaults to a tabulator; *newl* specifies the string emitted at the end of each line and defaults to ``\n``. - There's also an *encoding* argument; see :meth:`toxml`. + The *encoding* argument behaves like the corresponding argument of + :meth:`toxml`. .. _dom-example: @@ -243,7 +246,9 @@ utility to most DOM users. .. rubric:: Footnotes -.. [#] The encoding string included in XML output should conform to the - appropriate standards. For example, "UTF-8" is valid, but "UTF8" is - not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl +.. [#] The encoding name included in the XML output should conform to + the appropriate standards. For example, "UTF-8" is valid, but + "UTF8" is not valid in an XML document's declaration, even though + Python accepts it as an encoding name. + See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl and http://www.iana.org/assignments/character-sets . diff --git a/Doc/library/xml.dom.pulldom.rst b/Doc/library/xml.dom.pulldom.rst index 1893fba..4a5ef4c 100644 --- a/Doc/library/xml.dom.pulldom.rst +++ b/Doc/library/xml.dom.pulldom.rst @@ -5,6 +5,9 @@ :synopsis: Support for building partial DOM trees from SAX events. .. moduleauthor:: Paul Prescod <paul@prescod.net> +**Source code:** :source:`Lib/xml/dom/pulldom.py` + +-------------- :mod:`xml.dom.pulldom` allows building only selected portions of a Document Object Model representation of a document from SAX events. diff --git a/Doc/library/xml.dom.rst b/Doc/library/xml.dom.rst index 45d30d6..c1786a6 100644 --- a/Doc/library/xml.dom.rst +++ b/Doc/library/xml.dom.rst @@ -693,18 +693,27 @@ Attr Objects .. attribute:: Attr.name - The attribute name. In a namespace-using document it may have colons in it. + The attribute name. + In a namespace-using document it may include a colon. .. attribute:: Attr.localName - The part of the name following the colon if there is one, else the entire name. + The part of the name following the colon if there is one, else the + entire name. This is a read-only attribute. .. attribute:: Attr.prefix - The part of the name preceding the colon if there is one, else the empty string. + The part of the name preceding the colon if there is one, else the + empty string. + + +.. attribute:: Attr.value + + The text value of the attribute. This is a synonym for the + :attr:`nodeValue` attribute. .. _dom-attributelist-objects: @@ -967,29 +976,24 @@ Python. Type Mapping ^^^^^^^^^^^^ -The primitive IDL types used in the DOM specification are mapped to Python types +The IDL types used in the DOM specification are mapped to Python types according to the following table. +------------------+-------------------------------------------+ | IDL Type | Python Type | +==================+===========================================+ -| ``boolean`` | ``IntegerType`` (with a value of ``0`` or | -| | ``1``) | +| ``boolean`` | ``bool`` or ``int`` | +------------------+-------------------------------------------+ -| ``int`` | ``IntegerType`` | +| ``int`` | ``int`` | +------------------+-------------------------------------------+ -| ``long int`` | ``IntegerType`` | +| ``long int`` | ``int`` | +------------------+-------------------------------------------+ -| ``unsigned int`` | ``IntegerType`` | +| ``unsigned int`` | ``int`` | ++------------------+-------------------------------------------+ +| ``DOMString`` | ``str`` or ``bytes`` | ++------------------+-------------------------------------------+ +| ``null`` | ``None`` | +------------------+-------------------------------------------+ - -Additionally, the :class:`DOMString` defined in the recommendation is mapped to -a bytes or string object. Applications should be able to handle -Unicode whenever a string is returned from the DOM. - -The IDL ``null`` value is mapped to ``None``, which may be accepted or -provided by the implementation whenever ``null`` is allowed by the API. - .. _dom-accessor-methods: diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst index bc04a53..18c35aa 100644 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -5,10 +5,13 @@ :synopsis: Implementation of the ElementTree API. .. moduleauthor:: Fredrik Lundh <fredrik@pythonware.com> +**Source code:** :source:`Lib/xml/etree/ElementTree.py` -The Element type is a flexible container object, designed to store hierarchical -data structures in memory. The type can be described as a cross between a list -and a dictionary. +-------------- + +The :class:`Element` type is a flexible container object, designed to store +hierarchical data structures in memory. The type can be described as a cross +between a list and a dictionary. Each element has a number of properties associated with it: @@ -23,7 +26,8 @@ Each element has a number of properties associated with it: * a number of child elements, stored in a Python sequence -To create an element instance, use the Element or SubElement factory functions. +To create an element instance, use the :class:`Element` constructor or the +:func:`SubElement` factory function. The :class:`ElementTree` class can be used to wrap an element structure, and convert it from and to XML. @@ -31,8 +35,14 @@ convert it from and to XML. A C implementation of this API is available as :mod:`xml.etree.cElementTree`. See http://effbot.org/zone/element-index.htm for tutorials and links to other -docs. Fredrik Lundh's page is also the location of the development version of the -xml.etree.ElementTree. +docs. Fredrik Lundh's page is also the location of the development version of +the xml.etree.ElementTree. + +.. versionchanged:: 3.2 + The ElementTree API is updated to 1.3. For more information, see + `Introducing ElementTree 1.3 + <http://effbot.org/zone/elementtree-13-intro.htm>`_. + .. _elementtree-functions: @@ -43,16 +53,16 @@ Functions .. function:: Comment(text=None) Comment element factory. This factory function creates a special element - that will be serialized as an XML comment. The comment string can be either - an ASCII-only :class:`bytes` object or a :class:`str` object. *text* is a - string containing the comment string. Returns an element instance + that will be serialized as an XML comment by the standard serializer. The + comment string can be either a bytestring or a Unicode string. *text* is a + string containing the comment string. Returns an element instance representing a comment. .. function:: dump(elem) - Writes an element tree or element structure to sys.stdout. This function should - be used for debugging only. + Writes an element tree or element structure to sys.stdout. This function + should be used for debugging only. The exact output format is implementation dependent. In this version, it's written as an ordinary XML file. @@ -60,39 +70,36 @@ Functions *elem* is an element tree or an individual element. -.. function:: Element(tag, attrib={}, **extra) +.. function:: fromstring(text) - Element factory. This function returns an object implementing the standard - Element interface. The exact class or type of that object is implementation - dependent, but it will always be compatible with the _ElementInterface class in - this module. + Parses an XML section from a string constant. Same as :func:`XML`. *text* + is a string containing XML data. Returns an :class:`Element` instance. - The element name, attribute names, and attribute values can be either an - ASCII-only :class:`bytes` object or a :class:`str` object. *tag* is the - element name. *attrib* is an optional dictionary, containing element - attributes. *extra* contains additional attributes, given as keyword - arguments. Returns an element instance. +.. function:: fromstringlist(sequence, parser=None) -.. function:: fromstring(text) + Parses an XML document from a sequence of string fragments. *sequence* is a + list or other sequence containing XML data fragments. *parser* is an + optional parser instance. If not given, the standard :class:`XMLParser` + parser is used. Returns an :class:`Element` instance. - Parses an XML section from a string constant. Same as XML. *text* is a string - containing XML data. Returns an Element instance. + .. versionadded:: 3.2 .. function:: iselement(element) - Checks if an object appears to be a valid element object. *element* is an - element instance. Returns a true value if this is an element object. + Checks if an object appears to be a valid element object. *element* is an + element instance. Returns a true value if this is an element object. -.. function:: iterparse(source, events=None) +.. function:: iterparse(source, events=None, parser=None) Parses an XML section into an element tree incrementally, and reports what's going on to the user. *source* is a filename or :term:`file object` containing XML data. *events* is a list of events to report back. If omitted, only "end" - events are reported. Returns an :term:`iterator` providing ``(event, elem)`` - pairs. + events are reported. *parser* is an optional parser instance. If not + given, the standard :class:`XMLParser` parser is used. Returns an + :term:`iterator` providing ``(event, elem)`` pairs. .. note:: @@ -107,196 +114,271 @@ Functions .. function:: parse(source, parser=None) - Parses an XML section into an element tree. *source* is a filename or file - object containing XML data. *parser* is an optional parser instance. If not - given, the standard XMLTreeBuilder parser is used. Returns an ElementTree - instance. + Parses an XML section into an element tree. *source* is a filename or file + object containing XML data. *parser* is an optional parser instance. If + not given, the standard :class:`XMLParser` parser is used. Returns an + :class:`ElementTree` instance. .. function:: ProcessingInstruction(target, text=None) - PI element factory. This factory function creates a special element that will - be serialized as an XML processing instruction. *target* is a string containing - the PI target. *text* is a string containing the PI contents, if given. Returns - an element instance, representing a processing instruction. + PI element factory. This factory function creates a special element that + will be serialized as an XML processing instruction. *target* is a string + containing the PI target. *text* is a string containing the PI contents, if + given. Returns an element instance, representing a processing instruction. + + +.. function:: register_namespace(prefix, uri) + + Registers a namespace prefix. The registry is global, and any existing + mapping for either the given prefix or the namespace URI will be removed. + *prefix* is a namespace prefix. *uri* is a namespace uri. Tags and + attributes in this namespace will be serialized with the given prefix, if at + all possible. + + .. versionadded:: 3.2 .. function:: SubElement(parent, tag, attrib={}, **extra) - Subelement factory. This function creates an element instance, and appends it - to an existing element. + Subelement factory. This function creates an element instance, and appends + it to an existing element. + + The element name, attribute names, and attribute values can be either + bytestrings or Unicode strings. *parent* is the parent element. *tag* is + the subelement name. *attrib* is an optional dictionary, containing element + attributes. *extra* contains additional attributes, given as keyword + arguments. Returns an element instance. + + +.. function:: tostring(element, encoding="us-ascii", method="xml") + + Generates a string representation of an XML element, including all + subelements. *element* is an :class:`Element` instance. *encoding* [1]_ is + the output encoding (default is US-ASCII). Use ``encoding="unicode"`` to + generate a Unicode string. *method* is either ``"xml"``, + ``"html"`` or ``"text"`` (default is ``"xml"``). Returns an (optionally) + encoded string containing the XML data. - The element name, attribute names, and attribute values can be an ASCII-only - :class:`bytes` object or a :class:`str` object. *parent* is the parent - element. *tag* is the subelement name. *attrib* is an optional dictionary, - containing element attributes. *extra* contains additional attributes, given - as keyword arguments. Returns an element instance. +.. function:: tostringlist(element, encoding="us-ascii", method="xml") -.. function:: tostring(element, encoding=None) + Generates a string representation of an XML element, including all + subelements. *element* is an :class:`Element` instance. *encoding* [1]_ is + the output encoding (default is US-ASCII). Use ``encoding="unicode"`` to + generate a Unicode string. *method* is either ``"xml"``, + ``"html"`` or ``"text"`` (default is ``"xml"``). Returns a list of + (optionally) encoded strings containing the XML data. It does not guarantee + any specific sequence, except that ``"".join(tostringlist(element)) == + tostring(element)``. - Generates a string representation of an XML element, including all subelements. - *element* is an Element instance. *encoding* is the output encoding (default is - US-ASCII). Returns an encoded string containing the XML data. + .. versionadded:: 3.2 -.. function:: XML(text) +.. function:: XML(text, parser=None) Parses an XML section from a string constant. This function can be used to - embed "XML literals" in Python code. *text* is a string containing XML data. - Returns an Element instance. + embed "XML literals" in Python code. *text* is a string containing XML + data. *parser* is an optional parser instance. If not given, the standard + :class:`XMLParser` parser is used. Returns an :class:`Element` instance. -.. function:: XMLID(text) +.. function:: XMLID(text, parser=None) Parses an XML section from a string constant, and also returns a dictionary - which maps from element id:s to elements. *text* is a string containing XML - data. Returns a tuple containing an Element instance and a dictionary. + which maps from element id:s to elements. *text* is a string containing XML + data. *parser* is an optional parser instance. If not given, the standard + :class:`XMLParser` parser is used. Returns a tuple containing an + :class:`Element` instance and a dictionary. + + +.. _elementtree-element-objects: + +Element Objects +--------------- + + +.. class:: Element(tag, attrib={}, **extra) + + Element class. This class defines the Element interface, and provides a + reference implementation of this interface. + + The element name, attribute names, and attribute values can be either + bytestrings or Unicode strings. *tag* is the element name. *attrib* is + an optional dictionary, containing element attributes. *extra* contains + additional attributes, given as keyword arguments. + + + .. attribute:: tag + + A string identifying what kind of data this element represents (the + element type, in other words). + + + .. attribute:: text + The *text* attribute can be used to hold additional data associated with + the element. As the name implies this attribute is usually a string but + may be any application-specific object. If the element is created from + an XML file the attribute will contain any text found between the element + tags. -.. _elementtree-element-interface: -The Element Interface ---------------------- + .. attribute:: tail -Element objects returned by Element or SubElement have the following methods -and attributes. + The *tail* attribute can be used to hold additional data associated with + the element. This attribute is usually a string but may be any + application-specific object. If the element is created from an XML file + the attribute will contain any text found after the element's end tag and + before the next tag. -.. attribute:: Element.tag + .. attribute:: attrib - A string identifying what kind of data this element represents (the element - type, in other words). + A dictionary containing the element's attributes. Note that while the + *attrib* value is always a real mutable Python dictionary, an ElementTree + implementation may choose to use another internal representation, and + create the dictionary only if someone asks for it. To take advantage of + such implementations, use the dictionary methods below whenever possible. + The following dictionary-like methods work on the element attributes. -.. attribute:: Element.text - The *text* attribute can be used to hold additional data associated with the - element. As the name implies this attribute is usually a string but may be any - application-specific object. If the element is created from an XML file the - attribute will contain any text found between the element tags. + .. method:: clear() + Resets an element. This function removes all subelements, clears all + attributes, and sets the text and tail attributes to None. -.. attribute:: Element.tail - The *tail* attribute can be used to hold additional data associated with the - element. This attribute is usually a string but may be any application-specific - object. If the element is created from an XML file the attribute will contain - any text found after the element's end tag and before the next tag. + .. method:: get(key, default=None) + Gets the element attribute named *key*. -.. attribute:: Element.attrib + Returns the attribute value, or *default* if the attribute was not found. - A dictionary containing the element's attributes. Note that while the *attrib* - value is always a real mutable Python dictionary, an ElementTree implementation - may choose to use another internal representation, and create the dictionary - only if someone asks for it. To take advantage of such implementations, use the - dictionary methods below whenever possible. -The following dictionary-like methods work on the element attributes. + .. method:: items() + Returns the element attributes as a sequence of (name, value) pairs. The + attributes are returned in an arbitrary order. -.. method:: Element.clear() - Resets an element. This function removes all subelements, clears all - attributes, and sets the text and tail attributes to None. + .. method:: keys() + Returns the elements attribute names as a list. The names are returned + in an arbitrary order. -.. method:: Element.get(key, default=None) - Gets the element attribute named *key*. + .. method:: set(key, value) - Returns the attribute value, or *default* if the attribute was not found. + Set the attribute *key* on the element to *value*. + The following methods work on the element's children (subelements). -.. method:: Element.items() - Returns the element attributes as a sequence of (name, value) pairs. The - attributes are returned in an arbitrary order. + .. method:: append(subelement) + Adds the element *subelement* to the end of this elements internal list + of subelements. -.. method:: Element.keys() - Returns the elements attribute names as a list. The names are returned in an - arbitrary order. + .. method:: extend(subelements) + Appends *subelements* from a sequence object with zero or more elements. + Raises :exc:`AssertionError` if a subelement is not a valid object. -.. method:: Element.set(key, value) + .. versionadded:: 3.2 - Set the attribute *key* on the element to *value*. -The following methods work on the element's children (subelements). + .. method:: find(match) + Finds the first subelement matching *match*. *match* may be a tag name + or path. Returns an element instance or ``None``. -.. method:: Element.append(subelement) - Adds the element *subelement* to the end of this elements internal list of - subelements. + .. method:: findall(match) + Finds all matching subelements, by tag name or path. Returns a list + containing all matching elements in document order. -.. method:: Element.find(match) - Finds the first subelement matching *match*. *match* may be a tag name or path. - Returns an element instance or ``None``. + .. method:: findtext(match, default=None) + Finds text for the first subelement matching *match*. *match* may be + a tag name or path. Returns the text content of the first matching + element, or *default* if no element was found. Note that if the matching + element has no text content an empty string is returned. -.. method:: Element.findall(match) - Finds all subelements matching *match*. *match* may be a tag name or path. - Returns an iterable yielding all matching elements in document order. + .. method:: getchildren() + .. deprecated:: 3.2 + Use ``list(elem)`` or iteration. -.. method:: Element.findtext(condition, default=None) - Finds text for the first subelement matching *condition*. *condition* may be a - tag name or path. Returns the text content of the first matching element, or - *default* if no element was found. Note that if the matching element has no - text content an empty string is returned. + .. method:: getiterator(tag=None) + + .. deprecated:: 3.2 + Use method :meth:`Element.iter` instead. + + + .. method:: insert(index, element) + + Inserts a subelement at the given position in this element. + + + .. method:: iter(tag=None) + Creates a tree :term:`iterator` with the current element as the root. + The iterator iterates over this element and all elements below it, in + document (depth first) order. If *tag* is not ``None`` or ``'*'``, only + elements whose tag equals *tag* are returned from the iterator. If the + tree structure is modified during iteration, the result is undefined. -.. method:: Element.getchildren() - Returns all subelements. The elements are returned in document order. + .. method:: iterfind(match) + Finds all matching subelements, by tag name or path. Returns an iterable + yielding all matching elements in document order. -.. method:: Element.getiterator(tag=None) + .. versionadded:: 3.2 - Creates a tree iterator with the current element as the root. The iterator - iterates over this element and all elements below it, in document (depth first) - order. If *tag* is not ``None`` or ``'*'``, only elements whose tag equals - *tag* are returned from the iterator. + .. method:: itertext() -.. method:: Element.insert(index, element) + Creates a text iterator. The iterator loops over this element and all + subelements, in document order, and returns all inner text. - Inserts a subelement at the given position in this element. + .. versionadded:: 3.2 -.. method:: Element.makeelement(tag, attrib) + .. method:: makeelement(tag, attrib) - Creates a new element object of the same type as this element. Do not call this - method, use the SubElement factory function instead. + Creates a new element object of the same type as this element. Do not + call this method, use the :func:`SubElement` factory function instead. -.. method:: Element.remove(subelement) + .. method:: remove(subelement) - Removes *subelement* from the element. Unlike the findXYZ methods this method - compares elements based on the instance identity, not on tag value or contents. + Removes *subelement* from the element. Unlike the find\* methods this + method compares elements based on the instance identity, not on tag value + or contents. -Element objects also support the following sequence type methods for working -with subelements: :meth:`__delitem__`, :meth:`__getitem__`, :meth:`__setitem__`, -:meth:`__len__`. + :class:`Element` objects also support the following sequence type methods + for working with subelements: :meth:`__delitem__`, :meth:`__getitem__`, + :meth:`__setitem__`, :meth:`__len__`. -Caution: Because Element objects do not define a :meth:`__bool__` method, -elements with no subelements will test as ``False``. :: + Caution: Elements with no subelements will test as ``False``. This behavior + will change in future versions. Use specific ``len(elem)`` or ``elem is + None`` test instead. :: - element = root.find('foo') + element = root.find('foo') - if not element: # careful! - print("element not found, or element has no subelements") + if not element: # careful! + print("element not found, or element has no subelements") - if element is None: - print("element not found") + if element is None: + print("element not found") .. _elementtree-elementtree-objects: @@ -307,50 +389,49 @@ ElementTree Objects .. class:: ElementTree(element=None, file=None) - ElementTree wrapper class. This class represents an entire element hierarchy, - and adds some extra support for serialization to and from standard XML. + ElementTree wrapper class. This class represents an entire element + hierarchy, and adds some extra support for serialization to and from + standard XML. - *element* is the root element. The tree is initialized with the contents of the - XML *file* if given. + *element* is the root element. The tree is initialized with the contents + of the XML *file* if given. .. method:: _setroot(element) Replaces the root element for this tree. This discards the current contents of the tree, and replaces it with the given element. Use with - care. *element* is an element instance. + care. *element* is an element instance. - .. method:: find(path) + .. method:: find(match) - Finds the first toplevel element with given tag. Same as - getroot().find(path). *path* is the element to look for. Returns the - first matching element, or ``None`` if no element was found. + Finds the first toplevel element matching *match*. *match* may be a tag + name or path. Same as getroot().find(match). Returns the first matching + element, or ``None`` if no element was found. - .. method:: findall(path) + .. method:: findall(match) - Finds all toplevel elements with the given tag. Same as - getroot().findall(path). *path* is the element to look for. Returns a - list or :term:`iterator` containing all matching elements, in document - order. + Finds all matching subelements, by tag name or path. Same as + getroot().findall(match). *match* may be a tag name or path. Returns a + list containing all matching elements, in document order. - .. method:: findtext(path, default=None) + .. method:: findtext(match, default=None) Finds the element text for the first toplevel element with given tag. - Same as getroot().findtext(path). *path* is the toplevel element to look - for. *default* is the value to return if the element was not - found. Returns the text content of the first matching element, or the - default value no element was found. Note that if the element has is - found, but has no text content, this method returns an empty string. + Same as getroot().findtext(match). *match* may be a tag name or path. + *default* is the value to return if the element was not found. Returns + the text content of the first matching element, or the default value no + element was found. Note that if the element is found, but has no text + content, this method returns an empty string. .. method:: getiterator(tag=None) - Creates and returns a tree iterator for the root element. The iterator - loops over all elements in this tree, in section order. *tag* is the tag - to look for (default is to return all elements) + .. deprecated:: 3.2 + Use method :meth:`ElementTree.iter` instead. .. method:: getroot() @@ -358,19 +439,40 @@ ElementTree Objects Returns the root element for this tree. + .. method:: iter(tag=None) + + Creates and returns a tree iterator for the root element. The iterator + loops over all elements in this tree, in section order. *tag* is the tag + to look for (default is to return all elements) + + + .. method:: iterfind(match) + + Finds all matching subelements, by tag name or path. Same as + getroot().iterfind(match). Returns an iterable yielding all matching + elements in document order. + + .. versionadded:: 3.2 + + .. method:: parse(source, parser=None) Loads an external XML section into this element tree. *source* is a file name or :term:`file object`. *parser* is an optional parser instance. - If not given, the standard XMLTreeBuilder parser is used. Returns the section + If not given, the standard XMLParser parser is used. Returns the section root element. - .. method:: write(file, encoding=None) + .. method:: write(file, encoding="us-ascii", xml_declaration=None, method="xml") Writes the element tree to a file, as XML. *file* is a file name, or a :term:`file object` opened for writing. *encoding* [1]_ is the output encoding - (default is US-ASCII). + (default is US-ASCII). Use ``encoding="unicode"`` to write a Unicode string. + *xml_declaration* controls if an XML declaration + should be added to the file. Use False for never, True for always, None + for only if not US-ASCII or UTF-8 or Unicode (default is None). *method* is + either ``"xml"``, ``"html"`` or ``"text"`` (default is ``"xml"``). + Returns an (optionally) encoded string. This is the XML file that is going to be manipulated:: @@ -389,13 +491,13 @@ Example of changing the attribute "target" of every link in first paragraph:: >>> from xml.etree.ElementTree import ElementTree >>> tree = ElementTree() >>> tree.parse("index.xhtml") - <Element html at b7d3f1ec> + <Element 'html' at 0xb77e6fac> >>> p = tree.find("body/p") # Finds first occurrence of tag p in body >>> p - <Element p at 8416e0c> - >>> links = p.getiterator("a") # Returns list of all links + <Element 'p' at 0xb77ec26c> + >>> links = list(p.iter("a")) # Returns list of all links >>> links - [<Element a at b7d4f9ec>, <Element a at b7d4fb0c>] + [<Element 'a' at 0xb77ec2ac>, <Element 'a' at 0xb77ec1cc>] >>> for i in links: # Iterates through all found links ... i.attrib["target"] = "blank" >>> tree.write("output.xhtml") @@ -408,12 +510,12 @@ QName Objects .. class:: QName(text_or_uri, tag=None) - QName wrapper. This can be used to wrap a QName attribute value, in order to - get proper namespace handling on output. *text_or_uri* is a string containing - the QName value, in the form {uri}local, or, if the tag argument is given, the - URI part of a QName. If *tag* is given, the first argument is interpreted as an - URI, and this argument is interpreted as a local name. :class:`QName` instances - are opaque. + QName wrapper. This can be used to wrap a QName attribute value, in order + to get proper namespace handling on output. *text_or_uri* is a string + containing the QName value, in the form {uri}local, or, if the tag argument + is given, the URI part of a QName. If *tag* is given, the first argument is + interpreted as an URI, and this argument is interpreted as a local name. + :class:`QName` instances are opaque. .. _elementtree-treebuilder-objects: @@ -424,74 +526,89 @@ TreeBuilder Objects .. class:: TreeBuilder(element_factory=None) - Generic element structure builder. This builder converts a sequence of start, - data, and end method calls to a well-formed element structure. You can use this - class to build an element structure using a custom XML parser, or a parser for - some other XML-like format. The *element_factory* is called to create new - Element instances when given. + Generic element structure builder. This builder converts a sequence of + start, data, and end method calls to a well-formed element structure. You + can use this class to build an element structure using a custom XML parser, + or a parser for some other XML-like format. The *element_factory* is called + to create new :class:`Element` instances when given. .. method:: close() - Flushes the parser buffers, and returns the toplevel document - element. Returns an Element instance. + Flushes the builder buffers, and returns the toplevel document + element. Returns an :class:`Element` instance. .. method:: data(data) - Adds text to the current element. *data* is a string. This should be - either an ASCII-only :class:`bytes` object or a :class:`str` object. + Adds text to the current element. *data* is a string. This should be + either a bytestring, or a Unicode string. .. method:: end(tag) - Closes the current element. *tag* is the element name. Returns the closed - element. + Closes the current element. *tag* is the element name. Returns the + closed element. .. method:: start(tag, attrs) - Opens a new element. *tag* is the element name. *attrs* is a dictionary - containing element attributes. Returns the opened element. + Opens a new element. *tag* is the element name. *attrs* is a dictionary + containing element attributes. Returns the opened element. + + In addition, a custom :class:`TreeBuilder` object can provide the + following method: -.. _elementtree-xmltreebuilder-objects: + .. method:: doctype(name, pubid, system) + + Handles a doctype declaration. *name* is the doctype name. *pubid* is + the public identifier. *system* is the system identifier. This method + does not exist on the default :class:`TreeBuilder` class. + + .. versionadded:: 3.2 -XMLTreeBuilder Objects ----------------------- +.. _elementtree-xmlparser-objects: -.. class:: XMLTreeBuilder(html=0, target=None) +XMLParser Objects +----------------- - Element structure builder for XML source data, based on the expat parser. *html* - are predefined HTML entities. This flag is not supported by the current - implementation. *target* is the target object. If omitted, the builder uses an - instance of the standard TreeBuilder class. + +.. class:: XMLParser(html=0, target=None, encoding=None) + + :class:`Element` structure builder for XML source data, based on the expat + parser. *html* are predefined HTML entities. This flag is not supported by + the current implementation. *target* is the target object. If omitted, the + builder uses an instance of the standard TreeBuilder class. *encoding* [1]_ + is optional. If given, the value overrides the encoding specified in the + XML file. .. method:: close() - Finishes feeding data to the parser. Returns an element structure. + Finishes feeding data to the parser. Returns an element structure. .. method:: doctype(name, pubid, system) - Handles a doctype declaration. *name* is the doctype name. *pubid* is the - public identifier. *system* is the system identifier. + .. deprecated:: 3.2 + Define the :meth:`TreeBuilder.doctype` method on a custom TreeBuilder + target. .. method:: feed(data) - Feeds data to the parser. *data* is encoded data. + Feeds data to the parser. *data* is encoded data. -:meth:`XMLTreeBuilder.feed` calls *target*\'s :meth:`start` method +:meth:`XMLParser.feed` calls *target*\'s :meth:`start` method for each opening tag, its :meth:`end` method for each closing tag, -and data is processed by method :meth:`data`. :meth:`XMLTreeBuilder.close` +and data is processed by method :meth:`data`. :meth:`XMLParser.close` calls *target*\'s method :meth:`close`. -:class:`XMLTreeBuilder` can be used not only for building a tree structure. +:class:`XMLParser` can be used not only for building a tree structure. This is an example of counting the maximum depth of an XML file:: - >>> from xml.etree.ElementTree import XMLTreeBuilder + >>> from xml.etree.ElementTree import XMLParser >>> class MaxDepth: # The target object of the parser ... maxDepth = 0 ... depth = 0 @@ -507,7 +624,7 @@ This is an example of counting the maximum depth of an XML file:: ... return self.maxDepth ... >>> target = MaxDepth() - >>> parser = XMLTreeBuilder(target=target) + >>> parser = XMLParser(target=target) >>> exampleXml = """ ... <a> ... <b> @@ -527,7 +644,6 @@ This is an example of counting the maximum depth of an XML file:: .. rubric:: Footnotes .. [#] The encoding string included in XML output should conform to the - appropriate standards. For example, "UTF-8" is valid, but "UTF8" is - not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl + appropriate standards. For example, "UTF-8" is valid, but "UTF8" is + not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl and http://www.iana.org/assignments/character-sets. - diff --git a/Doc/library/xml.etree.rst b/Doc/library/xml.etree.rst deleted file mode 100644 index a944765..0000000 --- a/Doc/library/xml.etree.rst +++ /dev/null @@ -1,23 +0,0 @@ -:mod:`xml.etree` --- The ElementTree API for XML -================================================ - -.. module:: xml.etree - :synopsis: Package containing common ElementTree modules. -.. moduleauthor:: Fredrik Lundh <fredrik@pythonware.com> - - -The ElementTree package is a simple, efficient, and quite popular library for -XML manipulation in Python. The :mod:`xml.etree` package contains the most -common components from the ElementTree API library. In the current release, -this package contains the :mod:`ElementTree`, :mod:`ElementPath`, and -:mod:`ElementInclude` modules from the full ElementTree distribution. - -.. XXX To be continued! - - -.. seealso:: - - `ElementTree Overview <http://effbot.org/tag/elementtree>`_ - The home page for :mod:`ElementTree`. This includes links to additional - documentation, alternative implementations, and other add-ons. - diff --git a/Doc/library/xml.sax.utils.rst b/Doc/library/xml.sax.utils.rst index 95099f6..ff36fd8 100644 --- a/Doc/library/xml.sax.utils.rst +++ b/Doc/library/xml.sax.utils.rst @@ -50,13 +50,19 @@ or as base classes. using the reference concrete syntax. -.. class:: XMLGenerator(out=None, encoding='iso-8859-1') +.. class:: XMLGenerator(out=None, encoding='iso-8859-1', short_empty_elements=False) This class implements the :class:`ContentHandler` interface by writing SAX events back into an XML document. In other words, using an :class:`XMLGenerator` as the content handler will reproduce the original document being parsed. *out* should be a file-like object which will default to *sys.stdout*. *encoding* is the encoding of the output stream which defaults to ``'iso-8859-1'``. + *short_empty_elements* controls the formatting of elements that contain no + content: if *False* (the default) they are emitted as a pair of start/end + tags, if set to *True* they are emitted as a single self-closed tag. + + .. versionadded:: 3.2 + short_empty_elements .. class:: XMLFilterBase(base) diff --git a/Doc/library/xmlrpc.client.rst b/Doc/library/xmlrpc.client.rst index d25cbaf..a62021d 100644 --- a/Doc/library/xmlrpc.client.rst +++ b/Doc/library/xmlrpc.client.rst @@ -10,6 +10,10 @@ .. XXX Not everything is documented yet. It might be good to describe Marshaller, Unmarshaller, getparser, dumps, loads, and Transport. +**Source code:** :source:`Lib/xmlrpc/client.py` + +-------------- + XML-RPC is a Remote Procedure Call method that uses XML passed via HTTP as a transport. With it, a client can call methods with parameters on a remote server (the server is named by a URI) and get back structured data. This module diff --git a/Doc/library/xmlrpc.server.rst b/Doc/library/xmlrpc.server.rst index 3cb2c3a..67feba6 100644 --- a/Doc/library/xmlrpc.server.rst +++ b/Doc/library/xmlrpc.server.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Brian Quinlan <brianq@activestate.com> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/xmlrpc/server.py` + +-------------- The :mod:`xmlrpc.server` module provides a basic server framework for XML-RPC servers written in Python. Servers can either be free standing, using diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index 827b6b1..3282054 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -6,6 +6,10 @@ .. moduleauthor:: James C. Ahlstrom <jim@interet.com> .. sectionauthor:: James C. Ahlstrom <jim@interet.com> +**Source code:** :source:`Lib/zipfile.py` + +-------------- + The ZIP file format is a common archive and compression standard. This module provides tools to create, read, write, append, and list a ZIP file. Any advanced use of this module will require an understanding of the format, as @@ -24,10 +28,18 @@ For other archive formats, see the :mod:`bz2`, :mod:`gzip`, and The module defines the following items: -.. exception:: BadZipfile +.. exception:: BadZipFile The error raised for bad ZIP files (old name: ``zipfile.error``). + .. versionadded:: 3.2 + + +.. exception:: BadZipfile + + This is an alias for :exc:`BadZipFile` that exists for compatibility with + Python versions prior to 3.2. Usage is deprecated. + .. exception:: LargeZipFile @@ -43,6 +55,7 @@ The module defines the following items: .. class:: PyZipFile + :noindex: Class for creating ZIP archives containing Python libraries. @@ -101,30 +114,37 @@ ZipFile Objects Open a ZIP file, where *file* can be either a path to a file (a string) or a file-like object. The *mode* parameter should be ``'r'`` to read an existing file, ``'w'`` to truncate and write a new file, or ``'a'`` to append to an - existing file. If *mode* is ``'a'`` and *file* refers to an existing ZIP file, - then additional files are added to it. If *file* does not refer to a ZIP file, - then a new ZIP archive is appended to the file. This is meant for adding a ZIP - archive to another file, such as :file:`python.exe`. Using :: - - cat myzip.zip >> python.exe - - also works, and at least :program:`WinZip` can read such files. If *mode* is - ``a`` and the file does not exist at all, it is created. *compression* is the - ZIP compression method to use when writing the archive, and should be - :const:`ZIP_STORED` or :const:`ZIP_DEFLATED`; unrecognized values will cause - :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED` is specified but the - :mod:`zlib` module is not available, :exc:`RuntimeError` is also raised. The - default is :const:`ZIP_STORED`. If *allowZip64* is ``True`` zipfile will create - ZIP files that use the ZIP64 extensions when the zipfile is larger than 2 GB. If - it is false (the default) :mod:`zipfile` will raise an exception when the ZIP - file would require ZIP64 extensions. ZIP64 extensions are disabled by default - because the default :program:`zip` and :program:`unzip` commands on Unix (the - InfoZIP utilities) don't support these extensions. + existing file. If *mode* is ``'a'`` and *file* refers to an existing ZIP + file, then additional files are added to it. If *file* does not refer to a + ZIP file, then a new ZIP archive is appended to the file. This is meant for + adding a ZIP archive to another file (such as :file:`python.exe`). If + *mode* is ``a`` and the file does not exist at all, it is created. + *compression* is the ZIP compression method to use when writing the archive, + and should be :const:`ZIP_STORED` or :const:`ZIP_DEFLATED`; unrecognized + values will cause :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED` + is specified but the :mod:`zlib` module is not available, :exc:`RuntimeError` + is also raised. The default is :const:`ZIP_STORED`. If *allowZip64* is + ``True`` zipfile will create ZIP files that use the ZIP64 extensions when + the zipfile is larger than 2 GB. If it is false (the default) :mod:`zipfile` + will raise an exception when the ZIP file would require ZIP64 extensions. + ZIP64 extensions are disabled by default because the default :program:`zip` + and :program:`unzip` commands on Unix (the InfoZIP utilities) don't support + these extensions. If the file is created with mode ``'a'`` or ``'w'`` and then :meth:`close`\ d without adding any files to the archive, the appropriate ZIP structures for an empty archive will be written to the file. + ZipFile is also a context manager and therefore supports the + :keyword:`with` statement. In the example, *myzip* is closed after the + :keyword:`with` statement's suite is finished---even if an exception occurs:: + + with ZipFile('spam.zip', 'w') as myzip: + myzip.write('eggs.txt') + + .. versionadded:: 3.2 + Added the ability to use :class:`ZipFile` as a context manager. + .. method:: ZipFile.close() @@ -263,7 +283,7 @@ ZipFile Objects byte, the name of the file in the archive will be truncated at the null byte. -.. method:: ZipFile.writestr(zinfo_or_arcname, bytes) +.. method:: ZipFile.writestr(zinfo_or_arcname, bytes[, compress_type]) Write the string *bytes* to the archive; *zinfo_or_arcname* is either the file name it will be given in the archive, or a :class:`ZipInfo` instance. If it's @@ -273,6 +293,10 @@ ZipFile Objects created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling :meth:`writestr` on a closed ZipFile will raise a :exc:`RuntimeError`. + If given, *compress_type* overrides the value given for the *compression* + parameter to the constructor for the new entry, or in the *zinfo_or_arcname* + (if that is a :class:`ZipInfo` instance). + .. note:: When passing a :class:`ZipInfo` instance as the *zinfo_or_arcname* parameter, @@ -280,6 +304,9 @@ ZipFile Objects member of the given :class:`ZipInfo` instance. By default, the :class:`ZipInfo` constructor sets this member to :const:`ZIP_STORED`. + .. versionchanged:: 3.2 + The *compression_type* argument. + The following data attributes are also available: @@ -296,37 +323,53 @@ The following data attributes are also available: string no longer than 65535 bytes. Comments longer than this will be truncated in the written archive when :meth:`ZipFile.close` is called. + .. _pyzipfile-objects: PyZipFile Objects ----------------- The :class:`PyZipFile` constructor takes the same parameters as the -:class:`ZipFile` constructor. Instances have one method in addition to those of -:class:`ZipFile` objects. - - -.. method:: PyZipFile.writepy(pathname, basename='') - - Search for files :file:`\*.py` and add the corresponding file to the archive. - The corresponding file is a :file:`\*.pyo` file if available, else a - :file:`\*.pyc` file, compiling if necessary. If the pathname is a file, the - filename must end with :file:`.py`, and just the (corresponding - :file:`\*.py[co]`) file is added at the top level (no path information). If the - pathname is a file that does not end with :file:`.py`, a :exc:`RuntimeError` - will be raised. If it is a directory, and the directory is not a package - directory, then all the files :file:`\*.py[co]` are added at the top level. If - the directory is a package directory, then all :file:`\*.py[co]` are added under - the package name as a file path, and if any subdirectories are package - directories, all of these are added recursively. *basename* is intended for - internal use only. The :meth:`writepy` method makes archives with file names - like this:: - - string.pyc # Top level name - test/__init__.pyc # Package directory - test/testall.pyc # Module test.testall - test/bogus/__init__.pyc # Subpackage directory - test/bogus/myfile.pyc # Submodule test.bogus.myfile +:class:`ZipFile` constructor, and one additional parameter, *optimize*. + +.. class:: PyZipFile(file, mode='r', compression=ZIP_STORED, allowZip64=False, \ + optimize=-1) + + .. versionadded:: 3.2 + The *optimize* parameter. + + Instances have one method in addition to those of :class:`ZipFile` objects: + + .. method:: PyZipFile.writepy(pathname, basename='') + + Search for files :file:`\*.py` and add the corresponding file to the + archive. + + If the *optimize* parameter to :class:`PyZipFile` was not given or ``-1``, + the corresponding file is a :file:`\*.pyo` file if available, else a + :file:`\*.pyc` file, compiling if necessary. + + If the *optimize* parameter to :class:`PyZipFile` was ``0``, ``1`` or + ``2``, only files with that optimization level (see :func:`compile`) are + added to the archive, compiling if necessary. + + If the pathname is a file, the filename must end with :file:`.py`, and + just the (corresponding :file:`\*.py[co]`) file is added at the top level + (no path information). If the pathname is a file that does not end with + :file:`.py`, a :exc:`RuntimeError` will be raised. If it is a directory, + and the directory is not a package directory, then all the files + :file:`\*.py[co]` are added at the top level. If the directory is a + package directory, then all :file:`\*.py[co]` are added under the package + name as a file path, and if any subdirectories are package directories, + all of these are added recursively. *basename* is intended for internal + use only. The :meth:`writepy` method makes archives with file names like + this:: + + string.pyc # Top level name + test/__init__.pyc # Package directory + test/testall.pyc # Module test.testall + test/bogus/__init__.pyc # Subpackage directory + test/bogus/myfile.pyc # Submodule test.bogus.myfile .. _zipinfo-objects: diff --git a/Doc/library/zipimport.rst b/Doc/library/zipimport.rst index 57ac1e4..2f0fa38 100644 --- a/Doc/library/zipimport.rst +++ b/Doc/library/zipimport.rst @@ -26,18 +26,20 @@ Any files may be present in the ZIP archive, but only files :file:`.py` and corresponding :file:`.pyc` or :file:`.pyo` file, meaning that if a ZIP archive doesn't contain :file:`.pyc` files, importing may be rather slow. +ZIP archives with an archive comment are currently not supported. + .. seealso:: `PKZIP Application Note <http://www.pkware.com/documents/casestudies/APPNOTE.TXT>`_ Documentation on the ZIP file format by Phil Katz, the creator of the format and algorithms used. - :pep:`0273` - Import Modules from Zip Archives + :pep:`273` - Import Modules from Zip Archives Written by James C. Ahlstrom, who also provided an implementation. Python 2.3 follows the specification in PEP 273, but uses an implementation written by Just van Rossum that uses the import hooks described in PEP 302. - :pep:`0302` - New Import Hooks + :pep:`302` - New Import Hooks The PEP to add the import hooks that help this module work. diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst index 862cf91..a7b8343 100644 --- a/Doc/library/zlib.rst +++ b/Doc/library/zlib.rst @@ -96,20 +96,24 @@ The available exception and functions in this module are: Decompresses the bytes in *data*, returning a bytes object containing the uncompressed data. The *wbits* parameter controls the size of the window - buffer. If *bufsize* is given, it is used as the initial size of the output + buffer, and is discussed further below. + If *bufsize* is given, it is used as the initial size of the output buffer. Raises the :exc:`error` exception if any error occurs. The absolute value of *wbits* is the base two logarithm of the size of the history buffer (the "window size") used when compressing data. Its absolute value should be between 8 and 15 for the most recent versions of the zlib library, larger values resulting in better compression at the expense of greater - memory usage. The default value is 15. When *wbits* is negative, the standard + memory usage. When decompressing a stream, *wbits* must not be smaller + than the size originally used to compress the stream; using a too-small + value will result in an exception. The default value is therefore the + highest value, 15. When *wbits* is negative, the standard :program:`gzip` header is suppressed. *bufsize* is the initial size of the buffer used to hold decompressed data. If more space is required, the buffer size will be increased as needed, so you don't have to get this value exactly right; tuning it will only save a few calls - to :cfunc:`malloc`. The default size is 16384. + to :c:func:`malloc`. The default size is 16384. .. function:: decompressobj([wbits]) |
