diff options
Diffstat (limited to 'Doc/library')
139 files changed, 13954 insertions, 1716 deletions
diff --git a/Doc/library/_thread.rst b/Doc/library/_thread.rst index 369e9cd..751c3e8 100644 --- a/Doc/library/_thread.rst +++ b/Doc/library/_thread.rst @@ -35,6 +35,9 @@ It defines the following constants and functions: Raised on thread-specific errors. + .. versionchanged:: 3.3 + This is now a synonym of the built-in :exc:`RuntimeError`. + .. data:: LockType @@ -91,7 +94,7 @@ It defines the following constants and functions: *size* argument specifies the stack size to be used for subsequently created threads, and must be 0 (use platform or configured default) or a positive integer value of at least 32,768 (32kB). If changing the thread stack size is - unsupported, a :exc:`ThreadError` is raised. If the specified stack size is + unsupported, a :exc:`RuntimeError` is raised. If the specified stack size is invalid, a :exc:`ValueError` is raised and the stack size is unmodified. 32kB is currently the minimum supported stack size value to guarantee sufficient stack space for the interpreter itself. Note that some platforms may have diff --git a/Doc/library/abc.rst b/Doc/library/abc.rst index 1048b24..6f42222 100644 --- a/Doc/library/abc.rst +++ b/Doc/library/abc.rst @@ -18,7 +18,7 @@ regarding a type hierarchy for numbers based on ABCs.) The :mod:`collections` module has some concrete classes that derive from ABCs; these can, of course, be further derived. In addition the -:mod:`collections` module has some ABCs that can be used to test whether +:mod:`collections.abc` submodule has some ABCs that can be used to test whether a class or instance provides a particular interface, for example, is it hashable or a mapping. @@ -55,6 +55,9 @@ This module provides the following class: assert issubclass(tuple, MyABC) assert isinstance((), MyABC) + .. versionchanged:: 3.3 + Returns the registered subclass, to allow usage as a class decorator. + You can also override this method in an abstract base class: .. method:: __subclasshook__(subclass) @@ -124,19 +127,18 @@ This module provides the following class: available as a method of ``Foo``, so it is provided separately. -It also provides the following decorators: +The :mod:`abc` module also provides the following decorators: .. decorator:: abstractmethod(function) A decorator indicating abstract methods. - Using this decorator requires that the class's metaclass is :class:`ABCMeta` or - is derived from it. - A class that has a metaclass derived from :class:`ABCMeta` - cannot be instantiated unless all of its abstract methods and - properties are overridden. - The abstract methods can be called using any of the normal 'super' call - mechanisms. + Using this decorator requires that the class's metaclass is :class:`ABCMeta` + or is derived from it. A class that has a metaclass derived from + :class:`ABCMeta` cannot be instantiated unless all of its abstract methods + and properties are overridden. The abstract methods can be called using any + of the normal 'super' call mechanisms. :func:`abstractmethod` may be used + to declare abstract methods for properties and descriptors. Dynamically adding abstract methods to a class, or attempting to modify the abstraction status of a method or class once it is created, are not @@ -144,12 +146,52 @@ It also provides the following decorators: regular inheritance; "virtual subclasses" registered with the ABC's :meth:`register` method are not affected. - Usage:: + When :func:`abstractmethod` is applied in combination with other method + descriptors, it should be applied as the innermost decorator, as shown in + the following usage examples:: class C(metaclass=ABCMeta): @abstractmethod def my_abstract_method(self, ...): ... + @classmethod + @abstractmethod + def my_abstract_classmethod(cls, ...): + ... + @staticmethod + @abstractmethod + def my_abstract_staticmethod(...): + ... + + @property + @abstractmethod + def my_abstract_property(self): + ... + @my_abstract_property.setter + @abstractmethod + def my_abstract_property(self, val): + ... + + @abstractmethod + def _get_x(self): + ... + @abstractmethod + def _set_x(self, val): + ... + x = property(_get_x, _set_x) + + In order to correctly interoperate with the abstract base class machinery, + the descriptor must identify itself as abstract using + :attr:`__isabstractmethod__`. In general, this attribute should be ``True`` + if any of the methods used to compose the descriptor are abstract. For + example, Python's built-in property does the equivalent of:: + + class Descriptor: + ... + @property + def __isabstractmethod__(self): + return any(getattr(f, '__isabstractmethod__', False) for + f in (self._fget, self._fset, self._fdel)) .. note:: @@ -174,6 +216,8 @@ It also provides the following decorators: ... .. versionadded:: 3.2 + .. deprecated:: 3.3 + Use :class:`classmethod` with :func:`abstractmethod` instead .. decorator:: abstractstaticmethod(function) @@ -189,18 +233,19 @@ It also provides the following decorators: ... .. versionadded:: 3.2 + .. deprecated:: 3.3 + Use :class:`staticmethod` with :func:`abstractmethod` instead -.. function:: abstractproperty(fget=None, fset=None, fdel=None, doc=None) +.. decorator:: abstractproperty(fget=None, fset=None, fdel=None, doc=None) A subclass of the built-in :func:`property`, indicating an abstract property. - Using this function requires that the class's metaclass is :class:`ABCMeta` or - is derived from it. - A class that has a metaclass derived from :class:`ABCMeta` cannot be - instantiated unless all of its abstract methods and properties are overridden. - The abstract properties can be called using any of the normal - 'super' call mechanisms. + Using this function requires that the class's metaclass is :class:`ABCMeta` + or is derived from it. A class that has a metaclass derived from + :class:`ABCMeta` cannot be instantiated unless all of its abstract methods + and properties are overridden. The abstract properties can be called using + any of the normal 'super' call mechanisms. Usage:: @@ -217,6 +262,9 @@ It also provides the following decorators: def setx(self, value): ... x = abstractproperty(getx, setx) + .. deprecated:: 3.3 + Use :class:`property` with :func:`abstractmethod` instead + .. rubric:: Footnotes diff --git a/Doc/library/archiving.rst b/Doc/library/archiving.rst index 75d137c..c928494 100644 --- a/Doc/library/archiving.rst +++ b/Doc/library/archiving.rst @@ -5,8 +5,9 @@ Data Compression and Archiving ****************************** The modules described in this chapter support data compression with the zlib, -gzip, and bzip2 algorithms, and the creation of ZIP- and tar-format archives. -See also :ref:`archiving-operations` provided by the :mod:`shutil` module. +gzip, bzip2 and lzma algorithms, and the creation of ZIP- and tar-format +archives. See also :ref:`archiving-operations` provided by the :mod:`shutil` +module. .. toctree:: @@ -14,5 +15,6 @@ See also :ref:`archiving-operations` provided by the :mod:`shutil` module. zlib.rst gzip.rst bz2.rst + lzma.rst zipfile.rst tarfile.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 186e12d..8e7fa5e 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -368,16 +368,16 @@ formatter_class ^^^^^^^^^^^^^^^ :class:`ArgumentParser` objects allow the help formatting to be customized by -specifying an alternate formatting class. Currently, there are three such +specifying an alternate formatting class. Currently, there are four such classes: .. class:: RawDescriptionHelpFormatter RawTextHelpFormatter ArgumentDefaultsHelpFormatter + MetavarTypeHelpFormatter -The first two allow more control over how textual descriptions are displayed, -while the last automatically adds information about argument default values. - +:class:`RawDescriptionHelpFormatter` and :class:`RawTextHelpFormatter` give +more control over how textual descriptions are displayed. By default, :class:`ArgumentParser` objects line-wrap the description_ and epilog_ texts in command-line help messages:: @@ -430,8 +430,8 @@ should not be line-wrapped:: :class:`RawTextHelpFormatter` maintains whitespace for all sorts of help text, including argument descriptions. -The other formatter class available, :class:`ArgumentDefaultsHelpFormatter`, -will add information about the default value of each of the arguments:: +:class:`ArgumentDefaultsHelpFormatter` automatically adds information about +default values to each of the argument help messages:: >>> parser = argparse.ArgumentParser( ... prog='PROG', @@ -448,6 +448,25 @@ will add information about the default value of each of the arguments:: -h, --help show this help message and exit --foo FOO FOO! (default: 42) +:class:`MetavarTypeHelpFormatter` uses the name of the type_ argument for each +argument as the display name for its values (rather than using the dest_ +as the regular formatter does):: + + >>> parser = argparse.ArgumentParser( + ... prog='PROG', + ... formatter_class=argparse.MetavarTypeHelpFormatter) + >>> parser.add_argument('--foo', type=int) + >>> parser.add_argument('bar', type=float) + >>> parser.print_help() + usage: PROG [-h] [--foo int] float + + positional arguments: + float + + optional arguments: + -h, --help show this help message and exit + --foo int + conflict_handler ^^^^^^^^^^^^^^^^ diff --git a/Doc/library/array.rst b/Doc/library/array.rst index d563cce..3e275a2 100644 --- a/Doc/library/array.rst +++ b/Doc/library/array.rst @@ -14,36 +14,44 @@ them is constrained. The type is specified at object creation time by using a :dfn:`type code`, which is a single character. The following type codes are defined: -+-----------+----------------+-------------------+-----------------------+ -| Type code | C Type | Python Type | Minimum size in bytes | -+===========+================+===================+=======================+ -| ``'b'`` | signed char | int | 1 | -+-----------+----------------+-------------------+-----------------------+ -| ``'B'`` | unsigned char | int | 1 | -+-----------+----------------+-------------------+-----------------------+ -| ``'u'`` | Py_UNICODE | Unicode character | 2 (see note) | -+-----------+----------------+-------------------+-----------------------+ -| ``'h'`` | signed short | int | 2 | -+-----------+----------------+-------------------+-----------------------+ -| ``'H'`` | unsigned short | int | 2 | -+-----------+----------------+-------------------+-----------------------+ -| ``'i'`` | signed int | int | 2 | -+-----------+----------------+-------------------+-----------------------+ -| ``'I'`` | unsigned int | int | 2 | -+-----------+----------------+-------------------+-----------------------+ -| ``'l'`` | signed long | int | 4 | -+-----------+----------------+-------------------+-----------------------+ -| ``'L'`` | unsigned long | int | 4 | -+-----------+----------------+-------------------+-----------------------+ -| ``'f'`` | float | float | 4 | -+-----------+----------------+-------------------+-----------------------+ -| ``'d'`` | double | float | 8 | -+-----------+----------------+-------------------+-----------------------+ - -.. note:: - - The ``'u'`` typecode corresponds to Python's unicode character. On narrow - Unicode builds this is 2-bytes, on wide builds this is 4-bytes. ++-----------+--------------------+-------------------+-----------------------+-------+ +| Type code | C Type | Python Type | Minimum size in bytes | Notes | ++===========+====================+===================+=======================+=======+ +| ``'b'`` | signed char | int | 1 | | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'B'`` | unsigned char | int | 1 | | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'u'`` | Py_UCS4 | Unicode character | 4 | | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'h'`` | signed short | int | 2 | | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'H'`` | unsigned short | int | 2 | | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'i'`` | signed int | int | 2 | | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'I'`` | unsigned int | int | 2 | | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'l'`` | signed long | int | 4 | | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'L'`` | unsigned long | int | 4 | | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'q'`` | signed long long | int | 8 | \(1) | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'Q'`` | unsigned long long | int | 8 | \(1) | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'f'`` | float | float | 4 | | ++-----------+--------------------+-------------------+-----------------------+-------+ +| ``'d'`` | double | float | 8 | | ++-----------+--------------------+-------------------+-----------------------+-------+ + +Notes: + +(1) + The ``'q'`` and ``'Q'`` type codes are available only if + the platform C compiler used to build Python supports C :c:type:`long long`, + or, on Windows, :c:type:`__int64`. + + .. versionadded:: 3.3 The actual representation of values is determined by the machine architecture (strictly speaking, by the C implementation). The actual size can be accessed diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index e2c0b6d..16de3ca 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -96,9 +96,6 @@ Node classes Abstract Grammar ---------------- -The module defines a string constant ``__version__`` which is the decimal -Subversion revision number of the file shown below. - The abstract grammar is currently defined as follows: .. literalinclude:: ../../Parser/Python.asdl diff --git a/Doc/library/asynchat.rst b/Doc/library/asynchat.rst index 75b3cda..55c61d7 100644 --- a/Doc/library/asynchat.rst +++ b/Doc/library/asynchat.rst @@ -197,6 +197,9 @@ The :meth:`handle_request` method is called once all relevant input has been marshalled, after setting the channel terminator to ``None`` to ensure that any extraneous data sent by the web client are ignored. :: + + import asynchat + class http_request_handler(asynchat.async_chat): def __init__(self, sock, addr, sessions, log): diff --git a/Doc/library/asyncore.rst b/Doc/library/asyncore.rst index 619b7bb..7cacca1 100644 --- a/Doc/library/asyncore.rst +++ b/Doc/library/asyncore.rst @@ -184,12 +184,15 @@ any that have been added to the map during asynchronous service) is closed. Most of these are nearly identical to their socket partners. - .. method:: create_socket(family, type) + .. method:: create_socket(family=socket.AF_INET, type=socket.SOCK_STREAM) This is identical to the creation of a normal socket, and will use the same options for creation. Refer to the :mod:`socket` documentation for information on creating sockets. + .. versionchanged:: 3.3 + *family* and *type* arguments can be omitted. + .. method:: connect(address) @@ -274,13 +277,13 @@ asyncore Example basic HTTP client Here is a very basic HTTP client that uses the :class:`dispatcher` class to implement its socket handling:: - import asyncore, socket + import asyncore class HTTPClient(asyncore.dispatcher): def __init__(self, host, path): asyncore.dispatcher.__init__(self) - self.create_socket(socket.AF_INET, socket.SOCK_STREAM) + self.create_socket() self.connect( (host, 80) ) self.buffer = bytes('GET %s HTTP/1.0\r\nHost: %s\r\n\r\n' % (path, host), 'ascii') @@ -314,7 +317,6 @@ Here is a basic echo server that uses the :class:`dispatcher` class to accept connections and dispatches the incoming connections to a handler:: import asyncore - import socket class EchoHandler(asyncore.dispatcher_with_send): @@ -327,7 +329,7 @@ connections and dispatches the incoming connections to a handler:: def __init__(self, host, port): asyncore.dispatcher.__init__(self) - self.create_socket(socket.AF_INET, socket.SOCK_STREAM) + self.create_socket() self.set_reuse_addr() self.bind((host, port)) self.listen(5) @@ -338,4 +340,3 @@ connections and dispatches the incoming connections to a handler:: server = EchoServer('localhost', 8080) asyncore.loop() - diff --git a/Doc/library/atexit.rst b/Doc/library/atexit.rst index 7c76bab..3d5c014 100644 --- a/Doc/library/atexit.rst +++ b/Doc/library/atexit.rst @@ -67,8 +67,9 @@ automatically when the program terminates without relying on the application making an explicit call into this module at termination. :: try: - _count = int(open("/tmp/counter").read()) - except IOError: + with open("/tmp/counter") as infile: + _count = int(infile.read()) + except FileNotFoundError: _count = 0 def incrcounter(n): @@ -76,7 +77,8 @@ making an explicit call into this module at termination. :: _count = _count + n def savecounter(): - open("/tmp/counter", "w").write("%d" % _count) + with open("/tmp/counter", "w") as outfile: + outfile.write("%d" % _count) import atexit atexit.register(savecounter) diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index 06f3ab1..afbedce 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -18,9 +18,14 @@ POST request. The encoding algorithm is not the same as the There are two interfaces provided by this module. The modern interface supports encoding and decoding ASCII byte string objects using all three -alphabets. The legacy interface provides for encoding and decoding to and from -file-like objects as well as byte strings, but only using the Base64 standard -alphabet. +alphabets. Additionally, the decoding functions of the modern interface also +accept Unicode strings containing only ASCII characters. The legacy interface +provides for encoding and decoding to and from file-like objects as well as +byte strings, but only using the Base64 standard alphabet. + +.. versionchanged:: 3.3 + ASCII-only Unicode strings are now accepted by the decoding functions of + the modern interface. The modern interface provides: diff --git a/Doc/library/binascii.rst b/Doc/library/binascii.rst index 2aa3702..baf430d 100644 --- a/Doc/library/binascii.rst +++ b/Doc/library/binascii.rst @@ -20,8 +20,13 @@ higher-level modules. .. note:: - Encoding and decoding functions do not accept Unicode strings. Only bytestring - and bytearray objects can be processed. + ``a2b_*`` functions accept Unicode strings containing only ASCII characters. + Other functions only accept bytes and bytes-compatible objects (such as + bytearray objects and other objects implementing the buffer API). + + .. versionchanged:: 3.3 + ASCII-only unicode strings are now accepted by the ``a2b_*`` functions. + The :mod:`binascii` module defines the following functions: diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index d13f6e0..9577f31 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -1,198 +1,169 @@ -:mod:`bz2` --- Compression compatible with :program:`bzip2` -=========================================================== +:mod:`bz2` --- Support for :program:`bzip2` compression +======================================================= .. module:: bz2 - :synopsis: Interface to compression and decompression routines - compatible with bzip2. + :synopsis: Interfaces for bzip2 compression and decompression. .. moduleauthor:: Gustavo Niemeyer <niemeyer@conectiva.com> +.. moduleauthor:: Nadeem Vawda <nadeem.vawda@gmail.com> .. sectionauthor:: Gustavo Niemeyer <niemeyer@conectiva.com> +.. sectionauthor:: Nadeem Vawda <nadeem.vawda@gmail.com> -This module provides a comprehensive interface for the bz2 compression library. -It implements a complete file interface, one-shot (de)compression functions, and -types for sequential (de)compression. +This module provides a comprehensive interface for compressing and +decompressing data using the bzip2 compression algorithm. -Here is a summary of the features offered by the bz2 module: +The :mod:`bz2` module contains: -* :class:`BZ2File` class implements a complete file interface, including - :meth:`~BZ2File.readline`, :meth:`~BZ2File.readlines`, - :meth:`~BZ2File.writelines`, :meth:`~BZ2File.seek`, etc; +* The :class:`BZ2File` class for reading and writing compressed files. +* The :class:`BZ2Compressor` and :class:`BZ2Decompressor` classes for + incremental (de)compression. +* The :func:`compress` and :func:`decompress` functions for one-shot + (de)compression. -* :class:`BZ2File` class implements emulated :meth:`~BZ2File.seek` support; - -* :class:`BZ2File` class implements universal newline support; - -* :class:`BZ2File` class offers an optimized line iteration using a readahead - algorithm; - -* Sequential (de)compression supported by :class:`BZ2Compressor` and - :class:`BZ2Decompressor` classes; - -* One-shot (de)compression supported by :func:`compress` and :func:`decompress` - functions; - -* Thread safety uses individual locking mechanism. +All of the classes in this module may safely be accessed from multiple threads. (De)compression of files ------------------------ -Handling of compressed files is offered by the :class:`BZ2File` class. - - -.. class:: BZ2File(filename, mode='r', buffering=0, compresslevel=9) - - Open a bz2 file. Mode can be either ``'r'`` or ``'w'``, for reading (default) - or writing. When opened for writing, the file will be created if it doesn't - exist, and truncated otherwise. If *buffering* is given, ``0`` means - unbuffered, and larger numbers specify the buffer size; the default is - ``0``. If *compresslevel* is given, it must be a number between ``1`` and - ``9``; the default is ``9``. Add a ``'U'`` to mode to open the file for input - with universal newline support. Any line ending in the input file will be - seen as a ``'\n'`` in Python. Also, a file so opened gains the attribute - :attr:`newlines`; the value for this attribute is one of ``None`` (no newline - read yet), ``'\r'``, ``'\n'``, ``'\r\n'`` or a tuple containing all the - newline types seen. Universal newlines are available only when - reading. Instances support iteration in the same way as normal :class:`file` - instances. - - :class:`BZ2File` supports the :keyword:`with` statement. - - .. versionchanged:: 3.1 - Support for the :keyword:`with` statement was added. - - - .. note:: +.. class:: BZ2File(filename=None, mode='r', buffering=None, compresslevel=9, \*, fileobj=None) - This class does not support input files containing multiple streams (such - as those produced by the :program:`pbzip2` tool). When reading such an - input file, only the first stream will be accessible. If you require - support for multi-stream files, consider using the third-party - :mod:`bz2file` module (available from - `PyPI <http://pypi.python.org/pypi/bz2file>`_). This module provides a - backport of Python 3.3's :class:`BZ2File` class, which does support - multi-stream files. + Open a bzip2-compressed file. + The :class:`BZ2File` can wrap an existing :term:`file object` (given by + *fileobj*), or operate directly on a named file (named by *filename*). + Exactly one of these two parameters should be provided. - .. method:: close() + The *mode* argument can be either ``'r'`` for reading (default), ``'w'`` for + overwriting, or ``'a'`` for appending. If *fileobj* is provided, a mode of + ``'w'`` does not truncate the file, and is instead equivalent to ``'a'``. - Close the file. Sets data attribute :attr:`closed` to true. A closed file - cannot be used for further I/O operations. :meth:`close` may be called - more than once without error. + The *buffering* argument is ignored. Its use is deprecated. + If *mode* is ``'w'`` or ``'a'``, *compresslevel* can be a number between + ``1`` and ``9`` specifying the level of compression: ``1`` produces the + least compression, and ``9`` (default) produces the most compression. - .. method:: read([size]) + If *mode* is ``'r'``, the input file may be the concatenation of multiple + compressed streams. - Read at most *size* uncompressed bytes, returned as a byte string. If the - *size* argument is negative or omitted, read until EOF is reached. + :class:`BZ2File` provides all of the members specified by the + :class:`io.BufferedIOBase`, except for :meth:`detach` and :meth:`truncate`. + Iteration and the :keyword:`with` statement are supported. + :class:`BZ2File` also provides the following method: - .. method:: readline([size]) + .. method:: peek([n]) - Return the next line from the file, as a byte string, retaining newline. - A non-negative *size* argument limits the maximum number of bytes to - return (an incomplete line may be returned then). Return an empty byte - string at EOF. + Return buffered data without advancing the file position. At least one + byte of data will be returned (unless at EOF). The exact number of bytes + returned is unspecified. + .. versionadded:: 3.3 - .. method:: readlines([size]) + .. versionchanged:: 3.1 + Support for the :keyword:`with` statement was added. - Return a list of lines read. The optional *size* argument, if given, is an - approximate bound on the total number of bytes in the lines returned. + .. versionchanged:: 3.3 + The :meth:`fileno`, :meth:`readable`, :meth:`seekable`, :meth:`writable`, + :meth:`read1` and :meth:`readinto` methods were added. + .. versionchanged:: 3.3 + The *fileobj* argument to the constructor was added. - .. method:: seek(offset[, whence]) + .. versionchanged:: 3.3 + The ``'a'`` (append) mode was added, along with support for reading + multi-stream files. - Move to new file position. Argument *offset* is a byte count. Optional - argument *whence* defaults to ``os.SEEK_SET`` or ``0`` (offset from start - of file; offset should be ``>= 0``); other values are ``os.SEEK_CUR`` or - ``1`` (move relative to current position; offset can be positive or - negative), and ``os.SEEK_END`` or ``2`` (move relative to end of file; - offset is usually negative, although many platforms allow seeking beyond - the end of a file). - Note that seeking of bz2 files is emulated, and depending on the - parameters the operation may be extremely slow. +Incremental (de)compression +--------------------------- +.. class:: BZ2Compressor(compresslevel=9) - .. method:: tell() + Create a new compressor object. This object may be used to compress data + incrementally. For one-shot compression, use the :func:`compress` function + instead. - Return the current file position, an integer. + *compresslevel*, if given, must be a number between ``1`` and ``9``. The + default is ``9``. + .. method:: compress(data) - .. method:: write(data) + Provide data to the compressor object. Returns a chunk of compressed data + if possible, or an empty byte string otherwise. - Write the byte string *data* to file. Note that due to buffering, - :meth:`close` may be needed before the file on disk reflects the data - written. + When you have finished providing data to the compressor, call the + :meth:`flush` method to finish the compression process. - .. method:: writelines(sequence_of_byte_strings) + .. method:: flush() - Write the sequence of byte strings to the file. Note that newlines are not - added. The sequence can be any iterable object producing byte strings. - This is equivalent to calling write() for each byte string. + Finish the compression process. Returns the compressed data left in + internal buffers. + The compressor object may not be used after this method has been called. -Sequential (de)compression --------------------------- -Sequential compression and decompression is done using the classes -:class:`BZ2Compressor` and :class:`BZ2Decompressor`. +.. class:: BZ2Decompressor() + Create a new decompressor object. This object may be used to decompress data + incrementally. For one-shot compression, use the :func:`decompress` function + instead. -.. class:: BZ2Compressor(compresslevel=9) + .. note:: + This class does not transparently handle inputs containing multiple + compressed streams, unlike :func:`decompress` and :class:`BZ2File`. If + you need to decompress a multi-stream input with :class:`BZ2Decompressor`, + you must use a new decompressor for each stream. - Create a new compressor object. This object may be used to compress data - sequentially. If you want to compress data in one shot, use the - :func:`compress` function instead. The *compresslevel* parameter, if given, - must be a number between ``1`` and ``9``; the default is ``9``. + .. method:: decompress(data) - .. method:: compress(data) + Provide data to the decompressor object. Returns a chunk of decompressed + data if possible, or an empty byte string otherwise. - Provide more data to the compressor object. It will return chunks of - compressed data whenever possible. When you've finished providing data to - compress, call the :meth:`flush` method to finish the compression process, - and return what is left in internal buffers. + Attempting to decompress data after the end of the current stream is + reached raises an :exc:`EOFError`. If any data is found after the end of + the stream, it is ignored and saved in the :attr:`unused_data` attribute. - .. method:: flush() + .. attribute:: eof - Finish the compression process and return what is left in internal - buffers. You must not use the compressor object after calling this method. + True if the end-of-stream marker has been reached. + .. versionadded:: 3.3 -.. class:: BZ2Decompressor() - Create a new decompressor object. This object may be used to decompress data - sequentially. If you want to decompress data in one shot, use the - :func:`decompress` function instead. + .. attribute:: unused_data - .. method:: decompress(data) + Data found after the end of the compressed stream. - Provide more data to the decompressor object. It will return chunks of - decompressed data whenever possible. If you try to decompress data after - the end of stream is found, :exc:`EOFError` will be raised. If any data - was found after the end of stream, it'll be ignored and saved in - :attr:`unused_data` attribute. + If this attribute is accessed before the end of the stream has been + reached, its value will be ``b''``. One-shot (de)compression ------------------------ -One-shot compression and decompression is provided through the :func:`compress` -and :func:`decompress` functions. +.. function:: compress(data, compresslevel=9) + Compress *data*. -.. function:: compress(data, compresslevel=9) + *compresslevel*, if given, must be a number between ``1`` and ``9``. The + default is ``9``. - Compress *data* in one shot. If you want to compress data sequentially, use - an instance of :class:`BZ2Compressor` instead. The *compresslevel* parameter, - if given, must be a number between ``1`` and ``9``; the default is ``9``. + For incremental compression, use a :class:`BZ2Compressor` instead. .. function:: decompress(data) - Decompress *data* in one shot. If you want to decompress data sequentially, - use an instance of :class:`BZ2Decompressor` instead. + Decompress *data*. + + If *data* is the concatenation of multiple compressed streams, decompress + all of the streams. + + For incremental decompression, use a :class:`BZ2Decompressor` instead. + + .. versionchanged:: 3.3 + Support for multi-stream inputs was added. diff --git a/Doc/library/chunk.rst b/Doc/library/chunk.rst index d3558a4..c1ba497 100644 --- a/Doc/library/chunk.rst +++ b/Doc/library/chunk.rst @@ -84,8 +84,9 @@ instance will fail with a :exc:`EOFError` exception. Close and skip to the end of the chunk. This does not close the underlying file. - The remaining methods will raise :exc:`IOError` if called after the - :meth:`close` method has been called. + The remaining methods will raise :exc:`OSError` if called after the + :meth:`close` method has been called. Before Python 3.3, they used to + raise :exc:`IOError`, now an alias of :exc:`OSError`. .. method:: isatty() diff --git a/Doc/library/cmd.rst b/Doc/library/cmd.rst index fd7f453..0c43bb8 100644 --- a/Doc/library/cmd.rst +++ b/Doc/library/cmd.rst @@ -285,8 +285,8 @@ immediate playback:: def do_playback(self, arg): 'Playback commands from a file: PLAYBACK rose.cmd' self.close() - cmds = open(arg).read().splitlines() - self.cmdqueue.extend(cmds) + with open(arg) as f: + self.cmdqueue.extend(f.read().splitlines()) def precmd(self, line): line = line.lower() if self.file and 'playback' not in line: diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 7747794..a9fae95 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -458,7 +458,8 @@ define in order to be compatible with the Python codec registry. .. method:: reset() - Reset the encoder to the initial state. + Reset the encoder to the initial state. The output is discarded: call + ``.encode('', final=True)`` to reset the encoder and to get the output. .. method:: IncrementalEncoder.getstate() @@ -786,11 +787,9 @@ methods and attributes from the underlying stream. Encodings and Unicode --------------------- -Strings are stored internally as sequences of codepoints (to be precise -as :c:type:`Py_UNICODE` arrays). Depending on the way Python is compiled (either -via ``--without-wide-unicode`` or ``--with-wide-unicode``, with the -former being the default) :c:type:`Py_UNICODE` is either a 16-bit or 32-bit data -type. Once a string object is used outside of CPU and memory, CPU endianness +Strings are stored internally as sequences of codepoints in range ``0 - 10FFFF`` +(see :pep:`393` for more details about the implementation). +Once a string object is used outside of CPU and memory, CPU endianness and how these arrays are stored as bytes become an issue. Transforming a string object into a sequence of bytes is called encoding and recreating the string object from the sequence of bytes is known as decoding. There are many @@ -901,6 +900,15 @@ is meant to be exhaustive. Notice that spelling alternatives that only differ in case or use a hyphen instead of an underscore are also valid aliases; therefore, e.g. ``'utf-8'`` is a valid alias for the ``'utf_8'`` codec. +.. impl-detail:: + + Some common encodings can bypass the codecs lookup machinery to + improve performance. These optimization opportunities are only + recognized by CPython for a limited set of aliases: utf-8, utf8, + latin-1, latin1, iso-8859-1, mbcs (Windows only), ascii, utf-16, + and utf-32. Using alternative spellings for these encodings may + result in slower execution. + Many of the character sets support the same languages. They vary in individual characters (e.g. whether the EURO SIGN is supported or not), and in the assignment of characters to code positions. For the European languages in @@ -1003,6 +1011,11 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | cp1258 | windows-1258 | Vietnamese | +-----------------+--------------------------------+--------------------------------+ +| cp65001 | | Windows only: Windows UTF-8 | +| | | (``CP_UTF8``) | +| | | | +| | | .. versionadded:: 3.3 | ++-----------------+--------------------------------+--------------------------------+ | euc_jp | eucjp, ujis, u-jis | Japanese | +-----------------+--------------------------------+--------------------------------+ | euc_jis_2004 | jisx0213, eucjis2004 | Japanese | @@ -1160,6 +1173,8 @@ particular, the following variants typically exist: | unicode_internal | | Return the internal | | | | representation of the | | | | operand | +| | | | +| | | .. deprecated:: 3.3 | +--------------------+---------+---------------------------+ The following codecs provide bytes-to-bytes mappings. @@ -1272,12 +1287,13 @@ functions can be used directly if desired. .. module:: encodings.mbcs :synopsis: Windows ANSI codepage -Encode operand according to the ANSI codepage (CP_ACP). This codec only -supports ``'strict'`` and ``'replace'`` error handlers to encode, and -``'strict'`` and ``'ignore'`` error handlers to decode. +Encode operand according to the ANSI codepage (CP_ACP). Availability: Windows only. +.. versionchanged:: 3.3 + Support any error handler. + .. versionchanged:: 3.2 Before 3.2, the *errors* argument was ignored; ``'replace'`` was always used to encode, and ``'ignore'`` to decode. diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst new file mode 100644 index 0000000..9873489 --- /dev/null +++ b/Doc/library/collections.abc.rst @@ -0,0 +1,182 @@ +:mod:`collections.abc` --- Abstract Base Classes for Containers +=============================================================== + +.. module:: collections.abc + :synopsis: Abstract base classes for containers +.. moduleauthor:: Raymond Hettinger <python at rcn.com> +.. sectionauthor:: Raymond Hettinger <python at rcn.com> + +.. versionadded:: 3.3 + Formerly, this module was part of the :mod:`collections` module. + +.. testsetup:: * + + from collections import * + import itertools + __name__ = '<doctest>' + +**Source code:** :source:`Lib/collections/abc.py` + +-------------- + +This module provides :term:`abstract base classes <abstract base class>` that +can be used to test whether a class provides a particular interface; for +example, whether it is hashable or whether it is a mapping. + + +.. _collections-abstract-base-classes: + +Collections Abstract Base Classes +--------------------------------- + +The collections module offers the following :term:`ABCs <abstract base class>`: + +========================= ===================== ====================== ==================================================== +ABC Inherits from Abstract Methods Mixin Methods +========================= ===================== ====================== ==================================================== +:class:`Container` ``__contains__`` +:class:`Hashable` ``__hash__`` +:class:`Iterable` ``__iter__`` +:class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__`` +:class:`Sized` ``__len__`` +:class:`Callable` ``__call__`` + +:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``, ``__iter__``, ``__reversed__``, + :class:`Iterable`, ``index``, and ``count`` + :class:`Container` + +:class:`MutableSequence` :class:`Sequence` ``__setitem__``, Inherited :class:`Sequence` methods and + ``__delitem__``, ``append``, ``reverse``, ``extend``, ``pop``, + ``insert`` ``remove``, ``clear``, and ``__iadd__`` + +:class:`Set` :class:`Sized`, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``, + :class:`Iterable`, ``__gt__``, ``__ge__``, ``__and__``, ``__or__``, + :class:`Container` ``__sub__``, ``__xor__``, and ``isdisjoint`` + +:class:`MutableSet` :class:`Set` ``add``, Inherited :class:`Set` methods and + ``discard`` ``clear``, ``pop``, ``remove``, ``__ior__``, + ``__iand__``, ``__ixor__``, and ``__isub__`` + +:class:`Mapping` :class:`Sized`, ``__getitem__`` ``__contains__``, ``keys``, ``items``, ``values``, + :class:`Iterable`, ``get``, ``__eq__``, and ``__ne__`` + :class:`Container` + +:class:`MutableMapping` :class:`Mapping` ``__setitem__``, Inherited :class:`Mapping` methods and + ``__delitem__`` ``pop``, ``popitem``, ``clear``, ``update``, + and ``setdefault`` + + +:class:`MappingView` :class:`Sized` ``__len__`` +:class:`ItemsView` :class:`MappingView`, ``__contains__``, + :class:`Set` ``__iter__`` +:class:`KeysView` :class:`MappingView`, ``__contains__``, + :class:`Set` ``__iter__`` +:class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__`` +========================= ===================== ====================== ==================================================== + + +.. class:: Container + Hashable + Sized + Callable + + ABCs for classes that provide respectively the methods :meth:`__contains__`, + :meth:`__hash__`, :meth:`__len__`, and :meth:`__call__`. + +.. class:: Iterable + + ABC for classes that provide the :meth:`__iter__` method. + See also the definition of :term:`iterable`. + +.. class:: Iterator + + ABC for classes that provide the :meth:`__iter__` and :meth:`next` methods. + See also the definition of :term:`iterator`. + +.. class:: Sequence + MutableSequence + + ABCs for read-only and mutable :term:`sequences <sequence>`. + +.. class:: Set + MutableSet + + ABCs for read-only and mutable sets. + +.. class:: Mapping + MutableMapping + + ABCs for read-only and mutable :term:`mappings <mapping>`. + +.. class:: MappingView + ItemsView + KeysView + ValuesView + + ABCs for mapping, items, keys, and values :term:`views <view>`. + + +These ABCs allow us to ask classes or instances if they provide +particular functionality, for example:: + + size = None + if isinstance(myvar, collections.Sized): + size = len(myvar) + +Several of the ABCs are also useful as mixins that make it easier to develop +classes supporting container APIs. For example, to write a class supporting +the full :class:`Set` API, it only necessary to supply the three underlying +abstract methods: :meth:`__contains__`, :meth:`__iter__`, and :meth:`__len__`. +The ABC supplies the remaining methods such as :meth:`__and__` and +:meth:`isdisjoint` :: + + class ListBasedSet(collections.Set): + ''' Alternate set implementation favoring space over speed + and not requiring the set elements to be hashable. ''' + def __init__(self, iterable): + self.elements = lst = [] + for value in iterable: + if value not in lst: + lst.append(value) + def __iter__(self): + return iter(self.elements) + def __contains__(self, value): + return value in self.elements + def __len__(self): + return len(self.elements) + + s1 = ListBasedSet('abcdef') + s2 = ListBasedSet('defghi') + overlap = s1 & s2 # The __and__() method is supported automatically + +Notes on using :class:`Set` and :class:`MutableSet` as a mixin: + +(1) + Since some set operations create new sets, the default mixin methods need + a way to create new instances from an iterable. The class constructor is + assumed to have a signature in the form ``ClassName(iterable)``. + That assumption is factored-out to an internal classmethod called + :meth:`_from_iterable` which calls ``cls(iterable)`` to produce a new set. + If the :class:`Set` mixin is being used in a class with a different + constructor signature, you will need to override :meth:`_from_iterable` + with a classmethod that can construct new instances from + an iterable argument. + +(2) + To override the comparisons (presumably for speed, as the + semantics are fixed), redefine :meth:`__le__` and + then the other operations will automatically follow suit. + +(3) + The :class:`Set` mixin provides a :meth:`_hash` method to compute a hash value + for the set; however, :meth:`__hash__` is not defined because not all sets + are hashable or immutable. To add set hashabilty using mixins, + inherit from both :meth:`Set` and :meth:`Hashable`, then define + ``__hash__ = Set._hash``. + +.. seealso:: + + * `OrderedSet recipe <http://code.activestate.com/recipes/576694/>`_ for an + example built on :class:`MutableSet`. + + * For more about ABCs, see the :mod:`abc` module and :pep:`3119`. diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index e90b25e..71c27ed 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -12,7 +12,7 @@ import itertools __name__ = '<doctest>' -**Source code:** :source:`Lib/collections.py` and :source:`Lib/_abcoll.py` +**Source code:** :source:`Lib/collections/__init__.py` -------------- @@ -23,6 +23,7 @@ Python's general purpose built-in containers, :class:`dict`, :class:`list`, ===================== ==================================================================== :func:`namedtuple` factory function for creating tuple subclasses with named fields :class:`deque` list-like container with fast appends and pops on either end +:class:`ChainMap` dict-like class for creating a single view of multiple mappings :class:`Counter` dict subclass for counting hashable objects :class:`OrderedDict` dict subclass that remembers the order entries were added :class:`defaultdict` dict subclass that calls a factory function to supply missing values @@ -31,10 +32,124 @@ Python's general purpose built-in containers, :class:`dict`, :class:`list`, :class:`UserString` wrapper around string objects for easier string subclassing ===================== ==================================================================== -In addition to the concrete container classes, the collections module provides -:ref:`abstract base classes <collections-abstract-base-classes>` that can be -used to test whether a class provides a particular interface, for example, -whether it is hashable or a mapping. +.. versionchanged:: 3.3 + Moved :ref:`collections-abstract-base-classes` to the :mod:`collections.abc` module. + For backwards compatibility, they continue to be visible in this module + as well. + + +:class:`ChainMap` objects +------------------------- + +.. versionadded:: 3.3 + +A :class:`ChainMap` class is provided for quickly linking a number of mappings +so they can be treated as a single unit. It is often much faster than creating +a new dictionary and running multiple :meth:`~dict.update` calls. + +The class can be used to simulate nested scopes and is useful in templating. + +.. class:: ChainMap(*maps) + + A :class:`ChainMap` groups multiple dicts or other mappings together to + create a single, updateable view. If no *maps* are specified, a single empty + dictionary is provided so that a new chain always has at least one mapping. + + The underlying mappings are stored in a list. That list is public and can + accessed or updated using the *maps* attribute. There is no other state. + + Lookups search the underlying mappings successively until a key is found. In + contrast, writes, updates, and deletions only operate on the first mapping. + + A :class:`ChainMap` incorporates the underlying mappings by reference. So, if + one of the underlying mappings gets updated, those changes will be reflected + in :class:`ChainMap`. + + All of the usual dictionary methods are supported. In addition, there is a + *maps* attribute, a method for creating new subcontexts, and a property for + accessing all but the first mapping: + + .. attribute:: maps + + A user updateable list of mappings. The list is ordered from + first-searched to last-searched. It is the only stored state and can + be modified to change which mappings are searched. The list should + always contain at least one mapping. + + .. method:: new_child() + + Returns a new :class:`ChainMap` containing a new :class:`dict` followed by + all of the maps in the current instance. A call to ``d.new_child()`` is + equivalent to: ``ChainMap({}, *d.maps)``. This method is used for + creating subcontexts that can be updated without altering values in any + of the parent mappings. + + .. method:: parents() + + Returns a new :class:`ChainMap` containing all of the maps in the current + instance except the first one. This is useful for skipping the first map + in the search. The use-cases are similar to those for the + :keyword:`nonlocal` keyword used in :term:`nested scopes <nested scope>`. + The use-cases also parallel those for the builtin :func:`super` function. + A reference to ``d.parents`` is equivalent to: ``ChainMap(*d.maps[1:])``. + + Example of simulating Python's internal lookup chain:: + + import builtins + pylookup = ChainMap(locals(), globals(), vars(builtins)) + + Example of letting user specified values take precedence over environment + variables which in turn take precedence over default values:: + + import os, argparse + defaults = {'color': 'red', 'user': guest} + parser = argparse.ArgumentParser() + parser.add_argument('-u', '--user') + parser.add_argument('-c', '--color') + user_specified = vars(parser.parse_args()) + combined = ChainMap(user_specified, os.environ, defaults) + + Example patterns for using the :class:`ChainMap` class to simulate nested + contexts:: + + c = ChainMap() # Create root context + d = c.new_child() # Create nested child context + e = c.new_child() # Child of c, independent from d + e.maps[0] # Current context dictionary -- like Python's locals() + e.maps[-1] # Root context -- like Python's globals() + e.parents # Enclosing context chain -- like Python's nonlocals + + d['x'] # Get first key in the chain of contexts + d['x'] = 1 # Set value in current context + del['x'] # Delete from current context + list(d) # All nested values + k in d # Check all nested values + len(d) # Number of nested values + d.items() # All nested items + dict(d) # Flatten into a regular dictionary + + .. seealso:: + + * The `MultiContext class + <http://svn.enthought.com/svn/enthought/CodeTools/trunk/enthought/contexts/multi_context.py>`_ + in the Enthought `CodeTools package + <https://github.com/enthought/codetools>`_ has options to support + writing to any mapping in the chain. + + * Django's `Context class + <http://code.djangoproject.com/browser/django/trunk/django/template/context.py>`_ + for templating is a read-only chain of mappings. It also features + pushing and popping of contexts similar to the + :meth:`~collections.ChainMap.new_child` method and the + :meth:`~collections.ChainMap.parents` property. + + * The `Nested Contexts recipe + <http://code.activestate.com/recipes/577434/>`_ has options to control + whether writes and other mutations apply only to the first mapping or to + any mapping in the chain. + + * A `greatly simplified read-only version of Chainmap + <http://code.activestate.com/recipes/305268/>`_. :class:`Counter` objects @@ -149,7 +264,7 @@ Common patterns for working with :class:`Counter` objects:: c.items() # convert to a list of (elem, cnt) pairs Counter(dict(list_of_pairs)) # convert from a list of (elem, cnt) pairs c.most_common()[:-n:-1] # n least common elements - c += Counter() # remove zero and negative counts + +c # remove zero and negative counts Several mathematical operations are provided for combining :class:`Counter` objects to produce multisets (counters that have counts greater than zero). @@ -169,6 +284,18 @@ counts, but the output will exclude results with counts of zero or less. >>> c | d # union: max(c[x], d[x]) Counter({'a': 3, 'b': 2}) +Unary addition and substraction are shortcuts for adding an empty counter +or subtracting from an empty counter. + + >>> c = Counter(a=2, b=-4) + >>> +c + Counter({'a': 2}) + >>> -c + Counter({'b': 4}) + +.. versionadded:: 3.3 + Added support for unary plus, unary minus, and in-place multiset operations. + .. note:: Counters were primarily designed to work with positive integers to represent @@ -398,7 +525,8 @@ in Unix:: def tail(filename, n=10): 'Return the last n lines of a file' - return deque(open(filename), n) + with open(filename) as f: + return deque(f, n) Another approach to using deques is to maintain a sequence of recently added elements by appending to the right and popping to the left:: @@ -550,7 +678,7 @@ Setting the :attr:`default_factory` to :class:`set` makes the ... d[k].add(v) ... >>> list(d.items()) - [('blue', set([2, 4])), ('red', set([1, 3]))] + [('blue', {2, 4}), ('red', {1, 3})] :func:`namedtuple` Factory Function for Tuples with Named Fields @@ -583,7 +711,9 @@ they add the ability to access fields by name instead of position index. converted to ``['abc', '_1', 'ghi', '_3']``, eliminating the keyword ``def`` and the duplicate fieldname ``abc``. - If *verbose* is true, the class definition is printed just before being built. + If *verbose* is true, the class definition is printed after it is + built. This option is outdated; instead, it is simpler to print the + :attr:`_source` attribute. Named tuple instances do not have per-instance dictionaries, so they are lightweight and require no more memory than regular tuples. @@ -597,53 +727,6 @@ they add the ability to access fields by name instead of position index. >>> # Basic example >>> Point = namedtuple('Point', ['x', 'y']) - >>> p = Point(x=10, y=11) - - >>> # Example using the verbose option to print the class definition - >>> Point = namedtuple('Point', 'x y', verbose=True) - class Point(tuple): - 'Point(x, y)' - <BLANKLINE> - __slots__ = () - <BLANKLINE> - _fields = ('x', 'y') - <BLANKLINE> - def __new__(_cls, x, y): - 'Create a new instance of Point(x, y)' - return _tuple.__new__(_cls, (x, y)) - <BLANKLINE> - @classmethod - def _make(cls, iterable, new=tuple.__new__, len=len): - 'Make a new Point object from a sequence or iterable' - result = new(cls, iterable) - if len(result) != 2: - raise TypeError('Expected 2 arguments, got %d' % len(result)) - return result - <BLANKLINE> - def __repr__(self): - 'Return a nicely formatted representation string' - return self.__class__.__name__ + '(x=%r, y=%r)' % self - <BLANKLINE> - def _asdict(self): - 'Return a new OrderedDict which maps field names to their values' - return OrderedDict(zip(self._fields, self)) - <BLANKLINE> - __dict__ = property(_asdict) - <BLANKLINE> - def _replace(_self, **kwds): - 'Return a new Point object replacing specified fields with new values' - result = _self._make(map(kwds.pop, ('x', 'y'), _self)) - if kwds: - raise ValueError('Got unexpected field names: %r' % list(kwds.keys())) - return result - <BLANKLINE> - def __getnewargs__(self): - 'Return self as a plain tuple. Used by copy and pickle.' - return tuple(self) - <BLANKLINE> - x = _property(_itemgetter(0), doc='Alias for field number 0') - y = _property(_itemgetter(1), doc='Alias for field number 1') - >>> p = Point(11, y=22) # instantiate with positional or keyword arguments >>> p[0] + p[1] # indexable like the plain tuple (11, 22) 33 @@ -672,7 +755,7 @@ by the :mod:`csv` or :mod:`sqlite3` modules:: print(emp.name, emp.title) In addition to the methods inherited from tuples, named tuples support -three additional methods and one attribute. To prevent conflicts with +three additional methods and two attributes. To prevent conflicts with field names, the method and attribute names start with an underscore. .. classmethod:: somenamedtuple._make(iterable) @@ -710,6 +793,15 @@ field names, the method and attribute names start with an underscore. >>> for partnum, record in inventory.items(): ... inventory[partnum] = record._replace(price=newprices[partnum], timestamp=time.now()) +.. attribute:: somenamedtuple._source + + A string with the pure Python source code used to create the named + tuple class. The source makes the named tuple self-documenting. + It can be printed, executed using :func:`exec`, or saved to a file + and imported. + + .. versionadded:: 3.3 + .. attribute:: somenamedtuple._fields Tuple of strings listing the field names. Useful for introspection @@ -758,7 +850,6 @@ a fixed-width print format: The subclass shown above sets ``__slots__`` to an empty tuple. This helps keep memory requirements low by preventing the creation of instance dictionaries. - Subclassing is not useful for adding new, stored fields. Instead, simply create a new named tuple type from the :attr:`_fields` attribute: @@ -770,6 +861,7 @@ customize a prototype instance: >>> Account = namedtuple('Account', 'owner balance transaction_count') >>> default_account = Account('<owner name>', 0.0, 0) >>> johns_account = default_account._replace(owner='John') + >>> janes_account = default_account._replace(owner='Jane') Enumerated constants can be implemented with named tuples, but it is simpler and more efficient to use a simple class declaration: @@ -988,161 +1080,3 @@ attribute. be an instance of :class:`bytes`, :class:`str`, :class:`UserString` (or a subclass) or an arbitrary sequence which can be converted into a string using the built-in :func:`str` function. - - -.. _collections-abstract-base-classes: - -ABCs - abstract base classes ----------------------------- - -The collections module offers the following :term:`ABCs <abstract base class>`: - -========================= ===================== ====================== ==================================================== -ABC Inherits from Abstract Methods Mixin Methods -========================= ===================== ====================== ==================================================== -:class:`Container` ``__contains__`` -:class:`Hashable` ``__hash__`` -:class:`Iterable` ``__iter__`` -:class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__`` -:class:`Sized` ``__len__`` -:class:`Callable` ``__call__`` - -:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``, ``__iter__``, ``__reversed__``, - :class:`Iterable`, ``index``, and ``count`` - :class:`Container` - -:class:`MutableSequence` :class:`Sequence` ``__setitem__``, Inherited :class:`Sequence` methods and - ``__delitem__``, ``append``, ``reverse``, ``extend``, ``pop``, - ``insert`` ``remove``, and ``__iadd__`` - -:class:`Set` :class:`Sized`, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``, - :class:`Iterable`, ``__gt__``, ``__ge__``, ``__and__``, ``__or__``, - :class:`Container` ``__sub__``, ``__xor__``, and ``isdisjoint`` - -:class:`MutableSet` :class:`Set` ``add``, Inherited :class:`Set` methods and - ``discard`` ``clear``, ``pop``, ``remove``, ``__ior__``, - ``__iand__``, ``__ixor__``, and ``__isub__`` - -:class:`Mapping` :class:`Sized`, ``__getitem__`` ``__contains__``, ``keys``, ``items``, ``values``, - :class:`Iterable`, ``get``, ``__eq__``, and ``__ne__`` - :class:`Container` - -:class:`MutableMapping` :class:`Mapping` ``__setitem__``, Inherited :class:`Mapping` methods and - ``__delitem__`` ``pop``, ``popitem``, ``clear``, ``update``, - and ``setdefault`` - - -:class:`MappingView` :class:`Sized` ``__len__`` -:class:`ItemsView` :class:`MappingView`, ``__contains__``, - :class:`Set` ``__iter__`` -:class:`KeysView` :class:`MappingView`, ``__contains__``, - :class:`Set` ``__iter__`` -:class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__`` -========================= ===================== ====================== ==================================================== - - -.. class:: Container - Hashable - Sized - Callable - - ABCs for classes that provide respectively the methods :meth:`__contains__`, - :meth:`__hash__`, :meth:`__len__`, and :meth:`__call__`. - -.. class:: Iterable - - ABC for classes that provide the :meth:`__iter__` method. - See also the definition of :term:`iterable`. - -.. class:: Iterator - - ABC for classes that provide the :meth:`__iter__` and :meth:`next` methods. - See also the definition of :term:`iterator`. - -.. class:: Sequence - MutableSequence - - ABCs for read-only and mutable :term:`sequences <sequence>`. - -.. class:: Set - MutableSet - - ABCs for read-only and mutable sets. - -.. class:: Mapping - MutableMapping - - ABCs for read-only and mutable :term:`mappings <mapping>`. - -.. class:: MappingView - ItemsView - KeysView - ValuesView - - ABCs for mapping, items, keys, and values :term:`views <view>`. - - -These ABCs allow us to ask classes or instances if they provide -particular functionality, for example:: - - size = None - if isinstance(myvar, collections.Sized): - size = len(myvar) - -Several of the ABCs are also useful as mixins that make it easier to develop -classes supporting container APIs. For example, to write a class supporting -the full :class:`Set` API, it only necessary to supply the three underlying -abstract methods: :meth:`__contains__`, :meth:`__iter__`, and :meth:`__len__`. -The ABC supplies the remaining methods such as :meth:`__and__` and -:meth:`isdisjoint` :: - - class ListBasedSet(collections.Set): - ''' Alternate set implementation favoring space over speed - and not requiring the set elements to be hashable. ''' - def __init__(self, iterable): - self.elements = lst = [] - for value in iterable: - if value not in lst: - lst.append(value) - def __iter__(self): - return iter(self.elements) - def __contains__(self, value): - return value in self.elements - def __len__(self): - return len(self.elements) - - s1 = ListBasedSet('abcdef') - s2 = ListBasedSet('defghi') - overlap = s1 & s2 # The __and__() method is supported automatically - -Notes on using :class:`Set` and :class:`MutableSet` as a mixin: - -(1) - Since some set operations create new sets, the default mixin methods need - a way to create new instances from an iterable. The class constructor is - assumed to have a signature in the form ``ClassName(iterable)``. - That assumption is factored-out to an internal classmethod called - :meth:`_from_iterable` which calls ``cls(iterable)`` to produce a new set. - If the :class:`Set` mixin is being used in a class with a different - constructor signature, you will need to override :meth:`_from_iterable` - with a classmethod that can construct new instances from - an iterable argument. - -(2) - To override the comparisons (presumably for speed, as the - semantics are fixed), redefine :meth:`__le__` and - then the other operations will automatically follow suit. - -(3) - The :class:`Set` mixin provides a :meth:`_hash` method to compute a hash value - for the set; however, :meth:`__hash__` is not defined because not all sets - are hashable or immutable. To add set hashabilty using mixins, - inherit from both :meth:`Set` and :meth:`Hashable`, then define - ``__hash__ = Set._hash``. - -.. seealso:: - - * `OrderedSet recipe <http://code.activestate.com/recipes/576694/>`_ for an - example built on :class:`MutableSet`. - - * For more about ABCs, see the :mod:`abc` module and :pep:`3119`. diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index 29ffc0d..a88f10c 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -169,6 +169,12 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. of at most *max_workers* processes. If *max_workers* is ``None`` or not given, it will default to the number of processors on the machine. + .. versionchanged:: 3.3 + When one of the worker processes terminates abruptly, a + :exc:`BrokenProcessPool` error is now raised. Previously, behaviour + was undefined but operations on the executor or its futures would often + freeze or deadlock. + .. _processpoolexecutor-example: @@ -369,3 +375,16 @@ Module Functions :pep:`3148` -- futures - execute computations asynchronously The proposal which described this feature for inclusion in the Python standard library. + + +Exception classes +----------------- + +.. exception:: BrokenProcessPool + + Derived from :exc:`RuntimeError`, this exception class is raised when + one of the workers of a :class:`ProcessPoolExecutor` has terminated + in a non-clean fashion (for example, if it was killed from the outside). + + .. versionadded:: 3.3 + diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index e8dc17f..b1bfa18 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -12,8 +12,11 @@ This module provides utilities for common tasks involving the :keyword:`with` statement. For more information see also :ref:`typecontextmanager` and :ref:`context-managers`. -Functions provided: +Utilities +--------- + +Functions and classes provided: .. decorator:: contextmanager @@ -168,6 +171,280 @@ Functions provided: .. versionadded:: 3.2 +.. class:: ExitStack() + + A context manager that is designed to make it easy to programmatically + combine other context managers and cleanup functions, especially those + that are optional or otherwise driven by input data. + + For example, a set of files may easily be handled in a single with + statement as follows:: + + with ExitStack() as stack: + files = [stack.enter_context(open(fname)) for fname in filenames] + # All opened files will automatically be closed at the end of + # the with statement, even if attempts to open files later + # in the list throw an exception + + Each instance maintains a stack of registered callbacks that are called in + reverse order when the instance is closed (either explicitly or implicitly + at the end of a ``with`` statement). Note that callbacks are *not* invoked + implicitly when the context stack instance is garbage collected. + + This stack model is used so that context managers that acquire their + resources in their ``__init__`` method (such as file objects) can be + handled correctly. + + Since registered callbacks are invoked in the reverse order of + registration, this ends up behaving as if multiple nested ``with`` + statements had been used with the registered set of callbacks. This even + extends to exception handling - if an inner callback suppresses or replaces + an exception, then outer callbacks will be passed arguments based on that + updated state. + + This is a relatively low level API that takes care of the details of + correctly unwinding the stack of exit callbacks. It provides a suitable + foundation for higher level context managers that manipulate the exit + stack in application specific ways. + + .. versionadded:: 3.3 + + .. method:: enter_context(cm) + + Enters a new context manager and adds its :meth:`__exit__` method to + the callback stack. The return value is the result of the context + manager's own :meth:`__enter__` method. + + These context managers may suppress exceptions just as they normally + would if used directly as part of a ``with`` statement. + + .. method:: push(exit) + + Adds a context manager's :meth:`__exit__` method to the callback stack. + + As ``__enter__`` is *not* invoked, this method can be used to cover + part of an :meth:`__enter__` implementation with a context manager's own + :meth:`__exit__` method. + + If passed an object that is not a context manager, this method assumes + it is a callback with the same signature as a context manager's + :meth:`__exit__` method and adds it directly to the callback stack. + + By returning true values, these callbacks can suppress exceptions the + same way context manager :meth:`__exit__` methods can. + + The passed in object is returned from the function, allowing this + method to be used is a function decorator. + + .. method:: callback(callback, *args, **kwds) + + Accepts an arbitrary callback function and arguments and adds it to + the callback stack. + + Unlike the other methods, callbacks added this way cannot suppress + exceptions (as they are never passed the exception details). + + The passed in callback is returned from the function, allowing this + method to be used is a function decorator. + + .. method:: pop_all() + + Transfers the callback stack to a fresh :class:`ExitStack` instance + and returns it. No callbacks are invoked by this operation - instead, + they will now be invoked when the new stack is closed (either + explicitly or implicitly). + + For example, a group of files can be opened as an "all or nothing" + operation as follows:: + + with ExitStack() as stack: + files = [stack.enter_context(open(fname)) for fname in filenames] + close_files = stack.pop_all().close + # If opening any file fails, all previously opened files will be + # closed automatically. If all files are opened successfully, + # they will remain open even after the with statement ends. + # close_files() can then be invoked explicitly to close them all + + .. method:: close() + + Immediately unwinds the callback stack, invoking callbacks in the + reverse order of registration. For any context managers and exit + callbacks registered, the arguments passed in will indicate that no + exception occurred. + + +Examples and Recipes +-------------------- + +This section describes some examples and recipes for making effective use of +the tools provided by :mod:`contextlib`. + + +Cleaning up in an ``__enter__`` implementation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +As noted in the documentation of :meth:`ExitStack.push`, this +method can be useful in cleaning up an already allocated resource if later +steps in the :meth:`__enter__` implementation fail. + +Here's an example of doing this for a context manager that accepts resource +acquisition and release functions, along with an optional validation function, +and maps them to the context management protocol:: + + from contextlib import contextmanager, ExitStack + + class ResourceManager(object): + + def __init__(self, acquire_resource, release_resource, check_resource_ok=None): + self.acquire_resource = acquire_resource + self.release_resource = release_resource + if check_resource_ok is None: + def check_resource_ok(resource): + return True + self.check_resource_ok = check_resource_ok + + @contextmanager + def _cleanup_on_error(self): + with ExitStack() as stack: + stack.push(self) + yield + # The validation check passed and didn't raise an exception + # Accordingly, we want to keep the resource, and pass it + # back to our caller + stack.pop_all() + + def __enter__(self): + resource = self.acquire_resource() + with self._cleanup_on_error(): + if not self.check_resource_ok(resource): + msg = "Failed validation for {!r}" + raise RuntimeError(msg.format(resource)) + return resource + + def __exit__(self, *exc_details): + # We don't need to duplicate any of our resource release logic + self.release_resource() + + +Replacing any use of ``try-finally`` and flag variables +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A pattern you will sometimes see is a ``try-finally`` statement with a flag +variable to indicate whether or not the body of the ``finally`` clause should +be executed. In its simplest form (that can't already be handled just by +using an ``except`` clause instead), it looks something like this:: + + cleanup_needed = True + try: + result = perform_operation() + if result: + cleanup_needed = False + finally: + if cleanup_needed: + cleanup_resources() + +As with any ``try`` statement based code, this can cause problems for +development and review, because the setup code and the cleanup code can end +up being separated by arbitrarily long sections of code. + +:class:`ExitStack` makes it possible to instead register a callback for +execution at the end of a ``with`` statement, and then later decide to skip +executing that callback:: + + from contextlib import ExitStack + + with ExitStack() as stack: + stack.callback(cleanup_resources) + result = perform_operation() + if result: + stack.pop_all() + +This allows the intended cleanup up behaviour to be made explicit up front, +rather than requiring a separate flag variable. + +If a particular application uses this pattern a lot, it can be simplified +even further by means of a small helper class:: + + from contextlib import ExitStack + + class Callback(ExitStack): + def __init__(self, callback, *args, **kwds): + super(Callback, self).__init__() + self.callback(callback, *args, **kwds) + + def cancel(self): + self.pop_all() + + with Callback(cleanup_resources) as cb: + result = perform_operation() + if result: + cb.cancel() + +If the resource cleanup isn't already neatly bundled into a standalone +function, then it is still possible to use the decorator form of +:meth:`ExitStack.callback` to declare the resource cleanup in +advance:: + + from contextlib import ExitStack + + with ExitStack() as stack: + @stack.callback + def cleanup_resources(): + ... + result = perform_operation() + if result: + stack.pop_all() + +Due to the way the decorator protocol works, a callback function +declared this way cannot take any parameters. Instead, any resources to +be released must be accessed as closure variables + + +Using a context manager as a function decorator +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +:class:`ContextDecorator` makes it possible to use a context manager in +both an ordinary ``with`` statement and also as a function decorator. + +For example, it is sometimes useful to wrap functions or groups of statements +with a logger that can track the time of entry and time of exit. Rather than +writing both a function decorator and a context manager for the task, +inheriting from :class:`ContextDecorator` provides both capabilities in a +single definition:: + + from contextlib import ContextDecorator + import logging + + logging.basicConfig(level=logging.INFO) + + class track_entry_and_exit(ContextDecorator): + def __init__(self, name): + self.name = name + + def __enter__(self): + logging.info('Entering: {}'.format(name)) + + def __exit__(self, exc_type, exc, exc_tb): + logging.info('Exiting: {}'.format(name)) + +Instances of this class can be used as both a context manager:: + + with track_entry_and_exit('widget loader'): + print('Some time consuming activity goes here') + load_widget() + +And also as a function decorator:: + + @track_entry_and_exit('widget loader') + def activity(): + print('Some time consuming activity goes here') + load_widget() + +Note that there is one additional limitation when using context managers +as function decorators: there's no way to access the return value of +:meth:`__enter__`. If that value is needed, then it is still necessary to use +an explicit ``with`` statement. + .. seealso:: :pep:`0343` - The "with" statement diff --git a/Doc/library/copyreg.rst b/Doc/library/copyreg.rst index a2d316e..41061e5 100644 --- a/Doc/library/copyreg.rst +++ b/Doc/library/copyreg.rst @@ -32,6 +32,8 @@ Such constructors may be factory functions or class instances. returned by *function* at pickling time. :exc:`TypeError` will be raised if *object* is a class or *constructor* is not callable. - See the :mod:`pickle` module for more details on the interface expected of - *function* and *constructor*. - + See the :mod:`pickle` module for more details on the interface + expected of *function* and *constructor*. Note that the + :attr:`~pickle.Pickler.dispatch_table` attribute of a pickler + object or subclass of :class:`pickle.Pickler` can also be used for + declaring reduction functions. diff --git a/Doc/library/crypt.rst b/Doc/library/crypt.rst index 0be571e..1ba2ed3 100644 --- a/Doc/library/crypt.rst +++ b/Doc/library/crypt.rst @@ -15,9 +15,9 @@ This module implements an interface to the :manpage:`crypt(3)` routine, which is a one-way hash function based upon a modified DES algorithm; see the Unix man -page for further details. Possible uses include allowing Python scripts to -accept typed passwords from the user, or attempting to crack Unix passwords with -a dictionary. +page for further details. Possible uses include storing hashed passwords +so you can check passwords without storing the actual password, or attempting +to crack Unix passwords with a dictionary. .. index:: single: crypt(3) @@ -26,15 +26,74 @@ the :manpage:`crypt(3)` routine in the running system. Therefore, any extensions available on the current implementation will also be available on this module. +Hashing Methods +--------------- -.. function:: crypt(word, salt) +.. versionadded:: 3.3 + +The :mod:`crypt` module defines the list of hashing methods (not all methods +are available on all platforms): + +.. data:: METHOD_SHA512 + + A Modular Crypt Format method with 16 character salt and 86 character + hash. This is the strongest method. + +.. data:: METHOD_SHA256 + + Another Modular Crypt Format method with 16 character salt and 43 + character hash. + +.. data:: METHOD_MD5 + + Another Modular Crypt Format method with 8 character salt and 22 + character hash. + +.. data:: METHOD_CRYPT + + The traditional method with a 2 character salt and 13 characters of + hash. This is the weakest method. + + +Module Attributes +----------------- + +.. versionadded:: 3.3 + +.. attribute:: methods + + A list of available password hashing algorithms, as + ``crypt.METHOD_*`` objects. This list is sorted from strongest to + weakest, and is guaranteed to have at least ``crypt.METHOD_CRYPT``. + + +Module Functions +---------------- + +The :mod:`crypt` module defines the following functions: + +.. function:: crypt(word, salt=None) *word* will usually be a user's password as typed at a prompt or in a graphical - interface. *salt* is usually a random two-character string which will be used - to perturb the DES algorithm in one of 4096 ways. The characters in *salt* must - be in the set ``[./a-zA-Z0-9]``. Returns the hashed password as a string, which - will be composed of characters from the same alphabet as the salt (the first two - characters represent the salt itself). + interface. The optional *salt* is either a string as returned from + :func:`mksalt`, one of the ``crypt.METHOD_*`` values (though not all + may be available on all platforms), or a full encrypted password + including salt, as returned by this function. If *salt* is not + provided, the strongest method will be used (as returned by + :func:`methods`. + + Checking a password is usually done by passing the plain-text password + as *word* and the full results of a previous :func:`crypt` call, + which should be the same as the results of this call. + + *salt* (either a random 2 or 16 character string, possibly prefixed with + ``$digit$`` to indicate the method) which will be used to perturb the + encryption algorithm. The characters in *salt* must be in the set + ``[./a-zA-Z0-9]``, with the exception of Modular Crypt Format which + prefixes a ``$digit$``. + + Returns the hashed password as a string, which will be composed of + characters from the same alphabet as the salt. .. index:: single: crypt(3) @@ -42,18 +101,48 @@ this module. different sizes in the *salt*, it is recommended to use the full crypted password as salt when checking for a password. + .. versionchanged:: 3.3 + Accept ``crypt.METHOD_*`` values in addition to strings for *salt*. + + +.. function:: mksalt(method=None) + + Return a randomly generated salt of the specified method. If no + *method* is given, the strongest method available as returned by + :func:`methods` is used. + + The return value is a string either of 2 characters in length for + ``crypt.METHOD_CRYPT``, or 19 characters starting with ``$digit$`` and + 16 random characters from the set ``[./a-zA-Z0-9]``, suitable for + passing as the *salt* argument to :func:`crypt`. + + .. versionadded:: 3.3 + +Examples +-------- + A simple example illustrating typical use:: - import crypt, getpass, pwd + import pwd + import crypt + import getpass def login(): - username = input('Python login:') + username = input('Python login: ') cryptedpasswd = pwd.getpwnam(username)[1] if cryptedpasswd: if cryptedpasswd == 'x' or cryptedpasswd == '*': - raise "Sorry, currently no support for shadow passwords" + raise ValueError('no support for shadow passwords') cleartext = getpass.getpass() return crypt.crypt(cleartext, cryptedpasswd) == cryptedpasswd else: - return 1 + return True + +To generate a hash of a password using the strongest available method and +check it against the original:: + + import crypt + hashed = crypt.crypt(plaintext) + if hashed != crypt.crypt(plaintext, hashed): + raise ValueError("hashed version doesn't validate against original") diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index edbe726..ec0dfcc 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -11,15 +11,15 @@ pair: data; tabular The so-called CSV (Comma Separated Values) format is the most common import and -export format for spreadsheets and databases. There is no "CSV standard", so -the format is operationally defined by the many applications which read and -write it. The lack of a standard means that subtle differences often exist in -the data produced and consumed by different applications. These differences can -make it annoying to process CSV files from multiple sources. Still, while the -delimiters and quoting characters vary, the overall format is similar enough -that it is possible to write a single module which can efficiently manipulate -such data, hiding the details of reading and writing the data from the -programmer. +export format for spreadsheets and databases. CSV format was used for many +years prior to attempts to describe the format in a standardized way in +:rfc:`4180`. The lack of a well-defined standard means that subtle differences +often exist in the data produced and consumed by different applications. These +differences can make it annoying to process CSV files from multiple sources. +Still, while the delimiters and quoting characters vary, the overall format is +similar enough that it is possible to write a single module which can +efficiently manipulate such data, hiding the details of reading and writing the +data from the programmer. The :mod:`csv` module implements classes to read and write tabular data in CSV format. It allows programmers to say, "write this data in the format preferred diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index df39c28..8a2b196 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -39,9 +39,14 @@ loads libraries which export functions using the standard ``cdecl`` calling convention, while *windll* libraries call functions using the ``stdcall`` calling convention. *oledll* also uses the ``stdcall`` calling convention, and assumes the functions return a Windows :c:type:`HRESULT` error code. The error -code is used to automatically raise a :class:`WindowsError` exception when the +code is used to automatically raise a :class:`OSError` exception when the function call fails. +.. versionchanged:: 3.3 + Windows errors used to raise :exc:`WindowsError`, which is now an alias + of :exc:`OSError`. + + Here are some examples for Windows. Note that ``msvcrt`` is the MS standard C library containing most standard C functions, and uses the cdecl calling convention:: @@ -189,7 +194,7 @@ argument values:: >>> windll.kernel32.GetModuleHandleA(32) # doctest: +WINDOWS Traceback (most recent call last): File "<stdin>", line 1, in ? - WindowsError: exception: access violation reading 0x00000020 + OSError: exception: access violation reading 0x00000020 >>> There are, however, enough ways to crash Python with :mod:`ctypes`, so you @@ -491,7 +496,7 @@ useful to check for error return values and automatically raise an exception:: Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 3, in ValidHandle - WindowsError: [Errno 126] The specified module could not be found. + OSError: [Errno 126] The specified module could not be found. >>> ``WinError`` is a function which will call Windows ``FormatMessage()`` api to @@ -921,21 +926,21 @@ Callback functions :mod:`ctypes` allows to create C callable function pointers from Python callables. These are sometimes called *callback functions*. -First, you must create a class for the callback function, the class knows the +First, you must create a class for the callback function. The class knows the calling convention, the return type, and the number and types of arguments this function will receive. -The CFUNCTYPE factory function creates types for callback functions using the -normal cdecl calling convention, and, on Windows, the WINFUNCTYPE factory -function creates types for callback functions using the stdcall calling -convention. +The :func:`CFUNCTYPE` factory function creates types for callback functions +using the ``cdecl`` calling convention. On Windows, the :func:`WINFUNCTYPE` +factory function creates types for callback functions using the ``stdcall`` +calling convention. Both of these factory functions are called with the result type as first argument, and the callback functions expected argument types as the remaining arguments. I will present an example here which uses the standard C library's -:c:func:`qsort` function, this is used to sort items with the help of a callback +:c:func:`qsort` function, that is used to sort items with the help of a callback function. :c:func:`qsort` will be used to sort an array of integers:: >>> IntArray5 = c_int * 5 @@ -948,7 +953,7 @@ function. :c:func:`qsort` will be used to sort an array of integers:: items in the data array, the size of one item, and a pointer to the comparison function, the callback. The callback will then be called with two pointers to items, and it must return a negative integer if the first item is smaller than -the second, a zero if they are equal, and a positive integer else. +the second, a zero if they are equal, and a positive integer otherwise. So our callback function receives pointers to integers, and must return an integer. First we create the ``type`` for the callback function:: @@ -956,36 +961,8 @@ integer. First we create the ``type`` for the callback function:: >>> CMPFUNC = CFUNCTYPE(c_int, POINTER(c_int), POINTER(c_int)) >>> -For the first implementation of the callback function, we simply print the -arguments we get, and return 0 (incremental development ;-):: - - >>> def py_cmp_func(a, b): - ... print("py_cmp_func", a, b) - ... return 0 - ... - >>> - -Create the C callable callback:: - - >>> cmp_func = CMPFUNC(py_cmp_func) - >>> - -And we're ready to go:: - - >>> qsort(ia, len(ia), sizeof(c_int), cmp_func) # doctest: +WINDOWS - py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...> - py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...> - py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...> - py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...> - py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...> - py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...> - py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...> - py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...> - py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...> - py_cmp_func <ctypes.LP_c_long object at 0x00...> <ctypes.LP_c_long object at 0x00...> - >>> - -We know how to access the contents of a pointer, so lets redefine our callback:: +To get started, here is a simple callback that shows the values it gets +passed:: >>> def py_cmp_func(a, b): ... print("py_cmp_func", a[0], b[0]) @@ -994,23 +971,7 @@ We know how to access the contents of a pointer, so lets redefine our callback:: >>> cmp_func = CMPFUNC(py_cmp_func) >>> -Here is what we get on Windows:: - - >>> qsort(ia, len(ia), sizeof(c_int), cmp_func) # doctest: +WINDOWS - py_cmp_func 7 1 - py_cmp_func 33 1 - py_cmp_func 99 1 - py_cmp_func 5 1 - py_cmp_func 7 5 - py_cmp_func 33 5 - py_cmp_func 99 5 - py_cmp_func 7 99 - py_cmp_func 33 99 - py_cmp_func 7 33 - >>> - -It is funny to see that on linux the sort function seems to work much more -efficiently, it is doing less comparisons:: +The result:: >>> qsort(ia, len(ia), sizeof(c_int), cmp_func) # doctest: +LINUX py_cmp_func 5 1 @@ -1020,32 +981,13 @@ efficiently, it is doing less comparisons:: py_cmp_func 1 7 >>> -Ah, we're nearly done! The last step is to actually compare the two items and -return a useful result:: +Now we can actually compare the two items and return a useful result:: >>> def py_cmp_func(a, b): ... print("py_cmp_func", a[0], b[0]) ... return a[0] - b[0] ... >>> - -Final run on Windows:: - - >>> qsort(ia, len(ia), sizeof(c_int), CMPFUNC(py_cmp_func)) # doctest: +WINDOWS - py_cmp_func 33 7 - py_cmp_func 99 33 - py_cmp_func 5 99 - py_cmp_func 1 99 - py_cmp_func 33 7 - py_cmp_func 1 33 - py_cmp_func 5 33 - py_cmp_func 5 7 - py_cmp_func 1 7 - py_cmp_func 5 1 - >>> - -and on Linux:: - >>> qsort(ia, len(ia), sizeof(c_int), CMPFUNC(py_cmp_func)) # doctest: +LINUX py_cmp_func 5 1 py_cmp_func 33 99 @@ -1054,9 +996,6 @@ and on Linux:: py_cmp_func 5 7 >>> -It is quite interesting to see that the Windows :func:`qsort` function needs -more comparisons than the linux version! - As we can easily check, our array is sorted now:: >>> for i in ia: print(i, end=" ") @@ -1066,9 +1005,9 @@ As we can easily check, our array is sorted now:: **Important note for callback functions:** -Make sure you keep references to CFUNCTYPE objects as long as they are used from -C code. :mod:`ctypes` doesn't, and if you don't, they may be garbage collected, -crashing your program when a callback is made. +Make sure you keep references to :func:`CFUNCTYPE` objects as long as they are +used from C code. :mod:`ctypes` doesn't, and if you don't, they may be garbage +collected, crashing your program when a callback is made. .. _ctypes-accessing-values-exported-from-dlls: @@ -1345,7 +1284,10 @@ way is to instantiate one of the following classes: assumed to return the windows specific :class:`HRESULT` code. :class:`HRESULT` values contain information specifying whether the function call failed or succeeded, together with additional error code. If the return value signals a - failure, an :class:`WindowsError` is automatically raised. + failure, an :class:`OSError` is automatically raised. + + .. versionchanged:: 3.3 + :exc:`WindowsError` used to be raised. .. class:: WinDLL(name, mode=DEFAULT_MODE, handle=None, use_errno=False, use_last_error=False) @@ -1962,11 +1904,14 @@ Utility functions .. function:: WinError(code=None, descr=None) Windows only: this function is probably the worst-named thing in ctypes. It - creates an instance of WindowsError. If *code* is not specified, + creates an instance of OSError. If *code* is not specified, ``GetLastError`` is called to determine the error code. If *descr* is not specified, :func:`FormatError` is called to get a textual description of the error. + .. versionchanged:: 3.3 + An instance of :exc:`WindowsError` used to be created. + .. function:: wstring_at(address, size=-1) diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index f31b9c5..ff3a793 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -598,6 +598,17 @@ The module :mod:`curses` defines the following functions: Only one *ch* can be pushed before :meth:`getch` is called. +.. function:: unget_wch(ch) + + Push *ch* so the next :meth:`get_wch` will return it. + + .. note:: + + Only one *ch* can be pushed before :meth:`get_wch` is called. + + .. versionadded:: 3.3 + + .. function:: ungetmouse(id, x, y, z, bstate) Push a :const:`KEY_MOUSE` event onto the input queue, associating the given @@ -642,7 +653,7 @@ Window Objects -------------- Window objects, as returned by :func:`initscr` and :func:`newwin` above, have -the following methods: +the following methods and attributes: .. method:: window.addch([y, x,] ch[, attr]) @@ -823,6 +834,16 @@ the following methods: event. +.. attribute:: window.encoding + + Encoding used to encode method arguments (Unicode strings and characters). + The encoding attribute is inherited from by parent window when a subwindow + is created, for example with :meth:`window.subwin`. By default, the locale + encoding is used (see :func:`locale.getpreferredencoding`). + + .. versionadded:: 3.3 + + .. method:: window.erase() Clear the window. @@ -846,6 +867,14 @@ the following methods: until a key is pressed. +.. method:: window.get_wch([y, x]) + + Get a wide character. Like :meth:`getch`, but the integer returned is the + Unicode code point for the key pressed, so it can be passed to :func:`chr`. + + .. versionadded:: 3.3 + + .. method:: window.getkey([y, x]) Get a character, returning a string instead of an integer, as :meth:`getch` diff --git a/Doc/library/datatypes.rst b/Doc/library/datatypes.rst index 6b4a71a..8e33c1f 100644 --- a/Doc/library/datatypes.rst +++ b/Doc/library/datatypes.rst @@ -21,6 +21,7 @@ The following modules are documented in this chapter: datetime.rst calendar.rst collections.rst + collections.abc.rst heapq.rst bisect.rst array.rst diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index f7e8762..669236a 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -404,12 +404,19 @@ Other constructors, all class methods: .. classmethod:: date.fromtimestamp(timestamp) Return the local date corresponding to the POSIX timestamp, such as is returned - by :func:`time.time`. This may raise :exc:`ValueError`, if the timestamp is out - of the range of values supported by the platform C :c:func:`localtime` function. + by :func:`time.time`. This may raise :exc:`OverflowError`, if the timestamp is out + of the range of values supported by the platform C :c:func:`localtime` function, + and :exc:`OSError` on :c:func:`localtime` failure. It's common for this to be restricted to years from 1970 through 2038. Note that on non-POSIX systems that include leap seconds in their notion of a timestamp, leap seconds are ignored by :meth:`fromtimestamp`. + .. versionchanged:: 3.3 + Raise :exc:`OverflowError` instead of :exc:`ValueError` if the timestamp + is out of the range of values supported by the platform C + :c:func:`localtime` function. Raise :exc:`OSError` instead of + :exc:`ValueError` on :c:func:`localtime` failure. + .. classmethod:: date.fromordinal(ordinal) @@ -713,23 +720,55 @@ Other constructors, all class methods: equivalent to ``tz.fromutc(datetime.utcfromtimestamp(timestamp).replace(tzinfo=tz))``. - :meth:`fromtimestamp` may raise :exc:`ValueError`, if the timestamp is out of + :meth:`fromtimestamp` may raise :exc:`OverflowError`, if the timestamp is out of the range of values supported by the platform C :c:func:`localtime` or - :c:func:`gmtime` functions. It's common for this to be restricted to years in + :c:func:`gmtime` functions, and :exc:`OSError` on :c:func:`localtime` or + :c:func:`gmtime` failure. + It's common for this to be restricted to years in 1970 through 2038. Note that on non-POSIX systems that include leap seconds in their notion of a timestamp, leap seconds are ignored by :meth:`fromtimestamp`, and then it's possible to have two timestamps differing by a second that yield identical :class:`.datetime` objects. See also :meth:`utcfromtimestamp`. + .. versionchanged:: 3.3 + Raise :exc:`OverflowError` instead of :exc:`ValueError` if the timestamp + is out of the range of values supported by the platform C + :c:func:`localtime` or :c:func:`gmtime` functions. Raise :exc:`OSError` + instead of :exc:`ValueError` on :c:func:`localtime` or :c:func:`gmtime` + failure. + .. classmethod:: datetime.utcfromtimestamp(timestamp) Return the UTC :class:`.datetime` corresponding to the POSIX timestamp, with - :attr:`tzinfo` ``None``. This may raise :exc:`ValueError`, if the timestamp is - out of the range of values supported by the platform C :c:func:`gmtime` function. + :attr:`tzinfo` ``None``. This may raise :exc:`OverflowError`, if the timestamp is + out of the range of values supported by the platform C :c:func:`gmtime` function, + and :exc:`OSError` on :c:func:`gmtime` failure. It's common for this to be restricted to years in 1970 through 2038. See also :meth:`fromtimestamp`. + On the POSIX compliant platforms, ``utcfromtimestamp(timestamp)`` + is equivalent to the following expression:: + + datetime(1970, 1, 1) + timedelta(seconds=timestamp) + + There is no method to obtain the timestamp from a :class:`datetime` + instance, but POSIX timestamp corresponding to a :class:`datetime` + instance ``dt`` can be easily calculated as follows. For a naive + ``dt``:: + + timestamp = (dt - datetime(1970, 1, 1)) / timedelta(seconds=1) + + And for an aware ``dt``:: + + timestamp = (dt - datetime(1970, 1, 1, tzinfo=timezone.utc)) / timedelta(seconds=1) + + .. versionchanged:: 3.3 + Raise :exc:`OverflowError` instead of :exc:`ValueError` if the timestamp + is out of the range of values supported by the platform C + :c:func:`gmtime` function. Raise :exc:`OSError` instead of + :exc:`ValueError` on :c:func:`gmtime` failure. + .. classmethod:: datetime.fromordinal(ordinal) @@ -1588,11 +1627,12 @@ only EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). :class:`timezone` Objects -------------------------- -A :class:`timezone` object represents a timezone that is defined by a -fixed offset from UTC. Note that objects of this class cannot be used -to represent timezone information in the locations where different -offsets are used in different days of the year or where historical -changes have been made to civil time. +The :class:`timezone` class is a subclass of :class:`tzinfo`, each +instance of which represents a timezone defined by a fixed offset from +UTC. Note that objects of this class cannot be used to represent +timezone information in the locations where different offsets are used +in different days of the year or where historical changes have been +made to civil time. .. class:: timezone(offset[, name]) @@ -1761,8 +1801,7 @@ format codes. | | decimal number [00,99]. | | +-----------+--------------------------------+-------+ | ``%Y`` | Year with century as a decimal | \(5) | -| | number [0001,9999] (strptime), | | -| | [1000,9999] (strftime). | | +| | number [0001,9999]. | | +-----------+--------------------------------+-------+ | ``%z`` | UTC offset in the form +HHMM | \(6) | | | or -HHMM (empty string if the | | @@ -1796,10 +1835,7 @@ Notes: calculations when the day of the week and the year are specified. (5) - For technical reasons, :meth:`strftime` method does not support - dates before year 1000: ``t.strftime(format)`` will raise a - :exc:`ValueError` when ``t.year < 1000`` even if ``format`` does - not contain ``%Y`` directive. The :meth:`strptime` method can + The :meth:`strptime` method can parse years in the full [1, 9999] range, but years < 1000 must be zero-filled to 4-digit width. @@ -1807,6 +1843,10 @@ Notes: In previous versions, :meth:`strftime` method was restricted to years >= 1900. + .. versionchanged:: 3.3 + In version 3.2, :meth:`strftime` method was restricted to + years >= 1000. + (6) For example, if :meth:`utcoffset` returns ``timedelta(hours=-3, minutes=-30)``, ``%z`` is replaced with the string ``'-0330'``. diff --git a/Doc/library/debug.rst b/Doc/library/debug.rst index b2ee4fa..c69fb1c 100644 --- a/Doc/library/debug.rst +++ b/Doc/library/debug.rst @@ -10,7 +10,8 @@ allowing you to identify bottlenecks in your programs. .. toctree:: bdb.rst + faulthandler.rst pdb.rst profile.rst timeit.rst - trace.rst
\ No newline at end of file + trace.rst diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index ef8b43f..3fa006b 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -9,6 +9,7 @@ .. moduleauthor:: Raymond Hettinger <python at rcn.com> .. moduleauthor:: Aahz <aahz at pobox.com> .. moduleauthor:: Tim Peters <tim.one at comcast.net> +.. moduleauthor:: Stefan Krah <skrah at bytereef.org> .. sectionauthor:: Raymond D. Hettinger <python at rcn.com> .. import modules for testing inline doctests with the Sphinx doctest builder @@ -20,8 +21,9 @@ # make sure each group gets a fresh context setcontext(Context()) -The :mod:`decimal` module provides support for decimal floating point -arithmetic. It offers several advantages over the :class:`float` datatype: +The :mod:`decimal` module provides support for fast correctly-rounded +decimal floating point arithmetic. It offers several advantages over the +:class:`float` datatype: * Decimal "is based on a floating-point model which was designed with people in mind, and necessarily has a paramount guiding principle -- computers must @@ -92,7 +94,7 @@ computation. Depending on the needs of the application, signals may be ignored, considered as informational, or treated as exceptions. The signals in the decimal module are: :const:`Clamped`, :const:`InvalidOperation`, :const:`DivisionByZero`, :const:`Inexact`, :const:`Rounded`, :const:`Subnormal`, -:const:`Overflow`, and :const:`Underflow`. +:const:`Overflow`, :const:`Underflow` and :const:`FloatOperation`. For each signal there is a flag and a trap enabler. When a signal is encountered, its flag is set to one, then, if the trap enabler is @@ -122,7 +124,7 @@ precision, rounding, or enabled traps:: >>> from decimal import * >>> getcontext() - Context(prec=28, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999, + Context(prec=28, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999, capitals=1, clamp=0, flags=[], traps=[Overflow, DivisionByZero, InvalidOperation]) @@ -132,7 +134,7 @@ Decimal instances can be constructed from integers, strings, floats, or tuples. Construction from an integer or a float performs an exact conversion of the value of that integer or float. Decimal numbers include special values such as :const:`NaN` which stands for "Not a number", positive and negative -:const:`Infinity`, and :const:`-0`. +:const:`Infinity`, and :const:`-0`:: >>> getcontext().prec = 28 >>> Decimal(10) @@ -152,6 +154,25 @@ value of that integer or float. Decimal numbers include special values such as >>> Decimal('-Infinity') Decimal('-Infinity') +If the :exc:`FloatOperation` signal is trapped, accidental mixing of +decimals and floats in constructors or ordering comparisons raises +an exception:: + + >>> c = getcontext() + >>> c.traps[FloatOperation] = True + >>> Decimal(3.14) + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + decimal.FloatOperation: [<class 'decimal.FloatOperation'>] + >>> Decimal('3.5') < 3.7 + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + decimal.FloatOperation: [<class 'decimal.FloatOperation'>] + >>> Decimal('3.5') == 3.5 + True + +.. versionadded:: 3.3 + The significance of a new Decimal is determined solely by the number of digits input. Context precision and rounding only come into play during arithmetic operations. @@ -169,6 +190,16 @@ operations. >>> Decimal('3.1415926535') + Decimal('2.7182818285') Decimal('5.85988') +If the internal limits of the C version are exceeded, constructing +a decimal raises :class:`InvalidOperation`:: + + >>> Decimal("1e9999999999999999999") + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + decimal.InvalidOperation: [<class 'decimal.InvalidOperation'>] + +.. versionchanged:: 3.3 + Decimals interact well with much of the rest of Python. Here is a small decimal floating point flying circus: @@ -244,7 +275,7 @@ enabled: Decimal('0.142857142857142857142857142857142857142857142857142857142857') >>> ExtendedContext - Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999, + Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999, capitals=1, clamp=0, flags=[], traps=[]) >>> setcontext(ExtendedContext) >>> Decimal(1) / Decimal(7) @@ -269,7 +300,7 @@ using the :meth:`clear_flags` method. :: >>> Decimal(355) / Decimal(113) Decimal('3.14159292') >>> getcontext() - Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999999, Emax=999999999, + Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999, capitals=1, clamp=0, flags=[Inexact, Rounded], traps=[]) The *flags* entry shows that the rational approximation to :const:`Pi` was @@ -358,6 +389,10 @@ Decimal objects The argument to the constructor is now permitted to be a :class:`float` instance. + .. versionchanged:: 3.3 + :class:`float` arguments raise an exception if the :exc:`FloatOperation` + trap is set. By default the trap is off. + Decimal floating point objects share many properties with the other built-in numeric types such as :class:`float` and :class:`int`. All of the usual math operations and special methods apply. Likewise, decimal objects can be @@ -880,39 +915,33 @@ described below. In addition, the module provides three pre-made contexts: In single threaded environments, it is preferable to not use this context at all. Instead, simply create contexts explicitly as described below. - The default values are precision=28, rounding=ROUND_HALF_EVEN, and enabled traps - for Overflow, InvalidOperation, and DivisionByZero. + The default values are :attr:`prec`\ =\ :const:`28`, + :attr:`rounding`\ =\ :const:`ROUND_HALF_EVEN`, + and enabled traps for :class:`Overflow`, :class:`InvalidOperation`, and + :class:`DivisionByZero`. In addition to the three supplied contexts, new contexts can be created with the :class:`Context` constructor. -.. class:: Context(prec=None, rounding=None, traps=None, flags=None, Emin=None, Emax=None, capitals=None, clamp=None) +.. class:: Context(prec=None, rounding=None, Emin=None, Emax=None, capitals=None, clamp=None, flags=None, traps=None) Creates a new context. If a field is not specified or is :const:`None`, the default values are copied from the :const:`DefaultContext`. If the *flags* field is not specified or is :const:`None`, all flags are cleared. - The *prec* field is a positive integer that sets the precision for arithmetic - operations in the context. - - The *rounding* option is one of: + *prec* is an integer in the range [:const:`1`, :const:`MAX_PREC`] that sets + the precision for arithmetic operations in the context. - * :const:`ROUND_CEILING` (towards :const:`Infinity`), - * :const:`ROUND_DOWN` (towards zero), - * :const:`ROUND_FLOOR` (towards :const:`-Infinity`), - * :const:`ROUND_HALF_DOWN` (to nearest with ties going towards zero), - * :const:`ROUND_HALF_EVEN` (to nearest with ties going to nearest even integer), - * :const:`ROUND_HALF_UP` (to nearest with ties going away from zero), or - * :const:`ROUND_UP` (away from zero). - * :const:`ROUND_05UP` (away from zero if last digit after rounding towards zero - would have been 0 or 5; otherwise towards zero) + The *rounding* option is one of the constants listed in the section + `Rounding Modes`_. The *traps* and *flags* fields list any signals to be set. Generally, new contexts should only set traps and leave the flags clear. The *Emin* and *Emax* fields are integers specifying the outer limits allowable - for exponents. + for exponents. *Emin* must be in the range [:const:`MIN_EMIN`, :const:`0`], + *Emax* in the range [:const:`0`, :const:`MAX_EMAX`]. The *capitals* field is either :const:`0` or :const:`1` (the default). If set to :const:`1`, exponents are printed with a capital :const:`E`; otherwise, a @@ -951,6 +980,12 @@ In addition to the three supplied contexts, new contexts can be created with the Resets all of the flags to :const:`0`. + .. method:: clear_traps() + + Resets all of the traps to :const:`0`. + + .. versionadded:: 3.3 + .. method:: copy() Return a duplicate of the context. @@ -1250,8 +1285,13 @@ In addition to the three supplied contexts, new contexts can be created with the With two arguments, compute ``x**y``. If ``x`` is negative then ``y`` must be integral. The result will be inexact unless ``y`` is integral and the result is finite and can be expressed exactly in 'precision' digits. - The result should always be correctly rounded, using the rounding mode of - the current thread's context. + The rounding mode of the context is used. Results are always correctly-rounded + in the Python version. + + .. versionchanged:: 3.3 + The C module computes :meth:`power` in terms of the correctly-rounded + :meth:`exp` and :meth:`ln` functions. The result is well-defined but + only "almost always correctly-rounded". With three arguments, compute ``(x**y) % modulo``. For the three argument form, the following restrictions on the arguments hold: @@ -1339,6 +1379,69 @@ In addition to the three supplied contexts, new contexts can be created with the .. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +.. _decimal-rounding-modes: + +Constants +--------- + +The constants in this section are only relevant for the C module. They +are also included in the pure Python version for compatibility. + ++---------------------+---------------------+-------------------------------+ +| | 32-bit | 64-bit | ++=====================+=====================+===============================+ +| .. data:: MAX_PREC | :const:`425000000` | :const:`999999999999999999` | ++---------------------+---------------------+-------------------------------+ +| .. data:: MAX_EMAX | :const:`425000000` | :const:`999999999999999999` | ++---------------------+---------------------+-------------------------------+ +| .. data:: MIN_EMIN | :const:`-425000000` | :const:`-999999999999999999` | ++---------------------+---------------------+-------------------------------+ +| .. data:: MIN_ETINY | :const:`-849999999` | :const:`-1999999999999999997` | ++---------------------+---------------------+-------------------------------+ + + +.. data:: HAVE_THREADS + + The default value is True. If Python is compiled without threads, the + C version automatically disables the expensive thread local context + machinery. In this case, the value is False. + +Rounding modes +-------------- + +.. data:: ROUND_CEILING + + Round towards :const:`Infinity`. + +.. data:: ROUND_DOWN + + Round towards zero. + +.. data:: ROUND_FLOOR + + Round towards :const:`-Infinity`. + +.. data:: ROUND_HALF_DOWN + + Round to nearest with ties going towards zero. + +.. data:: ROUND_HALF_EVEN + + Round to nearest with ties going to nearest even integer. + +.. data:: ROUND_HALF_UP + + Round to nearest with ties going away from zero. + +.. data:: ROUND_UP + + Round away from zero. + +.. data:: ROUND_05UP + + Round away from zero if last digit after rounding towards zero would have + been 0 or 5; otherwise round towards zero. + .. _decimal-signals: @@ -1403,7 +1506,6 @@ condition. Infinity / Infinity x % 0 Infinity % x - x._rescale( non-integer ) sqrt(-x) and x > 0 0 ** 0 x ** (non-integer) @@ -1446,6 +1548,23 @@ condition. Occurs when a subnormal result is pushed to zero by rounding. :class:`Inexact` and :class:`Subnormal` are also signaled. + +.. class:: FloatOperation + + Enable stricter semantics for mixing floats and Decimals. + + If the signal is not trapped (default), mixing floats and Decimals is + permitted in the :class:`~decimal.Decimal` constructor, + :meth:`~decimal.Context.create_decimal` and all comparison operators. + Both conversion and comparisons are exact. Any occurrence of a mixed + operation is silently recorded by setting :exc:`FloatOperation` in the + context flags. Explicit conversions with :meth:`~decimal.Decimal.from_float` + or :meth:`~decimal.Context.create_decimal_from_float` do not set the flag. + + Otherwise (the signal is trapped), only equality comparisons and explicit + conversions are silent. All other mixed operations raise :exc:`FloatOperation`. + + The following table summarizes the hierarchy of signals:: exceptions.ArithmeticError(exceptions.Exception) @@ -1458,10 +1577,12 @@ The following table summarizes the hierarchy of signals:: InvalidOperation Rounded Subnormal + FloatOperation(DecimalException, exceptions.TypeError) .. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + .. _decimal-notes: Floating Point Notes @@ -1571,7 +1692,7 @@ normalized floating point representations, it is not immediately obvious that the following calculation returns a value equal to zero: >>> 1 / Decimal('Infinity') - Decimal('0E-1000000026') + Decimal('0E-1000026') .. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1583,7 +1704,7 @@ Working with threads The :func:`getcontext` function accesses a different :class:`Context` object for each thread. Having separate thread contexts means that threads may make -changes (such as ``getcontext.prec=10``) without interfering with other threads. +changes (such as ``getcontext().prec=10``) without interfering with other threads. Likewise, the :func:`setcontext` function automatically assigns its target to the current thread. diff --git a/Doc/library/depgraph-output.png b/Doc/library/depgraph-output.png Binary files differnew file mode 100644 index 0000000..960bb1b --- /dev/null +++ b/Doc/library/depgraph-output.png diff --git a/Doc/library/development.rst b/Doc/library/development.rst index c822e08..06e7048 100644 --- a/Doc/library/development.rst +++ b/Doc/library/development.rst @@ -19,5 +19,7 @@ The list of modules described in this chapter is: pydoc.rst doctest.rst unittest.rst + unittest.mock.rst + unittest.mock-examples.rst 2to3.rst test.rst diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index bdc37b3..836e240 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -752,8 +752,8 @@ It is also contained in the Python source distribution, as # we're passing these as arguments to the diff function fromdate = time.ctime(os.stat(fromfile).st_mtime) todate = time.ctime(os.stat(tofile).st_mtime) - fromlines = open(fromfile, 'U').readlines() - tolines = open(tofile, 'U').readlines() + with open(fromlines) as fromf, open(tofile) as tof: + fromlines, tolines = list(fromf), list(tof) if options.u: diff = difflib.unified_diff(fromlines, tolines, fromfile, tofile, diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 79cc583..6d08a0c 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -171,11 +171,6 @@ The Python compiler currently generates the following bytecode instructions. **General instructions** -.. opcode:: STOP_CODE - - Indicates end-of-code to the compiler, not used by the interpreter. - - .. opcode:: NOP Do nothing code. Used as a placeholder by the bytecode optimizer. @@ -436,6 +431,13 @@ the stack so that it is available for further iterations of the loop. Pops ``TOS`` and yields it from a :term:`generator`. +.. opcode:: YIELD_FROM + + Pops ``TOS`` and delegates to it as a subiterator from a :term:`generator`. + + .. versionadded:: 3.3 + + .. opcode:: IMPORT_STAR Loads all symbols not starting with ``'_'`` directly from the module TOS to the @@ -752,9 +754,10 @@ the more significant byte last. .. opcode:: MAKE_FUNCTION (argc) - Pushes a new function object on the stack. TOS is the code associated with the - function. The function object is defined to have *argc* default parameters, - which are found below TOS. + Pushes a new function object on the stack. TOS is the + :term:`qualified name` of the function; TOS1 is the code associated with + the function. The function object is defined to have *argc* default parameters, + which are found below TOS1. .. opcode:: MAKE_CLOSURE (argc) diff --git a/Doc/library/distutils.rst b/Doc/library/distutils.rst index 238b79d..53a69ae 100644 --- a/Doc/library/distutils.rst +++ b/Doc/library/distutils.rst @@ -12,18 +12,26 @@ additional modules into a Python installation. The new modules may be either 100%-pure Python, or may be extension modules written in C, or may be collections of Python packages which include modules coded in both Python and C. -This package is discussed in two separate chapters: +.. deprecated:: 3.3 + :mod:`packaging` replaces Distutils. See :ref:`packaging-index` and + :ref:`packaging-install-index`. +User documentation and API reference are provided in another document: + .. seealso:: :ref:`distutils-index` The manual for developers and packagers of Python modules. This describes how to prepare :mod:`distutils`\ -based packages so that they may be - easily installed into an existing Python installation. + easily installed into an existing Python installation. If also contains + instructions for end-users wanting to install a distutils-based package, + :ref:`install-index`. + + +.. trick to silence a Sphinx warning - :ref:`install-index` - An "administrators" manual which includes information on installing - modules into an existing Python installation. You do not need to be a - Python programmer to read this manual. +.. toctree:: + :hidden: + ../distutils/index diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst index 3be04a2..03733ee 100644 --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -32,7 +32,8 @@ Here are the public methods of the :class:`Generator` class, imported from the :mod:`email.generator` module: -.. class:: Generator(outfp, mangle_from_=True, maxheaderlen=78) +.. class:: Generator(outfp, mangle_from_=True, maxheaderlen=78, *, \ + policy=policy.default) The constructor for the :class:`Generator` class takes a :term:`file-like object` called *outfp* for an argument. *outfp* must support the :meth:`write` method @@ -53,10 +54,16 @@ Here are the public methods of the :class:`Generator` class, imported from the :class:`~email.header.Header` class. Set to zero to disable header wrapping. The default is 78, as recommended (but not required) by :rfc:`2822`. + The *policy* keyword specifies a :mod:`~email.policy` object that controls a + number of aspects of the generator's operation. The default policy + maintains backward compatibility. + + .. versionchanged:: 3.3 Added the *policy* keyword. + The other public :class:`Generator` methods are: - .. method:: flatten(msg, unixfrom=False, linesep='\\n') + .. method:: flatten(msg, unixfrom=False, linesep=None) Print the textual representation of the message object structure rooted at *msg* to the output file specified when the :class:`Generator` instance @@ -72,12 +79,13 @@ Here are the public methods of the :class:`Generator` class, imported from the Note that for subparts, no envelope header is ever printed. Optional *linesep* specifies the line separator character used to - terminate lines in the output. It defaults to ``\n`` because that is - the most useful value for Python application code (other library packages - expect ``\n`` separated lines). ``linesep=\r\n`` can be used to - generate output with RFC-compliant line separators. + terminate lines in the output. If specified it overrides the value + specified by the ``Generator``\'s ``policy``. - Messages parsed with a Bytes parser that have a + Because strings cannot represent non-ASCII bytes, ``Generator`` ignores + the value of the :attr:`~email.policy.Policy.must_be_7bit` + :mod:`~email.policy` setting and operates as if it were set ``True``. + This means that messages parsed with a Bytes parser that have a :mailheader:`Content-Transfer-Encoding` of 8bit will be converted to a use a 7bit Content-Transfer-Encoding. Non-ASCII bytes in the headers will be :rfc:`2047` encoded with a charset of `unknown-8bit`. @@ -103,7 +111,8 @@ As a convenience, see the :class:`~email.message.Message` methods formatted string representation of a message object. For more detail, see :mod:`email.message`. -.. class:: BytesGenerator(outfp, mangle_from_=True, maxheaderlen=78) +.. class:: BytesGenerator(outfp, mangle_from_=True, maxheaderlen=78, *, \ + policy=policy.default) The constructor for the :class:`BytesGenerator` class takes a binary :term:`file-like object` called *outfp* for an argument. *outfp* must @@ -125,19 +134,31 @@ formatted string representation of a message object. For more detail, see wrapping. The default is 78, as recommended (but not required) by :rfc:`2822`. + The *policy* keyword specifies a :mod:`~email.policy` object that controls a + number of aspects of the generator's operation. The default policy + maintains backward compatibility. + + .. versionchanged:: 3.3 Added the *policy* keyword. + The other public :class:`BytesGenerator` methods are: - .. method:: flatten(msg, unixfrom=False, linesep='\n') + .. method:: flatten(msg, unixfrom=False, linesep=None) Print the textual representation of the message object structure rooted at *msg* to the output file specified when the :class:`BytesGenerator` instance was created. Subparts are visited depth-first and the resulting - text will be properly MIME encoded. If the input that created the *msg* - contained bytes with the high bit set and those bytes have not been - modified, they will be copied faithfully to the output, even if doing so - is not strictly RFC compliant. (To produce strictly RFC compliant - output, use the :class:`Generator` class.) + text will be properly MIME encoded. If the :mod:`~email.policy` option + :attr:`~email.policy.Policy.must_be_7bit` is ``False`` (the default), + then any bytes with the high bit set in the original parsed message that + have not been modified will be copied faithfully to the output. If + ``must_be_7bit`` is true, the bytes will be converted as needed using an + ASCII content-transfer-encoding. In particular, RFC-invalid non-ASCII + bytes in headers will be encoded using the MIME ``unknown-8bit`` + character set, thus rendering them RFC-compliant. + + .. XXX: There should be a complementary option that just does the RFC + compliance transformation but leaves CTE 8bit parts alone. Messages parsed with a Bytes parser that have a :mailheader:`Content-Transfer-Encoding` of 8bit will be reconstructed @@ -152,10 +173,8 @@ formatted string representation of a message object. For more detail, see Note that for subparts, no envelope header is ever printed. Optional *linesep* specifies the line separator character used to - terminate lines in the output. It defaults to ``\n`` because that is - the most useful value for Python application code (other library packages - expect ``\n`` separated lines). ``linesep=\r\n`` can be used to - generate output with RFC-compliant line separators. + terminate lines in the output. If specified it overrides the value + specified by the ``Generator``\ 's ``policy``. .. method:: clone(fp) diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst index 9cd94d8..f3f34c8 100644 --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -175,7 +175,7 @@ Here are the classes: .. currentmodule:: email.mime.text -.. class:: MIMEText(_text, _subtype='plain', _charset='us-ascii') +.. class:: MIMEText(_text, _subtype='plain', _charset=None) Module: :mod:`email.mime.text` @@ -185,5 +185,5 @@ Here are the classes: minor type and defaults to :mimetype:`plain`. *_charset* is the character set of the text and is passed as a parameter to the :class:`~email.mime.nonmultipart.MIMENonMultipart` constructor; it defaults - to ``us-ascii``. No guessing or encoding is performed on the text data. - + to ``us-ascii`` if the string contains only ``ascii`` codepoints, and + ``utf-8`` otherwise. diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst index 384c5c9..a0303a4 100644 --- a/Doc/library/email.parser.rst +++ b/Doc/library/email.parser.rst @@ -58,12 +58,18 @@ list of defects that it can find. Here is the API for the :class:`FeedParser`: -.. class:: FeedParser(_factory=email.message.Message) +.. class:: FeedParser(_factory=email.message.Message, *, policy=policy.default) Create a :class:`FeedParser` instance. Optional *_factory* is a no-argument callable that will be called whenever a new message object is needed. It defaults to the :class:`email.message.Message` class. + The *policy* keyword specifies a :mod:`~email.policy` object that controls a + number of aspects of the parser's operation. The default policy maintains + backward compatibility. + + .. versionchanged:: 3.3 Added the *policy* keyword. + .. method:: feed(data) Feed the :class:`FeedParser` some more data. *data* should be a string @@ -94,15 +100,17 @@ Parser class API The :class:`Parser` class, imported from the :mod:`email.parser` module, provides an API that can be used to parse a message when the complete contents of the message are available in a string or file. The :mod:`email.parser` -module also provides a second class, called :class:`HeaderParser` which can be -used if you're only interested in the headers of the message. -:class:`HeaderParser` can be much faster in these situations, since it does not -attempt to parse the message body, instead setting the payload to the raw body -as a string. :class:`HeaderParser` has the same API as the :class:`Parser` -class. +module also provides header-only parsers, called :class:`HeaderParser` and +:class:`BytesHeaderParser`, which can be used if you're only interested in the +headers of the message. :class:`HeaderParser` and :class:`BytesHeaderParser` +can be much faster in these situations, since they do not attempt to parse the +message body, instead setting the payload to the raw body as a string. They +have the same API as the :class:`Parser` and :class:`BytesParser` classes. +.. versionadded:: 3.3 BytesHeaderParser -.. class:: Parser(_class=email.message.Message, strict=None) + +.. class:: Parser(_class=email.message.Message, *, policy=policy.default) The constructor for the :class:`Parser` class takes an optional argument *_class*. This must be a callable factory (such as a function or a class), and @@ -110,13 +118,13 @@ class. :class:`~email.message.Message` (see :mod:`email.message`). The factory will be called without arguments. - The optional *strict* flag is ignored. + The *policy* keyword specifies a :mod:`~email.policy` object that controls a + number of aspects of the parser's operation. The default policy maintains + backward compatibility. - .. deprecated:: 2.4 - Because the :class:`Parser` class is a backward compatible API wrapper - around the new-in-Python 2.4 :class:`FeedParser`, *all* parsing is - effectively non-strict. You should simply stop passing a *strict* flag to - the :class:`Parser` constructor. + .. versionchanged:: 3.3 + Removed the *strict* argument that was deprecated in 2.4. Added the + *policy* keyword. The other public :class:`Parser` methods are: @@ -147,12 +155,18 @@ class. Optional *headersonly* is as with the :meth:`parse` method. -.. class:: BytesParser(_class=email.message.Message, strict=None) +.. class:: BytesParser(_class=email.message.Message, *, policy=policy.default) This class is exactly parallel to :class:`Parser`, but handles bytes input. The *_class* and *strict* arguments are interpreted in the same way as for - the :class:`Parser` constructor. *strict* is supported only to make porting - code easier; it is deprecated. + the :class:`Parser` constructor. + + The *policy* keyword specifies a :mod:`~email.policy` object that + controls a number of aspects of the parser's operation. The default + policy maintains backward compatibility. + + .. versionchanged:: 3.3 + Removed the *strict* argument. Added the *policy* keyword. .. method:: parse(fp, headeronly=False) @@ -190,34 +204,48 @@ in the top-level :mod:`email` package namespace. .. currentmodule:: email -.. function:: message_from_string(s, _class=email.message.Message, strict=None) +.. function:: message_from_string(s, _class=email.message.Message, *, \ + policy=policy.default) Return a message object structure from a string. This is exactly equivalent to - ``Parser().parsestr(s)``. Optional *_class* and *strict* are interpreted as + ``Parser().parsestr(s)``. *_class* and *policy* are interpreted as with the :class:`Parser` class constructor. -.. function:: message_from_bytes(s, _class=email.message.Message, strict=None) + .. versionchanged:: 3.3 + Removed the *strict* argument. Added the *policy* keyword. + +.. function:: message_from_bytes(s, _class=email.message.Message, *, \ + policy=policy.default) Return a message object structure from a byte string. This is exactly equivalent to ``BytesParser().parsebytes(s)``. Optional *_class* and *strict* are interpreted as with the :class:`Parser` class constructor. .. versionadded:: 3.2 + .. versionchanged:: 3.3 + Removed the *strict* argument. Added the *policy* keyword. -.. function:: message_from_file(fp, _class=email.message.Message, strict=None) +.. function:: message_from_file(fp, _class=email.message.Message, *, \ + policy=policy.default) Return a message object structure tree from an open :term:`file object`. - This is exactly equivalent to ``Parser().parse(fp)``. Optional *_class* - and *strict* are interpreted as with the :class:`Parser` class constructor. + This is exactly equivalent to ``Parser().parse(fp)``. *_class* + and *policy* are interpreted as with the :class:`Parser` class constructor. + + .. versionchanged:: + Removed the *strict* argument. Added the *policy* keyword. -.. function:: message_from_binary_file(fp, _class=email.message.Message, strict=None) +.. function:: message_from_binary_file(fp, _class=email.message.Message, *, \ + policy=policy.default) Return a message object structure tree from an open binary :term:`file object`. This is exactly equivalent to ``BytesParser().parse(fp)``. - Optional *_class* and *strict* are interpreted as with the :class:`Parser` + *_class* and *policy* are interpreted as with the :class:`Parser` class constructor. .. versionadded:: 3.2 + .. versionchanged:: 3.3 + Removed the *strict* argument. Added the *policy* keyword. Here's an example of how you might use this at an interactive Python prompt:: diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst new file mode 100644 index 0000000..d9a292c --- /dev/null +++ b/Doc/library/email.policy.rst @@ -0,0 +1,184 @@ +:mod:`email`: Policy Objects +---------------------------- + +.. module:: email.policy + :synopsis: Controlling the parsing and generating of messages + +.. versionadded:: 3.3 + + +The :mod:`email` package's prime focus is the handling of email messages as +described by the various email and MIME RFCs. However, the general format of +email messages (a block of header fields each consisting of a name followed by +a colon followed by a value, the whole block followed by a blank line and an +arbitrary 'body'), is a format that has found utility outside of the realm of +email. Some of these uses conform fairly closely to the main RFCs, some do +not. And even when working with email, there are times when it is desirable to +break strict compliance with the RFCs. + +Policy objects give the email package the flexibility to handle all these +disparate use cases. + +A :class:`Policy` object encapsulates a set of attributes and methods that +control the behavior of various components of the email package during use. +:class:`Policy` instances can be passed to various classes and methods in the +email package to alter the default behavior. The settable values and their +defaults are described below. The :mod:`policy` module also provides some +pre-created :class:`Policy` instances. In addition to a :const:`default` +instance, there are instances tailored for certain applications. For example +there is an :const:`SMTP` :class:`Policy` with defaults appropriate for +generating output to be sent to an SMTP server. These are listed `below +<Policy Instances>`. + +In general an application will only need to deal with setting the policy at the +input and output boundaries. Once parsed, a message is represented by a +:class:`~email.message.Message` object, which is designed to be independent of +the format that the message has "on the wire" when it is received, transmitted, +or displayed. Thus, a :class:`Policy` can be specified when parsing a message +to create a :class:`~email.message.Message`, and again when turning the +:class:`~email.message.Message` into some other representation. While often a +program will use the same :class:`Policy` for both input and output, the two +can be different. + +As an example, the following code could be used to read an email message from a +file on disk and pass it to the system ``sendmail`` program on a Unix system:: + + >>> from email import msg_from_binary_file + >>> from email.generator import BytesGenerator + >>> import email.policy + >>> from subprocess import Popen, PIPE + >>> with open('mymsg.txt', 'b') as f: + ... msg = msg_from_binary_file(f, policy=email.policy.mbox) + >>> p = Popen(['sendmail', msg['To'][0].address], stdin=PIPE) + >>> g = BytesGenerator(p.stdin, policy=email.policy.SMTP) + >>> g.flatten(msg) + >>> p.stdin.close() + >>> rc = p.wait() + +.. XXX email.policy.mbox/MBOX does not exist yet + +Some email package methods accept a *policy* keyword argument, allowing the +policy to be overridden for that method. For example, the following code uses +the :meth:`~email.message.Message.as_string` method of the *msg* object from the +previous example and re-write it to a file using the native line separators for +the platform on which it is running:: + + >>> import os + >>> mypolicy = email.policy.Policy(linesep=os.linesep) + >>> with open('converted.txt', 'wb') as f: + ... f.write(msg.as_string(policy=mypolicy)) + +Policy instances are immutable, but they can be cloned, accepting the same +keyword arguments as the class constructor and returning a new :class:`Policy` +instance that is a copy of the original but with the specified attributes +values changed. For example, the following creates an SMTP policy that will +raise any defects detected as errors:: + + >>> strict_SMTP = email.policy.SMTP.clone(raise_on_defect=True) + +Policy objects can also be combined using the addition operator, producing a +policy object whose settings are a combination of the non-default values of the +summed objects:: + + >>> strict_SMTP = email.policy.SMTP + email.policy.strict + +This operation is not commutative; that is, the order in which the objects are +added matters. To illustrate:: + + >>> Policy = email.policy.Policy + >>> apolicy = Policy(max_line_length=100) + Policy(max_line_length=80) + >>> apolicy.max_line_length + 80 + >>> apolicy = Policy(max_line_length=80) + Policy(max_line_length=100) + >>> apolicy.max_line_length + 100 + + +.. class:: Policy(**kw) + + The valid constructor keyword arguments are any of the attributes listed + below. + + .. attribute:: max_line_length + + The maximum length of any line in the serialized output, not counting the + end of line character(s). Default is 78, per :rfc:`5322`. A value of + ``0`` or :const:`None` indicates that no line wrapping should be + done at all. + + .. attribute:: linesep + + The string to be used to terminate lines in serialized output. The + default is ``\n`` because that's the internal end-of-line discipline used + by Python, though ``\r\n`` is required by the RFCs. See `Policy + Instances`_ for policies that use an RFC conformant linesep. Setting it + to :attr:`os.linesep` may also be useful. + + .. attribute:: must_be_7bit + + If ``True``, data output by a bytes generator is limited to ASCII + characters. If :const:`False` (the default), then bytes with the high + bit set are preserved and/or allowed in certain contexts (for example, + where possible a content transfer encoding of ``8bit`` will be used). + String generators act as if ``must_be_7bit`` is ``True`` regardless of + the policy in effect, since a string cannot represent non-ASCII bytes. + + .. attribute:: raise_on_defect + + If :const:`True`, any defects encountered will be raised as errors. If + :const:`False` (the default), defects will be passed to the + :meth:`register_defect` method. + + :mod:`Policy` object also have the following methods: + + .. method:: handle_defect(obj, defect) + + *obj* is the object on which to register the defect. *defect* should be + an instance of a subclass of :class:`~email.errors.Defect`. + If :attr:`raise_on_defect` + is ``True`` the defect is raised as an exception. Otherwise *obj* and + *defect* are passed to :meth:`register_defect`. This method is intended + to be called by parsers when they encounter defects, and will not be + called by code that uses the email library unless that code is + implementing an alternate parser. + + .. method:: register_defect(obj, defect) + + *obj* is the object on which to register the defect. *defect* should be + a subclass of :class:`~email.errors.Defect`. This method is part of the + public API so that custom ``Policy`` subclasses can implement alternate + handling of defects. The default implementation calls the ``append`` + method of the ``defects`` attribute of *obj*. + + .. method:: clone(obj, *kw) + + Return a new :class:`Policy` instance whose attributes have the same + values as the current instance, except where those attributes are + given new values by the keyword arguments. + + +Policy Instances +^^^^^^^^^^^^^^^^ + +The following instances of :class:`Policy` provide defaults suitable for +specific common application domains. + +.. data:: default + + An instance of :class:`Policy` with all defaults unchanged. + +.. data:: SMTP + + Output serialized from a message will conform to the email and SMTP + RFCs. The only changed attribute is :attr:`linesep`, which is set to + ``\r\n``. + +.. data:: HTTP + + Suitable for use when serializing headers for use in HTTP traffic. + :attr:`linesep` is set to ``\r\n``, and :attr:`max_line_length` is set to + :const:`None` (unlimited). + +.. data:: strict + + :attr:`raise_on_defect` is set to :const:`True`. diff --git a/Doc/library/email.rst b/Doc/library/email.rst index 4530b95..fc206f4 100644 --- a/Doc/library/email.rst +++ b/Doc/library/email.rst @@ -51,6 +51,7 @@ Contents of the :mod:`email` package documentation: email.message.rst email.parser.rst email.generator.rst + email.policy.rst email.mime.rst email.header.rst email.charset.rst diff --git a/Doc/library/email.util.rst b/Doc/library/email.util.rst index f7b777a..2f9ef89 100644 --- a/Doc/library/email.util.rst +++ b/Doc/library/email.util.rst @@ -29,13 +29,20 @@ There are several useful utilities provided in the :mod:`email.utils` module: fails, in which case a 2-tuple of ``('', '')`` is returned. -.. function:: formataddr(pair) +.. function:: formataddr(pair, charset='utf-8') The inverse of :meth:`parseaddr`, this takes a 2-tuple of the form ``(realname, email_address)`` and returns the string value suitable for a :mailheader:`To` or :mailheader:`Cc` header. If the first element of *pair* is false, then the second element is returned unmodified. + Optional *charset* is the character set that will be used in the :rfc:`2047` + encoding of the ``realname`` if the ``realname`` contains non-ASCII + characters. Can be an instance of :class:`str` or a + :class:`~email.charset.Charset`. Defaults to ``utf-8``. + + .. versionchanged: 3.3 added the *charset* option + .. function:: getaddresses(fieldvalues) @@ -74,6 +81,20 @@ There are several useful utilities provided in the :mod:`email.utils` module: indexes 6, 7, and 8 of the result tuple are not usable. +.. function:: parsedate_to_datetime(date) + + The inverse of :func:`format_datetime`. Performs the same function as + :func:`parsedate`, but on success returns a :mod:`~datetime.datetime`. If + the input date has a timezone of ``-0000``, the ``datetime`` will be a naive + ``datetime``, and if the date is conforming to the RFCs it will represent a + time in UTC but with no indication of the actual source timezone of the + message the date comes from. If the input date has any other valid timezone + offset, the ``datetime`` will be an aware ``datetime`` with the + corresponding a :class:`~datetime.timezone` :class:`~datetime.tzinfo`. + + .. versionadded:: 3.3 + + .. function:: mktime_tz(tuple) Turn a 10-tuple as returned by :func:`parsedate_tz` into a UTC timestamp. It @@ -105,6 +126,20 @@ There are several useful utilities provided in the :mod:`email.utils` module: ``False``. The default is ``False``. +.. function:: format_datetime(dt, usegmt=False) + + Like ``formatdate``, but the input is a :mod:`datetime` instance. If it is + a naive datetime, it is assumed to be "UTC with no information about the + source timezone", and the conventional ``-0000`` is used for the timezone. + If it is an aware ``datetime``, then the numeric timezone offset is used. + If it is an aware timezone with offset zero, then *usegmt* may be set to + ``True``, in which case the string ``GMT`` is used instead of the numeric + timezone offset. This provides a way to generate standards conformant HTTP + date headers. + + .. versionadded:: 3.3 + + .. function:: make_msgid(idstring=None, domain=None) Returns a string suitable for an :rfc:`2822`\ -compliant diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index ca3ad3e..faf6732 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -34,6 +34,27 @@ programmers are encouraged to at least derive new exceptions from the defining exceptions is available in the Python Tutorial under :ref:`tut-userexceptions`. +When raising (or re-raising) an exception in an :keyword:`except` clause +:attr:`__context__` is automatically set to the last exception caught; if the +new exception is not handled the traceback that is eventually displayed will +include the originating exception(s) and the final exception. + +This implicit exception chain can be made explicit by using :keyword:`from` with +:keyword:`raise`. The single argument to :keyword:`from` must be an exception +or ``None``. It will be set as :attr:`__cause__` on the raised exception. +Setting :attr:`__cause__` implicitly sets the :attr:`__suppress_context__` to +``True``. If :attr:`__cause__` is an exception, it will be displayed. If +:attr:`__cause__` is present or :attr:`__suppress_context__` has a true value, +:attr:`__context__` will not be displayed. + +In either case, the default exception handling code will not display any of the +remaining links in the :attr:`__context__` chain if :attr:`__cause__` has been +set. + + +Base classes +------------ + The following exceptions are used mostly as base classes for other exceptions. .. exception:: BaseException @@ -90,27 +111,8 @@ The following exceptions are used mostly as base classes for other exceptions. can be raised directly by :func:`codecs.lookup`. -.. exception:: EnvironmentError - - The base class for exceptions that can occur outside the Python system: - :exc:`IOError`, :exc:`OSError`. When exceptions of this type are created with a - 2-tuple, the first item is available on the instance's :attr:`errno` attribute - (it is assumed to be an error number), and the second item is available on the - :attr:`strerror` attribute (it is usually the associated error message). The - tuple itself is also available on the :attr:`args` attribute. - - When an :exc:`EnvironmentError` exception is instantiated with a 3-tuple, the - first two items are available as above, while the third item is available on the - :attr:`filename` attribute. However, for backwards compatibility, the - :attr:`args` attribute contains only a 2-tuple of the first two constructor - arguments. - - The :attr:`filename` attribute is ``None`` when this exception is created with - other than 3 arguments. The :attr:`errno` and :attr:`strerror` attributes are - also ``None`` when the instance was created with other than 2 or 3 arguments. - In this last case, :attr:`args` contains the verbatim constructor arguments as a - tuple. - +Concrete exceptions +------------------- The following exceptions are the exceptions that are usually raised. @@ -151,21 +153,19 @@ The following exceptions are the exceptions that are usually raised. it is technically not an error. -.. exception:: IOError - - Raised when an I/O operation (such as the built-in :func:`print` or - :func:`open` functions or a method of a :term:`file object`) fails for an - I/O-related reason, e.g., "file not found" or "disk full". - - This class is derived from :exc:`EnvironmentError`. See the discussion above - for more information on exception instance attributes. - - .. exception:: ImportError Raised when an :keyword:`import` statement fails to find the module definition or when a ``from ... import`` fails to find a name that is to be imported. + The :attr:`name` and :attr:`path` attributes can be set using keyword-only + arguments to the constructor. When set they represent the name of the module + that was attempted to be imported and the path to any file which triggered + the exception, respectively. + + .. versionchanged:: 3.3 + Added the :attr:`name` and :attr:`path` attributes. + .. exception:: IndexError @@ -221,17 +221,30 @@ The following exceptions are the exceptions that are usually raised. .. index:: module: errno - This exception is derived from :exc:`EnvironmentError`. It is raised when a - function returns a system-related error (not for illegal argument types or - other incidental errors). The :attr:`errno` attribute is a numeric error - code from :c:data:`errno`, and the :attr:`strerror` attribute is the - corresponding string, as would be printed by the C function :c:func:`perror`. - See the module :mod:`errno`, which contains names for the error codes defined - by the underlying operating system. + This exception is raised when a system function returns a system-related + error, including I/O failures such as "file not found" or "disk full" + (not for illegal argument types or other incidental errors). Often a + subclass of :exc:`OSError` will actually be raised as described in + `OS exceptions`_ below. The :attr:`errno` attribute is a numeric error + code from the C variable :c:data:`errno`. + + Under Windows, the :attr:`winerror` attribute gives you the native + Windows error code. The :attr:`errno` attribute is then an approximate + translation, in POSIX terms, of that native error code. + + Under all platforms, the :attr:`strerror` attribute is the corresponding + error message as provided by the operating system (as formatted by the C + functions :c:func:`perror` under POSIX, and :c:func:`FormatMessage` + Windows). - For exceptions that involve a file system path (such as :func:`chdir` or - :func:`unlink`), the exception instance will contain a third attribute, - :attr:`filename`, which is the file name passed to the function. + For exceptions that involve a file system path (such as :func:`open` or + :func:`os.unlink`), the exception instance will contain an additional + attribute, :attr:`filename`, which is the file name passed to the function. + + .. versionchanged:: 3.3 + :exc:`EnvironmentError`, :exc:`IOError`, :exc:`WindowsError`, + :exc:`VMSError`, :exc:`socket.error`, :exc:`select.error` and + :exc:`mmap.error` have been merged into :exc:`OSError`. .. exception:: OverflowError @@ -262,8 +275,20 @@ The following exceptions are the exceptions that are usually raised. .. exception:: StopIteration Raised by built-in function :func:`next` and an :term:`iterator`\'s - :meth:`__next__` method to signal that there are no further values. + :meth:`__next__` method to signal that there are no further items to be + produced by the iterator. + + The exception object has a single attribute :attr:`value`, which is + given as an argument when constructing the exception, and defaults + to :const:`None`. + + When a generator function returns, a new :exc:`StopIteration` instance is + raised, and the value returned by the function is used as the + :attr:`value` parameter to the constructor of the exception. + .. versionchanged:: 3.3 + Added ``value`` attribute and the ability for generator functions to + use it to return a value. .. exception:: SyntaxError @@ -372,27 +397,142 @@ The following exceptions are the exceptions that are usually raised. more precise exception such as :exc:`IndexError`. -.. exception:: VMSError +.. exception:: ZeroDivisionError - Only available on VMS. Raised when a VMS-specific error occurs. + Raised when the second argument of a division or modulo operation is zero. The + associated value is a string indicating the type of the operands and the + operation. +The following exceptions are kept for compatibility with previous versions; +starting from Python 3.3, they are aliases of :exc:`OSError`. + +.. exception:: EnvironmentError + +.. exception:: IOError + +.. exception:: VMSError + + Only available on VMS. + .. exception:: WindowsError - Raised when a Windows-specific error occurs or when the error number does not - correspond to an :c:data:`errno` value. The :attr:`winerror` and - :attr:`strerror` values are created from the return values of the - :c:func:`GetLastError` and :c:func:`FormatMessage` functions from the Windows - Platform API. The :attr:`errno` value maps the :attr:`winerror` value to - corresponding ``errno.h`` values. This is a subclass of :exc:`OSError`. + Only available on Windows. -.. exception:: ZeroDivisionError +OS exceptions +^^^^^^^^^^^^^ + +The following exceptions are subclasses of :exc:`OSError`, they get raised +depending on the system error code. + +.. exception:: BlockingIOError + + Raised when an operation would block on an object (e.g. socket) set + for non-blocking operation. + Corresponds to :c:data:`errno` ``EAGAIN``, ``EALREADY``, + ``EWOULDBLOCK`` and ``EINPROGRESS``. + + In addition to those of :exc:`OSError`, :exc:`BlockingIOError` can have + one more attribute: + + .. attribute:: characters_written + + An integer containing the number of characters written to the stream + before it blocked. This attribute is available when using the + buffered I/O classes from the :mod:`io` module. + +.. exception:: ChildProcessError + + Raised when an operation on a child process failed. + Corresponds to :c:data:`errno` ``ECHILD``. + +.. exception:: ConnectionError + + A base class for connection-related issues. Subclasses are + :exc:`BrokenPipeError`, :exc:`ConnectionAbortedError`, + :exc:`ConnectionRefusedError` and :exc:`ConnectionResetError`. + + .. exception:: BrokenPipeError + + A subclass of :exc:`ConnectionError`, raised when trying to write on a + pipe while the other end has been closed, or trying to write on a socket + which has been shutdown for writing. + Corresponds to :c:data:`errno` ``EPIPE`` and ``ESHUTDOWN``. + + .. exception:: ConnectionAbortedError + + A subclass of :exc:`ConnectionError`, raised when a connection attempt + is aborted by the peer. + Corresponds to :c:data:`errno` ``ECONNABORTED``. + + .. exception:: ConnectionRefusedError + + A subclass of :exc:`ConnectionError`, raised when a connection attempt + is refused by the peer. + Corresponds to :c:data:`errno` ``ECONNREFUSED``. + + .. exception:: ConnectionResetError + + A subclass of :exc:`ConnectionError`, raised when a connection is + reset by the peer. + Corresponds to :c:data:`errno` ``ECONNRESET``. + +.. exception:: FileExistsError + + Raised when trying to create a file or directory which already exists. + Corresponds to :c:data:`errno` ``EEXIST``. + +.. exception:: FileNotFoundError + + Raised when a file or directory is requested but doesn't exist. + Corresponds to :c:data:`errno` ``ENOENT``. + +.. exception:: InterruptedError + + Raised when a system call is interrupted by an incoming signal. + Corresponds to :c:data:`errno` ``EEINTR``. + +.. exception:: IsADirectoryError + + Raised when a file operation (such as :func:`os.remove`) is requested + on a directory. + Corresponds to :c:data:`errno` ``EISDIR``. + +.. exception:: NotADirectoryError + + Raised when a directory operation (such as :func:`os.listdir`) is requested + on something which is not a directory. + Corresponds to :c:data:`errno` ``ENOTDIR``. + +.. exception:: PermissionError + + Raised when trying to run an operation without the adequate access + rights - for example filesystem permissions. + Corresponds to :c:data:`errno` ``EACCES`` and ``EPERM``. + +.. exception:: ProcessLookupError + + Raised when a given process doesn't exist. + Corresponds to :c:data:`errno` ``ESRCH``. + +.. exception:: TimeoutError + + Raised when a system function timed out at the system level. + Corresponds to :c:data:`errno` ``ETIMEDOUT``. + +.. versionadded:: 3.3 + All the above :exc:`OSError` subclasses were added. + + +.. seealso:: + + :pep:`3151` - Reworking the OS and IO exception hierarchy + PEP written and implemented by Antoine Pitrou. - Raised when the second argument of a division or modulo operation is zero. The - associated value is a string indicating the type of the operands and the - operation. +Warnings +-------- The following exceptions are used as warning categories; see the :mod:`warnings` module for more information. diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst new file mode 100644 index 0000000..c9b9546 --- /dev/null +++ b/Doc/library/faulthandler.rst @@ -0,0 +1,136 @@ +:mod:`faulthandler` --- Dump the Python traceback +================================================= + +.. module:: faulthandler + :synopsis: Dump the Python traceback. + +This module contains functions to dump Python tracebacks explicitly, on a fault, +after a timeout, or on a user signal. Call :func:`faulthandler.enable` to +install fault handlers for the :const:`SIGSEGV`, :const:`SIGFPE`, +:const:`SIGABRT`, :const:`SIGBUS`, and :const:`SIGILL` signals. You can also +enable them at startup by setting the :envvar:`PYTHONFAULTHANDLER` environment +variable or by using :option:`-X` ``faulthandler`` command line option. + +The fault handler is compatible with system fault handlers like Apport or the +Windows fault handler. The module uses an alternative stack for signal handlers +if the :c:func:`sigaltstack` function is available. This allows it to dump the +traceback even on a stack overflow. + +The fault handler is called on catastrophic cases and therefore can only use +signal-safe functions (e.g. it cannot allocate memory on the heap). Because of +this limitation traceback dumping is minimal compared to normal Python +tracebacks: + +* Only ASCII is supported. The ``backslashreplace`` error handler is used on + encoding. +* Each string is limited to 100 characters. +* Only the filename, the function name and the line number are + displayed. (no source code) +* It is limited to 100 frames and 100 threads. + +By default, the Python traceback is written to :data:`sys.stderr`. To see +tracebacks, applications must be run in the terminal. A log file can +alternatively be passed to :func:`faulthandler.enable`. + +The module is implemented in C, so tracebacks can be dumped on a crash or when +Python is deadlocked. + +.. versionadded:: 3.3 + + +Dump the traceback +------------------ + +.. function:: dump_traceback(file=sys.stderr, all_threads=True) + + Dump the tracebacks of all threads into *file*. If *all_threads* is + ``False``, dump only the current thread. + + +Fault handler state +------------------- + +.. function:: enable(file=sys.stderr, all_threads=True) + + Enable the fault handler: install handlers for the :const:`SIGSEGV`, + :const:`SIGFPE`, :const:`SIGABRT`, :const:`SIGBUS` and :const:`SIGILL` + signals to dump the Python traceback. If *all_threads* is ``True``, + produce tracebacks for every running thread. Otherwise, dump only the current + thread. + +.. function:: disable() + + Disable the fault handler: uninstall the signal handlers installed by + :func:`enable`. + +.. function:: is_enabled() + + Check if the fault handler is enabled. + + +Dump the tracebacks after a timeout +----------------------------------- + +.. function:: dump_tracebacks_later(timeout, repeat=False, file=sys.stderr, exit=False) + + Dump the tracebacks of all threads, after a timeout of *timeout* seconds, or + every *timeout* seconds if *repeat* is ``True``. If *exit* is ``True``, call + :c:func:`_exit` with status=1 after dumping the tracebacks. (Note + :c:func:`_exit` exits the process immediately, which means it doesn't do any + cleanup like flushing file buffers.) If the function is called twice, the new + call replaces previous parameters and resets the timeout. The timer has a + sub-second resolution. + + This function is implemented using a watchdog thread and therefore is not + available if Python is compiled with threads disabled. + +.. function:: cancel_dump_tracebacks_later() + + Cancel the last call to :func:`dump_tracebacks_later`. + + +Dump the traceback on a user signal +----------------------------------- + +.. function:: register(signum, file=sys.stderr, all_threads=True, chain=False) + + Register a user signal: install a handler for the *signum* signal to dump + the traceback of all threads, or of the current thread if *all_threads* is + ``False``, into *file*. Call the previous handler if chain is ``True``. + + Not available on Windows. + +.. function:: unregister(signum) + + Unregister a user signal: uninstall the handler of the *signum* signal + installed by :func:`register`. Return ``True`` if the signal was registered, + ``False`` otherwise. + + Not available on Windows. + + +File descriptor issue +--------------------- + +:func:`enable`, :func:`dump_tracebacks_later` and :func:`register` keep the +file descriptor of their *file* argument. If the file is closed and its file +descriptor is reused by a new file, or if :func:`os.dup2` is used to replace +the file descriptor, the traceback will be written into a different file. Call +these functions again each time that the file is replaced. + + +Example +------- + +Example of a segmentation fault on Linux: :: + + $ python -q -X faulthandler + >>> import ctypes + >>> ctypes.string_at(0) + Fatal Python error: Segmentation fault + + Current thread 0x00007fb899f39700: + File "/home/python/cpython/Lib/ctypes/__init__.py", line 486 in string_at + File "<stdin>", line 1 in <module> + Segmentation fault + diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index 6192400..9a9cdc1 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -19,6 +19,11 @@ argument. This can be an integer file descriptor, such as returned by ``sys.stdin.fileno()``, or a :class:`io.IOBase` object, such as ``sys.stdin`` itself, which provides a :meth:`fileno` that returns a genuine file descriptor. +.. versionchanged:: 3.3 + Operations in this module used to raise a :exc:`IOError` where they now + raise a :exc:`OSError`. + + The module defines the following functions: @@ -40,7 +45,7 @@ The module defines the following functions: larger than 1024 bytes, this is most likely to result in a segmentation violation or a more subtle data corruption. - If the :c:func:`fcntl` fails, an :exc:`IOError` is raised. + If the :c:func:`fcntl` fails, an :exc:`OSError` is raised. .. function:: ioctl(fd, op[, arg[, mutate_flag]]) @@ -107,7 +112,7 @@ The module defines the following functions: When *operation* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be bitwise ORed with :const:`LOCK_NB` to avoid blocking on lock acquisition. If :const:`LOCK_NB` is used and the lock cannot be acquired, an - :exc:`IOError` will be raised and the exception will have an *errno* + :exc:`OSError` will be raised and the exception will have an *errno* attribute set to :const:`EACCES` or :const:`EAGAIN` (depending on the operating system; for portability, check for both values). On at least some systems, :const:`LOCK_EX` can only be used if the file descriptor refers to a diff --git a/Doc/library/fileinput.rst b/Doc/library/fileinput.rst index ac44311..f8ec436 100644 --- a/Doc/library/fileinput.rst +++ b/Doc/library/fileinput.rst @@ -28,7 +28,10 @@ as the first argument to :func:`.input`. A single file name is also allowed. All files are opened in text mode by default, but you can override this by specifying the *mode* parameter in the call to :func:`.input` or :class:`FileInput`. If an I/O error occurs during opening or reading a file, -:exc:`IOError` is raised. +:exc:`OSError` is raised. + +.. versionchanged:: 3.3 + :exc:`IOError` used to be raised; it is now an alias of :exc:`OSError`. If ``sys.stdin`` is used more than once, the second and further use will return no lines, except perhaps for interactive use, or if it has been explicitly reset @@ -168,10 +171,6 @@ and the backup file remains around; by default, the extension is ``'.bak'`` and it is deleted when the output file is closed. In-place filtering is disabled when standard input is read. -.. note:: - - The current implementation does not work for MS-DOS 8+3 filesystems. - The two following opening hooks are provided by this module: diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst index 5bbef4f..3cc295a 100644 --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -40,7 +40,7 @@ Here's a sample session using the :mod:`ftplib` module:: The module defines the following items: -.. class:: FTP(host='', user='', passwd='', acct=''[, timeout]) +.. class:: FTP(host='', user='', passwd='', acct='', timeout=None, source_address=None) Return a new instance of the :class:`FTP` class. When *host* is given, the method call ``connect(host)`` is made. When *user* is given, additionally @@ -48,7 +48,8 @@ The module defines the following items: *acct* default to the empty string when not given). The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if is not specified, the global default timeout setting - will be used). + will be used). *source_address* is a 2-tuple ``(host, port)`` for the socket + to bind to as its source address before connecting. :class:`FTP` class supports the :keyword:`with` statement. Here is a sample on how using it: @@ -68,8 +69,11 @@ The module defines the following items: .. versionchanged:: 3.2 Support for the :keyword:`with` statement was added. + .. versionchanged:: 3.3 + *source_address* parameter was added. -.. class:: FTP_TLS(host='', user='', passwd='', acct='', [keyfile[, certfile[, context[, timeout]]]]) + +.. class:: FTP_TLS(host='', user='', passwd='', acct='', keyfile=None, certfile=None, context=None, timeout=None, source_address=None) A :class:`FTP` subclass which adds TLS support to FTP as described in :rfc:`4217`. @@ -80,10 +84,15 @@ The module defines the following items: private key and certificate chain file name for the SSL connection. *context* parameter is a :class:`ssl.SSLContext` object which allows bundling SSL configuration options, certificates and private keys into a - single (potentially long-lived) structure. + single (potentially long-lived) structure. *source_address* is a 2-tuple + ``(host, port)`` for the socket to bind to as its source address before + connecting. .. versionadded:: 3.2 + .. versionchanged:: 3.3 + *source_address* parameter was added. + Here's a sample session using the :class:`FTP_TLS` class: >>> from ftplib import FTP_TLS @@ -135,8 +144,7 @@ The module defines the following items: The set of all exceptions (as a tuple) that methods of :class:`FTP` instances may raise as a result of problems with the FTP connection (as opposed to programming errors made by the caller). This set includes the - four exceptions listed above as well as :exc:`socket.error` and - :exc:`IOError`. + four exceptions listed above as well as :exc:`OSError`. .. seealso:: @@ -174,7 +182,7 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. debugging output, logging each line sent and received on the control connection. -.. method:: FTP.connect(host='', port=0[, timeout]) +.. method:: FTP.connect(host='', port=0, timeout=None, source_address=None) Connect to the given host and port. The default port number is ``21``, as specified by the FTP protocol specification. It is rarely needed to specify a @@ -182,10 +190,14 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. instance; it should not be called at all if a host was given when the instance was created. All other methods can only be used after a connection has been made. - The optional *timeout* parameter specifies a timeout in seconds for the connection attempt. If no *timeout* is passed, the global default timeout setting will be used. + *source_address* is a 2-tuple ``(host, port)`` for the socket to bind to as + its source address before connecting. + + .. versionchanged:: 3.3 + *source_address* parameter was added. .. method:: FTP.getwelcome() @@ -241,13 +253,12 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. Retrieve a file or directory listing in ASCII transfer mode. *cmd* should be an appropriate ``RETR`` command (see :meth:`retrbinary`) or a command such as - ``LIST``, ``NLST`` or ``MLSD`` (usually just the string ``'LIST'``). + ``LIST`` or ``NLST`` (usually just the string ``'LIST'``). ``LIST`` retrieves a list of files and information about those files. - ``NLST`` retrieves a list of file names. On some servers, ``MLSD`` retrieves - a machine readable list of files and information about those files. The - *callback* function is called for each line with a string argument containing - the line with the trailing CRLF stripped. The default *callback* prints the - line to ``sys.stdout``. + ``NLST`` retrieves a list of file names. + The *callback* function is called for each line with a string argument + containing the line with the trailing CRLF stripped. The default *callback* + prints the line to ``sys.stdout``. .. method:: FTP.set_pasv(boolean) @@ -307,6 +318,20 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. in :meth:`transfercmd`. +.. method:: FTP.mlsd(path="", facts=[]) + + List a directory in a standardized format by using MLSD command + (:rfc:`3659`). If *path* is omitted the current directory is assumed. + *facts* is a list of strings representing the type of information desired + (e.g. ``["type", "size", "perm"]``). Return a generator object yielding a + tuple of two elements for every file found in path. First element is the + file name, the second one is a dictionary containing facts about the file + name. Content of this dictionary might be limited by the *facts* argument + but server is not guaranteed to return all requested facts. + + .. versionadded:: 3.3 + + .. method:: FTP.nlst(argument[, ...]) Return a list of file names as returned by the ``NLST`` command. The @@ -314,6 +339,8 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. directory). Multiple arguments can be used to pass non-standard options to the ``NLST`` command. + .. deprecated:: 3.3 use :meth:`mlsd` instead. + .. method:: FTP.dir(argument[, ...]) @@ -324,6 +351,8 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. as a *callback* function as for :meth:`retrlines`; the default prints to ``sys.stdout``. This method returns ``None``. + .. deprecated:: 3.3 use :meth:`mlsd` instead. + .. method:: FTP.rename(fromname, toname) @@ -396,6 +425,14 @@ FTP_TLS Objects Set up secure control connection by using TLS or SSL, depending on what specified in :meth:`ssl_version` attribute. +.. method:: FTP_TLS.ccc() + + Revert control channel back to plaintext. This can be useful to take + advantage of firewalls that know how to handle NAT with non-secure FTP + without opening fixed ports. + + .. versionadded:: 3.3 + .. method:: FTP_TLS.prot_p() Set up secure data connection. diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 3fcd694..d5ac23e 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -152,10 +152,6 @@ are always available. They are listed here in alphabetical order. 1,114,111 (0x10FFFF in base 16). :exc:`ValueError` will be raised if *i* is outside that range. - Note that on narrow Unicode builds, the result is a string of - length two for *i* greater than 65,535 (0xFFFF in hexadecimal). - - .. function:: classmethod(function) @@ -787,10 +783,10 @@ are always available. They are listed here in alphabetical order. :meth:`__index__` method that returns an integer. -.. function:: open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True) +.. function:: open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None) Open *file* and return a corresponding stream. If the file cannot be opened, - an :exc:`IOError` is raised. + an :exc:`OSError` is raised. *file* is either a string or bytes object giving the pathname (absolute or relative to the current working directory) of the file to be opened or @@ -801,17 +797,19 @@ are always available. They are listed here in alphabetical order. *mode* is an optional string that specifies the mode in which the file is opened. It defaults to ``'r'`` which means open for reading in text mode. Other common values are ``'w'`` for writing (truncating the file if it - already exists), and ``'a'`` for appending (which on *some* Unix systems, - means that *all* writes append to the end of the file regardless of the - current seek position). In text mode, if *encoding* is not specified the - encoding used is platform dependent. (For reading and writing raw bytes use - binary mode and leave *encoding* unspecified.) The available modes are: + already exists), ``'x'`` for exclusive creation and ``'a'`` for appending + (which on *some* Unix systems, means that *all* writes append to the end of + the file regardless of the current seek position). In text mode, if + *encoding* is not specified the encoding used is platform dependent. (For + reading and writing raw bytes use binary mode and leave *encoding* + unspecified.) The available modes are: ========= =============================================================== Character Meaning --------- --------------------------------------------------------------- ``'r'`` open for reading (default) ``'w'`` open for writing, truncating the file first + ``'x'`` open for exclusive creation, failing if the file already exists ``'a'`` open for writing, appending to the end of the file if it exists ``'b'`` binary mode ``'t'`` text mode (default) @@ -894,6 +892,16 @@ are always available. They are listed here in alphabetical order. closed. If a filename is given *closefd* has no effect and must be ``True`` (the default). + A custom opener can be used by passing a callable as *opener*. The underlying + file descriptor for the file object is then obtained by calling *opener* with + (*file*, *flags*). *opener* must return an open file descriptor (passing + :mod:`os.open` as *opener* results in functionality similar to passing + ``None``). + + .. versionchanged:: 3.3 + The *opener* parameter was added. + The ``'x'`` mode was added. + The type of file object returned by the :func:`open` function depends on the mode. When :func:`open` is used to open a file in a text mode (``'w'``, ``'r'``, ``'wt'``, ``'rt'``, etc.), it returns a subclass of @@ -919,6 +927,11 @@ are always available. They are listed here in alphabetical order. (where :func:`open` is declared), :mod:`os`, :mod:`os.path`, :mod:`tempfile`, and :mod:`shutil`. + .. versionchanged:: 3.3 + :exc:`IOError` used to be raised, it is now an alias of :exc:`OSError`. + :exc:`FileExistsError` is now raised if the file opened in exclusive + creation mode (``'x'``) already exists. + .. XXX works for bytes too, but should it? .. function:: ord(c) @@ -928,9 +941,6 @@ are always available. They are listed here in alphabetical order. point of that character. For example, ``ord('a')`` returns the integer ``97`` and ``ord('\u2020')`` returns ``8224``. This is the inverse of :func:`chr`. - On wide Unicode builds, if the argument length is not one, a - :exc:`TypeError` will be raised. On narrow Unicode builds, strings - of length two are accepted when they form a UTF-16 surrogate pair. .. function:: pow(x, y[, z]) @@ -948,7 +958,7 @@ are always available. They are listed here in alphabetical order. must be of integer types, and *y* must be non-negative. -.. function:: print([object, ...], *, sep=' ', end='\\n', file=sys.stdout) +.. function:: print([object, ...], *, sep=' ', end='\\n', file=sys.stdout, flush=False) Print *object*\(s) to the stream *file*, separated by *sep* and followed by *end*. *sep*, *end* and *file*, if present, must be given as keyword @@ -961,9 +971,12 @@ are always available. They are listed here in alphabetical order. *end*. The *file* argument must be an object with a ``write(string)`` method; if it - is not present or ``None``, :data:`sys.stdout` will be used. Output buffering - is determined by *file*. Use ``file.flush()`` to ensure, for instance, - immediate appearance on a screen. + is not present or ``None``, :data:`sys.stdout` will be used. Whether output + is buffered is usually determined by *file*, but if the *flush* keyword + argument is true, the stream is forcibly flushed. + + .. versionchanged:: 3.3 + Added the *flush* keyword argument. .. function:: property(fget=None, fset=None, fdel=None, doc=None) @@ -1046,7 +1059,9 @@ are always available. They are listed here in alphabetical order. ...]``. If *step* is positive, the last element is the largest ``start + i * step`` less than *stop*; if *step* is negative, the last element is the smallest ``start + i * step`` greater than *stop*. *step* must not be zero - (or else :exc:`ValueError` is raised). Example: + (or else :exc:`ValueError` is raised). Range objects have read-only data + attributes :attr:`start`, :attr:`stop` and :attr:`step` which return the + argument values (or their default). Example: >>> list(range(10)) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] @@ -1083,6 +1098,13 @@ are always available. They are listed here in alphabetical order. >>> r[-1] 18 + Testing range objects for equality with ``==`` and ``!=`` compares + them as sequences. That is, two range objects are considered equal if + they represent the same sequence of values. (Note that two range + objects that compare equal might have different :attr:`start`, + :attr:`stop` and :attr:`step` attributes, for example ``range(0) == + range(2, 1, 3)`` or ``range(0, 3, 2) == range(0, 4, 2)``.) + Ranges containing absolute values larger than :data:`sys.maxsize` are permitted but some features (such as :func:`len`) will raise :exc:`OverflowError`. @@ -1092,6 +1114,14 @@ are always available. They are listed here in alphabetical order. Test integers for membership in constant time instead of iterating through all items. + .. versionchanged:: 3.3 + Define '==' and '!=' to compare range objects based on the + sequence of values they define (instead of comparing based on + object identity). + + .. versionadded:: 3.3 + The :attr:`start`, :attr:`stop` and :attr:`step` attributes. + .. function:: repr(object) @@ -1299,10 +1329,12 @@ are always available. They are listed here in alphabetical order. Accordingly, :func:`super` is undefined for implicit lookups using statements or operators such as ``super()[name]``. - Also note that :func:`super` is not limited to use inside methods. The two - argument form specifies the arguments exactly and makes the appropriate - references. The zero argument form automatically searches the stack frame - for the class (``__class__``) and the first argument. + Also note that, aside from the zero argument form, :func:`super` is not + limited to use inside methods. The two argument form specifies the + arguments exactly and makes the appropriate references. The zero + argument form only works inside a class definition, as the compiler fills + in the necessary details to correctly retrieve the class being defined, + as well as accessing the current instance for ordinary methods. For practical suggestions on how to design cooperative classes using :func:`super`, see `guide to using super() @@ -1422,8 +1454,9 @@ are always available. They are listed here in alphabetical order. replaced (by importing the :mod:`builtins` module and assigning to ``builtins.__import__``) in order to change semantics of the :keyword:`import` statement, but nowadays it is usually simpler to use import - hooks (see :pep:`302`). Direct use of :func:`__import__` is rare, except in - cases where you want to import a module whose name is only known at runtime. + hooks (see :pep:`302`) to attain the same goals. Direct use of + :func:`__import__` is entirely discouraged in favor of + :func:`importlib.import_module`. The function imports the module *name*, potentially using the given *globals* and *locals* to determine how to interpret the name in a package context. @@ -1435,7 +1468,8 @@ are always available. They are listed here in alphabetical order. *level* specifies whether to use absolute or relative imports. ``0`` (the default) means only perform absolute imports. Positive values for *level* indicate the number of parent directories to search relative to the - directory of the module calling :func:`__import__`. + directory of the module calling :func:`__import__` (see :pep:`328` for the + details). When the *name* variable is of the form ``package.module``, normally, the top-level package (the name up till the first dot) is returned, *not* the diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst index 04743d3..1b0d82a 100644 --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -40,7 +40,7 @@ The :mod:`functools` module defines the following functions: .. versionadded:: 3.2 -.. decorator:: lru_cache(maxsize=100) +.. decorator:: lru_cache(maxsize=128, typed=False) Decorator to wrap a function with a memoizing callable that saves up to the *maxsize* most recent calls. It can save time when an expensive or I/O bound @@ -52,6 +52,10 @@ The :mod:`functools` module defines the following functions: If *maxsize* is set to None, the LRU feature is disabled and the cache can grow without bound. + If *typed* is set to True, function arguments of different types will be + cached separately. For example, ``f(3)`` and ``f(3.0)`` will be treated + as distinct calls with distinct results. + To help measure the effectiveness of the cache and tune the *maxsize* parameter, the wrapped function is instrumented with a :func:`cache_info` function that returns a :term:`named tuple` showing *hits*, *misses*, @@ -67,8 +71,8 @@ The :mod:`functools` module defines the following functions: An `LRU (least recently used) cache <http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used>`_ works - best when more recent calls are the best predictors of upcoming calls (for - example, the most popular articles on a news server tend to change daily). + best when the most recent calls are the best predictors of upcoming calls (for + example, the most popular articles on a news server tend to change each day). The cache's size limit assures that the cache does not grow without bound on long-running processes such as web servers. @@ -111,6 +115,9 @@ The :mod:`functools` module defines the following functions: .. versionadded:: 3.2 + .. versionchanged:: 3.3 + Added the *typed* option. + .. decorator:: total_ordering Given a class defining one or more rich comparison ordering methods, this diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 0281bb7..3d76411 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -153,8 +153,8 @@ The :mod:`gc` module provides the following functions: .. versionadded:: 3.1 -The following variable is provided for read-only access (you can mutate its -value but should not rebind it): +The following variables are provided for read-only access (you can mutate the +values but should not rebind them): .. data:: garbage @@ -183,6 +183,41 @@ value but should not rebind it): :const:`DEBUG_UNCOLLECTABLE` is set, in addition all uncollectable objects are printed. +.. data:: callbacks + + A list of callbacks that will be invoked by the garbage collector before and + after collection. The callbacks will be called with two arguments, + *phase* and *info*. + + *phase* can one of two values: + + "start": The garbage collection is about to start. + + "stop": The garbage collection has finished. + + *info* provides more information for the callback. The following + keys are currently defined: + + "generation": The oldest generation being collected. + + "collected": When *phase* is "stop", the number of objects + successfully collected. + + "uncollectable": when *phase* is "stop", the number of objects + that could not be collected and were put in :data:`garbage`. + + Applications can add their own callbacks to this list. The primary + use cases are: + + Gathering statistics about garbage collection, such as how often + various generations are collected, and how long the collection + takes. + + Allowing applications to identify and clear their own uncollectable + types when they appear in :data:`garbage`. + + .. versionadded:: 3.3 + The following constants are provided for use with :func:`set_debug`: diff --git a/Doc/library/gettext.rst b/Doc/library/gettext.rst index 0fa022c..825311b 100644 --- a/Doc/library/gettext.rst +++ b/Doc/library/gettext.rst @@ -185,10 +185,13 @@ class can also install themselves in the built-in namespace as the function translation object from the cache; the actual instance data is still shared with the cache. - If no :file:`.mo` file is found, this function raises :exc:`IOError` if + If no :file:`.mo` file is found, this function raises :exc:`OSError` if *fallback* is false (which is the default), and returns a :class:`NullTranslations` instance if *fallback* is true. + .. versionchanged:: 3.3 + :exc:`IOError` used to be raised instead of :exc:`OSError`. + .. function:: install(domain, localedir=None, codeset=None, names=None) @@ -342,7 +345,7 @@ The entire set of key/value pairs are placed into a dictionary and set as the If the :file:`.mo` file's magic number is invalid, or if other problems occur while reading the file, instantiating a :class:`GNUTranslations` class can raise -:exc:`IOError`. +:exc:`OSError`. The following methods are overridden from the base class implementation: diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst index 9422ea9..861a59c 100644 --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -13,9 +13,11 @@ like the GNU programs :program:`gzip` and :program:`gunzip` would. The data compression is provided by the :mod:`zlib` module. -The :mod:`gzip` module provides the :class:`GzipFile` class. The :class:`GzipFile` -class reads and writes :program:`gzip`\ -format files, automatically compressing -or decompressing the data so that it looks like an ordinary :term:`file object`. +The :mod:`gzip` module provides the :class:`GzipFile` class, as well as the +:func:`gzip.open`, :func:`compress` and :func:`decompress` convenience +functions. The :class:`GzipFile` class reads and writes :program:`gzip`\ -format +files, automatically compressing or decompressing the data so that it looks like +an ordinary :term:`file object`. Note that additional file formats which can be decompressed by the :program:`gzip` and :program:`gunzip` programs, such as those produced by @@ -24,6 +26,32 @@ Note that additional file formats which can be decompressed by the The module defines the following items: +.. function:: open(filename, mode='rb', compresslevel=9, encoding=None, errors=None, newline=None) + + Open *filename* as a gzip-compressed file in binary or text mode. + + Returns a :term:`file object`. + + The *mode* argument can be any of ``'r'``, ``'rb'``, ``'a'``, ``'ab'``, + ``'w'``, or ``'wb'`` for binary mode, or ``'rt'``, ``'at'``, or ``'wt'`` for + text mode. The default is ``'rb'``. + + The *compresslevel* argument is an integer from 1 to 9, as for the + :class:`GzipFile` constructor. + + For binary mode, this function is equivalent to the :class:`GzipFile` + constructor: ``GzipFile(filename, mode, compresslevel)``. In this case, the + *encoding*, *errors* and *newline* arguments must not be provided. + + For text mode, a :class:`GzipFile` object is created, and wrapped in an + :class:`io.TextIOWrapper` instance with the specified encoding, error + handling behavior, and line ending(s). + + .. versionchanged:: 3.3 + Support for text mode was added, along with the *encoding*, *errors* and + *newline* arguments. + + .. class:: GzipFile(filename=None, mode=None, compresslevel=9, fileobj=None, mtime=None) Constructor for the :class:`GzipFile` class, which simulates most of the @@ -46,9 +74,9 @@ The module defines the following items: or ``'wb'``, depending on whether the file will be read or written. The default is the mode of *fileobj* if discernible; otherwise, the default is ``'rb'``. - Note that the file is always opened in binary mode; text mode is not - supported. If you need to read a compressed file in text mode, wrap your - :class:`GzipFile` with an :class:`io.TextIOWrapper`. + Note that the file is always opened in binary mode. To open a compressed file + in text mode, use :func:`gzip.open` (or wrap your :class:`GzipFile` with an + :class:`io.TextIOWrapper`). The *compresslevel* argument is an integer from ``1`` to ``9`` controlling the level of compression; ``1`` is fastest and produces the least compression, and @@ -71,7 +99,7 @@ The module defines the following items: :class:`GzipFile` supports the :class:`io.BufferedIOBase` interface, including iteration and the :keyword:`with` statement. Only the - :meth:`read1` and :meth:`truncate` methods aren't implemented. + :meth:`truncate` method isn't implemented. :class:`GzipFile` also provides the following method: @@ -93,12 +121,9 @@ The module defines the following items: .. versionchanged:: 3.2 Support for unseekable files was added. + .. versionchanged:: 3.3 + The :meth:`io.BufferedIOBase.read1` method is now implemented. -.. function:: open(filename, mode='rb', compresslevel=9) - - This is a shorthand for ``GzipFile(filename,`` ``mode,`` ``compresslevel)``. - The *filename* argument is required; *mode* defaults to ``'rb'`` and - *compresslevel* defaults to ``9``. .. function:: compress(data, compresslevel=9) diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst index eff2724..8274bb1 100644 --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -38,6 +38,13 @@ An HMAC object has the following methods: given to the constructor. It may contain non-ASCII bytes, including NUL bytes. + .. warning:: + + When comparing the output of :meth:`digest` to an externally-supplied + digest during a verification routine, it is recommended to use the + :func:`hmac.secure_compare` function instead of the ``==`` operator + to avoid potential timing attacks. + .. method:: HMAC.hexdigest() @@ -45,6 +52,13 @@ An HMAC object has the following methods: length containing only hexadecimal digits. This may be used to exchange the value safely in email or other non-binary environments. + .. warning:: + + When comparing the output of :meth:`hexdigest` to an externally-supplied + digest during a verification routine, it is recommended to use the + :func:`hmac.secure_compare` function instead of the ``==`` operator + to avoid potential timing attacks. + .. method:: HMAC.copy() @@ -52,6 +66,25 @@ An HMAC object has the following methods: compute the digests of strings that share a common initial substring. +This module also provides the following helper function: + +.. function:: secure_compare(a, b) + + Returns the equivalent of ``a == b``, but using a time-independent + comparison method. Comparing the full lengths of the inputs *a* and *b*, + instead of short-circuiting the comparison upon the first unequal byte, + prevents leaking information about the inputs being compared and mitigates + potential timing attacks. The inputs must be either :class:`str` or + :class:`bytes` instances. + + .. note:: + + While the :func:`hmac.secure_compare` function prevents leaking the + contents of the inputs via a timing attack, it does leak the length + of the inputs. However, this generally is not a security risk. + + .. versionadded:: 3.3 + .. seealso:: Module :mod:`hashlib` diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index 52fbe57..efb4c41 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -339,6 +339,15 @@ and also the following constants for integer status codes: | :const:`UPGRADE_REQUIRED` | ``426`` | HTTP Upgrade to TLS, | | | | :rfc:`2817`, Section 6 | +------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`PRECONDITION_REQUIRED` | ``428`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 3 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`TOO_MANY_REQUESTS` | ``429`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 4 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`REQUEST_HEADER_FIELDS_TOO_LARGE` | ``431`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 5 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ | :const:`INTERNAL_SERVER_ERROR` | ``500`` | HTTP/1.1, `RFC 2616, Section | | | | 10.5.1 | | | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.1>`_ | @@ -369,6 +378,12 @@ and also the following constants for integer status codes: | :const:`NOT_EXTENDED` | ``510`` | An HTTP Extension Framework, | | | | :rfc:`2774`, Section 7 | +------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`NETWORK_AUTHENTICATION_REQUIRED` | ``511`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 6 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ + +.. versionchanged:: 3.3 + Added codes ``428``, ``429``, ``431`` and ``511`` from :rfc:`6585`. .. data:: responses @@ -502,6 +517,12 @@ statement. Reads and returns the response body, or up to the next *amt* bytes. +.. method:: HTTPResponse.readinto(b) + + Reads up to the next len(b) bytes of the response body into the buffer *b*. + Returns the number of bytes read. + + .. versionadded:: 3.3 .. method:: HTTPResponse.getheader(name, default=None) diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst index cc8f251..40f24f2 100644 --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -40,7 +40,11 @@ The module defines the following exception: .. exception:: LoadError Instances of :class:`FileCookieJar` raise this exception on failure to load - cookies from a file. :exc:`LoadError` is a subclass of :exc:`IOError`. + cookies from a file. :exc:`LoadError` is a subclass of :exc:`OSError`. + + .. versionchanged:: 3.3 + LoadError was made a subclass of :exc:`OSError` instead of + :exc:`IOError`. The following classes are provided: @@ -257,9 +261,12 @@ contained :class:`Cookie` objects. Arguments are as for :meth:`save`. The named file must be in the format understood by the class, or - :exc:`LoadError` will be raised. Also, :exc:`IOError` may be raised, for + :exc:`LoadError` will be raised. Also, :exc:`OSError` may be raised, for example if the file does not exist. + .. versionchanged:: 3.3 + :exc:`IOError` used to be raised, it is now an alias of :exc:`OSError`. + .. method:: FileCookieJar.revert(filename=None, ignore_discard=False, ignore_expires=False) diff --git a/Doc/library/http.cookies.rst b/Doc/library/http.cookies.rst index 5ae3fd4..646f2e8 100644 --- a/Doc/library/http.cookies.rst +++ b/Doc/library/http.cookies.rst @@ -22,9 +22,12 @@ many current day browsers and servers have relaxed parsing rules when comes to Cookie handling. As a result, the parsing rules used are a bit less strict. The character set, :data:`string.ascii_letters`, :data:`string.digits` and -``!#$%&'*+-.^_`|~`` denote the set of valid characters allowed by this module +``!#$%&'*+-.^_`|~:`` denote the set of valid characters allowed by this module in Cookie name (as :attr:`~Morsel.key`). +.. versionchanged:: 3.3 + Allowed ':' as a valid Cookie name character. + .. note:: diff --git a/Doc/library/http.server.rst b/Doc/library/http.server.rst index 300e332..d947ed2 100644 --- a/Doc/library/http.server.rst +++ b/Doc/library/http.server.rst @@ -179,19 +179,29 @@ of which this module provides three different variants: .. method:: send_response(code, message=None) - Sends a response header and logs the accepted request. The HTTP response - line is sent, followed by *Server* and *Date* headers. The values for - these two headers are picked up from the :meth:`version_string` and - :meth:`date_time_string` methods, respectively. + Adds a response header to the headers buffer and logs the accepted + request. The HTTP response line is written to the internal buffer, + followed by *Server* and *Date* headers. The values for these two headers + are picked up from the :meth:`version_string` and + :meth:`date_time_string` methods, respectively. If the server does not + intend to send any other headers using the :meth:`send_header` method, + then :meth:`send_response` should be followed by a :meth:`end_headers` + call. + + .. versionchanged:: 3.3 + Headers are stored to an internal buffer and :meth:`end_headers` + needs to be called explicitly. + .. method:: send_header(keyword, value) - Stores the HTTP header to an internal buffer which will be written to the - output stream when :meth:`end_headers` method is invoked. - *keyword* should specify the header keyword, with *value* - specifying its value. + Adds the HTTP header to an internal buffer which will be written to the + output stream when either :meth:`end_headers` or :meth:`flush_headers` is + invoked. *keyword* should specify the header keyword, with *value* + specifying its value. Note that, after the send_header calls are done, + :meth:`end_headers` MUST BE called in order to complete the operation. - .. versionchanged:: 3.2 Storing the headers in an internal buffer + .. versionchanged:: 3.2 Headers are stored in an internal buffer. .. method:: send_response_only(code, message=None) @@ -205,10 +215,19 @@ of which this module provides three different variants: .. method:: end_headers() - Write the buffered HTTP headers to the output stream and send a blank - line, indicating the end of the HTTP headers in the response. + Adds a blank line + (indicating the end of the HTTP headers in the response) + to the headers buffer and calls :meth:`flush_headers()`. + + .. versionchanged:: 3.2 + The buffered headers are written to the output stream. - .. versionchanged:: 3.2 Writing the buffered headers to the output stream. + .. method:: flush_headers() + + Finally send the headers to the output stream and flush the internal + headers buffer. + + .. versionadded:: 3.3 .. method:: log_request(code='-', size='-') @@ -250,8 +269,11 @@ of which this module provides three different variants: .. method:: address_string() - Returns the client address, formatted for logging. A name lookup is - performed on the client's IP address. + Returns the client address. + + .. versionchanged:: 3.3 + Previously, a name lookup was performed. To avoid name resolution + delays, it now always returns the IP address. .. class:: SimpleHTTPRequestHandler(request, client_address, server) @@ -299,7 +321,7 @@ of which this module provides three different variants: response if the :func:`listdir` fails. If the request was mapped to a file, it is opened and the contents are - returned. Any :exc:`IOError` exception in opening the requested file is + returned. Any :exc:`OSError` exception in opening the requested file is mapped to a ``404``, ``'File not found'`` error. Otherwise, the content type is guessed by calling the :meth:`guess_type` method, which in turn uses the *extensions_map* variable. diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index 3f45c95..038355c 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -64,14 +64,21 @@ Three exceptions are defined as attributes of the :class:`IMAP4` class: There's also a subclass for secure connections: -.. class:: IMAP4_SSL(host='', port=IMAP4_SSL_PORT, keyfile=None, certfile=None) +.. class:: IMAP4_SSL(host='', port=IMAP4_SSL_PORT, keyfile=None, certfile=None, ssl_context=None) This is a subclass derived from :class:`IMAP4` that connects over an SSL encrypted socket (to use this class you need a socket module that was compiled with SSL support). If *host* is not specified, ``''`` (the local host) is used. If *port* is omitted, the standard IMAP4-over-SSL port (993) is used. *keyfile* and *certfile* are also optional - they can contain a PEM formatted private key - and certificate chain file for the SSL connection. + and certificate chain file for the SSL connection. *ssl_context* parameter is a + :class:`ssl.SSLContext` object which allows bundling SSL configuration + options, certificates and private keys into a single (potentially long-lived) + structure. Note that the *keyfile*/*certfile* parameters are mutually exclusive with *ssl_context*, + a :class:`ValueError` is thrown if *keyfile*/*certfile* is provided along with *ssl_context*. + + .. versionchanged:: 3.3 + *ssl_context* parameter added. The second subclass allows for connections created by a child process: diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst index 1345b25..3e79936 100644 --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -30,6 +30,9 @@ This module provides an interface to the mechanisms used to implement the :const:`PY_SOURCE`, :const:`PY_COMPILED`, or :const:`C_EXTENSION`, described below. + .. deprecated:: 3.3 + Use the constants defined on :mod:`importlib.machinery` instead. + .. function:: find_module(name[, path]) @@ -69,6 +72,9 @@ This module provides an interface to the mechanisms used to implement the then use :func:`find_module` with the *path* argument set to ``P.__path__``. When *P* itself has a dotted name, apply this recipe recursively. + .. deprecated:: 3.3 + Use :func:`importlib.find_loader` instead. + .. function:: load_module(name, file, pathname, description) @@ -90,6 +96,10 @@ This module provides an interface to the mechanisms used to implement the it was not ``None``, even when an exception is raised. This is best done using a :keyword:`try` ... :keyword:`finally` statement. + .. deprecated:: 3.3 + Unneeded as loaders should be used to load modules and + :func:`find_module` is deprecated. + .. function:: new_module(name) @@ -97,37 +107,6 @@ This module provides an interface to the mechanisms used to implement the in ``sys.modules``. -.. function:: lock_held() - - Return ``True`` if the import lock is currently held, else ``False``. On - platforms without threads, always return ``False``. - - On platforms with threads, a thread executing an import holds an internal lock - until the import is complete. This lock blocks other threads from doing an - import until the original import completes, which in turn prevents other threads - from seeing incomplete module objects constructed by the original thread while - in the process of completing its import (and the imports, if any, triggered by - that). - - -.. function:: acquire_lock() - - Acquire the interpreter's import lock for the current thread. This lock should - be used by import hooks to ensure thread-safety when importing modules. - - Once a thread has acquired the import lock, the same thread may acquire it - again without blocking; the thread must release it once for each time it has - acquired it. - - On platforms without threads, this function does nothing. - - -.. function:: release_lock() - - Release the interpreter's import lock. On platforms without threads, this - function does nothing. - - .. function:: reload(module) Reload a previously imported *module*. The argument must be a module object, so @@ -225,6 +204,49 @@ file paths. magic number, as returned by :func:`get_magic`. +The following functions help interact with the import system's internal +locking mechanism. Locking semantics of imports are an implementation +detail which may vary from release to release. However, Python ensures +that circular imports work without any deadlocks. + +.. versionchanged:: 3.3 + In Python 3.3, the locking scheme has changed to per-module locks for + the most part. A global import lock is kept for some critical tasks, + such as initializing the per-module locks. + + +.. function:: lock_held() + + Return ``True`` if the global import lock is currently held, else + ``False``. On platforms without threads, always return ``False``. + + On platforms with threads, a thread executing an import first holds a + global import lock, then sets up a per-module lock for the rest of the + import. This blocks other threads from importing the same module until + the original import completes, preventing other threads from seeing + incomplete module objects constructed by the original thread. An + exception is made for circular imports, which by construction have to + expose an incomplete module object at some point. + +.. function:: acquire_lock() + + Acquire the interpreter's global import lock for the current thread. + This lock should be used by import hooks to ensure thread-safety when + importing modules. + + Once a thread has acquired the import lock, the same thread may acquire it + again without blocking; the thread must release it once for each time it has + acquired it. + + On platforms without threads, this function does nothing. + + +.. function:: release_lock() + + Release the interpreter's global import lock. On platforms without + threads, this function does nothing. + + The following constants with integer values, defined in this module, are used to indicate the search result of :func:`find_module`. @@ -233,31 +255,43 @@ to indicate the search result of :func:`find_module`. The module was found as a source file. + .. deprecated:: 3.3 + .. data:: PY_COMPILED The module was found as a compiled code object file. + .. deprecated:: 3.3 + .. data:: C_EXTENSION The module was found as dynamically loadable shared library. + .. deprecated:: 3.3 + .. data:: PKG_DIRECTORY The module was found as a package directory. + .. deprecated:: 3.3 + .. data:: C_BUILTIN The module was found as a built-in module. + .. deprecated:: 3.3 + .. data:: PY_FROZEN The module was found as a frozen module. + .. deprecated:: 3.3 + .. class:: NullImporter(path_string) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index c9f742a..35a99bf 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -86,6 +86,31 @@ Functions that was imported (e.g. ``pkg.mod``), while :func:`__import__` returns the top-level package or module (e.g. ``pkg``). +.. function:: find_loader(name, path=None) + + Find the loader for a module, optionally within the specified *path*. If the + module is in :attr:`sys.modules`, then ``sys.modules[name].__loader__`` is + returned (unless the loader would be ``None``, in which case + :exc:`ValueError` is raised). Otherwise a search using :attr:`sys.meta_path` + is done. ``None`` is returned if no loader is found. + + A dotted name does not have its parent's implicitly imported. If that is + desired (although not nessarily required to find the loader, it will most + likely be needed if the loader actually is used to load the module), then + you will have to import the packages containing the module prior to calling + this function. + +.. function:: invalidate_caches() + + Invalidate the internal caches of the finders stored at + :data:`sys.path_importer_cache`. If a finder implements + :meth:`abc.Finder.invalidate_caches()` then it will be called to perform the + invalidation. This function may be needed if some modules are installed + while your program is running and you expect the program to notice the + changes. + + .. versionadded:: 3.3 + :mod:`importlib.abc` -- Abstract base classes related to import --------------------------------------------------------------- @@ -111,6 +136,12 @@ are also provided to help in implementing the core ABCs. be the value of :attr:`__path__` from the parent package. If a loader cannot be found, ``None`` is returned. + .. method:: invalidate_caches() + + An optional method which, when called, should invalidate any internal + cache used by the finder. Used by :func:`invalidate_caches()` when + invalidating the caches of all cached finders. + .. class:: Loader @@ -220,6 +251,39 @@ are also provided to help in implementing the core ABCs. module. +.. class:: FileLoader(fullname, path) + + An abstract base class which inherits from :class:`ResourceLoader` and + :class:`ExecutionLoader`, providing concreate implementations of + :meth:`ResourceLoader.get_data` and :meth:`ExecutionLoader.get_filename`. + + The *fullname* argument is a fully resolved name of the module the loader is + to handle. The *path* argument is the path to the file for the module. + + .. versionadded:: 3.3 + + .. attribute:: name + + The name of the module the loader can handle. + + .. attribute:: path + + Path to the file of the module. + + .. method:: load_module(fullname=None) + + Calls + ``super().load_module(fullname if fullname is not None else self.name)``. + + .. method:: get_filename(fullname) + + Returns :attr:`path` when ``fullname`` equals :attr:`name` or ``None``. + + .. method:: get_data(path) + + Returns the open, binary file for *path*. + + .. class:: SourceLoader An abstract base class for implementing source (and optionally bytecode) @@ -239,11 +303,30 @@ are also provided to help in implementing the core ABCs. optimization to speed up loading by removing the parsing step of Python's compiler, and so no bytecode-specific API is exposed. + .. method:: path_stats(self, path) + + Optional abstract method which returns a :class:`dict` containing + metadata about the specifed path. Supported dictionary keys are: + + - ``'mtime'`` (mandatory): an integer or floating-point number + representing the modification time of the source code; + - ``'size'`` (optional): the size in bytes of the source code. + + Any other keys in the dictionary are ignored, to allow for future + extensions. + + .. versionadded:: 3.3 + .. method:: path_mtime(self, path) Optional abstract method which returns the modification time for the specified path. + .. deprecated:: 3.3 + This method is deprecated in favour of :meth:`path_stats`. You don't + have to implement it, but it is still available for compatibility + purposes. + .. method:: set_data(self, path, data) Optional abstract method which writes the specified bytes to a file @@ -413,6 +496,43 @@ are also provided to help in implementing the core ABCs. This module contains the various objects that help :keyword:`import` find and load modules. +.. attribute:: SOURCE_SUFFIXES + + A list of strings representing the recognized file suffixes for source + modules. + + .. versionadded:: 3.3 + +.. attribute:: DEBUG_BYTECODE_SUFFIXES + + A list of strings representing the file suffixes for non-optimized bytecode + modules. + + .. versionadded:: 3.3 + +.. attribute:: OPTIMIZED_BYTECODE_SUFFIXES + + A list of strings representing the file suffixes for optimized bytecode + modules. + + .. versionadded:: 3.3 + +.. attribute:: BYTECODE_SUFFIXES + + A list of strings representing the recognized file suffixes for bytecode + modules. Set to either :attr:`DEBUG_BYTECODE_SUFFIXES` or + :attr:`OPTIMIZED_BYTECODE_SUFFIXES` based on whether ``__debug__`` is true. + + .. versionadded:: 3.3 + +.. attribute:: EXTENSION_SUFFIXES + + A list of strings representing the the recognized file suffixes for + extension modules. + + .. versionadded:: 3.3 + + .. class:: BuiltinImporter An :term:`importer` for built-in modules. All known built-in modules are @@ -441,7 +561,9 @@ find and load modules. This class does not perfectly mirror the semantics of :keyword:`import` in terms of :data:`sys.path`. No implicit path hooks are assumed for - simplification of the class and its semantics. + simplification of the class and its semantics. This implies that when + ``None`` is found in :data:`sys.path_importer_cache` that it is simply + ignored instead of implying a default finder. Only class methods are defined by this class to alleviate the need for instantiation. @@ -451,7 +573,7 @@ find and load modules. Class method that attempts to find a :term:`loader` for the module specified by *fullname* on :data:`sys.path` or, if defined, on *path*. For each path entry that is searched, - :data:`sys.path_importer_cache` is checked. If an non-false object is + :data:`sys.path_importer_cache` is checked. If a non-false object is found then it is used as the :term:`finder` to look for the module being searched for. If no entry is found in :data:`sys.path_importer_cache`, then :data:`sys.path_hooks` is @@ -460,15 +582,177 @@ find and load modules. module. If no finder is ever found then ``None`` is returned. +.. class:: FileFinder(path, \*loader_details) + + A concrete implementation of :class:`importlib.abc.Finder` which caches + results from the file system. + + The *path* argument is the directory for which the finder is in charge of + searching. + + The *loader_details* argument is a variable number of 3-item tuples each + containing a loader, file suffixes the loader recognizes, and a boolean + representing whether the loader handles packages. + + The finder will cache the directory contents as necessary, making stat calls + for each module search to verify the cache is not outdated. Because cache + staleness relies upon the granularity of the operating system's state + information of the file system, there is a potential race condition of + searching for a module, creating a new file, and then searching for the + module the new file represents. If the operations happen fast enough to fit + within the granularity of stat calls, then the module search will fail. To + prevent this from happening, when you create a module dynamically, make sure + to call :func:`importlib.invalidate_caches`. + + .. versionadded:: 3.3 + + .. attribute:: path + + The path the finder will search in. + + .. method:: find_module(fullname) + + Attempt to find the loader to handle *fullname* within :attr:`path`. + + .. method:: invalidate_caches() + + Clear out the internal cache. + + .. classmethod:: path_hook(\*loader_details) + + A class method which returns a closure for use on :attr:`sys.path_hooks`. + An instance of :class:`FileFinder` is returned by the closure using the + path argument given to the closure directly and *loader_details* + indirectly. + + If the argument to the closure is not an existing directory, + :exc:`ImportError` is raised. + + +.. class:: SourceFileLoader(fullname, path) + + A concrete implementation of :class:`importlib.abc.SourceLoader` by + subclassing :class:`importlib.abc.FileLoader` and providing some concrete + implementations of other methods. + + .. versionadded:: 3.3 + + .. attribute:: name + + The name of the module that this loader will handle. + + .. attribute:: path + + The path to the source file. + + .. method:: is_package(fullname) + + Return true if :attr:`path` appears to be for a package. + + .. method:: path_stats(path) + + Concrete implementation of :meth:`importlib.abc.SourceLoader.path_stats`. + + .. method:: set_data(path, data) + + Concrete implementation of :meth:`importlib.abc.SourceLoader.set_data`. + + +.. class:: SourcelessFileLoader(fullname, path) + + A concrete implementation of :class:`importlib.abc.FileLoader` which can + import bytecode files (i.e. no source code files exist). + + Please note that direct use of bytecode files (and thus not source code + files) inhibits your modules from being usable by all Python + implementations or new versions of Python which change the bytecode + format. + + .. versionadded:: 3.3 + + .. attribute:: name + + The name of the module the loader will handle. + + .. attribute:: path + + The path to the bytecode file. + + .. method:: is_package(fullname) + + Determines if the module is a package based on :attr:`path`. + + .. method:: get_code(fullname) + + Returns the code object for :attr:`name` created from :attr:`path`. + + .. method:: get_source(fullname) + + Returns ``None`` as bytecode files have no source when this loader is + used. + + +.. class:: ExtensionFileLoader(fullname, path) + + A concrete implementation of :class:`importlib.abc.InspectLoader` for + extension modules. + + The *fullname* argument specifies the name of the module the loader is to + support. The *path* argument is the path to the extension module's file. + + .. versionadded:: 3.3 + + .. attribute:: name + + Name of the module the loader supports. + + .. attribute:: path + + Path to the extension module. + + .. method:: load_module(fullname=None) + + Loads the extension module if and only if *fullname* is the same as + :attr:`name` or is ``None``. + + .. method:: is_package(fullname) + + Returns ``False`` as extension modules can never be packages. + + .. method:: get_code(fullname) + + Returns ``None`` as extension modules lack a code object. + + .. method:: get_source(fullname) + + Returns ``None`` as extension modules do not have source code. + + :mod:`importlib.util` -- Utility code for importers --------------------------------------------------- .. module:: importlib.util - :synopsis: Importers and path hooks + :synopsis: Utility code for importers This module contains the various objects that help in the construction of an :term:`importer`. +.. function:: resolve_name(name, package) + + Resolve a relative module name to an absolute one. + + If **name** has no leading dots, then **name** is simply returned. This + allows for usage such as + ``importlib.util.resolve_name('sys', __package__)`` without doing a + check to see if the **package** argument is needed. + + :exc:`ValueError` is raised if **name** is a relative module name but + package is a false value (e.g. ``None`` or the empty string). + :exc:`ValueError` is also raised a relative name would escape its containing + package (e.g. requesting ``..bacon`` from within the ``spam`` package). + + .. versionadded:: 3.3 + .. decorator:: module_for_loader A :term:`decorator` for a :term:`loader` method, @@ -477,22 +761,30 @@ an :term:`importer`. signature taking two positional arguments (e.g. ``load_module(self, module)``) for which the second argument will be the module **object** to be used by the loader. - Note that the decorator - will not work on static methods because of the assumption of two - arguments. + Note that the decorator will not work on static methods because of the + assumption of two arguments. The decorated method will take in the **name** of the module to be loaded as expected for a :term:`loader`. If the module is not found in :data:`sys.modules` then a new one is constructed with its - :attr:`__name__` attribute set. Otherwise the module found in - :data:`sys.modules` will be passed into the method. If an - exception is raised by the decorated method and a module was added to + :attr:`__name__` attribute set to **name**, :attr:`__loader__` set to + **self**, and :attr:`__package__` set if + :meth:`importlib.abc.InspectLoader.is_package` is defined for **self** and + does not raise :exc:`ImportError` for **name**. If a new module is not + needed then the module found in :data:`sys.modules` will be passed into the + method. + + If an exception is raised by the decorated method and a module was added to :data:`sys.modules` it will be removed to prevent a partially initialized module from being in left in :data:`sys.modules`. If the module was already in :data:`sys.modules` then it is left alone. Use of this decorator handles all the details of which module object a - loader should initialize as specified by :pep:`302`. + loader should initialize as specified by :pep:`302` as best as possible. + + .. versionchanged:: 3.3 + :attr:`__loader__` and :attr:`__package__` are automatically set + (when possible). .. decorator:: set_loader @@ -500,7 +792,13 @@ an :term:`importer`. to set the :attr:`__loader__` attribute on loaded modules. If the attribute is already set the decorator does nothing. It is assumed that the first positional argument to the - wrapped method is what :attr:`__loader__` should be set to. + wrapped method (i.e. ``self``) is what :attr:`__loader__` should be set to. + + .. note:: + + It is recommended that :func:`module_for_loader` be used over this + decorator as it subsumes this functionality. + .. decorator:: set_package @@ -511,8 +809,12 @@ an :term:`importer`. set on and not the module found in :data:`sys.modules`. Reliance on this decorator is discouraged when it is possible to set - :attr:`__package__` before the execution of the code is possible. By - setting it before the code for the module is executed it allows the - attribute to be used at the global level of the module during + :attr:`__package__` before importing. By + setting it beforehand the code for the module is executed with the + attribute set and thus can be used by global level code during initialization. + .. note:: + + It is recommended that :func:`module_for_loader` be used over this + decorator as it subsumes this functionality. diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index d127ce8..611f780 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -190,6 +190,10 @@ attributes: compared to the constants defined in the :mod:`imp` module; see the documentation for that module for more information on module types. + .. deprecated:: 3.3 + You may check the file path's suffix against the supported suffixes + listed in :mod:`importlib.machinery` to infer the same information. + .. function:: getmodulename(path) @@ -355,17 +359,25 @@ Retrieving source code argument may be a module, class, method, function, traceback, frame, or code object. The source code is returned as a list of the lines corresponding to the object and the line number indicates where in the original source file the first - line of code was found. An :exc:`IOError` is raised if the source code cannot + line of code was found. An :exc:`OSError` is raised if the source code cannot be retrieved. + .. versionchanged:: 3.3 + :exc:`OSError` is raised instead of :exc:`IOError`, now an alias of the + former. + .. function:: getsource(object) Return the text of the source code for an object. The argument may be a module, class, method, function, traceback, frame, or code object. The source code is - returned as a single string. An :exc:`IOError` is raised if the source code + returned as a single string. An :exc:`OSError` is raised if the source code cannot be retrieved. + .. versionchanged:: 3.3 + :exc:`OSError` is raised instead of :exc:`IOError`, now an alias of the + former. + .. function:: cleandoc(doc) @@ -432,11 +444,16 @@ Classes and functions locals dictionary of the given frame. -.. function:: formatargspec(args[, varargs, varkw, defaults, formatarg, formatvarargs, formatvarkw, formatvalue]) +.. function:: formatargspec(args[, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations, formatarg, formatvarargs, formatvarkw, formatvalue, formatreturns, formatannotations]) - Format a pretty argument spec from the four values returned by - :func:`getargspec`. The format\* arguments are the corresponding optional - formatting functions that are called to turn names and values into strings. + Format a pretty argument spec from the values returned by + :func:`getargspec` or :func:`getfullargspec`. + + The first seven arguments are (``args``, ``varargs``, ``varkw``, + ``defaults``, ``kwonlyargs``, ``kwonlydefaults``, ``annotations``). The + other five arguments are the corresponding optional formatting functions + that are called to turn names and values into strings. The last argument + is an optional function to format the sequence of arguments. .. function:: formatargvalues(args[, varargs, varkw, locals, formatarg, formatvarargs, formatvarkw, formatvalue]) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index becc4a2..4d564bb 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -33,6 +33,10 @@ giving a :class:`str` object to the ``write()`` method of a binary stream will raise a ``TypeError``. So will giving a :class:`bytes` object to the ``write()`` method of a text stream. +.. versionchanged:: 3.3 + Operations defined in this module used to raise :exc:`IOError`, which is + now an alias of :exc:`OSError`. + Text I/O ^^^^^^^^ @@ -109,21 +113,13 @@ High-level Module Interface .. exception:: BlockingIOError - Error raised when blocking would occur on a non-blocking stream. It inherits - :exc:`IOError`. - - In addition to those of :exc:`IOError`, :exc:`BlockingIOError` has one - attribute: - - .. attribute:: characters_written - - An integer containing the number of characters written to the stream - before it blocked. + This is a compatibility alias for the builtin :exc:`BlockingIOError` + exception. .. exception:: UnsupportedOperation - An exception inheriting :exc:`IOError` and :exc:`ValueError` that is raised + An exception inheriting :exc:`OSError` and :exc:`ValueError` that is raised when an unsupported operation is called on a stream. @@ -202,8 +198,8 @@ I/O Base Classes Even though :class:`IOBase` does not declare :meth:`read`, :meth:`readinto`, or :meth:`write` because their signatures will vary, implementations and clients should consider those methods part of the interface. Also, - implementations may raise a :exc:`IOError` when operations they do not - support are called. + implementations may raise a :exc:`ValueError` (or :exc:`UnsupportedOperation`) + when operations they do not support are called. The basic type used for binary data read from or written to a file is :class:`bytes`. :class:`bytearray`\s are accepted too, and in some cases @@ -211,7 +207,7 @@ I/O Base Classes :class:`str` data. Note that calling any method (even inquiries) on a closed stream is - undefined. Implementations may raise :exc:`IOError` in this case. + undefined. Implementations may raise :exc:`ValueError` in this case. IOBase (and its subclasses) support the iterator protocol, meaning that an :class:`IOBase` object can be iterated over yielding the lines in a stream. @@ -244,7 +240,7 @@ I/O Base Classes .. method:: fileno() Return the underlying file descriptor (an integer) of the stream if it - exists. An :exc:`IOError` is raised if the IO object does not use a file + exists. An :exc:`OSError` is raised if the IO object does not use a file descriptor. .. method:: flush() @@ -260,7 +256,7 @@ I/O Base Classes .. method:: readable() Return ``True`` if the stream can be read from. If False, :meth:`read` - will raise :exc:`IOError`. + will raise :exc:`OSError`. .. method:: readline(limit=-1) @@ -298,7 +294,7 @@ I/O Base Classes .. method:: seekable() Return ``True`` if the stream supports random access. If ``False``, - :meth:`seek`, :meth:`tell` and :meth:`truncate` will raise :exc:`IOError`. + :meth:`seek`, :meth:`tell` and :meth:`truncate` will raise :exc:`OSError`. .. method:: tell() @@ -316,7 +312,7 @@ I/O Base Classes .. method:: writable() Return ``True`` if the stream supports writing. If ``False``, - :meth:`write` and :meth:`truncate` will raise :exc:`IOError`. + :meth:`write` and :meth:`truncate` will raise :exc:`OSError`. .. method:: writelines(lines) @@ -450,7 +446,7 @@ I/O Base Classes Write the given bytes or bytearray object, *b* and return the number of bytes written (never less than ``len(b)``, since if the write fails - an :exc:`IOError` will be raised). Depending on the actual + an :exc:`OSError` will be raised). Depending on the actual implementation, these bytes may be readily written to the underlying stream, or held in a buffer for performance and latency reasons. @@ -462,7 +458,7 @@ I/O Base Classes Raw File I/O ^^^^^^^^^^^^ -.. class:: FileIO(name, mode='r', closefd=True) +.. class:: FileIO(name, mode='r', closefd=True, opener=None) :class:`FileIO` represents an OS-level file containing bytes data. It implements the :class:`RawIOBase` interface (and therefore the @@ -475,14 +471,27 @@ Raw File I/O * an integer representing the number of an existing OS-level file descriptor to which the resulting :class:`FileIO` object will give access. - The *mode* can be ``'r'``, ``'w'`` or ``'a'`` for reading (default), writing, - or appending. The file will be created if it doesn't exist when opened for - writing or appending; it will be truncated when opened for writing. Add a + The *mode* can be ``'r'``, ``'w'``, ``'x'`` or ``'a'`` for reading + (default), writing, exclusive creation or appending. The file will be + created if it doesn't exist when opened for writing or appending; it will be + truncated when opened for writing. :exc:`FileExistsError` will be raised if + it already exists when opened for creating. Opening a file for creating + implies writing, so this mode behaves in a similar way to ``'w'``. Add a ``'+'`` to the mode to allow simultaneous reading and writing. The :meth:`read` (when called with a positive argument), :meth:`readinto` and :meth:`write` methods on this class will only make one system call. + A custom opener can be used by passing a callable as *opener*. The underlying + file descriptor for the file object is then obtained by calling *opener* with + (*name*, *flags*). *opener* must return an open file descriptor (passing + :mod:`os.open` as *opener* results in functionality similar to passing + ``None``). + + .. versionchanged:: 3.3 + The *opener* parameter was added. + The ``'x'`` mode was added. + In addition to the attributes and methods from :class:`IOBase` and :class:`RawIOBase`, :class:`FileIO` provides the following data attributes and methods: @@ -736,7 +745,8 @@ Text I/O written. -.. class:: TextIOWrapper(buffer, encoding=None, errors=None, newline=None, line_buffering=False) +.. class:: TextIOWrapper(buffer, encoding=None, errors=None, newline=None, \ + line_buffering=False, write_through=False) A buffered text stream over a :class:`BufferedIOBase` binary stream. It inherits :class:`TextIOBase`. @@ -767,6 +777,13 @@ Text I/O If *line_buffering* is ``True``, :meth:`flush` is implied when a call to write contains a newline character. + If *write_through* is ``True``, calls to :meth:`write` are guaranteed + not to be buffered: any data written on the :class:`TextIOWrapper` + object is immediately handled to its underlying binary *buffer*. + + .. versionchanged:: 3.3 + The *write_through* argument has been added. + :class:`TextIOWrapper` provides one attribute in addition to those of :class:`TextIOBase` and its parents: diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index d1d1188..994a25a 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -46,7 +46,7 @@ Iterator Arguments Results ==================== ============================ ================================================= ============================================================= Iterator Arguments Results Example ==================== ============================ ================================================= ============================================================= -:func:`accumulate` p p0, p0+p1, p0+p1+p2, ... ``accumulate([1,2,3,4,5]) --> 1 3 6 10 15`` +:func:`accumulate` p [,func] p0, p0+p1, p0+p1+p2, ... ``accumulate([1,2,3,4,5]) --> 1 3 6 10 15`` :func:`chain` p, q, ... p0, p1, ... plast, q0, q1, ... ``chain('ABC', 'DEF') --> A B C D E F`` :func:`compress` data, selectors (d[0] if s[0]), (d[1] if s[1]), ... ``compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F`` :func:`dropwhile` pred, seq seq[n], seq[n+1], starting when pred fails ``dropwhile(lambda x: x<5, [1,4,6,4,1]) --> 6 4 1`` @@ -84,23 +84,61 @@ The following module functions all construct and return iterators. Some provide streams of infinite length, so they should only be accessed by functions or loops that truncate the stream. -.. function:: accumulate(iterable) +.. function:: accumulate(iterable[, func]) Make an iterator that returns accumulated sums. Elements may be any addable - type including :class:`Decimal` or :class:`Fraction`. Equivalent to:: + type including :class:`Decimal` or :class:`Fraction`. If the optional + *func* argument is supplied, it should be a function of two arguments + and it will be used instead of addition. - def accumulate(iterable): + Equivalent to:: + + def accumulate(iterable, func=operator.add): 'Return running totals' # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 + # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 it = iter(iterable) total = next(it) yield total for element in it: - total = total + element + total = func(total, element) yield total + There are a number of uses for the *func* argument. It can be set to + :func:`min` for a running minimum, :func:`max` for a running maximum, or + :func:`operator.mul` for a running product. Amortization tables can be + built by accumulating interest and applying payments. First-order + `recurrence relations <http://en.wikipedia.org/wiki/Recurrence_relation>`_ + can be modeled by supplying the initial value in the iterable and using only + the accumulated total in *func* argument:: + + >>> data = [3, 4, 6, 2, 1, 9, 0, 7, 5, 8] + >>> list(accumulate(data, operator.mul)) # running product + [3, 12, 72, 144, 144, 1296, 0, 0, 0, 0] + >>> list(accumulate(data, max)) # running maximum + [3, 4, 6, 6, 6, 9, 9, 9, 9, 9] + + # Amortize a 5% loan of 1000 with 4 annual payments of 90 + >>> cashflows = [1000, -90, -90, -90, -90] + >>> list(accumulate(cashflows, lambda bal, pmt: bal*1.05 + pmt)) + [1000, 960.0, 918.0, 873.9000000000001, 827.5950000000001] + + # Chaotic recurrence relation http://en.wikipedia.org/wiki/Logistic_map + >>> logistic_map = lambda x, _: r * x * (1 - x) + >>> r = 3.8 + >>> x0 = 0.4 + >>> inputs = repeat(x0, 36) # only the initial value is used + >>> [format(x, '.2f') for x in accumulate(inputs, logistic_map)] + ['0.40', '0.91', '0.30', '0.81', '0.60', '0.92', '0.29', '0.79', '0.63', + '0.88' ,'0.39', '0.90', '0.33', '0.84', '0.52', '0.95', '0.18', '0.57', + '0.93', '0.25', '0.71', '0.79', '0.63', '0.88', '0.39', '0.91', '0.32', + '0.83', '0.54', '0.95', '0.20', '0.60', '0.91', '0.30', '0.80', '0.60'] + .. versionadded:: 3.2 + .. versionchanged:: 3.3 + Added the optional *func* parameter. + .. function:: chain(*iterables) Make an iterator that returns elements from the first iterable until it is diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst index ef65cfa..537b8c7 100644 --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -164,6 +164,87 @@ this value. changed. If it has, the existing stream is flushed and closed and the file opened again, before outputting the record to the file. +.. _base-rotating-handler: + +BaseRotatingHandler +^^^^^^^^^^^^^^^^^^^ + +The :class:`BaseRotatingHandler` class, located in the :mod:`logging.handlers` +module, is the base class for the rotating file handlers, +:class:`RotatingFileHandler` and :class:`TimedRotatingFileHandler`. You should +not need to instantiate this class, but it has attributes and methods you may +need to override. + +.. class:: BaseRotatingHandler(filename, mode, encoding=None, delay=False) + + The parameters are as for :class:`FileHandler`. The attributes are: + + .. attribute:: namer + + If this attribute is set to a callable, the :meth:`rotation_filename` + method delegates to this callable. The parameters passed to the callable + are those passed to :meth:`rotation_filename`. + + .. note:: The namer function is called quite a few times during rollover, + so it should be as simple and as fast as possible. It should also + return the same output every time for a given input, otherwise the + rollover behaviour may not work as expected. + + .. versionadded:: 3.3 + + + .. attribute:: BaseRotatingHandler.rotator + + If this attribute is set to a callable, the :meth:`rotate` method + delegates to this callable. The parameters passed to the callable are + those passed to :meth:`rotate`. + + .. versionadded:: 3.3 + + .. method:: BaseRotatingHandler.rotation_filename(default_name) + + Modify the filename of a log file when rotating. + + This is provided so that a custom filename can be provided. + + The default implementation calls the 'namer' attribute of the handler, + if it's callable, passing the default name to it. If the attribute isn't + callable (the default is ``None``), the name is returned unchanged. + + :param default_name: The default name for the log file. + + .. versionadded:: 3.3 + + + .. method:: BaseRotatingHandler.rotate(source, dest) + + When rotating, rotate the current log. + + The default implementation calls the 'rotator' attribute of the handler, + if it's callable, passing the source and dest arguments to it. If the + attribute isn't callable (the default is ``None``), the source is simply + renamed to the destination. + + :param source: The source filename. This is normally the base + filename, e.g. 'test.log' + :param dest: The destination filename. This is normally + what the source is rotated to, e.g. 'test.log.1'. + + .. versionadded:: 3.3 + +The reason the attributes exist is to save you having to subclass - you can use +the same callables for instances of :class:`RotatingFileHandler` and +:class:`TimedRotatingFileHandler`. If either the namer or rotator callable +raises an exception, this will be handled in the same way as any other +exception during an :meth:`emit` call, i.e. via the :meth:`handleError` method +of the handler. + +If you need to make more significant changes to rotation processing, you can +override the methods. + +For an example, see :ref:`cookbook-rotator-namer`. + + .. _rotating-file-handler: RotatingFileHandler @@ -452,6 +533,15 @@ supports sending logging messages to a remote or local Unix syslog. behaviour) but can be set to ``False`` on a ``SysLogHandler`` instance in order for that instance to *not* append the NUL terminator. + .. versionchanged:: 3.3 + (See: :issue:`12419`.) In earlier versions, there was no facility for + an "ident" or "tag" prefix to identify the source of the message. This + can now be specified using a class-level attribute, defaulting to + ``""`` to preserve existing behaviour, but which can be overridden on + a ``SysLogHandler`` instance in order for that instance to prepend + the ident to every message handled. Note that the provided ident must + be text, not bytes, and is prepended to the message exactly as is. + .. method:: encodePriority(facility, priority) Encodes the facility and priority into an integer. You can pass in strings @@ -614,7 +704,7 @@ The :class:`SMTPHandler` class, located in the :mod:`logging.handlers` module, supports sending logging messages to an email address via SMTP. -.. class:: SMTPHandler(mailhost, fromaddr, toaddrs, subject, credentials=None, secure=None) +.. class:: SMTPHandler(mailhost, fromaddr, toaddrs, subject, credentials=None, secure=None, timeout=1.0) Returns a new instance of the :class:`SMTPHandler` class. The instance is initialized with the from and to addresses and subject line of the email. The @@ -630,6 +720,12 @@ supports sending logging messages to an email address via SMTP. and certificate file. (This tuple is passed to the :meth:`smtplib.SMTP.starttls` method.) + A timeout can be specified for communication with the SMTP server using the + *timeout* argument. + + .. versionadded:: 3.3 + The *timeout* argument was added. + .. method:: emit(record) Formats the record and sends it to the specified addressees. @@ -690,7 +786,7 @@ should, then :meth:`flush` is expected to do the flushing. .. method:: close() - Calls :meth:`flush`, sets the target to :const:`None` and clears the + Calls :meth:`flush`, sets the target to ``None`` and clears the buffer. @@ -859,6 +955,15 @@ possible, while any potentially slow operations (such as sending an email via Note that if you don't call this before your application exits, there may be some records still left on the queue, which won't be processed. + .. method:: enqueue_sentinel() + + Writes a sentinel to the queue to tell the listener to quit. This + implementation uses ``put_nowait()``. You may want to override this + method if you want to use timeouts or work with custom queue + implementations. + + .. versionadded:: 3.3 + .. seealso:: diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 14e00fe..fbbd305 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -213,6 +213,9 @@ is the module's name in the Python package namespace. Logs a message with level :const:`WARNING` on this logger. The arguments are interpreted as for :meth:`debug`. + .. note:: There is an obsolete method ``warn`` which is functionally + identical to ``warning``. As ``warn`` is deprecated, please do not use + it - use ``warning`` instead. .. method:: Logger.error(msg, *args, **kwargs) @@ -492,6 +495,19 @@ The useful mapping keys in a :class:`LogRecord` are given in the section on want all logging times to be shown in GMT, set the ``converter`` attribute in the ``Formatter`` class. + .. versionchanged:: 3.3 + Previously, the default ISO 8601 format was hard-coded as in this + example: ``2010-09-06 22:38:15,292`` where the part before the comma is + handled by a strptime format string (``'%Y-%m-%d %H:%M:%S'``), and the + part after the comma is a millisecond value. Because strptime does not + have a format placeholder for milliseconds, the millisecond value is + appended using another format string, ``'%s,%03d'`` – and both of these + format strings have been hardcoded into this method. With the change, + these strings are defined as class-level attributes which can be + overridden at the instance level when desired. The names of the + attributes are ``default_time_format`` (for the strptime format string) + and ``default_msec_format`` (for appending the millisecond value). + .. method:: formatException(exc_info) Formats the specified exception information (a standard exception tuple as @@ -898,8 +914,12 @@ functions. .. function:: warning(msg, *args, **kwargs) - Logs a message with level :const:`WARNING` on the root logger. The arguments are - interpreted as for :func:`debug`. + Logs a message with level :const:`WARNING` on the root logger. The arguments + are interpreted as for :func:`debug`. + + .. note:: There is an obsolete function ``warn`` which is functionally + identical to ``warning``. As ``warn`` is deprecated, please do not use + it - use ``warning`` instead. .. function:: error(msg, *args, **kwargs) @@ -1025,12 +1045,27 @@ functions. | ``stream`` | Use the specified stream to initialize the | | | StreamHandler. Note that this argument is | | | incompatible with 'filename' - if both are | - | | present, 'stream' is ignored. | + | | present, a ``ValueError`` is raised. | + +--------------+---------------------------------------------+ + | ``handlers`` | If specified, this should be an iterable of | + | | already created handlers to add to the root | + | | logger. Any handlers which don't already | + | | have a formatter set will be assigned the | + | | default formatter created in this function. | + | | Note that this argument is incompatible | + | | with 'filename' or 'stream' - if both are | + | | present, a ``ValueError`` is raised. | +--------------+---------------------------------------------+ .. versionchanged:: 3.2 The ``style`` argument was added. + .. versionchanged:: 3.3 + The ``handlers`` argument was added. Additional checks were added to + catch situations where incompatible arguments are specified (e.g. + ``handlers`` together with ``stream`` or ``filename``, or ``stream`` + together with ``filename``). + .. function:: shutdown() diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst new file mode 100644 index 0000000..33a5428 --- /dev/null +++ b/Doc/library/lzma.rst @@ -0,0 +1,375 @@ +:mod:`lzma` --- Compression using the LZMA algorithm +==================================================== + +.. module:: lzma + :synopsis: A Python wrapper for the liblzma compression library. +.. moduleauthor:: Nadeem Vawda <nadeem.vawda@gmail.com> +.. sectionauthor:: Nadeem Vawda <nadeem.vawda@gmail.com> + +.. versionadded:: 3.3 + + +This module provides classes and convenience functions for compressing and +decompressing data using the LZMA compression algorithm. Also included is a file +interface supporting the ``.xz`` and legacy ``.lzma`` file formats used by the +:program:`xz` utility, as well as raw compressed streams. + +The interface provided by this module is very similar to that of the :mod:`bz2` +module. However, note that :class:`LZMAFile` is *not* thread-safe, unlike +:class:`bz2.BZ2File`, so if you need to use a single :class:`LZMAFile` instance +from multiple threads, it is necessary to protect it with a lock. + + +.. exception:: LZMAError + + This exception is raised when an error occurs during compression or + decompression, or while initializing the compressor/decompressor state. + + +Reading and writing compressed files +------------------------------------ + +.. class:: LZMAFile(filename=None, mode="r", \*, fileobj=None, format=None, check=-1, preset=None, filters=None) + + Open an LZMA-compressed file. + + An :class:`LZMAFile` can wrap an existing :term:`file object` (given by + *fileobj*), or operate directly on a named file (named by *filename*). + Exactly one of these two parameters should be provided. If *fileobj* is + provided, it is not closed when the :class:`LZMAFile` is closed. + + The *mode* argument can be either ``"r"`` for reading (default), ``"w"`` for + overwriting, or ``"a"`` for appending. If *fileobj* is provided, a mode of + ``"w"`` does not truncate the file, and is instead equivalent to ``"a"``. + + When opening a file for reading, the input file may be the concatenation of + multiple separate compressed streams. These are transparently decoded as a + single logical stream. + + When opening a file for reading, the *format* and *filters* arguments have + the same meanings as for :class:`LZMADecompressor`. In this case, the *check* + and *preset* arguments should not be used. + + When opening a file for writing, the *format*, *check*, *preset* and + *filters* arguments have the same meanings as for :class:`LZMACompressor`. + + :class:`LZMAFile` supports all the members specified by + :class:`io.BufferedIOBase`, except for :meth:`detach` and :meth:`truncate`. + Iteration and the :keyword:`with` statement are supported. + + The following method is also provided: + + .. method:: peek(size=-1) + + Return buffered data without advancing the file position. At least one + byte of data will be returned, unless EOF has been reached. The exact + number of bytes returned is unspecified (the *size* argument is ignored). + + +Compressing and decompressing data in memory +-------------------------------------------- + +.. class:: LZMACompressor(format=FORMAT_XZ, check=-1, preset=None, filters=None) + + Create a compressor object, which can be used to compress data incrementally. + + For a more convenient way of compressing a single chunk of data, see + :func:`compress`. + + The *format* argument specifies what container format should be used. + Possible values are: + + * :const:`FORMAT_XZ`: The ``.xz`` container format. + This is the default format. + + * :const:`FORMAT_ALONE`: The legacy ``.lzma`` container format. + This format is more limited than ``.xz`` -- it does not support integrity + checks or multiple filters. + + * :const:`FORMAT_RAW`: A raw data stream, not using any container format. + This format specifier does not support integrity checks, and requires that + you always specify a custom filter chain (for both compression and + decompression). Additionally, data compressed in this manner cannot be + decompressed using :const:`FORMAT_AUTO` (see :class:`LZMADecompressor`). + + The *check* argument specifies the type of integrity check to include in the + compressed data. This check is used when decompressing, to ensure that the + data has not been corrupted. Possible values are: + + * :const:`CHECK_NONE`: No integrity check. + This is the default (and the only acceptable value) for + :const:`FORMAT_ALONE` and :const:`FORMAT_RAW`. + + * :const:`CHECK_CRC32`: 32-bit Cyclic Redundancy Check. + + * :const:`CHECK_CRC64`: 64-bit Cyclic Redundancy Check. + This is the default for :const:`FORMAT_XZ`. + + * :const:`CHECK_SHA256`: 256-bit Secure Hash Algorithm. + + If the specified check is not supported, an :class:`LZMAError` is raised. + + The compression settings can be specified either as a preset compression + level (with the *preset* argument), or in detail as a custom filter chain + (with the *filters* argument). + + The *preset* argument (if provided) should be an integer between ``0`` and + ``9`` (inclusive), optionally OR-ed with the constant + :const:`PRESET_EXTREME`. If neither *preset* nor *filters* are given, the + default behavior is to use :const:`PRESET_DEFAULT` (preset level ``6``). + Higher presets produce smaller output, but make the compression process + slower. + + .. note:: + + In addition to being more CPU-intensive, compression with higher presets + also requires much more memory (and produces output that needs more memory + to decompress). With preset ``9`` for example, the overhead for an + :class:`LZMACompressor` object can be as high as 800MiB. For this reason, + it is generally best to stick with the default preset. + + The *filters* argument (if provided) should be a filter chain specifier. + See :ref:`filter-chain-specs` for details. + + .. method:: compress(data) + + Compress *data* (a :class:`bytes` object), returning a :class:`bytes` + object containing compressed data for at least part of the input. Some of + *data* may be buffered internally, for use in later calls to + :meth:`compress` and :meth:`flush`. The returned data should be + concatenated with the output of any previous calls to :meth:`compress`. + + .. method:: flush() + + Finish the compression process, returning a :class:`bytes` object + containing any data stored in the compressor's internal buffers. + + The compressor cannot be used after this method has been called. + + +.. class:: LZMADecompressor(format=FORMAT_AUTO, memlimit=None, filters=None) + + Create a decompressor object, which can be used to decompress data + incrementally. + + For a more convenient way of decompressing an entire compressed stream at + once, see :func:`decompress`. + + The *format* argument specifies the container format that should be used. The + default is :const:`FORMAT_AUTO`, which can decompress both ``.xz`` and + ``.lzma`` files. Other possible values are :const:`FORMAT_XZ`, + :const:`FORMAT_ALONE`, and :const:`FORMAT_RAW`. + + The *memlimit* argument specifies a limit (in bytes) on the amount of memory + that the decompressor can use. When this argument is used, decompression will + fail with an :class:`LZMAError` if it is not possible to decompress the input + within the given memory limit. + + The *filters* argument specifies the filter chain that was used to create + the stream being decompressed. This argument is required if *format* is + :const:`FORMAT_RAW`, but should not be used for other formats. + See :ref:`filter-chain-specs` for more information about filter chains. + + .. note:: + This class does not transparently handle inputs containing multiple + compressed streams, unlike :func:`decompress` and :class:`LZMAFile`. To + decompress a multi-stream input with :class:`LZMADecompressor`, you must + create a new decompressor for each stream. + + .. method:: decompress(data) + + Decompress *data* (a :class:`bytes` object), returning a :class:`bytes` + object containing the decompressed data for at least part of the input. + Some of *data* may be buffered internally, for use in later calls to + :meth:`decompress`. The returned data should be concatenated with the + output of any previous calls to :meth:`decompress`. + + .. attribute:: check + + The ID of the integrity check used by the input stream. This may be + :const:`CHECK_UNKNOWN` until enough of the input has been decoded to + determine what integrity check it uses. + + .. attribute:: eof + + True if the end-of-stream marker has been reached. + + .. attribute:: unused_data + + Data found after the end of the compressed stream. + + Before the end of the stream is reached, this will be ``b""``. + + +.. function:: compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None) + + Compress *data* (a :class:`bytes` object), returning the compressed data as a + :class:`bytes` object. + + See :class:`LZMACompressor` above for a description of the *format*, *check*, + *preset* and *filters* arguments. + + +.. function:: decompress(data, format=FORMAT_AUTO, memlimit=None, filters=None) + + Decompress *data* (a :class:`bytes` object), returning the uncompressed data + as a :class:`bytes` object. + + If *data* is the concatenation of multiple distinct compressed streams, + decompress all of these streams, and return the concatenation of the results. + + See :class:`LZMADecompressor` above for a description of the *format*, + *memlimit* and *filters* arguments. + + +Miscellaneous +------------- + +.. function:: is_check_supported(check) + + Returns true if the given integrity check is supported on this system. + + :const:`CHECK_NONE` and :const:`CHECK_CRC32` are always supported. + :const:`CHECK_CRC64` and :const:`CHECK_SHA256` may be unavailable if you are + using a version of :program:`liblzma` that was compiled with a limited + feature set. + + +.. function:: encode_filter_properties(filter) + + Return a :class:`bytes` object encoding the options (properties) of the + filter specified by *filter* (a dictionary). + + *filter* is interpreted as a filter specifier, as described in + :ref:`filter-chain-specs`. + + The returned data does not include the filter ID itself, only the options. + + This function is primarily of interest to users implementing custom file + formats. + + +.. function:: decode_filter_properties(filter_id, encoded_props) + + Return a dictionary describing a filter with ID *filter_id*, and options + (properties) decoded from the :class:`bytes` object *encoded_props*. + + The returned dictionary is a filter specifier, as described in + :ref:`filter-chain-specs`. + + This function is primarily of interest to users implementing custom file + formats. + + +.. _filter-chain-specs: + +Specifying custom filter chains +------------------------------- + +A filter chain specifier is a sequence of dictionaries, where each dictionary +contains the ID and options for a single filter. Each dictionary must contain +the key ``"id"``, and may contain additional keys to specify filter-dependent +options. Valid filter IDs are as follows: + +* Compression filters: + * :const:`FILTER_LZMA1` (for use with :const:`FORMAT_ALONE`) + * :const:`FILTER_LZMA2` (for use with :const:`FORMAT_XZ` and :const:`FORMAT_RAW`) + +* Delta filter: + * :const:`FILTER_DELTA` + +* Branch-Call-Jump (BCJ) filters: + * :const:`FILTER_X86` + * :const:`FILTER_IA64` + * :const:`FILTER_ARM` + * :const:`FILTER_ARMTHUMB` + * :const:`FILTER_POWERPC` + * :const:`FILTER_SPARC` + +A filter chain can consist of up to 4 filters, and cannot be empty. The last +filter in the chain must be a compression filter, and any other filters must be +delta or BCJ filters. + +Compression filters support the following options (specified as additional +entries in the dictionary representing the filter): + + * ``preset``: A compression preset to use as a source of default values for + options that are not specified explicitly. + * ``dict_size``: Dictionary size in bytes. This should be between 4KiB and + 1.5GiB (inclusive). + * ``lc``: Number of literal context bits. + * ``lp``: Number of literal position bits. The sum ``lc + lp`` must be at + most 4. + * ``pb``: Number of position bits; must be at most 4. + * ``mode``: :const:`MODE_FAST` or :const:`MODE_NORMAL`. + * ``nice_len``: What should be considered a "nice length" for a match. + This should be 273 or less. + * ``mf``: What match finder to use -- :const:`MF_HC3`, :const:`MF_HC4`, + :const:`MF_BT2`, :const:`MF_BT3`, or :const:`MF_BT4`. + * ``depth``: Maximum search depth used by match finder. 0 (default) means to + select automatically based on other filter options. + +The delta filter stores the differences between bytes, producing more repetitive +input for the compressor in certain circumstances. It only supports a single +The delta filter supports only one option, ``dist``. This indicates the distance +between bytes to be subtracted. The default is 1, i.e. take the differences +between adjacent bytes. + +The BCJ filters are intended to be applied to machine code. They convert +relative branches, calls and jumps in the code to use absolute addressing, with +the aim of increasing the redundancy that can be exploited by the compressor. +These filters support one option, ``start_offset``. This specifies the address +that should be mapped to the beginning of the input data. The default is 0. + + +Examples +-------- + +Reading in a compressed file:: + + import lzma + with lzma.LZMAFile("file.xz") as f: + file_content = f.read() + +Creating a compressed file:: + + import lzma + data = b"Insert Data Here" + with lzma.LZMAFile("file.xz", "w") as f: + f.write(data) + +Compressing data in memory:: + + import lzma + data_in = b"Insert Data Here" + data_out = lzma.compress(data_in) + +Incremental compression:: + + import lzma + lzc = lzma.LZMACompressor() + out1 = lzc.compress(b"Some data\n") + out2 = lzc.compress(b"Another piece of data\n") + out3 = lzc.compress(b"Even more data\n") + out4 = lzc.flush() + # Concatenate all the partial results: + result = b"".join([out1, out2, out3, out4]) + +Writing compressed data to an already-open file:: + + import lzma + with open("file.xz", "wb") as f: + f.write(b"This data will not be compressed\n") + with lzma.LZMAFile(fileobj=f, mode="w") as lzf: + lzf.write(b"This *will* be compressed\n") + f.write(b"Not compressed\n") + +Creating a compressed file using a custom filter chain:: + + import lzma + my_filters = [ + {"id": lzma.FILTER_DELTA, "dist": 5}, + {"id": lzma.FILTER_LZMA2, "preset": 7 | lzma.PRESET_EXTREME}, + ] + with lzma.LZMAFile("file.xz", "w", filters=my_filters) as f: + f.write(b"blah blah blah") diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 98c5b33..62c0f34 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -184,6 +184,19 @@ Power and logarithmic functions result is calculated in a way which is accurate for *x* near zero. +.. function:: log2(x) + + Return the base-2 logarithm of *x*. This is usually more accurate than + ``log(x, 2)``. + + .. versionadded:: 3.3 + + .. seealso:: + + :meth:`int.bit_length` returns the number of bits necessary to represent + an integer in binary, excluding the sign and leading zeros. + + .. function:: log10(x) Return the base-10 logarithm of *x*. This is usually more accurate diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index 5f0f004..1a19a7e 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -196,12 +196,16 @@ To map anonymous memory, -1 should be passed as the fileno along with the length move will raise a :exc:`TypeError` exception. - .. method:: read(num) + .. method:: read([n]) - Return a :class:`bytes` containing up to *num* bytes starting from the - current file position; the file position is updated to point after the - bytes that were returned. + Return a :class:`bytes` containing up to *n* bytes starting from the + current file position. If the argument is omitted, *None* or negative, + return all bytes from the current file position to the end of the + mapping. The file position is updated to point after the bytes that were + returned. + .. versionchanged:: 3.3 + Argument can be omitted or *None*. .. method:: read_byte() diff --git a/Doc/library/msvcrt.rst b/Doc/library/msvcrt.rst index 889a0c5..9d23720 100644 --- a/Doc/library/msvcrt.rst +++ b/Doc/library/msvcrt.rst @@ -20,6 +20,11 @@ api. The normal API deals only with ASCII characters and is of limited use for internationalized applications. The wide char API should be used where ever possible +.. versionchanged:: 3.3 + Operations in this module now raise :exc:`OSError` where :exc:`IOError` + was raised. + + .. _msvcrt-files: File Operations @@ -29,7 +34,7 @@ File Operations .. function:: locking(fd, mode, nbytes) Lock part of a file based on file descriptor *fd* from the C runtime. Raises - :exc:`IOError` on failure. The locked region of the file extends from the + :exc:`OSError` on failure. The locked region of the file extends from the current file position for *nbytes* bytes, and may continue beyond the end of the file. *mode* must be one of the :const:`LK_\*` constants listed below. Multiple regions in a file may be locked at the same time, but may not overlap. Adjacent @@ -41,13 +46,13 @@ File Operations Locks the specified bytes. If the bytes cannot be locked, the program immediately tries again after 1 second. If, after 10 attempts, the bytes cannot - be locked, :exc:`IOError` is raised. + be locked, :exc:`OSError` is raised. .. data:: LK_NBLCK LK_NBRLCK - Locks the specified bytes. If the bytes cannot be locked, :exc:`IOError` is + Locks the specified bytes. If the bytes cannot be locked, :exc:`OSError` is raised. @@ -73,7 +78,7 @@ File Operations .. function:: get_osfhandle(fd) - Return the file handle for the file descriptor *fd*. Raises :exc:`IOError` if + Return the file handle for the file descriptor *fd*. Raises :exc:`OSError` if *fd* is not recognized. @@ -144,4 +149,4 @@ Other Functions .. function:: heapmin() Force the :c:func:`malloc` heap to clean itself up and return unused blocks to - the operating system. On failure, this raises :exc:`IOError`. + the operating system. On failure, this raises :exc:`OSError`. diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index a9512f3..722f00c 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -120,9 +120,7 @@ processes: print(q.get()) # prints "[42, None, 'hello']" p.join() - Queues are thread and process safe, but note that they must never - be instantiated as a side effect of importing a module: this can lead - to a deadlock! (see :ref:`threaded-imports`) + Queues are thread and process safe. **Pipes** @@ -297,7 +295,7 @@ The :mod:`multiprocessing` package mostly replicates the API of the :class:`Process` and exceptions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. class:: Process([group[, target[, name[, args[, kwargs]]]]]) +.. class:: Process([group[, target[, name[, args[, kwargs]]]]], *, daemon=None) Process objects represent activity that is run in a separate process. The :class:`Process` class has equivalents of all the methods of @@ -312,13 +310,19 @@ The :mod:`multiprocessing` package mostly replicates the API of the :sub:`1`,N\ :sub:`2`,...,N\ :sub:`k` is a sequence of integers whose length is determined by the *generation* of the process. *args* is the argument tuple for the target invocation. *kwargs* is a dictionary of keyword - arguments for the target invocation. By default, no arguments are passed to - *target*. + arguments for the target invocation. If provided, the keyword-only *daemon* argument + sets the process :attr:`daemon` flag to ``True`` or ``False``. If ``None`` + (the default), this flag will be inherited from the creating process. + + By default, no arguments are passed to *target*. If a subclass overrides the constructor, it must make sure it invokes the base class constructor (:meth:`Process.__init__`) before doing anything else to the process. + .. versionchanged:: 3.3 + Added the *daemon* argument. + .. method:: run() Method representing the process's activity. @@ -337,10 +341,9 @@ The :mod:`multiprocessing` package mostly replicates the API of the .. method:: join([timeout]) - Block the calling thread until the process whose :meth:`join` method is - called terminates or until the optional timeout occurs. - - If *timeout* is ``None`` then there is no timeout. + If the optional argument *timeout* is ``None`` (the default), the method + blocks until the process whose :meth:`join` method is called terminates. + If *timeout* is a positive number, it blocks at most *timeout* seconds. A process can be joined many times. @@ -405,6 +408,21 @@ The :mod:`multiprocessing` package mostly replicates the API of the See :ref:`multiprocessing-auth-keys`. + .. attribute:: sentinel + + A numeric handle of a system object which will become "ready" when + the process ends. + + You can use this value if you want to wait on several events at + once using :func:`multiprocessing.connection.wait`. Otherwise + calling :meth:`join()` is simpler. + + On Windows, this is an OS handle usable with the ``WaitForSingleObject`` + and ``WaitForMultipleObjects`` family of API calls. On Unix, this is + a file descriptor usable with primitives from the :mod:`select` module. + + .. versionadded:: 3.3 + .. method:: terminate() Terminate the process. On Unix this is done using the ``SIGTERM`` signal; @@ -464,7 +482,7 @@ primitives like locks. For passing messages one can use :func:`Pipe` (for a connection between two processes) or a queue (which allows multiple producers and consumers). -The :class:`Queue`, :class:`multiprocessing.queues.SimpleQueue` and :class:`JoinableQueue` types are multi-producer, +The :class:`Queue`, :class:`SimpleQueue` and :class:`JoinableQueue` types are multi-producer, multi-consumer FIFO queues modelled on the :class:`queue.Queue` class in the standard library. They differ in that :class:`Queue` lacks the :meth:`~queue.Queue.task_done` and :meth:`~queue.Queue.join` methods introduced @@ -610,7 +628,7 @@ For an example of the usage of queues for interprocess communication see exits -- see :meth:`join_thread`. -.. class:: multiprocessing.queues.SimpleQueue() +.. class:: SimpleQueue() It is a simplified :class:`Queue` type, very close to a locked :class:`Pipe`. @@ -766,6 +784,9 @@ Connection objects are usually created using :func:`Pipe` -- see also *timeout* is a number then this specifies the maximum time in seconds to block. If *timeout* is ``None`` then an infinite timeout is used. + Note that multiple connection objects may be polled at once by + using :func:`multiprocessing.connection.wait`. + .. method:: send_bytes(buffer[, offset[, size]]) Send byte data from an object supporting the buffer interface as a @@ -784,9 +805,14 @@ Connection objects are usually created using :func:`Pipe` -- see also to receive and the other end has closed. If *maxlength* is specified and the message is longer than *maxlength* - then :exc:`IOError` is raised and the connection will no longer be + then :exc:`OSError` is raised and the connection will no longer be readable. + .. versionchanged:: 3.3 + This function used to raise a :exc:`IOError`, which is now an + alias of :exc:`OSError`. + + .. method:: recv_bytes_into(buffer[, offset]) Read into *buffer* a complete message of byte data sent from the other end @@ -804,6 +830,10 @@ Connection objects are usually created using :func:`Pipe` -- see also raised and the complete message is available as ``e.args[0]`` where ``e`` is the exception instance. + .. versionchanged:: 3.3 + Connection objects themselves can now be transferred between processes + using :meth:`Connection.send` and :meth:`Connection.recv`. + For example: @@ -869,6 +899,9 @@ object -- see :ref:`multiprocessing-managers`. If *lock* is specified then it should be a :class:`Lock` or :class:`RLock` object from :mod:`multiprocessing`. + .. versionchanged:: 3.3 + The :meth:`wait_for` method was added. + .. class:: Event() A clone of :class:`threading.Event`. @@ -893,6 +926,12 @@ object -- see :ref:`multiprocessing-managers`. .. note:: + The :meth:`acquire` and :meth:`wait` methods of each of these types + treat negative timeouts as zero timeouts. This differs from + :mod:`threading` where, since version 3.2, the equivalent + :meth:`acquire` methods treat negative timeouts as infinite + timeouts. + On Mac OS X, ``sem_timedwait`` is unsupported, so calling ``acquire()`` with a timeout will emulate that function's behavior using a sleeping loop. @@ -1253,6 +1292,9 @@ their parent process exits. The manager classes are defined in the If *lock* is supplied then it should be a proxy for a :class:`threading.Lock` or :class:`threading.RLock` object. + .. versionchanged:: 3.3 + The :meth:`wait_for` method was added. + .. method:: Event() Create a shared :class:`threading.Event` object and return a proxy for it. @@ -1657,6 +1699,24 @@ with the :class:`Pool` class. returned iterator should be considered arbitrary. (Only when there is only one worker process is the order guaranteed to be "correct".) + .. method:: starmap(func, iterable[, chunksize]) + + Like :meth:`map` except that the elements of the `iterable` are expected + to be iterables that are unpacked as arguments. + + Hence an `iterable` of `[(1,2), (3, 4)]` results in `[func(1,2), + func(3,4)]`. + + .. versionadded:: 3.3 + + .. method:: starmap_async(func, iterable[, chunksize[, callback[, error_back]]]) + + A combination of :meth:`starmap` and :meth:`map_async` that iterates over + `iterable` of iterables and calls `func` with the iterables unpacked. + Returns a result object. + + .. versionadded:: 3.3 + .. method:: close() Prevents any more tasks from being submitted to the pool. Once all the @@ -1737,8 +1797,9 @@ Usually message passing between processes is done using queues or by using However, the :mod:`multiprocessing.connection` module allows some extra flexibility. It basically gives a high level message oriented API for dealing -with sockets or Windows named pipes, and also has support for *digest -authentication* using the :mod:`hmac` module. +with sockets or Windows named pipes. It also has support for *digest +authentication* using the :mod:`hmac` module, and for polling +multiple connections at the same time. .. function:: deliver_challenge(connection, authkey) @@ -1836,6 +1897,39 @@ authentication* using the :mod:`hmac` module. The address from which the last accepted connection came. If this is unavailable then it is ``None``. +.. function:: wait(object_list, timeout=None) + + Wait till an object in *object_list* is ready. Returns the list of + those objects in *object_list* which are ready. If *timeout* is a + float then the call blocks for at most that many seconds. If + *timeout* is ``None`` then it will block for an unlimited period. + A negative timeout is equivalent to a zero timeout. + + For both Unix and Windows, an object can appear in *object_list* if + it is + + * a readable :class:`~multiprocessing.Connection` object; + * a connected and readable :class:`socket.socket` object; or + * the :attr:`~multiprocessing.Process.sentinel` attribute of a + :class:`~multiprocessing.Process` object. + + A connection or socket object is ready when there is data available + to be read from it, or the other end has been closed. + + **Unix**: ``wait(object_list, timeout)`` almost equivalent + ``select.select(object_list, [], [], timeout)``. The difference is + that, if :func:`select.select` is interrupted by a signal, it can + raise :exc:`OSError` with an error number of ``EINTR``, whereas + :func:`wait` will not. + + **Windows**: An item in *object_list* must either be an integer + handle which is waitable (according to the definition used by the + documentation of the Win32 function ``WaitForMultipleObjects()``) + or it can be an object with a :meth:`fileno` method which returns a + socket handle or pipe handle. (Note that pipe handles and socket + handles are **not** waitable handles.) + + .. versionadded:: 3.3 The module defines two exceptions: @@ -1887,6 +1981,41 @@ server:: conn.close() +The following code uses :func:`~multiprocessing.connection.wait` to +wait for messages from multiple processes at once:: + + import time, random + from multiprocessing import Process, Pipe, current_process + from multiprocessing.connection import wait + + def foo(w): + for i in range(10): + w.send((i, current_process().name)) + w.close() + + if __name__ == '__main__': + readers = [] + + for i in range(4): + r, w = Pipe(duplex=False) + readers.append(r) + p = Process(target=foo, args=(w,)) + p.start() + # We close the writable end of the pipe now to be sure that + # p is the only process which owns a handle for it. This + # ensures that when p closes its handle for the writable end, + # wait() will promptly report the readable end as being ready. + w.close() + + while readers: + for r in wait(readers): + try: + msg = r.recv() + except EOFError: + readers.remove(r) + else: + print(msg) + .. _multiprocessing-address-formats: diff --git a/Doc/library/nntplib.rst b/Doc/library/nntplib.rst index ef507e1..62da72c 100644 --- a/Doc/library/nntplib.rst +++ b/Doc/library/nntplib.rst @@ -70,10 +70,23 @@ The module itself defines the following classes: connecting to an NNTP server on the local machine and intend to call reader-specific commands, such as ``group``. If you get unexpected :exc:`NNTPPermanentError`\ s, you might need to set *readermode*. + :class:`NNTP` class supports the :keyword:`with` statement to + unconditionally consume :exc:`socket.error` exceptions and to close the NNTP + connection when done. Here is a sample on how using it: + + >>> from nntplib import NNTP + >>> with NNTP('news.gmane.org') as n: + ... n.group('gmane.comp.python.committers') + ... + ('211 1755 1 1755 gmane.comp.python.committers', 1755, 1, 1755, 'gmane.comp.python.committers') + >>> + .. versionchanged:: 3.2 *usenetrc* is now False by default. + .. versionchanged:: 3.3 + Support for the :keyword:`with` statement was added. .. class:: NNTP_SSL(host, port=563, user=None, password=None, ssl_context=None, readermode=None, usenetrc=False, [timeout]) @@ -504,6 +517,9 @@ them have been superseded by newer commands in :rfc:`3977`. article with message ID *id*. Most of the time, this extension is not enabled by NNTP server administrators. + .. deprecated:: 3.3 + The XPATH extension is not actively used. + .. XXX deprecated: diff --git a/Doc/library/numeric.rst b/Doc/library/numeric.rst index ba22cb6..2732a84 100644 --- a/Doc/library/numeric.rst +++ b/Doc/library/numeric.rst @@ -8,9 +8,9 @@ Numeric and Mathematical Modules The modules described in this chapter provide numeric and math-related functions and data types. The :mod:`numbers` module defines an abstract hierarchy of numeric types. The :mod:`math` and :mod:`cmath` modules contain various -mathematical functions for floating-point and complex numbers. For users more -interested in decimal accuracy than in speed, the :mod:`decimal` module supports -exact representations of decimal numbers. +mathematical functions for floating-point and complex numbers. The :mod:`decimal` +module supports exact representations of decimal numbers, using arbitrary precision +arithmetic. The following modules are documented in this chapter: diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 3f43338..8cc43c5 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -221,6 +221,17 @@ process and user. Availability: Unix. +.. function:: getgrouplist(user, group) + + Return list of group ids that *user* belongs to. If *group* is not in the + list, it is included; typically, *group* is specified as the group ID + field from the password record for *user*. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: getgroups() Return list of supplemental group ids associated with the current process. @@ -302,6 +313,22 @@ process and user. .. versionchanged:: 3.2 Added support for Windows. +.. function:: getpriority(which, who) + + .. index:: single: process; scheduling priority + + Get program scheduling priority. The value *which* is one of + :const:`PRIO_PROCESS`, :const:`PRIO_PGRP`, or :const:`PRIO_USER`, and *who* + is interpreted relative to *which* (a process identifier for + :const:`PRIO_PROCESS`, process group identifier for :const:`PRIO_PGRP`, and a + user ID for :const:`PRIO_USER`). A zero value for *who* denotes + (respectively) the calling process, the process group of the calling process, + or the real user ID of the calling process. + + Availability: Unix + + .. versionadded:: 3.3 + .. function:: getresuid() Return a tuple (ruid, euid, suid) denoting the current process's @@ -352,6 +379,15 @@ process and user. .. versionadded:: 3.2 +.. data:: PRIO_PROCESS + PRIO_PGRP + PRIO_USER + + Parameters for :func:`getpriority` and :func:`setpriority` functions. + + Availability: Unix. + + .. versionadded:: 3.3 .. function:: putenv(key, value) @@ -425,6 +461,25 @@ process and user. Availability: Unix. +.. function:: setpriority(which, who, priority) + + .. index:: single: process; scheduling priority + + Set program scheduling priority. The value *which* is one of + :const:`PRIO_PROCESS`, :const:`PRIO_PGRP`, or :const:`PRIO_USER`, and *who* + is interpreted relative to *which* (a process identifier for + :const:`PRIO_PROCESS`, process group identifier for :const:`PRIO_PGRP`, and a + user ID for :const:`PRIO_USER`). A zero value for *who* denotes + (respectively) the calling process, the process group of the calling process, + or the real user ID of the calling process. + *priority* is a value in the range -20 to 19. The default priority is 0; + lower priorities cause more favorable scheduling. + + Availability: Unix + + .. versionadded:: 3.3 + + .. function:: setregid(rgid, egid) Set the current process's real and effective group ids. @@ -554,7 +609,8 @@ These functions create new :term:`file objects <file object>`. (See also :func:` the built-in :func:`open` function. When specified, the *mode* argument must start with one of the letters - ``'r'``, ``'w'``, or ``'a'``, otherwise a :exc:`ValueError` is raised. + ``'r'``, ``'w'``, ``'x'`` or ``'a'``, otherwise a :exc:`ValueError` is + raised. On Unix, when the *mode* argument starts with ``'a'``, the *O_APPEND* flag is set on the file descriptor (which the :c:func:`fdopen` implementation already @@ -562,6 +618,8 @@ These functions create new :term:`file objects <file object>`. (See also :func:` Availability: Unix, Windows. + .. versionchanged:: 3.3 + The ``'x'`` mode was added. .. _os-fd-ops: @@ -582,6 +640,21 @@ associated with a :term:`file object` when required. Note that using the file descriptor directly will bypass the file object methods, ignoring aspects such as internal buffering of data. +.. data:: AT_SYMLINK_NOFOLLOW + AT_EACCESS + AT_FDCWD + AT_REMOVEDIR + AT_SYMLINK_FOLLOW + UTIME_NOW + UTIME_OMIT + + These parameters are used as flags to the \*at family of functions. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: close(fd) Close file descriptor *fd*. @@ -630,6 +703,19 @@ as internal buffering of data. Availability: Unix, Windows. +.. function:: faccessat(dirfd, path, mode, flags=0) + + Like :func:`access` but if *path* is relative, it is taken as relative to *dirfd*. + *flags* is optional and can be constructed by ORing together zero or more + of these values: :data:`AT_SYMLINK_NOFOLLOW`, :data:`AT_EACCESS`. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: fchmod(fd, mode) Change the mode of the file given by *fd* to the numeric *mode*. See the docs @@ -638,6 +724,18 @@ as internal buffering of data. Availability: Unix. +.. function:: fchmodat(dirfd, path, mode, flags=0) + + Like :func:`chmod` but if *path* is relative, it is taken as relative to *dirfd*. + *flags* is optional and may be 0 or :data:`AT_SYMLINK_NOFOLLOW`. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: fchown(fd, uid, gid) Change the owner and group id of the file given by *fd* to the numeric *uid* @@ -646,6 +744,18 @@ as internal buffering of data. Availability: Unix. +.. function:: fchownat(dirfd, path, uid, gid, flags=0) + + Like :func:`chown` but if *path* is relative, it is taken as relative to *dirfd*. + *flags* is optional and may be 0 or :data:`AT_SYMLINK_NOFOLLOW`. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: fdatasync(fd) Force write of file with filedescriptor *fd* to disk. Does not force update of @@ -657,6 +767,47 @@ as internal buffering of data. This function is not available on MacOS. +.. function:: fgetxattr(fd, attr) + + This works exactly like :func:`getxattr` but operates on a file descriptor, + *fd*, instead of a path. + + Availability: Linux + + .. versionadded:: 3.3 + + +.. function:: flistxattr(fd) + + This is exactly like :func:`listxattr` but operates on a file descriptor, + *fd*, instead of a path. + + Availability: Linux + + .. versionadded:: 3.3 + + +.. function:: flistdir(fd) + + Like :func:`listdir`, but uses a file descriptor instead and always returns + strings. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: fexecve(fd, args, env) + + Execute the program specified by a file descriptor *fd* with arguments given + by *args* and environment given by *env*, replacing the current process. + *args* and *env* are given as in :func:`execve`. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: fpathconf(fd, name) Return system configuration information relevant to an open file. *name* @@ -681,6 +832,17 @@ as internal buffering of data. Availability: Unix, Windows. +.. function:: fstatat(dirfd, path, flags=0) + + Like :func:`stat` but if *path* is relative, it is taken as relative to *dirfd*. + *flags* is optional and may be 0 or :data:`AT_SYMLINK_NOFOLLOW`. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + .. function:: fstatvfs(fd) @@ -710,6 +872,78 @@ as internal buffering of data. Availability: Unix. +.. function:: fremovexattr(fd, attr) + + This works exactly like :func:`removexattr` but operates on a file + descriptor, *fd*, instead of a path. + + Availability: Linux + + .. versionadded:: 3.3 + + +.. function:: fsetxattr(fd, attr, value, flags=0) + + This works exactly like :func:`setxattr` but on a file descriptor, *fd*, + instead of a path. + + + Availability: Linux + + .. versionadded:: 3.3 + + +.. function:: futimesat(dirfd, path[, times]) + + Like :func:`utime` but if *path* is relative, it is taken as relative to *dirfd*. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. *times* must be a + 2-tuple of numbers, of the form ``(atime, mtime)``, or None. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: futimens(fd[, atimes, mtimes]) + + Updates the timestamps of a file specified by the file descriptor *fd*, with + nanosecond precision. + If no second argument is given, set *atime* and *mtime* to the current time. + *atimes* and *mtimes* must be 2-tuples of numbers, of the form + ``(atime_sec, atime_nsec)`` and ``(mtime_sec, mtime_nsec)`` respectively, + or ``None``. + If *atime_nsec* or *mtime_nsec* is specified as :data:`UTIME_NOW`, the corresponding + timestamp is updated to the current time. + If *atime_nsec* or *mtime_nsec* is specified as :data:`UTIME_OMIT`, the corresponding + timestamp is not updated. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. data:: UTIME_NOW + UTIME_OMIT + + Flags used with :func:`futimens` to specify that the timestamp must be + updated either to the current time or not updated at all. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: futimes(fd[, times, *, ns=times]) + + Set the access and modified time of the file specified by the file + descriptor *fd* to the given values. See :func:`utime` for proper + use of the *times* and *ns* arguments. + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: isatty(fd) Return ``True`` if the file descriptor *fd* is open and connected to a @@ -718,6 +952,44 @@ as internal buffering of data. Availability: Unix. +.. function:: linkat(srcfd, srcpath, dstfd, dstpath, flags=0) + + Like :func:`link` but if *srcpath* is relative, it is taken as relative to *srcfd* + and if *dstpath* is relative, it is taken as relative to *dstfd*. + *flags* is optional and may be 0 or :data:`AT_SYMLINK_FOLLOW`. + If *srcpath* is relative and *srcfd* is the special value :data:`AT_FDCWD`, then + *srcpath* is interpreted relative to the current working directory. This + also applies for *dstpath*. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: lockf(fd, cmd, len) + + Apply, test or remove a POSIX lock on an open file descriptor. + *fd* is an open file descriptor. + *cmd* specifies the command to use - one of :data:`F_LOCK`, :data:`F_TLOCK`, + :data:`F_ULOCK` or :data:`F_TEST`. + *len* specifies the section of the file to lock. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. data:: F_LOCK + F_TLOCK + F_ULOCK + F_TEST + + Flags that specify what action :func:`lockf` will take. + + Availability: Unix. + + .. versionadded:: 3.3 + .. function:: lseek(fd, pos, how) Set the current position of file descriptor *fd* to position *pos*, modified @@ -737,6 +1009,39 @@ as internal buffering of data. respectively. Availability: Windows, Unix. +.. function:: mkdirat(dirfd, path, mode=0o777) + + Like :func:`mkdir` but if *path* is relative, it is taken as relative to *dirfd*. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: mkfifoat(dirfd, path, mode=0o666) + + Like :func:`mkfifo` but if *path* is relative, it is taken as relative to *dirfd*. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: mknodat(dirfd, path, mode=0o600, device=0) + + Like :func:`mknod` but if *path* is relative, it is taken as relative to *dirfd*. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: open(file, flags[, mode]) Open the file *file* and set various flags according to *flags* and possibly @@ -759,6 +1064,17 @@ as internal buffering of data. wrap a file descriptor in a file object, use :func:`fdopen`. +.. function:: openat(dirfd, path, flags, mode=0o777) + + Like :func:`open` but if *path* is relative, it is taken as relative to *dirfd*. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: openpty() .. index:: module: pty @@ -778,6 +1094,79 @@ as internal buffering of data. Availability: Unix, Windows. +.. function:: pipe2(flags) + + Create a pipe with *flags* set atomically. + *flags* can be constructed by ORing together one or more of these values: + :data:`O_NONBLOCK`, :data:`O_CLOEXEC`. + Return a pair of file descriptors ``(r, w)`` usable for reading and writing, + respectively. + + Availability: some flavors of Unix. + + .. versionadded:: 3.3 + + +.. function:: posix_fallocate(fd, offset, len) + + Ensures that enough disk space is allocated for the file specified by *fd* + starting from *offset* and continuing for *len* bytes. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: posix_fadvise(fd, offset, len, advice) + + Announces an intention to access data in a specific pattern thus allowing + the kernel to make optimizations. + The advice applies to the region of the file specified by *fd* starting at + *offset* and continuing for *len* bytes. + *advice* is one of :data:`POSIX_FADV_NORMAL`, :data:`POSIX_FADV_SEQUENTIAL`, + :data:`POSIX_FADV_RANDOM`, :data:`POSIX_FADV_NOREUSE`, + :data:`POSIX_FADV_WILLNEED` or :data:`POSIX_FADV_DONTNEED`. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. data:: POSIX_FADV_NORMAL + POSIX_FADV_SEQUENTIAL + POSIX_FADV_RANDOM + POSIX_FADV_NOREUSE + POSIX_FADV_WILLNEED + POSIX_FADV_DONTNEED + + Flags that can be used in *advice* in :func:`posix_fadvise` that specify + the access pattern that is likely to be used. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: pread(fd, buffersize, offset) + + Read from a file descriptor, *fd*, at a position of *offset*. It will read up + to *buffersize* number of bytes. The file offset remains unchanged. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: pwrite(fd, string, offset) + + Write *string* to a file descriptor, *fd*, from *offset*, leaving the file + offset unchanged. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: read(fd, n) Read at most *n* bytes from file descriptor *fd*. Return a bytestring containing the @@ -795,6 +1184,93 @@ as internal buffering of data. :meth:`~file.readline` methods. +.. function:: sendfile(out, in, offset, nbytes) + sendfile(out, in, offset, nbytes, headers=None, trailers=None, flags=0) + + Copy *nbytes* bytes from file descriptor *in* to file descriptor *out* + starting at *offset*. + Return the number of bytes sent. When EOF is reached return 0. + + The first function notation is supported by all platforms that define + :func:`sendfile`. + + On Linux, if *offset* is given as ``None``, the bytes are read from the + current position of *in* and the position of *in* is updated. + + The second case may be used on Mac OS X and FreeBSD where *headers* and + *trailers* are arbitrary sequences of buffers that are written before and + after the data from *in* is written. It returns the same as the first case. + + On Mac OS X and FreeBSD, a value of 0 for *nbytes* specifies to send until + the end of *in* is reached. + + On Solaris, *out* may be the file descriptor of a regular file or the file + descriptor of a socket. On all other platforms, *out* must be the file + descriptor of an open socket. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. data:: SF_NODISKIO + SF_MNOWAIT + SF_SYNC + + Parameters to the :func:`sendfile` function, if the implementation supports + them. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: readlinkat(dirfd, path) + + Like :func:`readlink` but if *path* is relative, it is taken as relative to *dirfd*. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: renameat(olddirfd, oldpath, newdirfd, newpath) + + Like :func:`rename` but if *oldpath* is relative, it is taken as relative to + *olddirfd* and if *newpath* is relative, it is taken as relative to *newdirfd*. + If *oldpath* is relative and *olddirfd* is the special value :data:`AT_FDCWD`, then + *oldpath* is interpreted relative to the current working directory. This + also applies for *newpath*. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: symlinkat(src, dstfd, dst) + + Like :func:`symlink` but if *dst* is relative, it is taken as relative to *dstfd*. + If *dst* is relative and *dstfd* is the special value :data:`AT_FDCWD`, then *dst* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: readv(fd, buffers) + + Read from a file descriptor into a number of writable buffers. *buffers* is + an arbitrary sequence of writable buffers. Returns the total number of bytes + read. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: tcgetpgrp(fd) Return the process group associated with the terminal given by *fd* (an open @@ -820,6 +1296,38 @@ as internal buffering of data. Availability: Unix. +.. function:: unlinkat(dirfd, path, flags=0) + + Like :func:`unlink` but if *path* is relative, it is taken as relative to *dirfd*. + *flags* is optional and may be 0 or :data:`AT_REMOVEDIR`. If :data:`AT_REMOVEDIR` is + specified, :func:`unlinkat` behaves like :func:`rmdir`. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: utimensat(dirfd, path[, atime=(atime_sec, atime_nsec), mtime=(mtime_sec, mtime_nsec), flags=0]) + + Updates the timestamps of a file with nanosecond precision. + The *atime* and *mtime* tuples default to ``None``, which sets those + values to the current time. + If *atime_nsec* or *mtime_nsec* is specified as :data:`UTIME_NOW`, the corresponding + timestamp is updated to the current time. + If *atime_nsec* or *mtime_nsec* is specified as :data:`UTIME_OMIT`, the corresponding + timestamp is not updated. + If *path* is relative, it is taken as relative to *dirfd*. + *flags* is optional and may be 0 (the default) or :data:`AT_SYMLINK_NOFOLLOW`. + If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path* + is interpreted relative to the current working directory. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: write(fd, str) Write the bytestring in *str* to file descriptor *fd*. Return the number of @@ -836,6 +1344,17 @@ as internal buffering of data. :meth:`~file.write` method. +.. function:: writev(fd, buffers) + + Write the contents of *buffers* to file descriptor *fd*, where *buffers* + is an arbitrary sequence of buffers. + Returns the total number of bytes written. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. _open-constants: ``open()`` flag constants @@ -867,9 +1386,12 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window O_NOCTTY O_SHLOCK O_EXLOCK + O_CLOEXEC These constants are only available on Unix. + .. versionchanged:: 3.3 + Add :data:`O_CLOEXEC` constant. .. data:: O_BINARY O_NOINHERIT @@ -892,6 +1414,56 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window the C library. +.. data:: RTLD_LAZY + RTLD_NOW + RTLD_GLOBAL + RTLD_LOCAL + RTLD_NODELETE + RTLD_NOLOAD + RTLD_DEEPBIND + + See the Unix manual page :manpage:`dlopen(3)`. + + .. versionadded:: 3.3 + + +.. _terminal-size: + +Querying the size of a terminal +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. versionadded:: 3.3 + +.. function:: get_terminal_size(fd=STDOUT_FILENO) + + Return the size of the terminal window as ``(columns, lines)``, + tuple of type :class:`terminal_size`. + + The optional argument ``fd`` (default ``STDOUT_FILENO``, or standard + output) specifies which file descriptor should be queried. + + If the file descriptor is not connected to a terminal, an :exc:`OSError` + is thrown. + + :func:`shutil.get_terminal_size` is the high-level function which + should normally be used, ``os.get_terminal_size`` is the low-level + implementation. + + Availability: Unix, Windows. + +.. class:: terminal_size(tuple) + + A tuple of ``(columns, lines)`` for holding terminal window size. + + .. attribute:: columns + + Width of the terminal window in characters. + + .. attribute:: lines + + Height of the terminal window in characters. + + .. _os-file-dir: Files and Directories @@ -927,11 +1499,8 @@ Files and Directories try: fp = open("myfile") - except IOError as e: - if e.errno == errno.EACCES: - return "some default data" - # Not a permission error. - raise + except PermissionError: + return "some default data" else: with fp: return fp.read() @@ -1067,9 +1636,23 @@ Files and Directories Change the owner and group id of *path* to the numeric *uid* and *gid*. To leave one of the ids unchanged, set it to -1. + See :func:`shutil.chown` for a higher-level function that accepts names in + addition to numeric ids. + Availability: Unix. +.. function:: getxattr(path, attr) + + Return the value of the extended filesystem attribute *attr* for + *path*. *attr* can be bytes or str. If it is str, it is encoded with the + filesystem encoding. + + Availability: Linux + + .. versionadded:: 3.3 + + .. function:: lchflags(path, flags) Set the flags of *path* to the numeric *flags*, like :func:`chflags`, but do not @@ -1095,6 +1678,15 @@ Files and Directories Availability: Unix. +.. function:: lgetxattr(path, attr) + + This works exactly like :func:`getxattr` but doesn't follow symlinks. + + Availability: Linux + + .. versionadded:: 3.3 + + .. function:: link(source, link_name) Create a hard link pointing to *source* named *link_name*. @@ -1119,6 +1711,44 @@ Files and Directories .. versionchanged:: 3.2 The *path* parameter became optional. + +.. function:: listxattr(path) + + Return a list of the extended filesystem attributes on *path*. Attributes are + returned as string decoded with the filesystem encoding. + + Availability: Linux + + .. versionadded:: 3.3 + + +.. function:: llistxattr(path) + + This works exactly like :func:`listxattr` but doesn't follow symlinks. + + Availability: Linux + + .. versionadded:: 3.3 + + +.. function:: lremovexattr(path, attr) + + This works exactly like :func:`removexattr` but doesn't follow symlinks. + + Availability: Linux + + .. versionadded:: 3.3 + + +.. function:: lsetxattr(path, attr, value, flags=0) + + This works exactly like :func:`setxattr` but doesn't follow symlinks. + + Availability: Linux + + .. versionadded:: 3.3 + + .. function:: lstat(path) Perform the equivalent of an :c:func:`lstat` system call on the given path. @@ -1130,6 +1760,17 @@ Files and Directories Added support for Windows 6.0 (Vista) symbolic links. +.. function:: lutimes(path[, times, *, ns=times]) + + Like :func:`utime`, but if *path* is a symbolic link, it is not + dereferenced. See :func:`utime` for proper use of the + *times* and *ns* arguments. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: mkfifo(path[, mode]) Create a FIFO (a named pipe) named *path* with numeric mode *mode*. The @@ -1281,6 +1922,17 @@ Files and Directories successfully removed. +.. function:: removexattr(path, attr) + + Removes the extended filesystem attribute *attr* from *path*. *attr* should + be bytes or str. If it is a string, it is encoded with the filesystem + encoding. + + Availability: Linux + + .. versionadded:: 3.3 + + .. function:: rename(src, dst) Rename the file or directory *src* to *dst*. If *dst* is a directory, @@ -1289,8 +1941,9 @@ Files and Directories Unix flavors if *src* and *dst* are on different filesystems. If successful, the renaming will be an atomic operation (this is a POSIX requirement). On Windows, if *dst* already exists, :exc:`OSError` will be raised even if it is a - file; there may be no way to implement an atomic rename when *dst* names an - existing file. + file. + + If you want cross-platform overwriting of the destination, use :func:`replace`. Availability: Unix, Windows. @@ -1308,6 +1961,19 @@ Files and Directories permissions needed to remove the leaf directory or file. +.. function:: replace(src, dst) + + Rename the file or directory *src* to *dst*. If *dst* is a directory, + :exc:`OSError` will be raised. If *dst* exists and is a file, it will + be replaced silently if the user has permission. The operation may fail + if *src* and *dst* are on different filesystems. If successful, + the renaming will be an atomic operation (this is a POSIX requirement). + + Availability: Unix, Windows + + .. versionadded:: 3.3 + + .. function:: rmdir(path) Remove (delete) the directory *path*. Only works when the directory is @@ -1317,13 +1983,51 @@ Files and Directories Availability: Unix, Windows. +.. data:: XATTR_SIZE_MAX + + The maximum size the value of an extended attribute can be. Currently, this + is 64 kilobytes on Linux. + + +.. data:: XATTR_CREATE + + This is a possible value for the flags argument in :func:`setxattr`. It + indicates the operation must create an attribute. + + +.. data:: XATTR_REPLACE + + This is a possible value for the flags argument in :func:`setxattr`. It + indicates the operation must replace an existing attribute. + + +.. function:: setxattr(path, attr, value, flags=0) + + Set the extended filesystem attribute *attr* on *path* to *value*. *attr* + must be a bytes or str with no embedded NULs. If it is str, it is encoded + with the filesystem encoding. *flags* may be :data:`XATTR_REPLACE` or + :data:`XATTR_CREATE`. If :data:`XATTR_REPLACE` is given and the attribute + does not exist, ``EEXISTS`` will be raised. If :data:`XATTR_CREATE` is given + and the attribute already exists, the attribute will not be created and + ``ENODATA`` will be raised. + + Availability: Linux + + .. note:: + + A bug in Linux kernel versions less than 2.6.39 caused the flags argument + to be ignored on some filesystems. + + .. versionadded:: 3.3 + + .. function:: stat(path) Perform the equivalent of a :c:func:`stat` system call on the given path. (This function follows symlinks; to stat a symlink use :func:`lstat`.) - The return value is an object whose attributes correspond to the members - of the :c:type:`stat` structure, namely: + The return value is an object whose attributes correspond roughly + to the members of the :c:type:`stat` structure, namely: * :attr:`st_mode` - protection bits, * :attr:`st_ino` - inode number, @@ -1332,10 +2036,18 @@ Files and Directories * :attr:`st_uid` - user id of owner, * :attr:`st_gid` - group id of owner, * :attr:`st_size` - size of file, in bytes, - * :attr:`st_atime` - time of most recent access, - * :attr:`st_mtime` - time of most recent content modification, - * :attr:`st_ctime` - platform dependent; time of most recent metadata change on - Unix, or the time of creation on Windows) + * :attr:`st_atime` - time of most recent access expressed in seconds, + * :attr:`st_mtime` - time of most recent content modification + expressed in seconds, + * :attr:`st_ctime` - platform dependent; time of most recent metadata + change on Unix, or the time of creation on Windows, expressed in seconds + * :attr:`st_atime_ns` - time of most recent access + expressed in nanoseconds as an integer, + * :attr:`st_mtime_ns` - time of most recent content modification + expressed in nanoseconds as an integer, + * :attr:`st_ctime_ns` - platform dependent; time of most recent metadata + change on Unix, or the time of creation on Windows, + expressed in nanoseconds as an integer On some Unix systems (such as Linux), the following attributes may also be available: @@ -1365,6 +2077,14 @@ Files and Directories or FAT32 file systems, :attr:`st_mtime` has 2-second resolution, and :attr:`st_atime` has only 1-day resolution. See your operating system documentation for details. + Similarly, although :attr:`st_atime_ns`, :attr:`st_mtime_ns`, + and :attr:`st_ctime_ns` are always expressed in nanoseconds, many + systems do not provide nanosecond precision. On systems that do + provide nanosecond precision, the floating-point object used to + store :attr:`st_atime`, :attr:`st_mtime`, and :attr:`st_ctime` + cannot preserve all of it, and as such will be slightly inexact. + If you need the exact timestamps you should always use + :attr:`st_atime_ns`, :attr:`st_mtime_ns`, and :attr:`st_ctime_ns`. For backward compatibility, the return value of :func:`~os.stat` is also accessible as a tuple of at least 10 integers giving the most important (and portable) @@ -1392,6 +2112,10 @@ Files and Directories Availability: Unix, Windows. + .. versionadded:: 3.3 + The :attr:`st_atime_ns`, :attr:`st_mtime_ns`, + and :attr:`st_ctime_ns` members. + .. function:: stat_float_times([newvalue]) @@ -1471,6 +2195,25 @@ Files and Directories Added support for Windows 6.0 (Vista) symbolic links. +.. function:: sync() + + Force write of everything to disk. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: truncate(path, length) + + Truncate the file corresponding to *path*, so that it is at most + *length* bytes in size. + + Availability: Unix. + + .. versionadded:: 3.3 + + .. function:: unlink(path) Remove (delete) the file *path*. This is the same function as @@ -1480,21 +2223,43 @@ Files and Directories Availability: Unix, Windows. -.. function:: utime(path, times) +.. function:: utime(path[, times, *, ns=(atime_ns, mtime_ns)]) + + Set the access and modified times of the file specified by *path*. + + :func:`utime` takes two optional parameters, *times* and *ns*. + These specify the times set on *path* and are used as follows: + + - If *ns* is specified, + it must be a 2-tuple of the form ``(atime_ns, mtime_ns)`` + where each member is an int expressing nanoseconds. + - If *times* is specified and is not ``None``, + it must be a 2-tuple of the form ``(atime, mtime)`` + where each member is an int or float expressing seconds. + - If *times* is specified as ``None``, + this is equivalent to specifying an ``(atime, mtime)`` + where both times are the current time. + (The effect is similar to running the Unix program + :program:`touch` on *path*.) + - If neither *ns* nor *times* is specified, this is + equivalent to specifying *times* as ``None``. - Set the access and modified times of the file specified by *path*. If *times* - is ``None``, then the file's access and modified times are set to the current - time. (The effect is similar to running the Unix program :program:`touch` on - the path.) Otherwise, *times* must be a 2-tuple of numbers, of the form - ``(atime, mtime)`` which is used to set the access and modified times, - respectively. Whether a directory can be given for *path* depends on whether - the operating system implements directories as files (for example, Windows - does not). Note that the exact times you set here may not be returned by a - subsequent :func:`~os.stat` call, depending on the resolution with which your - operating system records access and modification times; see :func:`~os.stat`. + Specifying both *times* and *ns* simultaneously is an error. + + Whether a directory can be given for *path* + depends on whether the operating system implements directories as files + (for example, Windows does not). Note that the exact times you set here may + not be returned by a subsequent :func:`~os.stat` call, depending on the + resolution with which your operating system records access and modification + times; see :func:`~os.stat`. The best way to preserve exact times is to + use the *st_atime_ns* and *st_mtime_ns* fields from the :func:`os.stat` + result object with the *ns* parameter to `utime`. Availability: Unix, Windows. + .. versionadded:: 3.3 + The :attr:`ns` keyword parameter. + .. function:: walk(top, topdown=True, onerror=None, followlinks=False) @@ -1579,6 +2344,57 @@ Files and Directories os.rmdir(os.path.join(root, name)) +.. function:: fwalk(top, topdown=True, onerror=None, followlinks=False) + + .. index:: + single: directory; walking + single: directory; traversal + + This behaves exactly like :func:`walk`, except that it yields a 4-tuple + ``(dirpath, dirnames, filenames, dirfd)``. + + *dirpath*, *dirnames* and *filenames* are identical to :func:`walk` output, + and *dirfd* is a file descriptor referring to the directory *dirpath*. + + .. note:: + + Since :func:`fwalk` yields file descriptors, those are only valid until + the next iteration step, so you should duplicate them (e.g. with + :func:`dup`) if you want to keep them longer. + + This example displays the number of bytes taken by non-directory files in each + directory under the starting directory, except that it doesn't look under any + CVS subdirectory:: + + import os + for root, dirs, files, rootfd in os.fwalk('python/Lib/email'): + print(root, "consumes", end="") + print(sum([os.fstatat(rootfd, name).st_size for name in files]), + end="") + print("bytes in", len(files), "non-directory files") + if 'CVS' in dirs: + dirs.remove('CVS') # don't visit CVS directories + + In the next example, walking the tree bottom-up is essential: + :func:`unlinkat` doesn't allow deleting a directory before the directory is + empty:: + + # Delete everything reachable from the directory named in "top", + # assuming there are no symbolic links. + # CAUTION: This is dangerous! For example, if top == '/', it + # could delete all your disk files. + import os + for root, dirs, files, rootfd in os.fwalk(top, topdown=False): + for name in files: + os.unlinkat(rootfd, name) + for name in dirs: + os.unlinkat(rootfd, name, os.AT_REMOVEDIR) + + Availability: Unix. + + .. versionadded:: 3.3 + + .. _os-process: Process Management @@ -1843,6 +2659,8 @@ written in Python, such as a mail server's external command delivery program. will be set to *sig*. The Windows version of :func:`kill` additionally takes process handles to be killed. + See also :func:`signal.pthread_kill`. + .. versionadded:: 3.2 Windows support. @@ -2053,6 +2871,58 @@ written in Python, such as a mail server's external command delivery program. Availability: Unix. +.. function:: waitid(idtype, id, options) + + Wait for the completion of one or more child processes. + *idtype* can be :data:`P_PID`, :data:`P_PGID` or :data:`P_ALL`. + *id* specifies the pid to wait on. + *options* is constructed from the ORing of one or more of :data:`WEXITED`, + :data:`WSTOPPED` or :data:`WCONTINUED` and additionally may be ORed with + :data:`WNOHANG` or :data:`WNOWAIT`. The return value is an object + representing the data contained in the :c:type:`siginfo_t` structure, namely: + :attr:`si_pid`, :attr:`si_uid`, :attr:`si_signo`, :attr:`si_status`, + :attr:`si_code` or ``None`` if :data:`WNOHANG` is specified and there are no + children in a waitable state. + + Availability: Unix. + + .. versionadded:: 3.3 + +.. data:: P_PID + P_PGID + P_ALL + + These are the possible values for *idtype* in :func:`waitid`. They affect + how *id* is interpreted. + + Availability: Unix. + + .. versionadded:: 3.3 + +.. data:: WEXITED + WSTOPPED + WNOWAIT + + Flags that can be used in *options* in :func:`waitid` that specify what + child signal to wait for. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. data:: CLD_EXITED + CLD_DUMPED + CLD_TRAPPED + CLD_CONTINUED + + These are the possible values for :attr:`si_code` in the result returned by + :func:`waitid`. + + Availability: Unix. + + .. versionadded:: 3.3 + .. function:: waitpid(pid, options) @@ -2194,6 +3064,155 @@ used to determine the disposition of a process. Availability: Unix. +Interface to the scheduler +-------------------------- + +These functions control how a process is allocated CPU time by the operating +system. They are only available on some Unix platforms. For more detailed +information, consult your Unix manpages. + +.. versionadded:: 3.3 + +The following scheduling policies are exposed if they are a supported by the +operating system. + +.. data:: SCHED_OTHER + + The default scheduling policy. + +.. data:: SCHED_BATCH + + Scheduling policy for CPU-intensive processes that tries to preserve + interactivity on the rest of the computer. + +.. data:: SCHED_IDLE + + Scheduling policy for extremely low priority background tasks. + +.. data:: SCHED_SPORADIC + + Scheduling policy for sporadic server programs. + +.. data:: SCHED_FIFO + + A First In First Out scheduling policy. + +.. data:: SCHED_RR + + A round-robin scheduling policy. + +.. data:: SCHED_RESET_ON_FORK + + This flag can OR'ed with any other scheduling policy. When a process with + this flag set forks, its child's scheduling policy and priority are reset to + the default. + + +.. class:: sched_param(sched_priority) + + This class represents tunable scheduling parameters used in + :func:`sched_setparam`, :func:`sched_setscheduler`, and + :func:`sched_getparam`. It is immutable. + + At the moment, there is only one possible parameter: + + .. attribute:: sched_priority + + The scheduling priority for a scheduling policy. + + +.. function:: sched_get_priority_min(policy) + + Get the minimum priority value for *policy*. *policy* is one of the + scheduling policy constants above. + + +.. function:: sched_get_priority_max(policy) + + Get the maximum priority value for *policy*. *policy* is one of the + scheduling policy constants above. + + +.. function:: sched_setscheduler(pid, policy, param) + + Set the scheduling policy for the process with PID *pid*. A *pid* of 0 means + the calling process. *policy* is one of the scheduling policy constants + above. *param* is a :class:`sched_param` instance. + + +.. function:: sched_getscheduler(pid) + + Return the scheduling policy for the process with PID *pid*. A *pid* of 0 + means the calling process. The result is one of the scheduling policy + constants above. + + +.. function:: sched_setparam(pid, param) + + Set a scheduling parameters for the process with PID *pid*. A *pid* of 0 means + the calling process. *param* is a :class:`sched_param` instance. + + +.. function:: sched_getparam(pid) + + Return the scheduling parameters as a :class:`sched_param` instance for the + process with PID *pid*. A *pid* of 0 means the calling process. + + +.. function:: sched_rr_get_interval(pid) + + Return the round-robin quantum in seconds for the process with PID *pid*. A + *pid* of 0 means the calling process. + + +.. function:: sched_yield() + + Voluntarily relinquish the CPU. + + +.. class:: cpu_set(ncpus) + + :class:`cpu_set` represents a set of CPUs on which a process is eligible to + run. *ncpus* is the number of CPUs the set should describe. Methods on + :class:`cpu_set` allow CPUs to be add or removed. + + :class:`cpu_set` supports the AND, OR, and XOR bitwise operations. For + example, given two cpu_sets, ``one`` and ``two``, ``one | two`` returns a + :class:`cpu_set` containing the cpus enabled both in ``one`` and ``two``. + + .. method:: set(i) + + Enable CPU *i*. + + .. method:: clear(i) + + Remove CPU *i*. + + .. method:: isset(i) + + Return ``True`` if CPU *i* is enabled in the set. + + .. method:: count() + + Return the number of enabled CPUs in the set. + + .. method:: zero() + + Clear the set completely. + + +.. function:: sched_setaffinity(pid, mask) + + Restrict the process with PID *pid* to a set of CPUs. *mask* is a + :class:`cpu_set` instance. + + +.. function:: sched_getaffinity(pid, size) + + Return the :class:`cpu_set` the process with PID *pid* is restricted to. The + result will contain *size* CPUs. + + .. _os-path: Miscellaneous System Information diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst index cf302cc..9e5bec9 100644 --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -38,6 +38,10 @@ the standard audio interface for Linux and recent versions of FreeBSD. This probably all warrants a footnote or two, but I don't understand things well enough right now to write it! --GPW +.. versionchanged:: 3.3 + Operations in this module now raise :exc:`OSError` where :exc:`IOError` + was raised. + .. seealso:: @@ -56,7 +60,7 @@ the standard audio interface for Linux and recent versions of FreeBSD. what went wrong. (If :mod:`ossaudiodev` receives an error from a system call such as - :c:func:`open`, :c:func:`write`, or :c:func:`ioctl`, it raises :exc:`IOError`. + :c:func:`open`, :c:func:`write`, or :c:func:`ioctl`, it raises :exc:`OSError`. Errors detected directly by :mod:`ossaudiodev` result in :exc:`OSSAudioError`.) (For backwards compatibility, the exception class is also available as @@ -168,7 +172,7 @@ The following methods each map to exactly one :func:`ioctl` system call. The correspondence is obvious: for example, :meth:`setfmt` corresponds to the ``SNDCTL_DSP_SETFMT`` ioctl, and :meth:`sync` to ``SNDCTL_DSP_SYNC`` (this can be useful when consulting the OSS documentation). If the underlying -:func:`ioctl` fails, they all raise :exc:`IOError`. +:func:`ioctl` fails, they all raise :exc:`OSError`. .. method:: oss_audio_device.nonblock() @@ -344,7 +348,7 @@ The mixer object provides two file-like methods: .. method:: oss_mixer_device.close() This method closes the open mixer device file. Any further attempts to use the - mixer after this file is closed will raise an :exc:`IOError`. + mixer after this file is closed will raise an :exc:`OSError`. .. method:: oss_mixer_device.fileno() @@ -403,7 +407,7 @@ The remaining methods are specific to audio mixing: returned, but both volumes are the same. Raises :exc:`OSSAudioError` if an invalid control was is specified, or - :exc:`IOError` if an unsupported control is specified. + :exc:`OSError` if an unsupported control is specified. .. method:: oss_mixer_device.set(control, (left, right)) @@ -427,7 +431,7 @@ The remaining methods are specific to audio mixing: .. method:: oss_mixer_device.set_recsrc(bitmask) Call this function to specify a recording source. Returns a bitmask indicating - the new recording source (or sources) if successful; raises :exc:`IOError` if an + the new recording source (or sources) if successful; raises :exc:`OSError` if an invalid source was specified. To set the current recording source to the microphone input:: diff --git a/Doc/library/packaging-misc.rst b/Doc/library/packaging-misc.rst new file mode 100644 index 0000000..5e56247 --- /dev/null +++ b/Doc/library/packaging-misc.rst @@ -0,0 +1,27 @@ +.. temporary file for modules that don't need a dedicated file yet + +:mod:`packaging.errors` --- Packaging exceptions +================================================ + +.. module:: packaging.errors + :synopsis: Packaging exceptions. + + +Provides exceptions used by the Packaging modules. Note that Packaging modules +may raise standard exceptions; in particular, SystemExit is usually raised for +errors that are obviously the end-user's fault (e.g. bad command-line arguments). + +This module is safe to use in ``from ... import *`` mode; it only exports +symbols whose names start with ``Packaging`` and end with ``Error``. + + +:mod:`packaging.manifest` --- The Manifest class +================================================ + +.. module:: packaging.manifest + :synopsis: The Manifest class, used for poking about the file system and + building lists of files. + + +This module provides the :class:`Manifest` class, used for poking about the +filesystem and building lists of files. diff --git a/Doc/library/packaging.command.rst b/Doc/library/packaging.command.rst new file mode 100644 index 0000000..6a85351 --- /dev/null +++ b/Doc/library/packaging.command.rst @@ -0,0 +1,111 @@ +:mod:`packaging.command` --- Standard Packaging commands +======================================================== + +.. module:: packaging.command + :synopsis: Standard packaging commands. + + +This subpackage contains one module for each standard Packaging command, such as +:command:`build` or :command:`upload`. Each command is implemented as a +separate module, with the command name as the name of the module and of the +class defined therein. + + + +:mod:`packaging.command.cmd` --- Abstract base class for Packaging commands +=========================================================================== + +.. module:: packaging.command.cmd + :synopsis: Abstract base class for commands. + + +This module supplies the abstract base class :class:`Command`. This class is +subclassed by the modules in the packaging.command subpackage. + + +.. class:: Command(dist) + + Abstract base class for defining command classes, the "worker bees" of the + Packaging. A useful analogy for command classes is to think of them as + subroutines with local variables called *options*. The options are declared + in :meth:`initialize_options` and defined (given their final values) in + :meth:`finalize_options`, both of which must be defined by every command + class. The distinction between the two is necessary because option values + might come from the outside world (command line, config file, ...), and any + options dependent on other options must be computed after these outside + influences have been processed --- hence :meth:`finalize_options`. The body + of the subroutine, where it does all its work based on the values of its + options, is the :meth:`run` method, which must also be implemented by every + command class. + + The class constructor takes a single argument *dist*, a + :class:`~packaging.dist.Distribution` instance. + + +Creating a new Packaging command +-------------------------------- + +This section outlines the steps to create a new Packaging command. + +.. XXX the following paragraph is focused on the stdlib; expand it to document + how to write and register a command in third-party projects + +A new command lives in a module in the :mod:`packaging.command` package. There +is a sample template in that directory called :file:`command_template`. Copy +this file to a new module with the same name as the new command you're +implementing. This module should implement a class with the same name as the +module (and the command). So, for instance, to create the command +``peel_banana`` (so that users can run ``setup.py peel_banana``), you'd copy +:file:`command_template` to :file:`packaging/command/peel_banana.py`, then edit +it so that it's implementing the class :class:`peel_banana`, a subclass of +:class:`Command`. It must define the following methods: + +.. method:: Command.initialize_options() + + Set default values for all the options that this command supports. Note that + these defaults may be overridden by other commands, by the setup script, by + config files, or by the command line. Thus, this is not the place to code + dependencies between options; generally, :meth:`initialize_options` + implementations are just a bunch of ``self.foo = None`` assignments. + + +.. method:: Command.finalize_options() + + Set final values for all the options that this command supports. This is + always called as late as possible, i.e. after any option assignments from the + command line or from other commands have been done. Thus, this is the place + to code option dependencies: if *foo* depends on *bar*, then it is safe to + set *foo* from *bar* as long as *foo* still has the same value it was + assigned in :meth:`initialize_options`. + + +.. method:: Command.run() + + A command's raison d'etre: carry out the action it exists to perform, + controlled by the options initialized in :meth:`initialize_options`, + customized by other commands, the setup script, the command line, and config + files, and finalized in :meth:`finalize_options`. All terminal output and + filesystem interaction should be done by :meth:`run`. + + +Command classes may define this attribute: + + +.. attribute:: Command.sub_commands + + *sub_commands* formalizes the notion of a "family" of commands, + e.g. ``install_dist`` as the parent with sub-commands ``install_lib``, + ``install_headers``, etc. The parent of a family of commands defines + *sub_commands* as a class attribute; it's a list of 2-tuples ``(command_name, + predicate)``, with *command_name* a string and *predicate* a function, a + string or ``None``. *predicate* is a method of the parent command that + determines whether the corresponding command is applicable in the current + situation. (E.g. ``install_headers`` is only applicable if we have any C + header files to install.) If *predicate* is ``None``, that command is always + applicable. + + *sub_commands* is usually defined at the *end* of a class, because + predicates can be methods of the class, so they must already have been + defined. The canonical example is the :command:`install_dist` command. + +.. XXX document how to add a custom command to another one's subcommands diff --git a/Doc/library/packaging.compiler.rst b/Doc/library/packaging.compiler.rst new file mode 100644 index 0000000..ecf641e --- /dev/null +++ b/Doc/library/packaging.compiler.rst @@ -0,0 +1,681 @@ +:mod:`packaging.compiler` --- Compiler classes +============================================== + +.. module:: packaging.compiler + :synopsis: Compiler classes to build C/C++ extensions or libraries. + + +This subpackage contains an abstract base class representing a compiler and +concrete implementations for common compilers. The compiler classes should not +be instantiated directly, but created using the :func:`new_compiler` factory +function. Compiler types provided by Packaging are listed in +:ref:`packaging-standard-compilers`. + + +Public functions +---------------- + +.. function:: new_compiler(plat=None, compiler=None, dry_run=False, force=False) + + Factory function to generate an instance of some + :class:`~.ccompiler.CCompiler` subclass for the requested platform or + compiler type. + + If no argument is given for *plat* and *compiler*, the default compiler type + for the platform (:attr:`os.name`) will be used: ``'unix'`` for Unix and + Mac OS X, ``'msvc'`` for Windows. + + If *plat* is given, it must be one of ``'posix'``, ``'darwin'`` or ``'nt'``. + An invalid value will not raise an exception but use the default compiler + type for the current platform. + + .. XXX errors should never pass silently; this behavior is particularly + harmful when a compiler type is given as first argument + + If *compiler* is given, *plat* will be ignored, allowing you to get for + example a ``'unix'`` compiler object under Windows or an ``'msvc'`` compiler + under Unix. However, not all compiler types can be instantiated on every + platform. + + +.. function:: customize_compiler(compiler) + + Do any platform-specific customization of a CCompiler instance. Mainly + needed on Unix to plug in the information that varies across Unices and is + stored in CPython's Makefile. + + +.. function:: gen_lib_options(compiler, library_dirs, runtime_library_dirs, libraries) + + Generate linker options for searching library directories and linking with + specific libraries. *libraries* and *library_dirs* are, respectively, lists + of library names (not filenames!) and search directories. Returns a list of + command-line options suitable for use with some compiler (depending on the + two format strings passed in). + + +.. function:: gen_preprocess_options(macros, include_dirs) + + Generate C preprocessor options (:option:`-D`, :option:`-U`, :option:`-I`) as + used by at least two types of compilers: the typical Unix compiler and Visual + C++. *macros* is the usual thing, a list of 1- or 2-tuples, where ``(name,)`` + means undefine (:option:`-U`) macro *name*, and ``(name, value)`` means + define (:option:`-D`) macro *name* to *value*. *include_dirs* is just a list + of directory names to be added to the header file search path (:option:`-I`). + Returns a list of command-line options suitable for either Unix compilers or + Visual C++. + + +.. function:: get_default_compiler(osname, platform) + + Determine the default compiler to use for the given platform. + + *osname* should be one of the standard Python OS names (i.e. the ones + returned by ``os.name``) and *platform* the common value returned by + ``sys.platform`` for the platform in question. + + The default values are ``os.name`` and ``sys.platform``. + + +.. function:: set_compiler(location) + + Add or change a compiler + + +.. function:: show_compilers() + + Print list of available compilers (used by the :option:`--help-compiler` + options to :command:`build`, :command:`build_ext`, :command:`build_clib`). + + +.. _packaging-standard-compilers: + +Standard compilers +------------------ + +Concrete subclasses of :class:`~.ccompiler.CCompiler` are provided in submodules +of the :mod:`packaging.compiler` package. You do not need to import them, using +:func:`new_compiler` is the public API to use. This table documents the +standard compilers; be aware that they can be replaced by other classes on your +platform. + +=============== ======================================================== ======= +name description notes +=============== ======================================================== ======= +``'unix'`` typical Unix-style command-line C compiler [#]_ +``'msvc'`` Microsoft compiler [#]_ +``'bcpp'`` Borland C++ compiler +``'cygwin'`` Cygwin compiler (Windows port of GCC) +``'mingw32'`` Mingw32 port of GCC (same as Cygwin in no-Cygwin mode) +=============== ======================================================== ======= + + +.. [#] The Unix compiler class assumes this behavior: + + * macros defined with :option:`-Dname[=value]` + + * macros undefined with :option:`-Uname` + + * include search directories specified with :option:`-Idir` + + * libraries specified with :option:`-llib` + + * library search directories specified with :option:`-Ldir` + + * compile handled by :program:`cc` (or similar) executable with + :option:`-c` option: compiles :file:`.c` to :file:`.o` + + * link static library handled by :program:`ar` command (possibly with + :program:`ranlib`) + + * link shared library handled by :program:`cc` :option:`-shared` + + +.. [#] On Windows, extension modules typically need to be compiled with the same + compiler that was used to compile CPython (for example Microsoft Visual + Studio .NET 2003 for CPython 2.4 and 2.5). The AMD64 and Itanium + binaries are created using the Platform SDK. + + Under the hood, there are actually two different subclasses of + :class:`~.ccompiler.CCompiler` defined: one is compatible with MSVC 2005 + and 2008, the other works with older versions. This should not be a + concern for regular use of the functions in this module. + + Packaging will normally choose the right compiler, linker etc. on its + own. To override this choice, the environment variables + *DISTUTILS_USE_SDK* and *MSSdk* must be both set. *MSSdk* indicates that + the current environment has been setup by the SDK's ``SetEnv.Cmd`` + script, or that the environment variables had been registered when the + SDK was installed; *DISTUTILS_USE_SDK* indicates that the user has made + an explicit choice to override the compiler selection done by Packaging. + + .. TODO document the envvars in Doc/using and the man page + + +:mod:`packaging.compiler.ccompiler` --- CCompiler base class +============================================================ + +.. module:: packaging.compiler.ccompiler + :synopsis: Abstract CCompiler class. + + +This module provides the abstract base class for the :class:`CCompiler` +classes. A :class:`CCompiler` instance can be used for all the compile and +link steps needed to build a single project. Methods are provided to set +options for the compiler --- macro definitions, include directories, link path, +libraries and the like. + +.. class:: CCompiler(dry_run=False, force=False) + + The abstract base class :class:`CCompiler` defines the interface that must be + implemented by real compiler classes. The class also has some utility + methods used by several compiler classes. + + The basic idea behind a compiler abstraction class is that each instance can + be used for all the compile/link steps in building a single project. Thus, + attributes common to all of those compile and link steps --- include + directories, macros to define, libraries to link against, etc. --- are + attributes of the compiler instance. To allow for variability in how + individual files are treated, most of those attributes may be varied on a + per-compilation or per-link basis. + + The constructor for each subclass creates an instance of the Compiler object. + Flags are *dry_run* (don't actually execute + the steps) and *force* (rebuild everything, regardless of dependencies). All + of these flags default to ``False`` (off). Note that you probably don't want to + instantiate :class:`CCompiler` or one of its subclasses directly - use the + :func:`new_compiler` factory function instead. + + The following methods allow you to manually alter compiler options for the + instance of the Compiler class. + + + .. method:: CCompiler.add_include_dir(dir) + + Add *dir* to the list of directories that will be searched for header + files. The compiler is instructed to search directories in the order in + which they are supplied by successive calls to :meth:`add_include_dir`. + + + .. method:: CCompiler.set_include_dirs(dirs) + + Set the list of directories that will be searched to *dirs* (a list of + strings). Overrides any preceding calls to :meth:`add_include_dir`; + subsequent calls to :meth:`add_include_dir` add to the list passed to + :meth:`set_include_dirs`. This does not affect any list of standard + include directories that the compiler may search by default. + + + .. method:: CCompiler.add_library(libname) + + Add *libname* to the list of libraries that will be included in all links + driven by this compiler object. Note that *libname* should *not* be the + name of a file containing a library, but the name of the library itself: + the actual filename will be inferred by the linker, the compiler, or the + compiler class (depending on the platform). + + The linker will be instructed to link against libraries in the order they + were supplied to :meth:`add_library` and/or :meth:`set_libraries`. It is + perfectly valid to duplicate library names; the linker will be instructed + to link against libraries as many times as they are mentioned. + + + .. method:: CCompiler.set_libraries(libnames) + + Set the list of libraries to be included in all links driven by this + compiler object to *libnames* (a list of strings). This does not affect + any standard system libraries that the linker may include by default. + + + .. method:: CCompiler.add_library_dir(dir) + + Add *dir* to the list of directories that will be searched for libraries + specified to :meth:`add_library` and :meth:`set_libraries`. The linker + will be instructed to search for libraries in the order they are supplied + to :meth:`add_library_dir` and/or :meth:`set_library_dirs`. + + + .. method:: CCompiler.set_library_dirs(dirs) + + Set the list of library search directories to *dirs* (a list of strings). + This does not affect any standard library search path that the linker may + search by default. + + + .. method:: CCompiler.add_runtime_library_dir(dir) + + Add *dir* to the list of directories that will be searched for shared + libraries at runtime. + + + .. method:: CCompiler.set_runtime_library_dirs(dirs) + + Set the list of directories to search for shared libraries at runtime to + *dirs* (a list of strings). This does not affect any standard search path + that the runtime linker may search by default. + + + .. method:: CCompiler.define_macro(name, value=None) + + Define a preprocessor macro for all compilations driven by this compiler + object. The optional parameter *value* should be a string; if it is not + supplied, then the macro will be defined without an explicit value and the + exact outcome depends on the compiler used (XXX true? does ANSI say + anything about this?) + + + .. method:: CCompiler.undefine_macro(name) + + Undefine a preprocessor macro for all compilations driven by this compiler + object. If the same macro is defined by :meth:`define_macro` and + undefined by :meth:`undefine_macro` the last call takes precedence + (including multiple redefinitions or undefinitions). If the macro is + redefined/undefined on a per-compilation basis (i.e. in the call to + :meth:`compile`), then that takes precedence. + + + .. method:: CCompiler.add_link_object(object) + + Add *object* to the list of object files (or analogues, such as explicitly + named library files or the output of "resource compilers") to be included + in every link driven by this compiler object. + + + .. method:: CCompiler.set_link_objects(objects) + + Set the list of object files (or analogues) to be included in every link + to *objects*. This does not affect any standard object files that the + linker may include by default (such as system libraries). + + The following methods implement methods for autodetection of compiler + options, providing some functionality similar to GNU :program:`autoconf`. + + + .. method:: CCompiler.detect_language(sources) + + Detect the language of a given file, or list of files. Uses the instance + attributes :attr:`language_map` (a dictionary), and :attr:`language_order` + (a list) to do the job. + + + .. method:: CCompiler.find_library_file(dirs, lib, debug=0) + + Search the specified list of directories for a static or shared library file + *lib* and return the full path to that file. If *debug* is true, look for a + debugging version (if that makes sense on the current platform). Return + ``None`` if *lib* wasn't found in any of the specified directories. + + + .. method:: CCompiler.has_function(funcname, includes=None, include_dirs=None, libraries=None, library_dirs=None) + + Return a boolean indicating whether *funcname* is supported on the current + platform. The optional arguments can be used to augment the compilation + environment by providing additional include files and paths and libraries and + paths. + + + .. method:: CCompiler.library_dir_option(dir) + + Return the compiler option to add *dir* to the list of directories searched for + libraries. + + + .. method:: CCompiler.library_option(lib) + + Return the compiler option to add *dir* to the list of libraries linked into the + shared library or executable. + + + .. method:: CCompiler.runtime_library_dir_option(dir) + + Return the compiler option to add *dir* to the list of directories searched for + runtime libraries. + + + .. method:: CCompiler.set_executables(**args) + + Define the executables (and options for them) that will be run to perform the + various stages of compilation. The exact set of executables that may be + specified here depends on the compiler class (via the 'executables' class + attribute), but most will have: + + +--------------+------------------------------------------+ + | attribute | description | + +==============+==========================================+ + | *compiler* | the C/C++ compiler | + +--------------+------------------------------------------+ + | *linker_so* | linker used to create shared objects and | + | | libraries | + +--------------+------------------------------------------+ + | *linker_exe* | linker used to create binary executables | + +--------------+------------------------------------------+ + | *archiver* | static library creator | + +--------------+------------------------------------------+ + + On platforms with a command line (Unix, DOS/Windows), each of these is a string + that will be split into executable name and (optional) list of arguments. + (Splitting the string is done similarly to how Unix shells operate: words are + delimited by spaces, but quotes and backslashes can override this. See + :func:`packaging.util.split_quoted`.) + + The following methods invoke stages in the build process. + + + .. method:: CCompiler.compile(sources, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None) + + Compile one or more source files. Generates object files (e.g. transforms a + :file:`.c` file to a :file:`.o` file.) + + *sources* must be a list of filenames, most likely C/C++ files, but in reality + anything that can be handled by a particular compiler and compiler class (e.g. + an ``'msvc'`` compiler can handle resource files in *sources*). Return a list of + object filenames, one per source filename in *sources*. Depending on the + implementation, not all source files will necessarily be compiled, but all + corresponding object filenames will be returned. + + If *output_dir* is given, object files will be put under it, while retaining + their original path component. That is, :file:`foo/bar.c` normally compiles to + :file:`foo/bar.o` (for a Unix implementation); if *output_dir* is *build*, then + it would compile to :file:`build/foo/bar.o`. + + *macros*, if given, must be a list of macro definitions. A macro definition is + either a ``(name, value)`` 2-tuple or a ``(name,)`` 1-tuple. The former defines + a macro; if the value is ``None``, the macro is defined without an explicit + value. The 1-tuple case undefines a macro. Later + definitions/redefinitions/undefinitions take precedence. + + *include_dirs*, if given, must be a list of strings, the directories to add to + the default include file search path for this compilation only. + + *debug* is a boolean; if true, the compiler will be instructed to output debug + symbols in (or alongside) the object file(s). + + *extra_preargs* and *extra_postargs* are implementation-dependent. On platforms + that have the notion of a command line (e.g. Unix, DOS/Windows), they are most + likely lists of strings: extra command-line arguments to prepend/append to the + compiler command line. On other platforms, consult the implementation class + documentation. In any event, they are intended as an escape hatch for those + occasions when the abstract compiler framework doesn't cut the mustard. + + *depends*, if given, is a list of filenames that all targets depend on. If a + source file is older than any file in depends, then the source file will be + recompiled. This supports dependency tracking, but only at a coarse + granularity. + + Raises :exc:`CompileError` on failure. + + + .. method:: CCompiler.create_static_lib(objects, output_libname, output_dir=None, debug=0, target_lang=None) + + Link a bunch of stuff together to create a static library file. The "bunch of + stuff" consists of the list of object files supplied as *objects*, the extra + object files supplied to :meth:`add_link_object` and/or + :meth:`set_link_objects`, the libraries supplied to :meth:`add_library` and/or + :meth:`set_libraries`, and the libraries supplied as *libraries* (if any). + + *output_libname* should be a library name, not a filename; the filename will be + inferred from the library name. *output_dir* is the directory where the library + file will be put. XXX defaults to what? + + *debug* is a boolean; if true, debugging information will be included in the + library (note that on most platforms, it is the compile step where this matters: + the *debug* flag is included here just for consistency). + + *target_lang* is the target language for which the given objects are being + compiled. This allows specific linkage time treatment of certain languages. + + Raises :exc:`LibError` on failure. + + + .. method:: CCompiler.link(target_desc, objects, output_filename, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None) + + Link a bunch of stuff together to create an executable or shared library file. + + The "bunch of stuff" consists of the list of object files supplied as *objects*. + *output_filename* should be a filename. If *output_dir* is supplied, + *output_filename* is relative to it (i.e. *output_filename* can provide + directory components if needed). + + *libraries* is a list of libraries to link against. These are library names, + not filenames, since they're translated into filenames in a platform-specific + way (e.g. *foo* becomes :file:`libfoo.a` on Unix and :file:`foo.lib` on + DOS/Windows). However, they can include a directory component, which means the + linker will look in that specific directory rather than searching all the normal + locations. + + *library_dirs*, if supplied, should be a list of directories to search for + libraries that were specified as bare library names (i.e. no directory + component). These are on top of the system default and those supplied to + :meth:`add_library_dir` and/or :meth:`set_library_dirs`. *runtime_library_dirs* + is a list of directories that will be embedded into the shared library and used + to search for other shared libraries that \*it\* depends on at run-time. (This + may only be relevant on Unix.) + + *export_symbols* is a list of symbols that the shared library will export. + (This appears to be relevant only on Windows.) + + *debug* is as for :meth:`compile` and :meth:`create_static_lib`, with the + slight distinction that it actually matters on most platforms (as opposed to + :meth:`create_static_lib`, which includes a *debug* flag mostly for form's + sake). + + *extra_preargs* and *extra_postargs* are as for :meth:`compile` (except of + course that they supply command-line arguments for the particular linker being + used). + + *target_lang* is the target language for which the given objects are being + compiled. This allows specific linkage time treatment of certain languages. + + Raises :exc:`LinkError` on failure. + + + .. method:: CCompiler.link_executable(objects, output_progname, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, target_lang=None) + + Link an executable. *output_progname* is the name of the file executable, while + *objects* are a list of object filenames to link in. Other arguments are as for + the :meth:`link` method. + + + .. method:: CCompiler.link_shared_lib(objects, output_libname, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None) + + Link a shared library. *output_libname* is the name of the output library, + while *objects* is a list of object filenames to link in. Other arguments are + as for the :meth:`link` method. + + + .. method:: CCompiler.link_shared_object(objects, output_filename, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None) + + Link a shared object. *output_filename* is the name of the shared object that + will be created, while *objects* is a list of object filenames to link in. + Other arguments are as for the :meth:`link` method. + + + .. method:: CCompiler.preprocess(source, output_file=None, macros=None, include_dirs=None, extra_preargs=None, extra_postargs=None) + + Preprocess a single C/C++ source file, named in *source*. Output will be written + to file named *output_file*, or *stdout* if *output_file* not supplied. + *macros* is a list of macro definitions as for :meth:`compile`, which will + augment the macros set with :meth:`define_macro` and :meth:`undefine_macro`. + *include_dirs* is a list of directory names that will be added to the default + list, in the same way as :meth:`add_include_dir`. + + Raises :exc:`PreprocessError` on failure. + + The following utility methods are defined by the :class:`CCompiler` class, for + use by the various concrete subclasses. + + + .. method:: CCompiler.executable_filename(basename, strip_dir=0, output_dir='') + + Returns the filename of the executable for the given *basename*. Typically for + non-Windows platforms this is the same as the basename, while Windows will get + a :file:`.exe` added. + + + .. method:: CCompiler.library_filename(libname, lib_type='static', strip_dir=0, output_dir='') + + Returns the filename for the given library name on the current platform. On Unix + a library with *lib_type* of ``'static'`` will typically be of the form + :file:`liblibname.a`, while a *lib_type* of ``'dynamic'`` will be of the form + :file:`liblibname.so`. + + + .. method:: CCompiler.object_filenames(source_filenames, strip_dir=0, output_dir='') + + Returns the name of the object files for the given source files. + *source_filenames* should be a list of filenames. + + + .. method:: CCompiler.shared_object_filename(basename, strip_dir=0, output_dir='') + + Returns the name of a shared object file for the given file name *basename*. + + + .. method:: CCompiler.execute(func, args, msg=None, level=1) + + Invokes :func:`packaging.util.execute` This method invokes a Python function + *func* with the given arguments *args*, after logging and taking into account + the *dry_run* flag. XXX see also. + + + .. method:: CCompiler.spawn(cmd) + + Invokes :func:`packaging.util.spawn`. This invokes an external process to run + the given command. XXX see also. + + + .. method:: CCompiler.mkpath(name, mode=511) + + Invokes :func:`packaging.dir_util.mkpath`. This creates a directory and any + missing ancestor directories. XXX see also. + + + .. method:: CCompiler.move_file(src, dst) + + Invokes :meth:`packaging.file_util.move_file`. Renames *src* to *dst*. XXX see + also. + + +:mod:`packaging.compiler.extension` --- The Extension class +=========================================================== + +.. module:: packaging.compiler.extension + :synopsis: Class used to represent C/C++ extension modules. + + +This module provides the :class:`Extension` class, used to represent C/C++ +extension modules. + +.. class:: Extension + + The Extension class describes a single C or C++ extension module. It accepts + the following keyword arguments in its constructor: + + +------------------------+--------------------------------+---------------------------+ + | argument name | value | type | + +========================+================================+===========================+ + | *name* | the full name of the | string | + | | extension, including any | | + | | packages --- i.e. *not* a | | + | | filename or pathname, but | | + | | Python dotted name | | + +------------------------+--------------------------------+---------------------------+ + | *sources* | list of source filenames, | list of strings | + | | relative to the distribution | | + | | root (where the setup script | | + | | lives), in Unix form (slash- | | + | | separated) for portability. | | + | | Source files may be C, C++, | | + | | SWIG (.i), platform-specific | | + | | resource files, or whatever | | + | | else is recognized by the | | + | | :command:`build_ext` command | | + | | as source for a Python | | + | | extension. | | + +------------------------+--------------------------------+---------------------------+ + | *include_dirs* | list of directories to search | list of strings | + | | for C/C++ header files (in | | + | | Unix form for portability) | | + +------------------------+--------------------------------+---------------------------+ + | *define_macros* | list of macros to define; each | list of tuples | + | | macro is defined using a | | + | | 2-tuple ``(name, value)``, | | + | | where *value* is | | + | | either the string to define it | | + | | to or ``None`` to define it | | + | | without a particular value | | + | | (equivalent of ``#define FOO`` | | + | | in source or :option:`-DFOO` | | + | | on Unix C compiler command | | + | | line) | | + +------------------------+--------------------------------+---------------------------+ + | *undef_macros* | list of macros to undefine | list of strings | + | | explicitly | | + +------------------------+--------------------------------+---------------------------+ + | *library_dirs* | list of directories to search | list of strings | + | | for C/C++ libraries at link | | + | | time | | + +------------------------+--------------------------------+---------------------------+ + | *libraries* | list of library names (not | list of strings | + | | filenames or paths) to link | | + | | against | | + +------------------------+--------------------------------+---------------------------+ + | *runtime_library_dirs* | list of directories to search | list of strings | + | | for C/C++ libraries at run | | + | | time (for shared extensions, | | + | | this is when the extension is | | + | | loaded) | | + +------------------------+--------------------------------+---------------------------+ + | *extra_objects* | list of extra files to link | list of strings | + | | with (e.g. object files not | | + | | implied by 'sources', static | | + | | library that must be | | + | | explicitly specified, binary | | + | | resource files, etc.) | | + +------------------------+--------------------------------+---------------------------+ + | *extra_compile_args* | any extra platform- and | list of strings | + | | compiler-specific information | | + | | to use when compiling the | | + | | source files in 'sources'. For | | + | | platforms and compilers where | | + | | a command line makes sense, | | + | | this is typically a list of | | + | | command-line arguments, but | | + | | for other platforms it could | | + | | be anything. | | + +------------------------+--------------------------------+---------------------------+ + | *extra_link_args* | any extra platform- and | list of strings | + | | compiler-specific information | | + | | to use when linking object | | + | | files together to create the | | + | | extension (or to create a new | | + | | static Python interpreter). | | + | | Similar interpretation as for | | + | | 'extra_compile_args'. | | + +------------------------+--------------------------------+---------------------------+ + | *export_symbols* | list of symbols to be exported | list of strings | + | | from a shared extension. Not | | + | | used on all platforms, and not | | + | | generally necessary for Python | | + | | extensions, which typically | | + | | export exactly one symbol: | | + | | ``init`` + extension_name. | | + +------------------------+--------------------------------+---------------------------+ + | *depends* | list of files that the | list of strings | + | | extension depends on | | + +------------------------+--------------------------------+---------------------------+ + | *language* | extension language (i.e. | string | + | | ``'c'``, ``'c++'``, | | + | | ``'objc'``). Will be detected | | + | | from the source extensions if | | + | | not provided. | | + +------------------------+--------------------------------+---------------------------+ + | *optional* | specifies that a build failure | boolean | + | | in the extension should not | | + | | abort the build process, but | | + | | simply skip the extension. | | + +------------------------+--------------------------------+---------------------------+ + +To distribute extension modules that live in a package (e.g. ``package.ext``), +you need to create a :file:`{package}/__init__.py` file to let Python recognize +and import your module. diff --git a/Doc/library/packaging.database.rst b/Doc/library/packaging.database.rst new file mode 100644 index 0000000..9d750f0 --- /dev/null +++ b/Doc/library/packaging.database.rst @@ -0,0 +1,345 @@ +:mod:`packaging.database` --- Database of installed distributions +================================================================= + +.. module:: packaging.database + :synopsis: Functions to query and manipulate installed distributions. + + +This module provides an implementation of :PEP:`376`. It was originally +intended to land in :mod:`pkgutil`, but with the inclusion of Packaging in the +standard library, it was thought best to include it in a submodule of +:mod:`packaging`, leaving :mod:`pkgutil` to deal with imports. + +Installed Python distributions are represented by instances of +:class:`Distribution`, or :class:`EggInfoDistribution` for legacy egg formats. +Most functions also provide an extra argument ``use_egg_info`` to take legacy +distributions into account. + +For the purpose of this module, "installed" means that the distribution's +:file:`.dist-info`, :file:`.egg-info` or :file:`egg` directory or file is found +on :data:`sys.path`. For example, if the parent directory of a +:file:`dist-info` directory is added to :envvar:`PYTHONPATH`, then it will be +available in the database. + +Classes representing installed distributions +-------------------------------------------- + +.. class:: Distribution(path) + + Class representing an installed distribution. It is different from + :class:`packaging.dist.Distribution` which holds the list of files, the + metadata and options during the run of a Packaging command. + + Instantiate with the *path* to a ``.dist-info`` directory. Instances can be + compared and sorted. Other available methods are: + + .. XXX describe how comparison works + + .. method:: get_distinfo_file(path, binary=False) + + Return a read-only file object for a file located at + :file:`{project}-{version}.dist-info/{path}`. *path* should be a + ``'/'``-separated path relative to the ``.dist-info`` directory or an + absolute path; if it is an absolute path and doesn't start with the path + to the :file:`.dist-info` directory, a :class:`PackagingError` is raised. + + If *binary* is ``True``, the file is opened in binary mode. + + .. method:: get_resource_path(relative_path) + + .. TODO + + .. method:: list_distinfo_files(local=False) + + Return an iterator over all files located in the :file:`.dist-info` + directory. If *local* is ``True``, each returned path is transformed into + a local absolute path, otherwise the raw value found in the :file:`RECORD` + file is returned. + + .. method:: list_installed_files(local=False) + + Iterate over the files installed with the distribution and registered in + the :file:`RECORD` file and yield a tuple ``(path, md5, size)`` for each + line. If *local* is ``True``, the returned path is transformed into a + local absolute path, otherwise the raw value is returned. + + A local absolute path is an absolute path in which occurrences of ``'/'`` + have been replaced by :data:`os.sep`. + + .. method:: uses(path) + + Check whether *path* was installed by this distribution (i.e. if the path + is present in the :file:`RECORD` file). *path* can be a local absolute + path or a relative ``'/'``-separated path. Returns a boolean. + + Available attributes: + + .. attribute:: metadata + + Instance of :class:`packaging.metadata.Metadata` filled with the contents + of the :file:`{project}-{version}.dist-info/METADATA` file. + + .. attribute:: name + + Shortcut for ``metadata['Name']``. + + .. attribute:: version + + Shortcut for ``metadata['Version']``. + + .. attribute:: requested + + Boolean indicating whether this distribution was requested by the user of + automatically installed as a dependency. + + +.. class:: EggInfoDistribution(path) + + Class representing a legacy distribution. It is compatible with distutils' + and setuptools' :file:`.egg-info` and :file:`.egg` files and directories. + + .. FIXME should be named EggDistribution + + Instantiate with the *path* to an egg file or directory. Instances can be + compared and sorted. Other available methods are: + + .. method:: list_installed_files(local=False) + + .. method:: uses(path) + + Available attributes: + + .. attribute:: metadata + + Instance of :class:`packaging.metadata.Metadata` filled with the contents + of the :file:`{project-version}.egg-info/PKG-INFO` or + :file:`{project-version}.egg` file. + + .. attribute:: name + + Shortcut for ``metadata['Name']``. + + .. attribute:: version + + Shortcut for ``metadata['Version']``. + + +Functions to work with the database +----------------------------------- + +.. function:: get_distribution(name, use_egg_info=False, paths=None) + + Return an instance of :class:`Distribution` or :class:`EggInfoDistribution` + for the first installed distribution matching *name*. Egg distributions are + considered only if *use_egg_info* is true; if both a dist-info and an egg + file are found, the dist-info prevails. The directories to be searched are + given in *paths*, which defaults to :data:`sys.path`. Returns ``None`` if no + matching distribution is found. + + .. FIXME param should be named use_egg + + +.. function:: get_distributions(use_egg_info=False, paths=None) + + Return an iterator of :class:`Distribution` instances for all installed + distributions found in *paths* (defaults to :data:`sys.path`). If + *use_egg_info* is true, also return instances of :class:`EggInfoDistribution` + for legacy distributions found. + + +.. function:: get_file_users(path) + + Return an iterator over all distributions using *path*, a local absolute path + or a relative ``'/'``-separated path. + + .. XXX does this work with prefixes or full file path only? + + +.. function:: obsoletes_distribution(name, version=None, use_egg_info=False) + + Return an iterator over all distributions that declare they obsolete *name*. + *version* is an optional argument to match only specific releases (see + :mod:`packaging.version`). If *use_egg_info* is true, legacy egg + distributions will be considered as well. + + +.. function:: provides_distribution(name, version=None, use_egg_info=False) + + Return an iterator over all distributions that declare they provide *name*. + *version* is an optional argument to match only specific releases (see + :mod:`packaging.version`). If *use_egg_info* is true, legacy egg + distributions will be considered as well. + + +Utility functions +----------------- + +.. function:: distinfo_dirname(name, version) + + Escape *name* and *version* into a filename-safe form and return the + directory name built from them, for example + :file:`{safename}-{safeversion}.dist-info.` In *name*, runs of + non-alphanumeric characters are replaced with one ``'_'``; in *version*, + spaces become dots, and runs of other non-alphanumeric characters (except + dots) a replaced by one ``'-'``. + + .. XXX wth spaces in version numbers? + +For performance purposes, the list of distributions is being internally +cached. Caching is enabled by default, but you can control it with these +functions: + +.. function:: clear_cache() + + Clear the cache. + +.. function:: disable_cache() + + Disable the cache, without clearing it. + +.. function:: enable_cache() + + Enable the internal cache, without clearing it. + + +Examples +-------- + +Printing all information about a distribution +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Given the name of an installed distribution, we shall print out all +information that can be obtained using functions provided in this module:: + + import sys + import packaging.database + + try: + name = sys.argv[1] + except ValueError: + sys.exit('Not enough arguments') + + # first create the Distribution instance + dist = packaging.database.Distribution(path) + if dist is None: + sys.exit('No such distribution') + + print('Information about %r' % dist.name) + print() + + print('Files') + print('=====') + for path, md5, size in dist.list_installed_files(): + print('* Path: %s' % path) + print(' Hash %s, Size: %s bytes' % (md5, size)) + print() + + print('Metadata') + print('========') + for key, value in dist.metadata.items(): + print('%20s: %s' % (key, value)) + print() + + print('Extra') + print('=====') + if dist.requested: + print('* It was installed by user request') + else: + print('* It was installed as a dependency') + +If we save the script above as ``print_info.py``, we can use it to extract +information from a :file:`.dist-info` directory. By typing in the console: + +.. code-block:: sh + + python print_info.py choxie + +we get the following output: + +.. code-block:: none + + Information about 'choxie' + + Files + ===== + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9/truffles.py + Hash 5e052db6a478d06bad9ae033e6bc08af, Size: 111 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9/choxie/chocolate.py + Hash ac56bf496d8d1d26f866235b95f31030, Size: 214 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9/choxie/__init__.py + Hash 416aab08dfa846f473129e89a7625bbc, Size: 25 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/INSTALLER + Hash d41d8cd98f00b204e9800998ecf8427e, Size: 0 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/METADATA + Hash 696a209967fef3c8b8f5a7bb10386385, Size: 225 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/REQUESTED + Hash d41d8cd98f00b204e9800998ecf8427e, Size: 0 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/RECORD + Hash None, Size: None bytes + + Metadata + ======== + Metadata-Version: 1.2 + Name: choxie + Version: 2.0.0.9 + Platform: [] + Supported-Platform: UNKNOWN + Summary: Chocolate with a kick! + Description: UNKNOWN + Keywords: [] + Home-page: UNKNOWN + Author: UNKNOWN + Author-email: UNKNOWN + Maintainer: UNKNOWN + Maintainer-email: UNKNOWN + License: UNKNOWN + Classifier: [] + Download-URL: UNKNOWN + Obsoletes-Dist: ['truffles (<=0.8,>=0.5)', 'truffles (<=0.9,>=0.6)'] + Project-URL: [] + Provides-Dist: ['truffles (1.0)'] + Requires-Dist: ['towel-stuff (0.1)'] + Requires-Python: UNKNOWN + Requires-External: [] + + Extra + ===== + * It was installed as a dependency + + +Getting metadata about a distribution +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Sometimes you're not interested about the packaging information contained in a +full :class:`Distribution` object but just want to do something with its +:attr:`~Distribution.metadata`:: + + >>> from packaging.database import get_distribution + >>> info = get_distribution('chocolate').metadata + >>> info['Keywords'] + ['cooking', 'happiness'] + + +Finding out obsoleted distributions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now, we tackle a different problem, we are interested in finding out +which distributions have been obsoleted. This can be easily done as follows:: + + import packaging.database + + # iterate over all distributions in the system + for dist in packaging.database.get_distributions(): + name, version = dist.name, dist.version + # find out which distributions obsolete this name/version combination + replacements = packaging.database.obsoletes_distribution(name, version) + if replacements: + print('%r %s is obsoleted by' % (name, version), + ', '.join(repr(r.name) for r in replacements)) + +This is how the output might look like: + +.. code-block:: none + + 'strawberry' 0.6 is obsoleted by 'choxie' + 'grammar' 1.0a4 is obsoleted by 'towel-stuff' diff --git a/Doc/library/packaging.depgraph.rst b/Doc/library/packaging.depgraph.rst new file mode 100644 index 0000000..c384788 --- /dev/null +++ b/Doc/library/packaging.depgraph.rst @@ -0,0 +1,199 @@ +:mod:`packaging.depgraph` --- Dependency graph builder +====================================================== + +.. module:: packaging.depgraph + :synopsis: Graph builder for dependencies between releases. + + +This module provides the means to analyse the dependencies between various +distributions and to create a graph representing these dependency relationships. +In this document, "distribution" refers to an instance of +:class:`packaging.database.Distribution` or +:class:`packaging.database.EggInfoDistribution`. + +.. XXX terminology problem with dist vs. release: dists are installed, but deps + use releases + +.. XXX explain how to use it with dists not installed: Distribution can only be + instantiated with a path, but this module is useful for remote dist too + +.. XXX functions should accept and return iterators, not lists + + +The :class:`DependencyGraph` class +---------------------------------- + +.. class:: DependencyGraph + + Represent a dependency graph between releases. The nodes are distribution + instances; the edge model dependencies. An edge from ``a`` to ``b`` means + that ``a`` depends on ``b``. + + .. method:: add_distribution(distribution) + + Add *distribution* to the graph. + + .. method:: add_edge(x, y, label=None) + + Add an edge from distribution *x* to distribution *y* with the given + *label* (string). + + .. method:: add_missing(distribution, requirement) + + Add a missing *requirement* (string) for the given *distribution*. + + .. method:: repr_node(dist, level=1) + + Print a subgraph starting from *dist*. *level* gives the depth of the + subgraph. + + Direct access to the graph nodes and edges is provided through these + attributes: + + .. attribute:: adjacency_list + + Dictionary mapping distributions to a list of ``(other, label)`` tuples + where ``other`` is a distribution and the edge is labeled with ``label`` + (i.e. the version specifier, if such was provided). + + .. attribute:: reverse_list + + Dictionary mapping distributions to a list of predecessors. This allows + efficient traversal. + + .. attribute:: missing + + Dictionary mapping distributions to a list of requirements that were not + provided by any distribution. + + +Auxiliary functions +------------------- + +.. function:: dependent_dists(dists, dist) + + Recursively generate a list of distributions from *dists* that are dependent + on *dist*. + + .. XXX what does member mean here: "dist is a member of *dists* for which we + are interested" + +.. function:: generate_graph(dists) + + Generate a :class:`DependencyGraph` from the given list of distributions. + + .. XXX make this alternate constructor a DepGraph classmethod or rename; + 'generate' can suggest it creates a file or an image, use 'make' + +.. function:: graph_to_dot(graph, f, skip_disconnected=True) + + Write a DOT output for the graph to the file-like object *f*. + + If *skip_disconnected* is true, all distributions that are not dependent on + any other distribution are skipped. + + .. XXX why is this not a DepGraph method? + + +Example Usage +------------- + +Depict all dependenciess in the system +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +First, we shall generate a graph of all the distributions on the system +and then create an image out of it using the tools provided by +`Graphviz <http://www.graphviz.org/>`_:: + + from packaging.database import get_distributions + from packaging.depgraph import generate_graph + + dists = list(get_distributions()) + graph = generate_graph(dists) + +It would be interesting to print out the missing requirements. This can be done +as follows:: + + for dist, reqs in graph.missing.items(): + if reqs: + reqs = ' ,'.join(repr(req) for req in reqs) + print('Missing dependencies for %r: %s' % (dist.name, reqs)) + +Example output is: + +.. code-block:: none + + Missing dependencies for 'TurboCheetah': 'Cheetah' + Missing dependencies for 'TurboGears': 'ConfigObj', 'DecoratorTools', 'RuleDispatch' + Missing dependencies for 'jockey': 'PyKDE4.kdecore', 'PyKDE4.kdeui', 'PyQt4.QtCore', 'PyQt4.QtGui' + Missing dependencies for 'TurboKid': 'kid' + Missing dependencies for 'TurboJson: 'DecoratorTools', 'RuleDispatch' + +Now, we proceed with generating a graphical representation of the graph. First +we write it to a file, and then we generate a PNG image using the +:program:`dot` command-line tool:: + + from packaging.depgraph import graph_to_dot + with open('output.dot', 'w') as f: + # only show the interesting distributions, skipping the disconnected ones + graph_to_dot(graph, f, skip_disconnected=True) + +We can create the final picture using: + +.. code-block:: sh + + $ dot -Tpng output.dot > output.png + +An example result is: + +.. figure:: depgraph-output.png + :alt: Example PNG output from packaging.depgraph and dot + +If you want to include egg distributions as well, then the code requires only +one change, namely the line:: + + dists = list(packaging.database.get_distributions()) + +has to be replaced with:: + + dists = list(packaging.database.get_distributions(use_egg_info=True)) + +On many platforms, a richer graph is obtained because at the moment most +distributions are provided in the egg rather than the new standard +``.dist-info`` format. + +.. XXX missing image + + An example of a more involved graph for illustrative reasons can be seen + here: + + .. image:: depgraph_big.png + + +List all dependent distributions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We will list all distributions that are dependent on some given distibution. +This time, egg distributions will be considered as well:: + + import sys + from packaging.database import get_distribution, get_distributions + from packaging.depgraph import dependent_dists + + dists = list(get_distributions(use_egg_info=True)) + dist = get_distribution('bacon', use_egg_info=True) + if dist is None: + sys.exit('No such distribution in the system') + + deps = dependent_dists(dists, dist) + deps = ', '.join(repr(x.name) for x in deps) + print('Distributions depending on %r: %s' % (dist.name, deps)) + +And this is example output: + +.. with the dependency relationships as in the previous section + (depgraph_big) + +.. code-block:: none + + Distributions depending on 'bacon': 'towel-stuff', 'choxie', 'grammar' diff --git a/Doc/library/packaging.dist.rst b/Doc/library/packaging.dist.rst new file mode 100644 index 0000000..25cb62b --- /dev/null +++ b/Doc/library/packaging.dist.rst @@ -0,0 +1,108 @@ +:mod:`packaging.dist` --- The Distribution class +================================================ + +.. module:: packaging.dist + :synopsis: Core Distribution class. + + +This module provides the :class:`Distribution` class, which represents the +module distribution being built/packaged/distributed/installed. + +.. class:: Distribution(arguments) + + A :class:`Distribution` describes how to build, package, distribute and + install a Python project. + + The arguments accepted by the constructor are laid out in the following + table. Some of them will end up in a metadata object, the rest will become + data attributes of the :class:`Distribution` instance. + + .. TODO improve constructor to take a Metadata object + named params? + (i.e. Distribution(metadata, cmdclass, py_modules, etc) + .. TODO also remove obsolete(?) script_name, etc. parameters? see what + py2exe and other tools need + + +--------------------+--------------------------------+-------------------------------------------------------------+ + | argument name | value | type | + +====================+================================+=============================================================+ + | *name* | The name of the project | a string | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *version* | The version number of the | a string | + | | release; see | | + | | :mod:`packaging.version` | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *summary* | A single line describing the | a string | + | | project | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *description* | Longer description of the | a string | + | | project | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *author* | The name of the project author | a string | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *author_email* | The email address of the | a string | + | | project author | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *maintainer* | The name of the current | a string | + | | maintainer, if different from | | + | | the author | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *maintainer_email* | The email address of the | a string | + | | current maintainer, if | | + | | different from the author | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *home_page* | A URL for the proejct | a string | + | | (homepage) | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *download_url* | A URL to download the project | a string | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *packages* | A list of Python packages that | a list of strings | + | | packaging will manipulate | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *py_modules* | A list of Python modules that | a list of strings | + | | packaging will manipulate | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *scripts* | A list of standalone scripts | a list of strings | + | | to be built and installed | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *ext_modules* | A list of Python extensions to | a list of instances of | + | | be built | :class:`packaging.compiler.extension.Extension` | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *classifiers* | A list of categories for the | a list of strings; valid classifiers are listed on `PyPi | + | | distribution | <http://pypi.python.org/pypi?:action=list_classifiers>`_. | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *distclass* | the :class:`Distribution` | a subclass of | + | | class to use | :class:`packaging.dist.Distribution` | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *script_name* | The name of the setup.py | a string | + | | script - defaults to | | + | | ``sys.argv[0]`` | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *script_args* | Arguments to supply to the | a list of strings | + | | setup script | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *options* | default options for the setup | a string | + | | script | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *license* | The license for the | a string | + | | distribution; should be used | | + | | when there is no suitable | | + | | License classifier, or to | | + | | refine a classifier | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *keywords* | Descriptive keywords; used by | a list of strings or a comma-separated string | + | | catalogs such as PyPI | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *platforms* | Platforms compatible with this | a list of strings or a comma-separated string | + | | distribution; should be used | | + | | when there is no suitable | | + | | Platform classifier | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *cmdclass* | A mapping of command names to | a dictionary | + | | :class:`Command` subclasses | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *data_files* | A list of data files to | a list | + | | install | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *package_dir* | A mapping of Python packages | a dictionary | + | | to directory names | | + +--------------------+--------------------------------+-------------------------------------------------------------+ diff --git a/Doc/library/packaging.fancy_getopt.rst b/Doc/library/packaging.fancy_getopt.rst new file mode 100644 index 0000000..199cbcd --- /dev/null +++ b/Doc/library/packaging.fancy_getopt.rst @@ -0,0 +1,75 @@ +:mod:`packaging.fancy_getopt` --- Wrapper around the getopt module +================================================================== + +.. module:: packaging.fancy_getopt + :synopsis: Additional getopt functionality. + + +.. warning:: + This module is deprecated and will be replaced with :mod:`optparse`. + +This module provides a wrapper around the standard :mod:`getopt` module that +provides the following additional features: + +* short and long options are tied together + +* options have help strings, so :func:`fancy_getopt` could potentially create a + complete usage summary + +* options set attributes of a passed-in object + +* boolean options can have "negative aliases" --- e.g. if :option:`--quiet` is + the "negative alias" of :option:`--verbose`, then :option:`--quiet` on the + command line sets *verbose* to false. + +.. function:: fancy_getopt(options, negative_opt, object, args) + + Wrapper function. *options* is a list of ``(long_option, short_option, + help_string)`` 3-tuples as described in the constructor for + :class:`FancyGetopt`. *negative_opt* should be a dictionary mapping option names + to option names, both the key and value should be in the *options* list. + *object* is an object which will be used to store values (see the :meth:`getopt` + method of the :class:`FancyGetopt` class). *args* is the argument list. Will use + ``sys.argv[1:]`` if you pass ``None`` as *args*. + + +.. class:: FancyGetopt(option_table=None) + + The option_table is a list of 3-tuples: ``(long_option, short_option, + help_string)`` + + If an option takes an argument, its *long_option* should have ``'='`` appended; + *short_option* should just be a single character, no ``':'`` in any case. + *short_option* should be ``None`` if a *long_option* doesn't have a + corresponding *short_option*. All option tuples must have long options. + +The :class:`FancyGetopt` class provides the following methods: + + +.. method:: FancyGetopt.getopt(args=None, object=None) + + Parse command-line options in args. Store as attributes on *object*. + + If *args* is ``None`` or not supplied, uses ``sys.argv[1:]``. If *object* is + ``None`` or not supplied, creates a new :class:`OptionDummy` instance, stores + option values there, and returns a tuple ``(args, object)``. If *object* is + supplied, it is modified in place and :func:`getopt` just returns *args*; in + both cases, the returned *args* is a modified copy of the passed-in *args* list, + which is left untouched. + + .. TODO and args returned are? + + +.. method:: FancyGetopt.get_option_order() + + Returns the list of ``(option, value)`` tuples processed by the previous run of + :meth:`getopt` Raises :exc:`RuntimeError` if :meth:`getopt` hasn't been called + yet. + + +.. method:: FancyGetopt.generate_help(header=None) + + Generate help text (a list of strings, one per suggested line of output) from + the option table for this :class:`FancyGetopt` object. + + If supplied, prints the supplied *header* at the top of the help. diff --git a/Doc/library/packaging.install.rst b/Doc/library/packaging.install.rst new file mode 100644 index 0000000..3e00750 --- /dev/null +++ b/Doc/library/packaging.install.rst @@ -0,0 +1,112 @@ +:mod:`packaging.install` --- Installation tools +=============================================== + +.. module:: packaging.install + :synopsis: Download and installation building blocks + + +Packaging provides a set of tools to deal with downloads and installation of +distributions. Their role is to download the distribution from indexes, resolve +the dependencies, and provide a safe way to install distributions. An operation +that fails will cleanly roll back, not leave half-installed distributions on the +system. Here's the basic process followed: + +#. Move all distributions that will be removed to a temporary location. + +#. Install all the distributions that will be installed in a temporary location. + +#. If the installation fails, move the saved distributions back to their + location and delete the installed distributions. + +#. Otherwise, move the installed distributions to the right location and delete + the temporary locations. + +This is a higher-level module built on :mod:`packaging.database` and +:mod:`packaging.pypi`. + + +Public functions +---------------- + +.. function:: get_infos(requirements, index=None, installed=None, \ + prefer_final=True) + + Return information about what's going to be installed and upgraded. + *requirements* is a string containing the requirements for this + project, for example ``'FooBar 1.1'`` or ``'BarBaz (<1.2)'``. + + .. XXX are requirements comma-separated? + + If you want to use another index than the main PyPI, give its URI as *index* + argument. + + *installed* is a list of already installed distributions used to find + satisfied dependencies, obsoleted distributions and eventual conflicts. + + By default, alpha, beta and candidate versions are not picked up. Set + *prefer_final* to false to accept them too. + + The results are returned in a dictionary containing all the information + needed to perform installation of the requirements with the + :func:`install_from_infos` function: + + >>> get_install_info("FooBar (<=1.2)") + {'install': [<FooBar 1.1>], 'remove': [], 'conflict': []} + + .. TODO should return tuple or named tuple, not dict + .. TODO use "predicate" or "requirement" consistently in version and here + .. FIXME "info" cannot be plural in English, s/infos/info/ + + +.. function:: install(project) + + +.. function:: install_dists(dists, path, paths=None) + + Safely install all distributions provided in *dists* into *path*. *paths* is + a list of paths where already-installed distributions will be looked for to + find satisfied dependencies and conflicts (default: :data:`sys.path`). + Returns a list of installed dists. + + .. FIXME dists are instances of what? + + +.. function:: install_from_infos(install_path=None, install=[], remove=[], \ + conflicts=[], paths=None) + + Safely install and remove given distributions. This function is designed to + work with the return value of :func:`get_infos`: *install*, *remove* and + *conflicts* should be list of distributions returned by :func:`get_infos`. + If *install* is not empty, *install_path* must be given to specify the path + where the distributions should be installed. *paths* is a list of paths + where already-installed distributions will be looked for (default: + :data:`sys.path`). + + This function is a very basic installer; if *conflicts* is not empty, the + system will be in a conflicting state after the function completes. It is a + building block for more sophisticated installers with conflict resolution + systems. + + .. TODO document typical value for install_path + .. TODO document integration with default schemes, esp. user site-packages + + +.. function:: install_local_project(path) + + Install a distribution from a source directory, which must contain either a + Packaging-compliant :file:`setup.cfg` file or a legacy Distutils + :file:`setup.py` script (in which case Distutils will be used under the hood + to perform the installation). + + +.. function:: remove(project_name, paths=None, auto_confirm=True) + + Remove one distribution from the system. + + .. FIXME this is the only function using "project" instead of dist/release + +.. + Example usage + -------------- + + Get the scheme of what's gonna be installed if we install "foobar": diff --git a/Doc/library/packaging.metadata.rst b/Doc/library/packaging.metadata.rst new file mode 100644 index 0000000..332d69d --- /dev/null +++ b/Doc/library/packaging.metadata.rst @@ -0,0 +1,122 @@ +:mod:`packaging.metadata` --- Metadata handling +=============================================== + +.. module:: packaging.metadata + :synopsis: Class holding the metadata of a release. + + +.. TODO use sphinx-autogen to generate basic doc from the docstrings + +.. class:: Metadata + + This class can read and write metadata files complying with any of the + defined versions: 1.0 (:PEP:`241`), 1.1 (:PEP:`314`) and 1.2 (:PEP:`345`). It + implements methods to parse Metadata files and write them, and a mapping + interface to its contents. + + The :PEP:`345` implementation supports the micro-language for the environment + markers, and displays warnings when versions that are supposed to be + :PEP:`386`-compliant are violating the specification. + + +Reading metadata +---------------- + +The :class:`Metadata` class can be instantiated +with the path of the metadata file, and provides a dict-like interface to the +values:: + + >>> from packaging.metadata import Metadata + >>> metadata = Metadata('PKG-INFO') + >>> metadata.keys()[:5] + ('Metadata-Version', 'Name', 'Version', 'Platform', 'Supported-Platform') + >>> metadata['Name'] + 'CLVault' + >>> metadata['Version'] + '0.5' + >>> metadata['Requires-Dist'] + ["pywin32; sys.platform == 'win32'", "Sphinx"] + + +The fields that support environment markers can be automatically ignored if +the object is instantiated using the ``platform_dependent`` option. +:class:`Metadata` will interpret in this case +the markers and will automatically remove the fields that are not compliant +with the running environment. Here's an example under Mac OS X. The win32 +dependency we saw earlier is ignored:: + + >>> from packaging.metadata import Metadata + >>> metadata = Metadata('PKG-INFO', platform_dependent=True) + >>> metadata['Requires-Dist'] + ['Sphinx'] + + +If you want to provide your own execution context, let's say to test the +metadata under a particular environment that is not the current environment, +you can provide your own values in the ``execution_context`` option, which +is the dict that may contain one or more keys of the context the micro-language +expects. + +Here's an example, simulating a win32 environment:: + + >>> from packaging.metadata import Metadata + >>> context = {'sys.platform': 'win32'} + >>> metadata = Metadata('PKG-INFO', platform_dependent=True, + ... execution_context=context) + ... + >>> metadata['Requires-Dist'] = ["pywin32; sys.platform == 'win32'", + ... "Sphinx"] + ... + >>> metadata['Requires-Dist'] + ['pywin32', 'Sphinx'] + + +Writing metadata +---------------- + +Writing metadata can be done using the ``write`` method:: + + >>> metadata.write('/to/my/PKG-INFO') + +The class will pick the best version for the metadata, depending on the values +provided. If all the values provided exist in all versions, the class will +use :attr:`PKG_INFO_PREFERRED_VERSION`. It is set by default to 1.0, the most +widespread version. + + +Conflict checking and best version +---------------------------------- + +Some fields in :PEP:`345` have to comply with the version number specification +defined in :PEP:`386`. When they don't comply, a warning is emitted:: + + >>> from packaging.metadata import Metadata + >>> metadata = Metadata() + >>> metadata['Requires-Dist'] = ['Funky (Groovie)'] + "Funky (Groovie)" is not a valid predicate + >>> metadata['Requires-Dist'] = ['Funky (1.2)'] + +See also :mod:`packaging.version`. + + +.. TODO talk about check() + + +:mod:`packaging.markers` --- Environment markers +================================================ + +.. module:: packaging.markers + :synopsis: Micro-language for environment markers + + +This is an implementation of environment markers `as defined in PEP 345 +<http://www.python.org/dev/peps/pep-0345/#environment-markers>`_. It is used +for some metadata fields. + +.. function:: interpret(marker, execution_context=None) + + Interpret a marker and return a boolean result depending on the environment. + Example: + + >>> interpret("python_version > '1.0'") + True diff --git a/Doc/library/packaging.pypi.dist.rst b/Doc/library/packaging.pypi.dist.rst new file mode 100644 index 0000000..aaaaab7 --- /dev/null +++ b/Doc/library/packaging.pypi.dist.rst @@ -0,0 +1,114 @@ +:mod:`packaging.pypi.dist` --- Classes representing query results +================================================================= + +.. module:: packaging.pypi.dist + :synopsis: Classes representing the results of queries to indexes. + + +Information coming from the indexes is held in instances of the classes defined +in this module. + +Keep in mind that each project (eg. FooBar) can have several releases +(eg. 1.1, 1.2, 1.3), and each of these releases can be provided in multiple +distributions (eg. a source distribution, a binary one, etc). + + +ReleaseInfo +----------- + +Each release has a project name, version, metadata, and related distributions. + +This information is stored in :class:`ReleaseInfo` +objects. + +.. class:: ReleaseInfo + + +DistInfo +--------- + +:class:`DistInfo` is a simple class that contains +information related to distributions; mainly the URLs where distributions +can be found. + +.. class:: DistInfo + + +ReleasesList +------------ + +The :mod:`~packaging.pypi.dist` module provides a class which works +with lists of :class:`ReleaseInfo` classes; +used to filter and order results. + +.. class:: ReleasesList + + +Example usage +------------- + +Build a list of releases and order them +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Assuming we have a list of releases:: + + >>> from packaging.pypi.dist import ReleasesList, ReleaseInfo + >>> fb10 = ReleaseInfo("FooBar", "1.0") + >>> fb11 = ReleaseInfo("FooBar", "1.1") + >>> fb11a = ReleaseInfo("FooBar", "1.1a1") + >>> ReleasesList("FooBar", [fb11, fb11a, fb10]) + >>> releases.sort_releases() + >>> releases.get_versions() + ['1.1', '1.1a1', '1.0'] + >>> releases.add_release("1.2a1") + >>> releases.get_versions() + ['1.1', '1.1a1', '1.0', '1.2a1'] + >>> releases.sort_releases() + ['1.2a1', '1.1', '1.1a1', '1.0'] + >>> releases.sort_releases(prefer_final=True) + >>> releases.get_versions() + ['1.1', '1.0', '1.2a1', '1.1a1'] + + +Add distribution related information to releases +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It's easy to add distribution information to releases:: + + >>> from packaging.pypi.dist import ReleasesList, ReleaseInfo + >>> r = ReleaseInfo("FooBar", "1.0") + >>> r.add_distribution("sdist", url="http://example.org/foobar-1.0.tar.gz") + >>> r.dists + {'sdist': FooBar 1.0 sdist} + >>> r['sdist'].url + {'url': 'http://example.org/foobar-1.0.tar.gz', 'hashname': None, 'hashval': + None, 'is_external': True} + + +Getting attributes from the dist objects +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To abstract querying information returned from the indexes, attributes and +release information can be retrieved directly from dist objects. + +For instance, if you have a release instance that does not contain the metadata +attribute, it can be fetched by using the "fetch_metadata" method:: + + >>> r = Release("FooBar", "1.1") + >>> print r.metadata + None # metadata field is actually set to "None" + >>> r.fetch_metadata() + <Metadata for FooBar 1.1> + +.. XXX add proper roles to these constructs + + +It's possible to retrieve a project's releases (`fetch_releases`), +metadata (`fetch_metadata`) and distributions (`fetch_distributions`) using +a similar work flow. + +.. XXX what is possible? + +Internally, this is possible because while retrieving information about +projects, releases or distributions, a reference to the client used is +stored which can be accessed using the objects `_index` attribute. diff --git a/Doc/library/packaging.pypi.rst b/Doc/library/packaging.pypi.rst new file mode 100644 index 0000000..14602ce --- /dev/null +++ b/Doc/library/packaging.pypi.rst @@ -0,0 +1,74 @@ +:mod:`packaging.pypi` --- Interface to projects indexes +======================================================= + +.. module:: packaging.pypi + :synopsis: Low-level and high-level APIs to query projects indexes. + + +Packaging queries PyPI to get information about projects or download them. The +low-level facilities used internally are also part of the public API designed to +be used by other tools. + +The :mod:`packaging.pypi` package provides those facilities, which can be +used to access information about Python projects registered at indexes, the +main one being PyPI, located ad http://pypi.python.org/. + +There is two ways to retrieve data from these indexes: a screen-scraping +interface called the "simple API", and XML-RPC. The first one uses HTML pages +located under http://pypi.python.org/simple/, the second one makes XML-RPC +requests to http://pypi.python.org/pypi/. All functions and classes also work +with other indexes such as mirrors, which typically implement only the simple +interface. + +Packaging provides a class that wraps both APIs to provide full query and +download functionality: :class:`packaging.pypi.client.ClientWrapper`. If you +want more control, you can use the underlying classes +:class:`packaging.pypi.simple.Crawler` and :class:`packaging.pypi.xmlrpc.Client` +to connect to one specific interface. + + +:mod:`packaging.pypi.client` --- High-level query API +===================================================== + +.. module:: packaging.pypi.client + :synopsis: Wrapper around :mod;`packaging.pypi.xmlrpc` and + :mod:`packaging.pypi.simple` to query indexes. + + +This module provides a high-level API to query indexes and search +for releases and distributions. The aim of this module is to choose the best +way to query the API automatically, either using XML-RPC or the simple index, +with a preference toward the latter. + +.. class:: ClientWrapper + + Instances of this class will use the simple interface or XML-RPC requests to + query indexes and return :class:`packaging.pypi.dist.ReleaseInfo` and + :class:`packaging.pypi.dist.ReleasesList` objects. + + .. method:: find_projects + + .. method:: get_release + + .. method:: get_releases + + +:mod:`packaging.pypi.base` --- Base class for index crawlers +============================================================ + +.. module:: packaging.pypi.base + :synopsis: Base class used to implement crawlers. + + +.. class:: BaseClient(prefer_final, prefer_source) + + Base class containing common methods for the index crawlers or clients. One + method is currently defined: + + .. method:: download_distribution(requirements, temp_path=None, \ + prefer_source=None, prefer_final=None) + + Download a distribution from the last release according to the + requirements. If *temp_path* is provided, download to this path, + otherwise, create a temporary directory for the download. If a release is + found, the full path to the downloaded file is returned. diff --git a/Doc/library/packaging.pypi.simple.rst b/Doc/library/packaging.pypi.simple.rst new file mode 100644 index 0000000..f579b18 --- /dev/null +++ b/Doc/library/packaging.pypi.simple.rst @@ -0,0 +1,218 @@ +:mod:`packaging.pypi.simple` --- Crawler using the PyPI "simple" interface +========================================================================== + +.. module:: packaging.pypi.simple + :synopsis: Crawler using the screen-scraping "simple" interface to fetch info + and distributions. + + +The class provided by :mod:`packaging.pypi.simple` can access project indexes +and provide useful information about distributions. PyPI, other indexes and +local indexes are supported. + +You should use this module to search distributions by name and versions, process +index external pages and download distributions. It is not suited for things +that will end up in too long index processing (like "finding all distributions +with a specific version, no matter the name"); use :mod:`packaging.pypi.xmlrpc` +for that. + + +API +--- + +.. class:: Crawler(index_url=DEFAULT_SIMPLE_INDEX_URL, \ + prefer_final=False, prefer_source=True, \ + hosts=('*',), follow_externals=False, \ + mirrors_url=None, mirrors=None, timeout=15, \ + mirrors_max_tries=0) + + *index_url* is the address of the index to use for requests. + + The first two parameters control the query results. *prefer_final* + indicates whether a final version (not alpha, beta or candidate) is to be + preferred over a newer but non-final version (for example, whether to pick + up 1.0 over 2.0a3). It is used only for queries that don't give a version + argument. Likewise, *prefer_source* tells whether to prefer a source + distribution over a binary one, if no distribution argument was prodived. + + Other parameters are related to external links (that is links that go + outside the simple index): *hosts* is a list of hosts allowed to be + processed if *follow_externals* is true (default behavior is to follow all + hosts), *follow_externals* enables or disables following external links + (default is false, meaning disabled). + + The remaining parameters are related to the mirroring infrastructure + defined in :PEP:`381`. *mirrors_url* gives a URL to look on for DNS + records giving mirror adresses; *mirrors* is a list of mirror URLs (see + the PEP). If both *mirrors* and *mirrors_url* are given, *mirrors_url* + will only be used if *mirrors* is set to ``None``. *timeout* is the time + (in seconds) to wait before considering a URL has timed out; + *mirrors_max_tries"* is the number of times to try requesting informations + on mirrors before switching. + + The following methods are defined: + + .. method:: get_distributions(project_name, version) + + Return the distributions found in the index for the given release. + + .. method:: get_metadata(project_name, version) + + Return the metadata found on the index for this project name and + version. Currently downloads and unpacks a distribution to read the + PKG-INFO file. + + .. method:: get_release(requirements, prefer_final=None) + + Return one release that fulfills the given requirements. + + .. method:: get_releases(requirements, prefer_final=None, force_update=False) + + Search for releases and return a + :class:`~packaging.pypi.dist.ReleasesList` object containing the + results. + + .. method:: search_projects(name=None) + + Search the index for projects containing the given name and return a + list of matching names. + + See also the base class :class:`packaging.pypi.base.BaseClient` for inherited + methods. + + +.. data:: DEFAULT_SIMPLE_INDEX_URL + + The address used by default by the crawler class. It is currently + ``'http://a.pypi.python.org/simple/'``, the main PyPI installation. + + + + +Usage Examples +--------------- + +To help you understand how using the `Crawler` class, here are some basic +usages. + +Request the simple index to get a specific distribution +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Supposing you want to scan an index to get a list of distributions for +the "foobar" project. You can use the "get_releases" method for that. +The get_releases method will browse the project page, and return +:class:`ReleaseInfo` objects for each found link that rely on downloads. :: + + >>> from packaging.pypi.simple import Crawler + >>> crawler = Crawler() + >>> crawler.get_releases("FooBar") + [<ReleaseInfo "Foobar 1.1">, <ReleaseInfo "Foobar 1.2">] + + +Note that you also can request the client about specific versions, using version +specifiers (described in `PEP 345 +<http://www.python.org/dev/peps/pep-0345/#version-specifiers>`_):: + + >>> client.get_releases("FooBar < 1.2") + [<ReleaseInfo "FooBar 1.1">, ] + + +`get_releases` returns a list of :class:`ReleaseInfo`, but you also can get the +best distribution that fullfil your requirements, using "get_release":: + + >>> client.get_release("FooBar < 1.2") + <ReleaseInfo "FooBar 1.1"> + + +Download distributions +^^^^^^^^^^^^^^^^^^^^^^ + +As it can get the urls of distributions provided by PyPI, the `Crawler` +client also can download the distributions and put it for you in a temporary +destination:: + + >>> client.download("foobar") + /tmp/temp_dir/foobar-1.2.tar.gz + + +You also can specify the directory you want to download to:: + + >>> client.download("foobar", "/path/to/my/dir") + /path/to/my/dir/foobar-1.2.tar.gz + + +While downloading, the md5 of the archive will be checked, if not matches, it +will try another time, then if fails again, raise `MD5HashDoesNotMatchError`. + +Internally, that's not the Crawler which download the distributions, but the +`DistributionInfo` class. Please refer to this documentation for more details. + + +Following PyPI external links +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default behavior for packaging is to *not* follow the links provided +by HTML pages in the "simple index", to find distributions related +downloads. + +It's possible to tell the PyPIClient to follow external links by setting the +`follow_externals` attribute, on instantiation or after:: + + >>> client = Crawler(follow_externals=True) + +or :: + + >>> client = Crawler() + >>> client.follow_externals = True + + +Working with external indexes, and mirrors +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default `Crawler` behavior is to rely on the Python Package index stored +on PyPI (http://pypi.python.org/simple). + +As you can need to work with a local index, or private indexes, you can specify +it using the index_url parameter:: + + >>> client = Crawler(index_url="file://filesystem/path/") + +or :: + + >>> client = Crawler(index_url="http://some.specific.url/") + + +You also can specify mirrors to fallback on in case the first index_url you +provided doesnt respond, or not correctly. The default behavior for +`Crawler` is to use the list provided by Python.org DNS records, as +described in the :PEP:`381` about mirroring infrastructure. + +If you don't want to rely on these, you could specify the list of mirrors you +want to try by specifying the `mirrors` attribute. It's a simple iterable:: + + >>> mirrors = ["http://first.mirror","http://second.mirror"] + >>> client = Crawler(mirrors=mirrors) + + +Searching in the simple index +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It's possible to search for projects with specific names in the package index. +Assuming you want to find all projects containing the "distutils" keyword:: + + >>> c.search_projects("distutils") + [<Project "collective.recipe.distutils">, <Project "Distutils">, <Project + "Packaging">, <Project "distutilscross">, <Project "lpdistutils">, <Project + "taras.recipe.distutils">, <Project "zerokspot.recipe.distutils">] + + +You can also search the projects starting with a specific text, or ending with +that text, using a wildcard:: + + >>> c.search_projects("distutils*") + [<Project "Distutils">, <Project "Packaging">, <Project "distutilscross">] + + >>> c.search_projects("*distutils") + [<Project "collective.recipe.distutils">, <Project "Distutils">, <Project + "lpdistutils">, <Project "taras.recipe.distutils">, <Project + "zerokspot.recipe.distutils">] diff --git a/Doc/library/packaging.pypi.xmlrpc.rst b/Doc/library/packaging.pypi.xmlrpc.rst new file mode 100644 index 0000000..5242e4c --- /dev/null +++ b/Doc/library/packaging.pypi.xmlrpc.rst @@ -0,0 +1,143 @@ +:mod:`packaging.pypi.xmlrpc` --- Crawler using the PyPI XML-RPC interface +========================================================================= + +.. module:: packaging.pypi.xmlrpc + :synopsis: Client using XML-RPC requests to fetch info and distributions. + + +Indexes can be queried using XML-RPC calls, and Packaging provides a simple +way to interface with XML-RPC. + +You should **use** XML-RPC when: + +* Searching the index for projects **on other fields than project + names**. For instance, you can search for projects based on the + author_email field. +* Searching all the versions that have existed for a project. +* you want to retrieve METADATAs information from releases or + distributions. + + +You should **avoid using** XML-RPC method calls when: + +* Retrieving the last version of a project +* Getting the projects with a specific name and version. +* The simple index can match your needs + + +When dealing with indexes, keep in mind that the index queries will always +return you :class:`packaging.pypi.dist.ReleaseInfo` and +:class:`packaging.pypi.dist.ReleasesList` objects. + +Some methods here share common APIs with the one you can find on +:class:`packaging.pypi.simple`, internally, :class:`packaging.pypi.client` +is inherited by :class:`Client` + + +API +--- + +.. class:: Client + + +Usage examples +-------------- + +Use case described here are use case that are not common to the other clients. +If you want to see all the methods, please refer to API or to usage examples +described in :class:`packaging.pypi.client.Client` + + +Finding releases +^^^^^^^^^^^^^^^^ + +It's a common use case to search for "things" within the index. We can +basically search for projects by their name, which is the most used way for +users (eg. "give me the last version of the FooBar project"). + +This can be accomplished using the following syntax:: + + >>> client = xmlrpc.Client() + >>> client.get_release("Foobar (<= 1.3)) + <FooBar 1.2.1> + >>> client.get_releases("FooBar (<= 1.3)") + [FooBar 1.1, FooBar 1.1.1, FooBar 1.2, FooBar 1.2.1] + + +And we also can find for specific fields:: + + >>> client.search_projects(field=value) + + +You could specify the operator to use, default is "or":: + + >>> client.search_projects(field=value, operator="and") + + +The specific fields you can search are: + +* name +* version +* author +* author_email +* maintainer +* maintainer_email +* home_page +* license +* summary +* description +* keywords +* platform +* download_url + + +Getting metadata information +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +XML-RPC is a preferred way to retrieve metadata information from indexes. +It's really simple to do so:: + + >>> client = xmlrpc.Client() + >>> client.get_metadata("FooBar", "1.1") + <ReleaseInfo FooBar 1.1> + + +Assuming we already have a :class:`packaging.pypi.ReleaseInfo` object defined, +it's possible to pass it to the xmlrpc client to retrieve and complete its +metadata:: + + >>> foobar11 = ReleaseInfo("FooBar", "1.1") + >>> client = xmlrpc.Client() + >>> returned_release = client.get_metadata(release=foobar11) + >>> returned_release + <ReleaseInfo FooBar 1.1> + + +Get all the releases of a project +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To retrieve all the releases for a project, you can build them using +`get_releases`:: + + >>> client = xmlrpc.Client() + >>> client.get_releases("FooBar") + [<ReleaseInfo FooBar 0.9>, <ReleaseInfo FooBar 1.0>, <ReleaseInfo 1.1>] + + +Get information about distributions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Indexes have information about projects, releases **and** distributions. +If you're not familiar with those, please refer to the documentation of +:mod:`packaging.pypi.dist`. + +It's possible to retrieve information about distributions, e.g "what are the +existing distributions for this release ? How to retrieve them ?":: + + >>> client = xmlrpc.Client() + >>> release = client.get_distributions("FooBar", "1.1") + >>> release.dists + {'sdist': <FooBar 1.1 sdist>, 'bdist': <FooBar 1.1 bdist>} + +As you see, this does not return a list of distributions, but a release, +because a release can be used like a list of distributions. diff --git a/Doc/library/packaging.rst b/Doc/library/packaging.rst new file mode 100644 index 0000000..c6bff47 --- /dev/null +++ b/Doc/library/packaging.rst @@ -0,0 +1,75 @@ +:mod:`packaging` --- Packaging support +====================================== + +.. module:: packaging + :synopsis: Packaging system and building blocks for other packaging systems. +.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>, distutils and packaging + contributors + + +The :mod:`packaging` package provides support for building, packaging, +distributing and installing additional projects into a Python installation. +Projects may include Python modules, extension modules, packages and scripts. +:mod:`packaging` also provides building blocks for other packaging systems +that are not tied to the command system. + +This manual is the reference documentation for those standalone building +blocks and for extending Packaging. If you're looking for the user-centric +guides to install a project or package your own code, head to `See also`__. + + +Building blocks +--------------- + +.. toctree:: + :maxdepth: 2 + + packaging-misc + packaging.version + packaging.metadata + packaging.database + packaging.depgraph + packaging.pypi + packaging.pypi.dist + packaging.pypi.simple + packaging.pypi.xmlrpc + packaging.install + + +The command machinery +--------------------- + +.. toctree:: + :maxdepth: 2 + + packaging.dist + packaging.command + packaging.compiler + packaging.fancy_getopt + + +Other utilities +---------------- + +.. toctree:: + :maxdepth: 2 + + packaging.util + packaging.tests.pypi_server + +.. XXX missing: compat config create (dir_util) run pypi.{base,mirrors} + + +.. __: + +.. seealso:: + + :ref:`packaging-index` + The manual for developers of Python projects who want to package and + distribute them. This describes how to use :mod:`packaging` to make + projects easily found and added to an existing Python installation. + + :ref:`packaging-install-index` + A user-centered manual which includes information on adding projects + into an existing Python installation. You do not need to be a Python + programmer to read this manual. diff --git a/Doc/library/packaging.tests.pypi_server.rst b/Doc/library/packaging.tests.pypi_server.rst new file mode 100644 index 0000000..f3b7720 --- /dev/null +++ b/Doc/library/packaging.tests.pypi_server.rst @@ -0,0 +1,105 @@ +:mod:`packaging.tests.pypi_server` --- PyPI mock server +======================================================= + +.. module:: packaging.tests.pypi_server + :synopsis: Mock server used to test PyPI-related modules and commands. + + +When you are testing code that works with Packaging, you might find these tools +useful. + + +The mock server +--------------- + +.. class:: PyPIServer + + PyPIServer is a class that implements an HTTP server running in a separate + thread. All it does is record the requests for further inspection. The recorded + data is available under ``requests`` attribute. The default + HTTP response can be overridden with the ``default_response_status``, + ``default_response_headers`` and ``default_response_data`` attributes. + + By default, when accessing the server with urls beginning with `/simple/`, + the server also record your requests, but will look for files under + the `/tests/pypiserver/simple/` path. + + You can tell the sever to serve static files for other paths. This could be + accomplished by using the `static_uri_paths` parameter, as below:: + + server = PyPIServer(static_uri_paths=["first_path", "second_path"]) + + + You need to create the content that will be served under the + `/tests/pypiserver/default` path. If you want to serve content from another + place, you also can specify another filesystem path (which needs to be under + `tests/pypiserver/`. This will replace the default behavior of the server, and + it will not serve content from the `default` dir :: + + server = PyPIServer(static_filesystem_paths=["path/to/your/dir"]) + + + If you just need to add some paths to the existing ones, you can do as shown, + keeping in mind that the server will always try to load paths in reverse order + (e.g here, try "another/super/path" then the default one) :: + + server = PyPIServer(test_static_path="another/super/path") + server = PyPIServer("another/super/path") + # or + server.static_filesystem_paths.append("another/super/path") + + + As a result of what, in your tests, while you need to use the PyPIServer, in + order to isolates the test cases, the best practice is to place the common files + in the `default` folder, and to create a directory for each specific test case:: + + server = PyPIServer(static_filesystem_paths = ["default", "test_pypi_server"], + static_uri_paths=["simple", "external"]) + + +Base class and decorator for tests +---------------------------------- + +.. class:: PyPIServerTestCase + + ``PyPIServerTestCase`` is a test case class with setUp and tearDown methods that + take care of a single PyPIServer instance attached as a ``pypi`` attribute on + the test class. Use it as one of the base classes in your test case:: + + + class UploadTestCase(PyPIServerTestCase): + + def test_something(self): + cmd = self.prepare_command() + cmd.ensure_finalized() + cmd.repository = self.pypi.full_address + cmd.run() + + environ, request_data = self.pypi.requests[-1] + self.assertEqual(request_data, EXPECTED_REQUEST_DATA) + + +.. decorator:: use_pypi_server + + You also can use a decorator for your tests, if you do not need the same server + instance along all you test case. So, you can specify, for each test method, + some initialisation parameters for the server. + + For this, you need to add a `server` parameter to your method, like this:: + + class SampleTestCase(TestCase): + + @use_pypi_server() + def test_something(self, server): + ... + + + The decorator will instantiate the server for you, and run and stop it just + before and after your method call. You also can pass the server initializer, + just like this:: + + class SampleTestCase(TestCase): + + @use_pypi_server("test_case_name") + def test_something(self, server): + ... diff --git a/Doc/library/packaging.util.rst b/Doc/library/packaging.util.rst new file mode 100644 index 0000000..e628c32 --- /dev/null +++ b/Doc/library/packaging.util.rst @@ -0,0 +1,155 @@ +:mod:`packaging.util` --- Miscellaneous utility functions +========================================================= + +.. module:: packaging.util + :synopsis: Miscellaneous utility functions. + + +This module contains various helpers for the other modules. + +.. XXX a number of functions are missing, but the module may be split first + (it's ginormous right now, some things could go to compat for example) + +.. function:: get_platform() + + Return a string that identifies the current platform. This is used mainly to + distinguish platform-specific build directories and platform-specific built + distributions. Typically includes the OS name and version and the + architecture (as supplied by 'os.uname()'), although the exact information + included depends on the OS; e.g. for IRIX the architecture isn't particularly + important (IRIX only runs on SGI hardware), but for Linux the kernel version + isn't particularly important. + + Examples of returned values: + + * ``linux-i586`` + * ``linux-alpha`` + * ``solaris-2.6-sun4u`` + * ``irix-5.3`` + * ``irix64-6.2`` + + For non-POSIX platforms, currently just returns ``sys.platform``. + + For Mac OS X systems the OS version reflects the minimal version on which + binaries will run (that is, the value of ``MACOSX_DEPLOYMENT_TARGET`` + during the build of Python), not the OS version of the current system. + + For universal binary builds on Mac OS X the architecture value reflects + the univeral binary status instead of the architecture of the current + processor. For 32-bit universal binaries the architecture is ``fat``, + for 64-bit universal binaries the architecture is ``fat64``, and + for 4-way universal binaries the architecture is ``universal``. Starting + from Python 2.7 and Python 3.2 the architecture ``fat3`` is used for + a 3-way universal build (ppc, i386, x86_64) and ``intel`` is used for + a univeral build with the i386 and x86_64 architectures + + Examples of returned values on Mac OS X: + + * ``macosx-10.3-ppc`` + + * ``macosx-10.3-fat`` + + * ``macosx-10.5-universal`` + + * ``macosx-10.6-intel`` + + .. XXX reinvention of platform module? + + +.. function:: convert_path(pathname) + + Return 'pathname' as a name that will work on the native filesystem, i.e. + split it on '/' and put it back together again using the current directory + separator. Needed because filenames in the setup script are always supplied + in Unix style, and have to be converted to the local convention before we + can actually use them in the filesystem. Raises :exc:`ValueError` on + non-Unix-ish systems if *pathname* either starts or ends with a slash. + + +.. function:: change_root(new_root, pathname) + + Return *pathname* with *new_root* prepended. If *pathname* is relative, this + is equivalent to ``os.path.join(new_root,pathname)`` Otherwise, it requires + making *pathname* relative and then joining the two, which is tricky on + DOS/Windows. + + +.. function:: check_environ() + + Ensure that 'os.environ' has all the environment variables we guarantee that + users can use in config files, command-line options, etc. Currently this + includes: + + * :envvar:`HOME` - user's home directory (Unix only) + * :envvar:`PLAT` - description of the current platform, including hardware + and OS (see :func:`get_platform`) + + +.. function:: find_executable(executable, path=None) + + Search the path for a given executable name. + + +.. function:: execute(func, args, msg=None, dry_run=False) + + Perform some action that affects the outside world (for instance, writing to + the filesystem). Such actions are special because they are disabled by the + *dry_run* flag. This method takes care of all that bureaucracy for you; + all you have to do is supply the function to call and an argument tuple for + it (to embody the "external action" being performed), and an optional message + to print. + + +.. function:: newer(source, target) + + Return true if *source* exists and is more recently modified than *target*, + or if *source* exists and *target* doesn't. Return false if both exist and + *target* is the same age or newer than *source*. Raise + :exc:`PackagingFileError` if *source* does not exist. + + +.. function:: strtobool(val) + + Convert a string representation of truth to true (1) or false (0). + + True values are ``y``, ``yes``, ``t``, ``true``, ``on`` and ``1``; false + values are ``n``, ``no``, ``f``, ``false``, ``off`` and ``0``. Raises + :exc:`ValueError` if *val* is anything else. + + +.. function:: byte_compile(py_files, optimize=0, force=0, prefix=None, \ + base_dir=None, dry_run=0, direct=None) + + Byte-compile a collection of Python source files to either :file:`.pyc` or + :file:`.pyo` files in a :file:`__pycache__` subdirectory (see :pep:`3147`), + or to the same directory when using the distutils2 backport on Python + versions older than 3.2. + + *py_files* is a list of files to compile; any files that don't end in + :file:`.py` are silently skipped. *optimize* must be one of the following: + + * ``0`` - don't optimize (generate :file:`.pyc`) + * ``1`` - normal optimization (like ``python -O``) + * ``2`` - extra optimization (like ``python -OO``) + + This function is independent from the running Python's :option:`-O` or + :option:`-B` options; it is fully controlled by the parameters passed in. + + If *force* is true, all files are recompiled regardless of timestamps. + + The source filename encoded in each :term:`bytecode` file defaults to the filenames + listed in *py_files*; you can modify these with *prefix* and *basedir*. + *prefix* is a string that will be stripped off of each source filename, and + *base_dir* is a directory name that will be prepended (after *prefix* is + stripped). You can supply either or both (or neither) of *prefix* and + *base_dir*, as you wish. + + If *dry_run* is true, doesn't actually do anything that would affect the + filesystem. + + Byte-compilation is either done directly in this interpreter process with the + standard :mod:`py_compile` module, or indirectly by writing a temporary + script and executing it. Normally, you should let :func:`byte_compile` + figure out to use direct compilation or not (see the source for details). + The *direct* flag is used by the script generated in indirect mode; unless + you know what you're doing, leave it set to ``None``. diff --git a/Doc/library/packaging.version.rst b/Doc/library/packaging.version.rst new file mode 100644 index 0000000..f36cdab --- /dev/null +++ b/Doc/library/packaging.version.rst @@ -0,0 +1,104 @@ +:mod:`packaging.version` --- Version number classes +=================================================== + +.. module:: packaging.version + :synopsis: Classes that represent project version numbers. + + +This module contains classes and functions useful to deal with version numbers. +It's an implementation of version specifiers `as defined in PEP 345 +<http://www.python.org/dev/peps/pep-0345/#version-specifiers>`_. + + +Version numbers +--------------- + +.. class:: NormalizedVersion(self, s, error_on_huge_major_num=True) + + A specific version of a distribution, as described in PEP 345. *s* is a + string object containing the version number (for example ``'1.2b1'``), + *error_on_huge_major_num* a boolean specifying whether to consider an + apparent use of a year or full date as the major version number an error. + + The rationale for the second argument is that there were projects using years + or full dates as version numbers, which could cause problems with some + packaging systems sorting. + + Instances of this class can be compared and sorted:: + + >>> NormalizedVersion('1.2b1') < NormalizedVersion('1.2') + True + + :class:`NormalizedVersion` is used internally by :class:`VersionPredicate` to + do its work. + + +.. class:: IrrationalVersionError + + Exception raised when an invalid string is given to + :class:`NormalizedVersion`. + + >>> NormalizedVersion("irrational_version_number") + ... + IrrationalVersionError: irrational_version_number + + +.. function:: suggest_normalized_version(s) + + Before standardization in PEP 386, various schemes were in use. Packaging + provides a function to try to convert any string to a valid, normalized + version:: + + >>> suggest_normalized_version('2.1-rc1') + 2.1c1 + + + If :func:`suggest_normalized_version` can't make sense of the given string, + it will return ``None``:: + + >>> print(suggest_normalized_version('not a version')) + None + + +Version predicates +------------------ + +.. class:: VersionPredicate(predicate) + + This class deals with the parsing of field values like + ``ProjectName (>=version)``. + + .. method:: match(version) + + Test if a version number matches the predicate: + + >>> version = VersionPredicate("ProjectName (<1.2, >1.0)") + >>> version.match("1.2.1") + False + >>> version.match("1.1.1") + True + + +Validation helpers +------------------ + +If you want to use :term:`LBYL`-style checks instead of instantiating the +classes and catching :class:`IrrationalVersionError` and :class:`ValueError`, +you can use these functions: + +.. function:: is_valid_version(predicate) + + Check whether the given string is a valid version number. Example of valid + strings: ``'1.2'``, ``'4.2.0.dev4'``, ``'2.5.4.post2'``. + + +.. function:: is_valid_versions(predicate) + + Check whether the given string is a valid value for specifying multiple + versions, such as in the Requires-Python field. Example: ``'2.7, >=3.2'``. + + +.. function:: is_valid_predicate(predicate) + + Check whether the given string is a valid version predicate. Examples: + ``'some.project == 4.5, <= 4.7'``, ``'speciallib (> 1.0, != 1.4.2, < 2.0)'``. diff --git a/Doc/library/pdb.rst b/Doc/library/pdb.rst index 1e9de63..f4e37ac 100644 --- a/Doc/library/pdb.rst +++ b/Doc/library/pdb.rst @@ -38,6 +38,11 @@ of the debugger is:: > <string>(1)?() (Pdb) +.. versionchanged:: 3.3 + Tab-completion via the :mod:`readline` module is available for commands and + command arguments, e.g. the current global and local names are offered as + arguments of the ``print`` command. + :file:`pdb.py` can also be invoked as a script to debug other scripts. For example:: diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index a1f9af2..287a08b 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -287,6 +287,29 @@ The :mod:`pickle` module exports two classes, :class:`Pickler` and See :ref:`pickle-persistent` for details and examples of uses. + .. attribute:: dispatch_table + + A pickler object's dispatch table is a registry of *reduction + functions* of the kind which can be declared using + :func:`copyreg.pickle`. It is a mapping whose keys are classes + and whose values are reduction functions. A reduction function + takes a single argument of the associated class and should + conform to the same interface as a :meth:`~object.__reduce__` + method. + + By default, a pickler object will not have a + :attr:`dispatch_table` attribute, and it will instead use the + global dispatch table managed by the :mod:`copyreg` module. + However, to customize the pickling for a specific pickler object + one can set the :attr:`dispatch_table` attribute to a dict-like + object. Alternatively, if a subclass of :class:`Pickler` has a + :attr:`dispatch_table` attribute then this will be used as the + default dispatch table for instances of that class. + + See :ref:`pickle-dispatch` for usage examples. + + .. versionadded:: 3.3 + .. attribute:: fast Deprecated. Enable fast mode if set to a true value. The fast mode @@ -577,6 +600,44 @@ pickle external objects by reference. .. literalinclude:: ../includes/dbpickle.py +.. _pickle-dispatch: + +Dispatch Tables +^^^^^^^^^^^^^^^ + +If one wants to customize pickling of some classes without disturbing +any other code which depends on pickling, then one can create a +pickler with a private dispatch table. + +The global dispatch table managed by the :mod:`copyreg` module is +available as :data:`copyreg.dispatch_table`. Therefore, one may +choose to use a modified copy of :data:`copyreg.dispatch_table` as a +private dispatch table. + +For example :: + + f = io.BytesIO() + p = pickle.Pickler(f) + p.dispatch_table = copyreg.dispatch_table.copy() + p.dispatch_table[SomeClass] = reduce_SomeClass + +creates an instance of :class:`pickle.Pickler` with a private dispatch +table which handles the ``SomeClass`` class specially. Alternatively, +the code :: + + class MyPickler(pickle.Pickler): + dispatch_table = copyreg.dispatch_table.copy() + dispatch_table[SomeClass] = reduce_SomeClass + f = io.BytesIO() + p = MyPickler(f) + +does the same, but all instances of ``MyPickler`` will by default +share the same dispatch table. The equivalent code using the +:mod:`copyreg` module is :: + + copyreg.pickle(SomeClass, reduce_SomeClass) + f = io.BytesIO() + p = pickle.Pickler(f) .. _pickle-state: diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index 85eca9a..938c270 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -214,6 +214,10 @@ Win95/98 specific preferring :func:`win32pipe.popen`. On Windows NT, :func:`win32pipe.popen` should work; on Windows 9x it hangs due to bugs in the MS C library. + .. deprecated:: 3.3 + This function is obsolete. Use the :mod:`subprocess` module. Check + especially the :ref:`subprocess-replacements` section. + Mac OS Platform --------------- diff --git a/Doc/library/python.rst b/Doc/library/python.rst index b67fbfc..07eadb4 100644 --- a/Doc/library/python.rst +++ b/Doc/library/python.rst @@ -25,4 +25,5 @@ overview: inspect.rst site.rst fpectl.rst + packaging.rst distutils.rst diff --git a/Doc/library/random.rst b/Doc/library/random.rst index 5aa1262..daf7a76 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -43,6 +43,12 @@ The :mod:`random` module also provides the :class:`SystemRandom` class which uses the system function :func:`os.urandom` to generate random numbers from sources provided by the operating system. +.. warning:: + + The generators of the :mod:`random` module should not be used for security + purposes. Use :func:`ssl.RAND_bytes` if you require a cryptographically + secure pseudorandom number generator. + Bookkeeping functions: diff --git a/Doc/library/re.rst b/Doc/library/re.rst index 8397aad..d8ead68 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -684,9 +684,12 @@ form. .. function:: escape(string) - Return *string* with all non-alphanumerics backslashed; this is useful if you - want to match an arbitrary literal string that may have regular expression - metacharacters in it. + Escape all the characters in pattern except ASCII letters, numbers and ``'_'``. + This is useful if you want to match an arbitrary literal string that may + have regular expression metacharacters in it. + + .. versionchanged:: 3.3 + The ``'_'`` character is no longer escaped. .. function:: purge() diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index ab55197..1134619 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -199,7 +199,7 @@ normally be executed automatically during interactive sessions from the user's histfile = os.path.join(os.path.expanduser("~"), ".pyhist") try: readline.read_history_file(histfile) - except IOError: + except FileNotFoundError: pass import atexit atexit.register(readline.write_history_file, histfile) @@ -224,7 +224,7 @@ support history save/restore. :: if hasattr(readline, "read_history_file"): try: readline.read_history_file(histfile) - except IOError: + except FileNotFoundError: pass atexit.register(self.save_history, histfile) diff --git a/Doc/library/resource.rst b/Doc/library/resource.rst index c16b013..03a7cb5 100644 --- a/Doc/library/resource.rst +++ b/Doc/library/resource.rst @@ -14,13 +14,15 @@ resources utilized by a program. Symbolic constants are used to specify particular system resources and to request usage information about either the current process or its children. -A single exception is defined for errors: +An :exc:`OSError` is raised on syscall failure. .. exception:: error - The functions described below may raise this error if the underlying system call - failures unexpectedly. + A deprecated alias of :exc:`OSError`. + + .. versionchanged:: 3.3 + Following :pep:`3151`, this class was made an alias of :exc:`OSError`. Resource Limits diff --git a/Doc/library/sched.rst b/Doc/library/sched.rst index 000dba0..d6c86c7 100644 --- a/Doc/library/sched.rst +++ b/Doc/library/sched.rst @@ -14,7 +14,7 @@ The :mod:`sched` module defines a class which implements a general purpose event scheduler: -.. class:: scheduler(timefunc, delayfunc) +.. class:: scheduler(timefunc=time.time, delayfunc=time.sleep) The :class:`scheduler` class defines a generic interface to scheduling events. It needs two functions to actually deal with the "outside world" --- *timefunc* @@ -25,6 +25,12 @@ scheduler: event is run to allow other threads an opportunity to run in multi-threaded applications. + .. versionchanged:: 3.3 + *timefunc* and *delayfunc* parameters are optional. + .. versionchanged:: 3.3 + :class:`scheduler` class can be safely used in multi-threaded + environments. + Example:: >>> import sched, time @@ -44,33 +50,6 @@ Example:: From print_time 930343700.273 930343700.276 -In multi-threaded environments, the :class:`scheduler` class has limitations -with respect to thread-safety, inability to insert a new task before -the one currently pending in a running scheduler, and holding up the main -thread until the event queue is empty. Instead, the preferred approach -is to use the :class:`threading.Timer` class instead. - -Example:: - - >>> import time - >>> from threading import Timer - >>> def print_time(): - ... print("From print_time", time.time()) - ... - >>> def print_some_times(): - ... print(time.time()) - ... Timer(5, print_time, ()).start() - ... Timer(10, print_time, ()).start() - ... time.sleep(11) # sleep while time-delay events execute - ... print(time.time()) - ... - >>> print_some_times() - 930343690.257 - From print_time 930343695.274 - From print_time 930343700.273 - 930343701.301 - - .. _scheduler-objects: Scheduler Objects @@ -79,26 +58,38 @@ Scheduler Objects :class:`scheduler` instances have the following methods and attributes: -.. method:: scheduler.enterabs(time, priority, action, argument) +.. method:: scheduler.enterabs(time, priority, action, argument=[], kwargs={}) Schedule a new event. The *time* argument should be a numeric type compatible with the return value of the *timefunc* function passed to the constructor. Events scheduled for the same *time* will be executed in the order of their *priority*. - Executing the event means executing ``action(*argument)``. *argument* must be a - sequence holding the parameters for *action*. + Executing the event means executing ``action(*argument, **kwargs)``. + *argument* must be a sequence holding the parameters for *action*. + *kwargs* must be a dictionary holding the keyword parameters for *action*. Return value is an event which may be used for later cancellation of the event (see :meth:`cancel`). + .. versionchanged:: 3.3 + *argument* parameter is optional. -.. method:: scheduler.enter(delay, priority, action, argument) + .. versionadded:: 3.3 + *kwargs* parameter was added. + + +.. method:: scheduler.enter(delay, priority, action, argument=[], kwargs={}) Schedule an event for *delay* more time units. Other than the relative time, the other arguments, the effect and the return value are the same as those for :meth:`enterabs`. + .. versionchanged:: 3.3 + *argument* parameter is optional. + + .. versionadded:: 3.3 + *kwargs* parameter was added. .. method:: scheduler.cancel(event) @@ -111,12 +102,16 @@ Scheduler Objects Return true if the event queue is empty. -.. method:: scheduler.run() +.. method:: scheduler.run(blocking=True) - Run all scheduled events. This function will wait (using the :func:`delayfunc` + Run all scheduled events. This method will wait (using the :func:`delayfunc` function passed to the constructor) for the next event, then execute it and so on until there are no more scheduled events. + If *blocking* is False executes the scheduled events due to expire soonest + (if any) and then return the deadline of the next scheduled call in the + scheduler (if any). + Either *action* or *delayfunc* can raise an exception. In either case, the scheduler will maintain a consistent state and propagate the exception. If an exception is raised by *action*, the event will not be attempted in future calls @@ -127,6 +122,9 @@ Scheduler Objects the calling code is responsible for canceling events which are no longer pertinent. + .. versionadded:: 3.3 + *blocking* parameter was added. + .. attribute:: scheduler.queue Read-only attribute returning a list of upcoming events in the order they diff --git a/Doc/library/select.rst b/Doc/library/select.rst index a450ec2..0e667fe 100644 --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -6,7 +6,8 @@ This module provides access to the :c:func:`select` and :c:func:`poll` functions -available in most operating systems, :c:func:`epoll` available on Linux 2.5+ and +available in most operating systems, :c:func:`devpoll` available on +Solaris and derivatives, :c:func:`epoll` available on Linux 2.5+ and :c:func:`kqueue` available on most BSD. Note that on Windows, it only works for sockets; on other operating systems, it also works for other file types (in particular, on Unix, it works on pipes). @@ -18,17 +19,39 @@ The module defines the following: .. exception:: error - The exception raised when an error occurs. The accompanying value is a pair - containing the numeric error code from :c:data:`errno` and the corresponding - string, as would be printed by the C function :c:func:`perror`. + A deprecated alias of :exc:`OSError`. + .. versionchanged:: 3.3 + Following :pep:`3151`, this class was made an alias of :exc:`OSError`. -.. function:: epoll(sizehint=-1) - (Only supported on Linux 2.5.44 and newer.) Returns an edge polling object, - which can be used as Edge or Level Triggered interface for I/O events; see - section :ref:`epoll-objects` below for the methods supported by epolling - objects. +.. function:: devpoll() + + (Only supported on Solaris and derivatives.) Returns a ``/dev/poll`` + polling object; see section :ref:`devpoll-objects` below for the + methods supported by devpoll objects. + + :c:func:`devpoll` objects are linked to the number of file + descriptors allowed at the time of instantiation. If your program + reduces this value, :c:func:`devpoll` will fail. If your program + increases this value, :c:func:`devpoll` may return an + incomplete list of active file descriptors. + + .. versionadded:: 3.3 + +.. function:: epoll(sizehint=-1, flags=0) + + (Only supported on Linux 2.5.44 and newer.) Return an edge polling object, + which can be used as Edge or Level Triggered interface for I/O + events. *sizehint* is deprecated and completely ignored. *flags* can be set + to :const:`EPOLL_CLOEXEC`, which causes the epoll descriptor to be closed + automatically when :func:`os.execve` is called. See section + :ref:`epoll-objects` below for the methods supported by epolling objects. + + + .. versionchanged:: 3.3 + + Added the *flags* parameter. .. function:: poll() @@ -106,6 +129,74 @@ The module defines the following: .. versionadded:: 3.2 +.. _devpoll-objects: + +``/dev/poll`` Polling Objects +---------------------------------------------- + + http://developers.sun.com/solaris/articles/using_devpoll.html + http://developers.sun.com/solaris/articles/polling_efficient.html + +Solaris and derivatives have ``/dev/poll``. While :c:func:`select` is +O(highest file descriptor) and :c:func:`poll` is O(number of file +descriptors), ``/dev/poll`` is O(active file descriptors). + +``/dev/poll`` behaviour is very close to the standard :c:func:`poll` +object. + + +.. method:: devpoll.register(fd[, eventmask]) + + Register a file descriptor with the polling object. Future calls to the + :meth:`poll` method will then check whether the file descriptor has any pending + I/O events. *fd* can be either an integer, or an object with a :meth:`fileno` + method that returns an integer. File objects implement :meth:`fileno`, so they + can also be used as the argument. + + *eventmask* is an optional bitmask describing the type of events you want to + check for. The constants are the same that with :c:func:`poll` + object. The default value is a combination of the constants :const:`POLLIN`, + :const:`POLLPRI`, and :const:`POLLOUT`. + + .. warning:: + + Registering a file descriptor that's already registered is not an + error, but the result is undefined. The appropiate action is to + unregister or modify it first. This is an important difference + compared with :c:func:`poll`. + + +.. method:: devpoll.modify(fd[, eventmask]) + + This method does an :meth:`unregister` followed by a + :meth:`register`. It is (a bit) more efficient that doing the same + explicitly. + + +.. method:: devpoll.unregister(fd) + + Remove a file descriptor being tracked by a polling object. Just like the + :meth:`register` method, *fd* can be an integer or an object with a + :meth:`fileno` method that returns an integer. + + Attempting to remove a file descriptor that was never registered is + safely ignored. + + +.. method:: devpoll.poll([timeout]) + + Polls the set of registered file descriptors, and returns a possibly-empty list + containing ``(fd, event)`` 2-tuples for the descriptors that have events or + errors to report. *fd* is the file descriptor, and *event* is a bitmask with + bits set for the reported events for that descriptor --- :const:`POLLIN` for + waiting input, :const:`POLLOUT` to indicate that the descriptor can be written + to, and so forth. An empty list indicates that the call timed out and no file + descriptors had any events to report. If *timeout* is given, it specifies the + length of time in milliseconds which the system will wait for events before + returning. If *timeout* is omitted, -1, or :const:`None`, the call will + block until there is an event for this poll object. + + .. _epoll-objects: Edge and Level Trigger Polling (epoll) Objects @@ -165,11 +256,6 @@ Edge and Level Trigger Polling (epoll) Objects Register a fd descriptor with the epoll object. - .. note:: - - Registering a file descriptor that's already registered raises an - IOError -- contrary to :ref:`poll-objects`'s register. - .. method:: epoll.modify(fd, eventmask) diff --git a/Doc/library/shlex.rst b/Doc/library/shlex.rst index 0113fb7..941e090 100644 --- a/Doc/library/shlex.rst +++ b/Doc/library/shlex.rst @@ -34,6 +34,40 @@ The :mod:`shlex` module defines the following functions: passing ``None`` for *s* will read the string to split from standard input. + +.. function:: quote(s) + + Return a shell-escaped version of the string *s*. The returned value is a + string that can safely be used as one token in a shell command line, for + cases where you cannot use a list. + + This idiom would be unsafe:: + + >>> filename = 'somefile; rm -rf ~' + >>> command = 'ls -l {}'.format(filename) + >>> print(command) # executed by a shell: boom! + ls -l somefile; rm -rf ~ + + :func:`quote` lets you plug the security hole:: + + >>> command = 'ls -l {}'.format(quote(filename)) + >>> print(command) + ls -l 'somefile; rm -rf ~' + >>> remote_command = 'ssh home {}'.format(quote(command)) + >>> print(remote_command) + ssh home 'ls -l '"'"'somefile; rm -rf ~'"'"'' + + The quoting is compatible with UNIX shells and with :func:`split`: + + >>> remote_command = split(remote_command) + >>> remote_command + ['ssh', 'home', "ls -l 'somefile; rm -rf ~'"] + >>> command = split(remote_command[-1]) + >>> command + ['ls', '-l', 'somefile; rm -rf ~'] + + .. versionadded:: 3.3 + The :mod:`shlex` module defines the following class: @@ -282,5 +316,4 @@ parsing rules. * EOF is signaled with a :const:`None` value; -* Quoted empty strings (``''``) are allowed; - +* Quoted empty strings (``''``) are allowed. diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 18f6485..27d374a 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -47,45 +47,69 @@ Directory and files operations be copied. -.. function:: copyfile(src, dst) +.. function:: copyfile(src, dst, symlinks=False) Copy the contents (no metadata) of the file named *src* to a file named *dst*. *dst* must be the complete target file name; look at :func:`shutil.copy` for a copy that accepts a target directory path. If *src* and *dst* are the same files, :exc:`Error` is raised. - The destination location must be writable; otherwise, an :exc:`IOError` exception + + The destination location must be writable; otherwise, an :exc:`OSError` exception will be raised. If *dst* already exists, it will be replaced. Special files such as character or block devices and pipes cannot be copied with this function. *src* and *dst* are path names given as strings. + If *symlinks* is true and *src* is a symbolic link, a new symbolic link will + be created instead of copying the file *src* points to. + + .. versionchanged:: 3.3 + :exc:`IOError` used to be raised instead of :exc:`OSError`. + Added *symlinks* argument. -.. function:: copymode(src, dst) + +.. function:: copymode(src, dst, symlinks=False) Copy the permission bits from *src* to *dst*. The file contents, owner, and - group are unaffected. *src* and *dst* are path names given as strings. + group are unaffected. *src* and *dst* are path names given as strings. If + *symlinks* is true, *src* a symbolic link and the operating system supports + modes for symbolic links (for example BSD-based ones), the mode of the link + will be copied. + .. versionchanged:: 3.3 + Added *symlinks* argument. -.. function:: copystat(src, dst) +.. function:: copystat(src, dst, symlinks=False) Copy the permission bits, last access time, last modification time, and flags from *src* to *dst*. The file contents, owner, and group are unaffected. *src* - and *dst* are path names given as strings. + and *dst* are path names given as strings. If *src* and *dst* are both + symbolic links and *symlinks* true, the stats of the link will be copied as + far as the platform allows. + .. versionchanged:: 3.3 + Added *symlinks* argument. -.. function:: copy(src, dst) +.. function:: copy(src, dst, symlinks=False)) Copy the file *src* to the file or directory *dst*. If *dst* is a directory, a file with the same basename as *src* is created (or overwritten) in the directory specified. Permission bits are copied. *src* and *dst* are path - names given as strings. + names given as strings. If *symlinks* is true, symbolic links won't be + followed but recreated instead -- this resembles GNU's :program:`cp -P`. + .. versionchanged:: 3.3 + Added *symlinks* argument. -.. function:: copy2(src, dst) +.. function:: copy2(src, dst, symlinks=False) - Similar to :func:`shutil.copy`, but metadata is copied as well -- in fact, - this is just :func:`shutil.copy` followed by :func:`copystat`. This is - similar to the Unix command :program:`cp -p`. + Similar to :func:`shutil.copy`, but metadata is copied as well. This is + similar to the Unix command :program:`cp -p`. If *symlinks* is true, + symbolic links won't be followed but recreated instead -- this resembles + GNU's :program:`cp -P`. + .. versionchanged:: 3.3 + Added *symlinks* argument, try to copy extended file system attributes + too (currently Linux only). .. function:: ignore_patterns(\*patterns) @@ -103,9 +127,9 @@ Directory and files operations :func:`shutil.copy2`. If *symlinks* is true, symbolic links in the source tree are represented as - symbolic links in the new tree, but the metadata of the original links is NOT - copied; if false or omitted, the contents and metadata of the linked files - are copied to the new tree. + symbolic links in the new tree and the metadata of the original links will + be copied as far as the platform allows; if false or omitted, the contents + and metadata of the linked files are copied to the new tree. When *symlinks* is false, if the file pointed by the symlink doesn't exist, a exception will be added in the list of errors raised in @@ -129,7 +153,7 @@ Directory and files operations If *copy_function* is given, it must be a callable that will be used to copy each file. It will be called with the source path and the destination path as arguments. By default, :func:`shutil.copy2` is used, but any function - that supports the same signature (like :func:`copy`) can be used. + that supports the same signature (like :func:`shutil.copy`) can be used. .. versionchanged:: 3.2 Added the *copy_function* argument to be able to provide a custom copy @@ -139,6 +163,9 @@ Directory and files operations Added the *ignore_dangling_symlinks* argument to silent dangling symlinks errors when *symlinks* is false. + .. versionchanged:: 3.3 + Copy metadata when *symlinks* is false. + .. function:: rmtree(path, ignore_errors=False, onerror=None) @@ -173,7 +200,35 @@ Directory and files operations If the destination is on the current filesystem, then :func:`os.rename` is used. Otherwise, *src* is copied (using :func:`shutil.copy2`) to *dst* and - then removed. + then removed. In case of symlinks, a new symlink pointing to the target of + *src* will be created in or as *dst* and *src* will be removed. + + .. versionchanged:: 3.3 + Added explicit symlink handling for foreign filesystems, thus adapting + it to the behavior of GNU's :program:`mv`. + +.. function:: disk_usage(path) + + Return disk usage statistics about the given path as a :term:`named tuple` + with the attributes *total*, *used* and *free*, which are the amount of + total, used and free space, in bytes. + + .. versionadded:: 3.3 + + Availability: Unix, Windows. + +.. function:: chown(path, user=None, group=None) + + Change owner *user* and/or *group* of the given *path*. + + *user* can be a system user name or a uid; the same applies to *group*. At + least one argument is required. + + See also :func:`os.chown`, the underlying function. + + Availability: Unix. + + .. versionadded:: 3.3 .. exception:: Error @@ -406,3 +461,36 @@ The resulting archive contains:: -rw------- tarek/staff 1675 2008-06-09 13:26:54 ./id_rsa -rw-r--r-- tarek/staff 397 2008-06-09 13:26:54 ./id_rsa.pub -rw-r--r-- tarek/staff 37192 2010-02-06 18:23:10 ./known_hosts + + +Querying the size of the output terminal +---------------------------------------- + +.. versionadded:: 3.3 + +.. function:: get_terminal_size(fallback=(columns, lines)) + + Get the size of the terminal window. + + For each of the two dimensions, the environment variable, ``COLUMNS`` + and ``LINES`` respectively, is checked. If the variable is defined and + the value is a positive integer, it is used. + + When ``COLUMNS`` or ``LINES`` is not defined, which is the common case, + the terminal connected to :data:`sys.__stdout__` is queried + by invoking :func:`os.get_terminal_size`. + + If the terminal size cannot be successfully queried, either because + the system doesn't support querying, or because we are not + connected to a terminal, the value given in ``fallback`` parameter + is used. ``fallback`` defaults to ``(80, 24)`` which is the default + size used by many terminal emulators. + + The value returned is a named tuple of type :class:`os.terminal_size`. + + See also: The Single UNIX Specification, Version 2, + `Other Environment Variables`_. + +.. _`Other Environment Variables`: + http://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html#tag_002_003 + diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index d1cae13..2a472fe 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -44,6 +44,9 @@ This has consequences: signal handlers will be called when the calculation finishes. +.. _signals-and-threads: + + Signals and threads ^^^^^^^^^^^^^^^^^^^ @@ -131,6 +134,28 @@ The variables defined in the :mod:`signal` module are: in user and kernel space. SIGPROF is delivered upon expiration. +.. data:: SIG_BLOCK + + A possible value for the *how* parameter to :func:`pthread_sigmask` + indicating that signals are to be blocked. + + .. versionadded:: 3.3 + +.. data:: SIG_UNBLOCK + + A possible value for the *how* parameter to :func:`pthread_sigmask` + indicating that signals are to be unblocked. + + .. versionadded:: 3.3 + +.. data:: SIG_SETMASK + + A possible value for the *how* parameter to :func:`pthread_sigmask` + indicating that the signal mask is to be replaced. + + .. versionadded:: 3.3 + + The :mod:`signal` module defines one exception: .. exception:: ItimerError @@ -138,7 +163,11 @@ The :mod:`signal` module defines one exception: Raised to signal an error from the underlying :func:`setitimer` or :func:`getitimer` implementation. Expect this error if an invalid interval timer or a negative time is passed to :func:`setitimer`. - This error is a subtype of :exc:`IOError`. + This error is a subtype of :exc:`OSError`. + + .. versionadded:: 3.3 + This error used to be a subtype of :exc:`IOError`, which is now an + alias of :exc:`OSError`. The :mod:`signal` module defines the following functions: @@ -172,6 +201,65 @@ The :mod:`signal` module defines the following functions: will then be called. Returns nothing. Not on Windows. (See the Unix man page :manpage:`signal(2)`.) + See also :func:`sigwait`, :func:`sigwaitinfo`, :func:`sigtimedwait` and + :func:`sigpending`. + + +.. function:: pthread_kill(thread_id, signum) + + Send the signal *signum* to the thread *thread_id*, another thread in the + same process as the caller. The target thread can be executing any code + (Python or not). However, if the target thread is executing the Python + interpreter, the Python signal handlers will be :ref:`executed by the main + thread <signals-and-threads>`. Therefore, the only point of sending a signal to a particular + Python thread would be to force a running system call to fail with + :exc:`InterruptedError`. + + Use :func:`threading.get_ident()` or the :attr:`~threading.Thread.ident` + attribute of :class:`threading.Thread` objects to get a suitable value + for *thread_id*. + + If *signum* is 0, then no signal is sent, but error checking is still + performed; this can be used to check if the target thread is still running. + + Availability: Unix (see the man page :manpage:`pthread_kill(3)` for further + information). + + See also :func:`os.kill`. + + .. versionadded:: 3.3 + + +.. function:: pthread_sigmask(how, mask) + + Fetch and/or change the signal mask of the calling thread. The signal mask + is the set of signals whose delivery is currently blocked for the caller. + Return the old signal mask as a set of signals. + + The behavior of the call is dependent on the value of *how*, as follows. + + * :data:`SIG_BLOCK`: The set of blocked signals is the union of the current + set and the *mask* argument. + * :data:`SIG_UNBLOCK`: The signals in *mask* are removed from the current + set of blocked signals. It is permissible to attempt to unblock a + signal which is not blocked. + * :data:`SIG_SETMASK`: The set of blocked signals is set to the *mask* + argument. + + *mask* is a set of signal numbers (e.g. {:const:`signal.SIGINT`, + :const:`signal.SIGTERM`}). Use ``range(1, signal.NSIG)`` for a full mask + including all signals. + + For example, ``signal.pthread_sigmask(signal.SIG_BLOCK, [])`` reads the + signal mask of the calling thread. + + Availability: Unix. See the man page :manpage:`sigprocmask(3)` and + :manpage:`pthread_sigmask(3)` for further information. + + See also :func:`pause`, :func:`sigpending` and :func:`sigwait`. + + .. versionadded:: 3.3 + .. function:: setitimer(which, seconds[, interval]) @@ -201,13 +289,17 @@ The :mod:`signal` module defines the following functions: .. function:: set_wakeup_fd(fd) - Set the wakeup fd to *fd*. When a signal is received, a ``'\0'`` byte is - written to the fd. This can be used by a library to wakeup a poll or select - call, allowing the signal to be fully processed. + Set the wakeup file descriptor to *fd*. When a signal is received, the + signal number is written as a single byte into the fd. This can be used by + a library to wakeup a poll or select call, allowing the signal to be fully + processed. The old wakeup fd is returned. *fd* must be non-blocking. It is up to the library to remove any bytes before calling poll or select again. + Use for example ``struct.unpack('%uB' % len(data), data)`` to decode the + signal numbers list. + When threads are enabled, this function can only be called from the main thread; attempting to call it from other threads will cause a :exc:`ValueError` exception to be raised. @@ -247,6 +339,73 @@ The :mod:`signal` module defines the following functions: :const:`SIGTERM`. A :exc:`ValueError` will be raised in any other case. +.. function:: sigpending() + + Examine the set of signals that are pending for delivery to the calling + thread (i.e., the signals which have been raised while blocked). Return the + set of the pending signals. + + Availability: Unix (see the man page :manpage:`sigpending(2)` for further + information). + + See also :func:`pause`, :func:`pthread_sigmask` and :func:`sigwait`. + + .. versionadded:: 3.3 + + +.. function:: sigwait(sigset) + + Suspend execution of the calling thread until the delivery of one of the + signals specified in the signal set *sigset*. The function accepts the signal + (removes it from the pending list of signals), and returns the signal number. + + Availability: Unix (see the man page :manpage:`sigwait(3)` for further + information). + + See also :func:`pause`, :func:`pthread_sigmask`, :func:`sigpending`, + :func:`sigwaitinfo` and :func:`sigtimedwait`. + + .. versionadded:: 3.3 + + +.. function:: sigwaitinfo(sigset) + + Suspend execution of the calling thread until the delivery of one of the + signals specified in the signal set *sigset*. The function accepts the + signal and removes it from the pending list of signals. If one of the + signals in *sigset* is already pending for the calling thread, the function + will return immediately with information about that signal. The signal + handler is not called for the delivered signal. The function raises an + :exc:`InterruptedError` if it is interrupted by a signal that is not in + *sigset*. + + The return value is an object representing the data contained in the + :c:type:`siginfo_t` structure, namely: :attr:`si_signo`, :attr:`si_code`, + :attr:`si_errno`, :attr:`si_pid`, :attr:`si_uid`, :attr:`si_status`, + :attr:`si_band`. + + Availability: Unix (see the man page :manpage:`sigwaitinfo(2)` for further + information). + + See also :func:`pause`, :func:`sigwait` and :func:`sigtimedwait`. + + .. versionadded:: 3.3 + + +.. function:: sigtimedwait(sigset, timeout) + + Like :func:`sigwaitinfo`, but takes an additional *timeout* argument + specifying a timeout. If *timeout* is specified as :const:`0`, a poll is + performed. Returns :const:`None` if a timeout occurs. + + Availability: Unix (see the man page :manpage:`sigtimedwait(2)` for further + information). + + See also :func:`pause`, :func:`sigwait` and :func:`sigwaitinfo`. + + .. versionadded:: 3.3 + + .. _signal-example: Example @@ -263,7 +422,7 @@ be sent, and the handler raises an exception. :: def handler(signum, frame): print('Signal handler called with signal', signum) - raise IOError("Couldn't open device!") + raise OSError("Couldn't open device!") # Set the signal handler and a 5-second alarm signal.signal(signal.SIGALRM, handler) diff --git a/Doc/library/site.rst b/Doc/library/site.rst index db96add..b987897 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -16,7 +16,14 @@ import can be suppressed using the interpreter's :option:`-S` option. .. index:: triple: module; search; path Importing this module will append site-specific paths to the module search path -and add a few builtins. +and add a few builtins, unless :option:`-S` was used. In that case, this module +can be safely imported with no automatic modifications to the module search path +or additions to the builtins. To explicitly trigger the usual site-specific +additions, call the :func:`site.main` function. + +.. versionchanged:: 3.3 + Importing the module used to trigger paths manipulation even when using + :option:`-S`. .. index:: pair: site-python; directory @@ -127,10 +134,21 @@ empty, and the path manipulations are skipped; however the import of :func:`getuserbase` hasn't been called yet. Default value is :file:`~/.local` for UNIX and Mac OS X non-framework builds, :file:`~/Library/Python/{X.Y}` for Mac framework builds, and - :file:`{%APPDATA%}\\Python` for Windows. This value is used by Distutils to + :file:`{%APPDATA%}\\Python` for Windows. This value is used by Packaging to compute the installation directories for scripts, data files, Python modules, - etc. for the :ref:`user installation scheme <inst-alt-install-user>`. See - also :envvar:`PYTHONUSERBASE`. + etc. for the :ref:`user installation scheme <packaging-alt-install-user>`. + See also :envvar:`PYTHONUSERBASE`. + + +.. function:: main() + + Adds all the standard site-specific directories to the module search + path. This function is called automatically when this module is imported, + unless the :program:`python` interpreter was started with the :option:`-S` + flag. + + .. versionchanged:: 3.3 + This function used to be called unconditionnally. .. function:: addsitedir(sitedir, known_paths=None) diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index 3101ab7..45c5d6d 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -20,7 +20,7 @@ details of SMTP and ESMTP operation, consult :rfc:`821` (Simple Mail Transfer Protocol) and :rfc:`1869` (SMTP Service Extensions). -.. class:: SMTP(host='', port=0, local_hostname=None[, timeout]) +.. class:: SMTP(host='', port=0, local_hostname=None[, timeout], source_address=None) A :class:`SMTP` instance encapsulates an SMTP connection. It has methods that support a full repertoire of SMTP and ESMTP operations. If the optional @@ -29,13 +29,34 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). raised if the specified host doesn't respond correctly. The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout - setting will be used). + setting will be used). The optional source_address parameter allows to bind to some + specific source address in a machine with multiple network interfaces, + and/or to some specific source TCP port. It takes a 2-tuple (host, port), + for the socket to bind to as its source address before connecting. If + omitted (or if host or port are ``''`` and/or 0 respectively) the OS default + behavior will be used. For normal use, you should only require the initialization/connect, :meth:`sendmail`, and :meth:`quit` methods. An example is included below. + The :class:`SMTP` class supports the :keyword:`with` statement. When used + like this, the SMTP ``QUIT`` command is issued automatically when the + :keyword:`with` statement exits. E.g.:: -.. class:: SMTP_SSL(host='', port=0, local_hostname=None, keyfile=None, certfile=None[, timeout]) + >>> from smtplib import SMTP + >>> with SMTP("domain.org") as smtp: + ... smtp.noop() + ... + (250, b'Ok') + >>> + + .. versionchanged:: 3.3 + Support for the :keyword:`with` statement was added. + + .. versionchanged:: 3.3 + source_address argument was added. + +.. class:: SMTP_SSL(host='', port=0, local_hostname=None, keyfile=None, certfile=None[, timeout], context=None, source_address=None) A :class:`SMTP_SSL` instance behaves exactly the same as instances of :class:`SMTP`. :class:`SMTP_SSL` should be used for situations where SSL is @@ -43,18 +64,33 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). not appropriate. If *host* is not specified, the local host is used. If *port* is zero, the standard SMTP-over-SSL port (465) is used. *keyfile* and *certfile* are also optional, and can contain a PEM formatted private key - and certificate chain file for the SSL connection. The optional *timeout* + and certificate chain file for the SSL connection. *context* also optional, can contain + a SSLContext, and is an alternative to keyfile and certfile; If it is specified both + keyfile and certfile must be None. The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting - will be used). + will be used). The optional source_address parameter allows to bind to some + specific source address in a machine with multiple network interfaces, + and/or to some specific source tcp port. It takes a 2-tuple (host, port), + for the socket to bind to as its source address before connecting. If + omitted (or if host or port are ``''`` and/or 0 respectively) the OS default + behavior will be used. + + .. versionchanged:: 3.3 + *context* was added. + .. versionchanged:: 3.3 + source_address argument was added. -.. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None) + +.. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None, source_address=None) The LMTP protocol, which is very similar to ESMTP, is heavily based on the - standard SMTP client. It's common to use Unix sockets for LMTP, so our :meth:`connect` - method must support that as well as a regular host:port server. To specify a - Unix socket, you must use an absolute path for *host*, starting with a '/'. + standard SMTP client. It's common to use Unix sockets for LMTP, so our + :meth:`connect` method must support that as well as a regular host:port + server. The optional arguments local_hostname and source_address have the + same meaning as that of SMTP client. To specify a Unix socket, you must use + an absolute path for *host*, starting with a '/'. Authentication is supported, using the regular SMTP mechanism. When using a Unix socket, LMTP generally don't support or require any authentication, but your @@ -242,7 +278,7 @@ An :class:`SMTP` instance has the following methods: No suitable authentication method was found. -.. method:: SMTP.starttls(keyfile=None, certfile=None) +.. method:: SMTP.starttls(keyfile=None, certfile=None, context=None) Put the SMTP connection in TLS (Transport Layer Security) mode. All SMTP commands that follow will be encrypted. You should then call :meth:`ehlo` @@ -251,6 +287,9 @@ An :class:`SMTP` instance has the following methods: If *keyfile* and *certfile* are provided, these are passed to the :mod:`socket` module's :func:`ssl` function. + Optional *context* parameter is a :class:`ssl.SSLContext` object; This is an alternative to + using a keyfile and a certfile and if specified both *keyfile* and *certfile* should be None. + If there has been no previous ``EHLO`` or ``HELO`` command this session, this method tries ESMTP ``EHLO`` first. @@ -263,6 +302,9 @@ An :class:`SMTP` instance has the following methods: :exc:`RuntimeError` SSL/TLS support is not available to your Python interpreter. + .. versionchanged:: 3.3 + *context* was added. + .. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index f236d30..0315507 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -40,9 +40,23 @@ Socket families Depending on the system and the build options, various socket families are supported by this module. -Socket addresses are represented as follows: - -- A single string is used for the :const:`AF_UNIX` address family. +The address format required by a particular socket object is automatically +selected based on the address family specified when the socket object was +created. Socket addresses are represented as follows: + +- The address of an :const:`AF_UNIX` socket bound to a file system node + is represented as a string, using the file system encoding and the + ``'surrogateescape'`` error handler (see :pep:`383`). An address in + Linux's abstract namespace is returned as a :class:`bytes` object with + an initial null byte; note that sockets in this namespace can + communicate with normal file system sockets, so programs intended to + run on Linux may need to deal with both types of address. A string or + :class:`bytes` object can be used for either type of address when + passing it as an argument. + + .. versionchanged:: 3.3 + Previously, :const:`AF_UNIX` socket paths were assumed to use UTF-8 + encoding. - A pair ``(host, port)`` is used for the :const:`AF_INET` address family, where *host* is a string representing either a hostname in Internet domain @@ -80,6 +94,19 @@ Socket addresses are represented as follows: If *addr_type* is :const:`TIPC_ADDR_ID`, then *v1* is the node, *v2* is the reference, and *v3* should be set to 0. +- A tuple ``(interface, )`` is used for the :const:`AF_CAN` address family, + where *interface* is a string representing a network interface name like + ``'can0'``. The network interface name ``''`` can be used to receive packets + from all network interfaces of this family. + +- A string or a tuple ``(id, unit)`` is used for the :const:`SYSPROTO_CONTROL` + protocol of the :const:`PF_SYSTEM` family. The string is the name of a + kernel control using a dynamically-assigned ID. The tuple can be used if ID + and unit number of the kernel control are known or if a registered ID is + used. + + .. versionadded:: 3.3 + - Certain other address families (:const:`AF_BLUETOOTH`, :const:`AF_PACKET`) support specific representations. @@ -99,8 +126,9 @@ resolution and/or the host configuration. For deterministic behavior use a numeric address in *host* portion. All errors raise exceptions. The normal exceptions for invalid argument types -and out-of-memory conditions can be raised; errors related to socket or address -semantics raise :exc:`socket.error` or one of its subclasses. +and out-of-memory conditions can be raised; starting from Python 3.3, errors +related to socket or address semantics raise :exc:`OSError` or one of its +subclasses (they used to raise :exc:`socket.error`). Non-blocking mode is supported through :meth:`~socket.setblocking`. A generalization of this based on timeouts is supported through @@ -115,20 +143,15 @@ The module :mod:`socket` exports the following constants and functions: .. exception:: error - .. index:: module: errno - - A subclass of :exc:`IOError`, this exception is raised for socket-related - errors. It is recommended that you inspect its ``errno`` attribute to - discriminate between different kinds of errors. + A deprecated alias of :exc:`OSError`. - .. seealso:: - The :mod:`errno` module contains symbolic names for the error codes - defined by the underlying operating system. + .. versionchanged:: 3.3 + Following :pep:`3151`, this class was made an alias of :exc:`OSError`. .. exception:: herror - A subclass of :exc:`socket.error`, this exception is raised for + A subclass of :exc:`OSError`, this exception is raised for address-related errors, i.e. for functions that use *h_errno* in the POSIX C API, including :func:`gethostbyname_ex` and :func:`gethostbyaddr`. The accompanying value is a pair ``(h_errno, string)`` representing an @@ -136,10 +159,12 @@ The module :mod:`socket` exports the following constants and functions: *string* represents the description of *h_errno*, as returned by the :c:func:`hstrerror` C function. + .. versionchanged:: 3.3 + This class was made a subclass of :exc:`OSError`. .. exception:: gaierror - A subclass of :exc:`socket.error`, this exception is raised for + A subclass of :exc:`OSError`, this exception is raised for address-related errors by :func:`getaddrinfo` and :func:`getnameinfo`. The accompanying value is a pair ``(error, string)`` representing an error returned by a library call. *string* represents the description of @@ -147,15 +172,19 @@ The module :mod:`socket` exports the following constants and functions: numeric *error* value will match one of the :const:`EAI_\*` constants defined in this module. + .. versionchanged:: 3.3 + This class was made a subclass of :exc:`OSError`. .. exception:: timeout - A subclass of :exc:`socket.error`, this exception is raised when a timeout + A subclass of :exc:`OSError`, this exception is raised when a timeout occurs on a socket which has had timeouts enabled via a prior call to :meth:`~socket.settimeout` (or implicitly through :func:`~socket.setdefaulttimeout`). The accompanying value is a string whose value is currently always "timed out". + .. versionchanged:: 3.3 + This class was made a subclass of :exc:`OSError`. .. data:: AF_UNIX AF_INET @@ -198,6 +227,7 @@ The module :mod:`socket` exports the following constants and functions: SOMAXCONN MSG_* SOL_* + SCM_* IPPROTO_* IPPORT_* INADDR_* @@ -215,6 +245,32 @@ The module :mod:`socket` exports the following constants and functions: in the Unix header files are defined; for a few symbols, default values are provided. +.. data:: AF_CAN + PF_CAN + SOL_CAN_* + CAN_* + + Many constants of these forms, documented in the Linux documentation, are + also defined in the socket module. + + Availability: Linux >= 2.6.25. + + .. versionadded:: 3.3 + + +.. data:: AF_RDS + PF_RDS + SOL_RDS + RDS_* + + Many constants of these forms, documented in the Linux documentation, are + also defined in the socket module. + + Availability: Linux >= 2.6.30. + + .. versionadded:: 3.3 + + .. data:: SIO_* RCVALL_* @@ -393,10 +449,15 @@ The module :mod:`socket` exports the following constants and functions: Create a new socket using the given address family, socket type and protocol number. The address family should be :const:`AF_INET` (the default), - :const:`AF_INET6` or :const:`AF_UNIX`. The socket type should be - :const:`SOCK_STREAM` (the default), :const:`SOCK_DGRAM` or perhaps one of the - other ``SOCK_`` constants. The protocol number is usually zero and may be - omitted in that case. + :const:`AF_INET6`, :const:`AF_UNIX`, :const:`AF_CAN` or :const:`AF_RDS`. The + socket type should be :const:`SOCK_STREAM` (the default), + :const:`SOCK_DGRAM`, :const:`SOCK_RAW` or perhaps one of the other ``SOCK_`` + constants. The protocol number is usually zero and may be omitted in that + case or :const:`CAN_RAW` in case the address family is :const:`AF_CAN`. + + .. versionchanged:: 3.3 + The AF_CAN family was added. + The AF_RDS family was added. .. function:: socketpair([family[, type[, proto]]]) @@ -464,7 +525,7 @@ The module :mod:`socket` exports the following constants and functions: Unix manual page :manpage:`inet(3)` for details. If the IPv4 address string passed to this function is invalid, - :exc:`socket.error` will be raised. Note that exactly what is valid depends on + :exc:`OSError` will be raised. Note that exactly what is valid depends on the underlying C implementation of :c:func:`inet_aton`. :func:`inet_aton` does not support IPv6, and :func:`inet_pton` should be used @@ -481,7 +542,7 @@ The module :mod:`socket` exports the following constants and functions: argument. If the byte sequence passed to this function is not exactly 4 bytes in - length, :exc:`socket.error` will be raised. :func:`inet_ntoa` does not + length, :exc:`OSError` will be raised. :func:`inet_ntoa` does not support IPv6, and :func:`inet_ntop` should be used instead for IPv4/v6 dual stack support. @@ -495,7 +556,7 @@ The module :mod:`socket` exports the following constants and functions: Supported values for *address_family* are currently :const:`AF_INET` and :const:`AF_INET6`. If the IP address string *ip_string* is invalid, - :exc:`socket.error` will be raised. Note that exactly what is valid depends on + :exc:`OSError` will be raised. Note that exactly what is valid depends on both the value of *address_family* and the underlying implementation of :c:func:`inet_pton`. @@ -513,11 +574,54 @@ The module :mod:`socket` exports the following constants and functions: Supported values for *address_family* are currently :const:`AF_INET` and :const:`AF_INET6`. If the string *packed_ip* is not the correct length for the specified address family, :exc:`ValueError` will be raised. A - :exc:`socket.error` is raised for errors from the call to :func:`inet_ntop`. + :exc:`OSError` is raised for errors from the call to :func:`inet_ntop`. Availability: Unix (maybe not all platforms). +.. + XXX: Are sendmsg(), recvmsg() and CMSG_*() available on any + non-Unix platforms? The old (obsolete?) 4.2BSD form of the + interface, in which struct msghdr has no msg_control or + msg_controllen members, is not currently supported. + +.. function:: CMSG_LEN(length) + + Return the total length, without trailing padding, of an ancillary + data item with associated data of the given *length*. This value + can often be used as the buffer size for :meth:`~socket.recvmsg` to + receive a single item of ancillary data, but :rfc:`3542` requires + portable applications to use :func:`CMSG_SPACE` and thus include + space for padding, even when the item will be the last in the + buffer. Raises :exc:`OverflowError` if *length* is outside the + permissible range of values. + + Availability: most Unix platforms, possibly others. + + .. versionadded:: 3.3 + + +.. function:: CMSG_SPACE(length) + + Return the buffer size needed for :meth:`~socket.recvmsg` to + receive an ancillary data item with associated data of the given + *length*, along with any trailing padding. The buffer space needed + to receive multiple items is the sum of the :func:`CMSG_SPACE` + values for their associated data lengths. Raises + :exc:`OverflowError` if *length* is outside the permissible range + of values. + + Note that some systems might support ancillary data without + providing this function. Also note that setting the buffer size + using the results of this function may not precisely limit the + amount of ancillary data that can be received, since additional + data may be able to fit into the padding area. + + Availability: most Unix platforms, possibly others. + + .. versionadded:: 3.3 + + .. function:: getdefaulttimeout() Return the default timeout in seconds (float) for new socket objects. A value @@ -533,6 +637,59 @@ The module :mod:`socket` exports the following constants and functions: meanings. +.. function:: sethostname(name) + + Set the machine's hostname to *name*. This will raise a + :exc:`OSError` if you don't have enough rights. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: if_nameindex() + + Return a list of network interface information + (index int, name string) tuples. + :exc:`OSError` if the system call fails. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: if_nametoindex(if_name) + + Return a network interface index number corresponding to an + interface name. + :exc:`OSError` if no interface with the given name exists. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: if_indextoname(if_index) + + Return a network interface name corresponding to a + interface index number. + :exc:`OSError` if no interface with the given index exists. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: fromshare(data) + + Instantiate a socket from data obtained from :meth:`~socket.share`. + The socket is assumed to be in blocking mode. + + Availability: Windows. + + .. versionadded:: 3.3 + + .. data:: SocketType This is a Python type object that represents the socket object type. It is the @@ -706,6 +863,109 @@ correspond to Unix system calls applicable to sockets. to zero. (The format of *address* depends on the address family --- see above.) +.. method:: socket.recvmsg(bufsize[, ancbufsize[, flags]]) + + Receive normal data (up to *bufsize* bytes) and ancillary data from + the socket. The *ancbufsize* argument sets the size in bytes of + the internal buffer used to receive the ancillary data; it defaults + to 0, meaning that no ancillary data will be received. Appropriate + buffer sizes for ancillary data can be calculated using + :func:`CMSG_SPACE` or :func:`CMSG_LEN`, and items which do not fit + into the buffer might be truncated or discarded. The *flags* + argument defaults to 0 and has the same meaning as for + :meth:`recv`. + + The return value is a 4-tuple: ``(data, ancdata, msg_flags, + address)``. The *data* item is a :class:`bytes` object holding the + non-ancillary data received. The *ancdata* item is a list of zero + or more tuples ``(cmsg_level, cmsg_type, cmsg_data)`` representing + the ancillary data (control messages) received: *cmsg_level* and + *cmsg_type* are integers specifying the protocol level and + protocol-specific type respectively, and *cmsg_data* is a + :class:`bytes` object holding the associated data. The *msg_flags* + item is the bitwise OR of various flags indicating conditions on + the received message; see your system documentation for details. + If the receiving socket is unconnected, *address* is the address of + the sending socket, if available; otherwise, its value is + unspecified. + + On some systems, :meth:`sendmsg` and :meth:`recvmsg` can be used to + pass file descriptors between processes over an :const:`AF_UNIX` + socket. When this facility is used (it is often restricted to + :const:`SOCK_STREAM` sockets), :meth:`recvmsg` will return, in its + ancillary data, items of the form ``(socket.SOL_SOCKET, + socket.SCM_RIGHTS, fds)``, where *fds* is a :class:`bytes` object + representing the new file descriptors as a binary array of the + native C :c:type:`int` type. If :meth:`recvmsg` raises an + exception after the system call returns, it will first attempt to + close any file descriptors received via this mechanism. + + Some systems do not indicate the truncated length of ancillary data + items which have been only partially received. If an item appears + to extend beyond the end of the buffer, :meth:`recvmsg` will issue + a :exc:`RuntimeWarning`, and will return the part of it which is + inside the buffer provided it has not been truncated before the + start of its associated data. + + On systems which support the :const:`SCM_RIGHTS` mechanism, the + following function will receive up to *maxfds* file descriptors, + returning the message data and a list containing the descriptors + (while ignoring unexpected conditions such as unrelated control + messages being received). See also :meth:`sendmsg`. :: + + import socket, array + + def recv_fds(sock, msglen, maxfds): + fds = array.array("i") # Array of ints + msg, ancdata, flags, addr = sock.recvmsg(msglen, socket.CMSG_LEN(maxfds * fds.itemsize)) + for cmsg_level, cmsg_type, cmsg_data in ancdata: + if (cmsg_level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS): + # Append data, ignoring any truncated integers at the end. + fds.fromstring(cmsg_data[:len(cmsg_data) - (len(cmsg_data) % fds.itemsize)]) + return msg, list(fds) + + Availability: most Unix platforms, possibly others. + + .. versionadded:: 3.3 + + +.. method:: socket.recvmsg_into(buffers[, ancbufsize[, flags]]) + + Receive normal data and ancillary data from the socket, behaving as + :meth:`recvmsg` would, but scatter the non-ancillary data into a + series of buffers instead of returning a new bytes object. The + *buffers* argument must be an iterable of objects that export + writable buffers (e.g. :class:`bytearray` objects); these will be + filled with successive chunks of the non-ancillary data until it + has all been written or there are no more buffers. The operating + system may set a limit (:func:`~os.sysconf` value ``SC_IOV_MAX``) + on the number of buffers that can be used. The *ancbufsize* and + *flags* arguments have the same meaning as for :meth:`recvmsg`. + + The return value is a 4-tuple: ``(nbytes, ancdata, msg_flags, + address)``, where *nbytes* is the total number of bytes of + non-ancillary data written into the buffers, and *ancdata*, + *msg_flags* and *address* are the same as for :meth:`recvmsg`. + + Example:: + + >>> import socket + >>> s1, s2 = socket.socketpair() + >>> b1 = bytearray(b'----') + >>> b2 = bytearray(b'0123456789') + >>> b3 = bytearray(b'--------------') + >>> s1.send(b'Mary had a little lamb') + 22 + >>> s2.recvmsg_into([b1, memoryview(b2)[2:9], b3]) + (22, [], 0, None) + >>> [b1, b2, b3] + [bytearray(b'Mary'), bytearray(b'01 had a 9'), bytearray(b'little lamb---')] + + Availability: most Unix platforms, possibly others. + + .. versionadded:: 3.3 + + .. method:: socket.recvfrom_into(buffer[, nbytes[, flags]]) Receive data from the socket, writing it into *buffer* instead of creating a @@ -754,6 +1014,41 @@ correspond to Unix system calls applicable to sockets. above.) +.. method:: socket.sendmsg(buffers[, ancdata[, flags[, address]]]) + + Send normal and ancillary data to the socket, gathering the + non-ancillary data from a series of buffers and concatenating it + into a single message. The *buffers* argument specifies the + non-ancillary data as an iterable of buffer-compatible objects + (e.g. :class:`bytes` objects); the operating system may set a limit + (:func:`~os.sysconf` value ``SC_IOV_MAX``) on the number of buffers + that can be used. The *ancdata* argument specifies the ancillary + data (control messages) as an iterable of zero or more tuples + ``(cmsg_level, cmsg_type, cmsg_data)``, where *cmsg_level* and + *cmsg_type* are integers specifying the protocol level and + protocol-specific type respectively, and *cmsg_data* is a + buffer-compatible object holding the associated data. Note that + some systems (in particular, systems without :func:`CMSG_SPACE`) + might support sending only one control message per call. The + *flags* argument defaults to 0 and has the same meaning as for + :meth:`send`. If *address* is supplied and not ``None``, it sets a + destination address for the message. The return value is the + number of bytes of non-ancillary data sent. + + The following function sends the list of file descriptors *fds* + over an :const:`AF_UNIX` socket, on systems which support the + :const:`SCM_RIGHTS` mechanism. See also :meth:`recvmsg`. :: + + import socket, array + + def send_fds(sock, msg, fds): + return sock.sendmsg([msg], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, array.array("i", fds))]) + + Availability: most Unix platforms, possibly others. + + .. versionadded:: 3.3 + + .. method:: socket.setblocking(flag) Set blocking or non-blocking mode of the socket: if *flag* is false, the @@ -795,9 +1090,22 @@ correspond to Unix system calls applicable to sockets. Shut down one or both halves of the connection. If *how* is :const:`SHUT_RD`, further receives are disallowed. If *how* is :const:`SHUT_WR`, further sends are disallowed. If *how* is :const:`SHUT_RDWR`, further sends and receives are - disallowed. Depending on the platform, shutting down one half of the connection - can also close the opposite half (e.g. on Mac OS X, ``shutdown(SHUT_WR)`` does - not allow further reads on the other end of the connection). + disallowed. + + +.. method:: socket.share(process_id) + + :platform: Windows + + Duplacet a socket and prepare it for sharing with a target process. The + target process must be provided with *process_id*. The resulting bytes object + can then be passed to the target process using some form of interprocess + communication and the socket can be recreated there using :func:`fromshare`. + Once this method has been called, it is safe to close the socket since + the operating system has already duplicated it for the target process. + + .. versionadded:: 3.3 + Note that there are no methods :meth:`read` or :meth:`write`; use :meth:`~socket.recv` and :meth:`~socket.send` without *flags* argument instead. @@ -942,13 +1250,13 @@ sends traffic to the first one connected successfully. :: af, socktype, proto, canonname, sa = res try: s = socket.socket(af, socktype, proto) - except socket.error as msg: + except OSError as msg: s = None continue try: s.bind(sa) s.listen(1) - except socket.error as msg: + except OSError as msg: s.close() s = None continue @@ -977,12 +1285,12 @@ sends traffic to the first one connected successfully. :: af, socktype, proto, canonname, sa = res try: s = socket.socket(af, socktype, proto) - except socket.error as msg: + except OSError as msg: s = None continue try: s.connect(sa) - except socket.error as msg: + except OSError as msg: s.close() s = None continue @@ -996,7 +1304,7 @@ sends traffic to the first one connected successfully. :: print('Received', repr(data)) -The last example shows how to write a very simple network sniffer with raw +The next example shows how to write a very simple network sniffer with raw sockets on Windows. The example requires administrator privileges to modify the interface:: @@ -1021,11 +1329,51 @@ the interface:: # disabled promiscuous mode s.ioctl(socket.SIO_RCVALL, socket.RCVALL_OFF) +The last example shows how to use the socket interface to communicate to a CAN +network. This example might require special priviledge:: + + import socket + import struct + + + # CAN frame packing/unpacking (see 'struct can_frame' in <linux/can.h>) + + can_frame_fmt = "=IB3x8s" + can_frame_size = struct.calcsize(can_frame_fmt) + + def build_can_frame(can_id, data): + can_dlc = len(data) + data = data.ljust(8, b'\x00') + return struct.pack(can_frame_fmt, can_id, can_dlc, data) + + def dissect_can_frame(frame): + can_id, can_dlc, data = struct.unpack(can_frame_fmt, frame) + return (can_id, can_dlc, data[:can_dlc]) + + + # create a raw socket and bind it to the 'vcan0' interface + s = socket.socket(socket.AF_CAN, socket.SOCK_RAW, socket.CAN_RAW) + s.bind(('vcan0',)) + + while True: + cf, addr = s.recvfrom(can_frame_size) + + print('Received: can_id=%x, can_dlc=%x, data=%s' % dissect_can_frame(cf)) + + try: + s.send(cf) + except OSError: + print('Error sending CAN frame') + + try: + s.send(build_can_frame(0x01, b'\x01\x02\x03')) + except OSError: + print('Error sending CAN frame') Running an example several times with too small delay between executions, could lead to this error:: - socket.error: [Errno 98] Address already in use + OSError: [Errno 98] Address already in use This is because the previous execution has left the socket in a ``TIME_WAIT`` state, and can't be immediately reused. diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index 5287f17..7dc0cc7 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -154,9 +154,21 @@ Server Objects .. method:: BaseServer.serve_forever(poll_interval=0.5) Handle requests until an explicit :meth:`shutdown` request. - Poll for shutdown every *poll_interval* seconds. Ignores :attr:`self.timeout`. - If you need to do periodic tasks, do them in another thread. + Poll for shutdown every *poll_interval* seconds. Ignores :attr:`self.timeout`. It also calls + :meth:`service_actions` which may be used by a subclass or Mixin to provide + various cleanup actions. For e.g. ForkingMixin class uses + :meth:`service_actions` to cleanup the zombie child processes. + .. versionchanged:: 3.3 + Added service_actions call to the serve_forever method. + + +.. method:: BaseServer.service_actions() + + This is called by the serve_forever loop. This method is can be overridden + by Mixin's to add cleanup or service specific actions. + + .. versionadded:: 3.3 .. method:: BaseServer.shutdown() diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index babbfa5..2418892 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -369,6 +369,22 @@ Connection Objects method with :const:`None` for *handler*. +.. method:: Connection.set_trace_callback(trace_callback) + + Registers *trace_callback* to be called for each SQL statement that is + actually executed by the SQLite backend. + + The only argument passed to the callback is the statement (as string) that + is being executed. The return value of the callback is ignored. Note that + the backend does not only run statements passed to the :meth:`Cursor.execute` + methods. Other sources include the transaction management of the Python + module and the execution of triggers defined in the current database. + + Passing :const:`None` as *trace_callback* will disable the trace callback. + + .. versionadded:: 3.3 + + .. method:: Connection.enable_load_extension(enabled) This routine allows/disallows the SQLite engine to load SQLite extensions @@ -420,10 +436,6 @@ Connection Objects :mod:`sqlite3` module will return Unicode objects for ``TEXT``. If you want to return bytestrings instead, you can set it to :class:`bytes`. - For efficiency reasons, there's also a way to return :class:`str` objects - only for non-ASCII data, and :class:`bytes` otherwise. To activate it, set - this attribute to :const:`sqlite3.OptimizedUnicode`. - You can also set it to any other callable that accepts a single bytestring parameter and returns the resulting object. diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 8cd07d7..8092581 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -53,9 +53,53 @@ Functions, Constants, and Exceptions (currently provided by the OpenSSL library). This signifies some problem in the higher-level encryption and authentication layer that's superimposed on the underlying network connection. This error - is a subtype of :exc:`socket.error`, which in turn is a subtype of - :exc:`IOError`. The error code and message of :exc:`SSLError` instances - are provided by the OpenSSL library. + is a subtype of :exc:`OSError`. The error code and message of + :exc:`SSLError` instances are provided by the OpenSSL library. + + .. versionchanged:: 3.3 + :exc:`SSLError` used to be a subtype of :exc:`socket.error`. + +.. exception:: SSLZeroReturnError + + A subclass of :exc:`SSLError` raised when trying to read or write and + the SSL connection has been closed cleanly. Note that this doesn't + mean that the underlying transport (read TCP) has been closed. + + .. versionadded:: 3.3 + +.. exception:: SSLWantReadError + + A subclass of :exc:`SSLError` raised by a :ref:`non-blocking SSL socket + <ssl-nonblocking>` when trying to read or write data, but more data needs + to be received on the underlying TCP transport before the request can be + fulfilled. + + .. versionadded:: 3.3 + +.. exception:: SSLWantWriteError + + A subclass of :exc:`SSLError` raised by a :ref:`non-blocking SSL socket + <ssl-nonblocking>` when trying to read or write data, but more data needs + to be sent on the underlying TCP transport before the request can be + fulfilled. + + .. versionadded:: 3.3 + +.. exception:: SSLSyscallError + + A subclass of :exc:`SSLError` raised when a system error was encountered + while trying to fulfill an operation on a SSL socket. Unfortunately, + there is no easy way to inspect the original errno number. + + .. versionadded:: 3.3 + +.. exception:: SSLEOFError + + A subclass of :exc:`SSLError` raised when the SSL connection has been + terminated abruptly. Generally, you shouldn't try to reuse the underlying + transport when this error is encountered. + + .. versionadded:: 3.3 .. exception:: CertificateError @@ -161,6 +205,35 @@ instead. Random generation ^^^^^^^^^^^^^^^^^ +.. function:: RAND_bytes(num) + + Returns *num* cryptographically strong pseudo-random bytes. Raises an + :class:`SSLError` if the PRNG has not been seeded with enough data or if the + operation is not supported by the current RAND method. :func:`RAND_status` + can be used to check the status of the PRNG and :func:`RAND_add` can be used + to seed the PRNG. + + Read the Wikipedia article, `Cryptographically secure pseudorandom number + generator (CSPRNG) + <http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator>`_, + to get the requirements of a cryptographically generator. + + .. versionadded:: 3.3 + +.. function:: RAND_pseudo_bytes(num) + + Returns (bytes, is_cryptographic): bytes are *num* pseudo-random bytes, + is_cryptographic is True if the bytes generated are cryptographically + strong. Raises an :class:`SSLError` if the operation is not supported by the + current RAND method. + + Generated pseudo-random byte sequences will be unique if they are of + sufficient length, but are not necessarily unpredictable. They can be used + for non-cryptographic purposes and for certain purposes in cryptographic + protocols, but usually not for key generation etc. + + .. versionadded:: 3.3 + .. function:: RAND_status() Returns True if the SSL pseudo-random number generator has been seeded with @@ -170,7 +243,7 @@ Random generation .. function:: RAND_egd(path) - If you are running an entropy-gathering daemon (EGD) somewhere, and ``path`` + If you are running an entropy-gathering daemon (EGD) somewhere, and *path* is the pathname of a socket connection open to it, this will read 256 bytes of randomness from the socket, and add it to the SSL pseudo-random number generator to increase the security of generated secret keys. This is @@ -181,8 +254,8 @@ Random generation .. function:: RAND_add(bytes, entropy) - Mixes the given ``bytes`` into the SSL pseudo-random number generator. The - parameter ``entropy`` (a float) is a lower bound on the entropy contained in + Mixes the given *bytes* into the SSL pseudo-random number generator. The + parameter *entropy* (a float) is a lower bound on the entropy contained in string (so you can always use :const:`0.0`). See :rfc:`1750` for more information on sources of entropy. @@ -238,6 +311,9 @@ Certificate handling will attempt to validate the server certificate against that set of root certificates, and will fail if the validation attempt fails. + .. versionchanged:: 3.3 + This function is now IPv6-compatible. + .. function:: DER_cert_to_PEM_cert(DER_cert_bytes) Given a certificate as a DER-encoded blob of bytes, returns a PEM-encoded @@ -345,6 +421,46 @@ Constants .. versionadded:: 3.2 +.. data:: OP_CIPHER_SERVER_PREFERENCE + + Use the server's cipher ordering preference, rather than the client's. + This option has no effect on client sockets and SSLv2 server sockets. + + .. versionadded:: 3.3 + +.. data:: OP_SINGLE_DH_USE + + Prevents re-use of the same DH key for distinct SSL sessions. This + improves forward secrecy but requires more computational resources. + This option only applies to server sockets. + + .. versionadded:: 3.3 + +.. data:: OP_SINGLE_ECDH_USE + + Prevents re-use of the same ECDH key for distinct SSL sessions. This + improves forward secrecy but requires more computational resources. + This option only applies to server sockets. + + .. versionadded:: 3.3 + +.. data:: OP_NO_COMPRESSION + + Disable compression on the SSL channel. This is useful if the application + protocol supports its own compression scheme. + + This option is only available with OpenSSL 1.0.0 and later. + + .. versionadded:: 3.3 + +.. data:: HAS_ECDH + + Whether the OpenSSL library has built-in support for Elliptic Curve-based + Diffie-Hellman key exchange. This should be true unless the feature was + explicitly disabled by the distributor. + + .. versionadded:: 3.3 + .. data:: HAS_SNI Whether the OpenSSL library has built-in support for the *Server Name @@ -354,6 +470,23 @@ Constants .. versionadded:: 3.2 +.. data:: HAS_NPN + + Whether the OpenSSL library has built-in support for *Next Protocol + Negotiation* as described in the `NPN draft specification + <http://tools.ietf.org/html/draft-agl-tls-nextprotoneg>`_. When true, + you can use the :meth:`SSLContext.set_npn_protocols` method to advertise + which protocols you want to support. + + .. versionadded:: 3.3 + +.. data:: CHANNEL_BINDING_TYPES + + List of supported TLS channel binding types. Strings in this list + can be used as arguments to :meth:`SSLSocket.get_channel_binding`. + + .. versionadded:: 3.3 + .. data:: OPENSSL_VERSION The version string of the OpenSSL library loaded by the interpreter:: @@ -463,6 +596,37 @@ SSL sockets also have the following additional methods and attributes: version of the SSL protocol that defines its use, and the number of secret bits being used. If no connection has been established, returns ``None``. +.. method:: SSLSocket.compression() + + Return the compression algorithm being used as a string, or ``None`` + if the connection isn't compressed. + + If the higher-level protocol supports its own compression mechanism, + you can use :data:`OP_NO_COMPRESSION` to disable SSL-level compression. + + .. versionadded:: 3.3 + +.. method:: SSLSocket.get_channel_binding(cb_type="tls-unique") + + Get channel binding data for current connection, as a bytes object. Returns + ``None`` if not connected or the handshake has not been completed. + + The *cb_type* parameter allow selection of the desired channel binding + type. Valid channel binding types are listed in the + :data:`CHANNEL_BINDING_TYPES` list. Currently only the 'tls-unique' channel + binding, defined by :rfc:`5929`, is supported. :exc:`ValueError` will be + raised if an unsupported channel binding type is requested. + + .. versionadded:: 3.3 + +.. method:: SSLSocket.selected_npn_protocol() + + Returns the protocol that was selected during the TLS/SSL handshake. If + :meth:`SSLContext.set_npn_protocols` was not called, or if the other party + does not support NPN, or if the handshake has not yet happened, this will + return ``None``. + + .. versionadded:: 3.3 .. method:: SSLSocket.unwrap() @@ -472,7 +636,6 @@ SSL sockets also have the following additional methods and attributes: returned socket should always be used for further communication with the other side of the connection, rather than the original socket. - .. attribute:: SSLSocket.context The :class:`SSLContext` object this SSL socket is tied to. If the SSL @@ -502,7 +665,7 @@ to speed up repeated connections from the same clients. :class:`SSLContext` objects have the following methods and attributes: -.. method:: SSLContext.load_cert_chain(certfile, keyfile=None) +.. method:: SSLContext.load_cert_chain(certfile, keyfile=None, password=None) Load a private key and the corresponding certificate. The *certfile* string must be the path to a single file in PEM format containing the @@ -513,9 +676,25 @@ to speed up repeated connections from the same clients. :ref:`ssl-certificates` for more information on how the certificate is stored in the *certfile*. + The *password* argument may be a function to call to get the password for + decrypting the private key. It will only be called if the private key is + encrypted and a password is necessary. It will be called with no arguments, + and it should return a string, bytes, or bytearray. If the return value is + a string it will be encoded as UTF-8 before using it to decrypt the key. + Alternatively a string, bytes, or bytearray value may be supplied directly + as the *password* argument. It will be ignored if the private key is not + encrypted and no password is needed. + + If the *password* argument is not specified and a password is required, + OpenSSL's built-in password prompting mechanism will be used to + interactively prompt the user for a password. + An :class:`SSLError` is raised if the private key doesn't match with the certificate. + .. versionchanged:: 3.3 + New optional argument *password*. + .. method:: SSLContext.load_verify_locations(cafile=None, capath=None) Load a set of "certification authority" (CA) certificates used to validate @@ -554,6 +733,53 @@ to speed up repeated connections from the same clients. when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will give the currently selected cipher. +.. method:: SSLContext.set_npn_protocols(protocols) + + Specify which protocols the socket should avertise during the SSL/TLS + handshake. It should be a list of strings, like ``['http/1.1', 'spdy/2']``, + ordered by preference. The selection of a protocol will happen during the + handshake, and will play out according to the `NPN draft specification + <http://tools.ietf.org/html/draft-agl-tls-nextprotoneg>`_. After a + successful handshake, the :meth:`SSLSocket.selected_npn_protocol` method will + return the agreed-upon protocol. + + This method will raise :exc:`NotImplementedError` if :data:`HAS_NPN` is + False. + + .. versionadded:: 3.3 + +.. method:: SSLContext.load_dh_params(dhfile) + + Load the key generation parameters for Diffie-Helman (DH) key exchange. + Using DH key exchange improves forward secrecy at the expense of + computational resources (both on the server and on the client). + The *dhfile* parameter should be the path to a file containing DH + parameters in PEM format. + + This setting doesn't apply to client sockets. You can also use the + :data:`OP_SINGLE_DH_USE` option to further improve security. + + .. versionadded:: 3.3 + +.. method:: SSLContext.set_ecdh_curve(curve_name) + + Set the curve name for Elliptic Curve-based Diffie-Hellman (ECDH) key + exchange. ECDH is significantly faster than regular DH while arguably + as secure. The *curve_name* parameter should be a string describing + a well-known elliptic curve, for example ``prime256v1`` for a widely + supported curve. + + This setting doesn't apply to client sockets. You can also use the + :data:`OP_SINGLE_ECDH_USE` option to further improve security. + + This method is not available if :data:`HAS_ECDH` is False. + + .. versionadded:: 3.3 + + .. seealso:: + `SSL/TLS & Perfect Forward Secrecy <http://vincent.bernat.im/en/blog/2011-ssl-perfect-forward-secrecy.html>`_ + Vincent Bernat. + .. method:: SSLContext.wrap_socket(sock, server_side=False, \ do_handshake_on_connect=True, suppress_ragged_eofs=True, \ server_hostname=None) @@ -968,13 +1194,10 @@ to be aware of: try: sock.do_handshake() break - except ssl.SSLError as err: - if err.args[0] == ssl.SSL_ERROR_WANT_READ: - select.select([sock], [], []) - elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE: - select.select([], [sock], []) - else: - raise + except ssl.SSLWantReadError: + select.select([sock], [], []) + except ssl.SSLWantWriteError: + select.select([], [sock], []) .. _ssl-security: diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index 7de98b6..f47f464 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -104,6 +104,16 @@ Example:: if __name__ == '__main__': walktree(sys.argv[1], visitfile) +An additional utility function is provided to covert a file's mode in a human +readable string: + +.. function:: filemode(mode) + + Convert a file's mode to a string of the form '-rwxrwxrwx'. + + .. versionadded:: 3.3 + + All the variables below are simply symbolic indexes into the 10-tuple returned by :func:`os.stat`, :func:`os.fstat` or :func:`os.lstat`. @@ -344,4 +354,3 @@ The following flags can be used in the *flags* argument of :func:`os.chflags`: The file is a snapshot file. See the \*BSD or Mac OS systems man page :manpage:`chflags(2)` for more information. - diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index b89387f..bf86d8eb 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -15,6 +15,10 @@ interpreter. The principal built-in types are numerics, sequences, mappings, classes, instances and exceptions. +Some collection classes are mutable. The methods that add, subtract, or +rearrange their members in place, and don't return a specific item, never return +the collection instance itself but ``None``. + Some operations are supported by several object types; in particular, practically all objects can be compared, tested for truth value, and converted to a string (with the :func:`repr` function or the slightly different @@ -641,30 +645,30 @@ made available to Python as the :attr:`modulus` attribute of Here are the rules in detail: - - If ``x = m / n`` is a nonnegative rational number and ``n`` is not divisible - by ``P``, define ``hash(x)`` as ``m * invmod(n, P) % P``, where ``invmod(n, - P)`` gives the inverse of ``n`` modulo ``P``. +- If ``x = m / n`` is a nonnegative rational number and ``n`` is not divisible + by ``P``, define ``hash(x)`` as ``m * invmod(n, P) % P``, where ``invmod(n, + P)`` gives the inverse of ``n`` modulo ``P``. - - If ``x = m / n`` is a nonnegative rational number and ``n`` is - divisible by ``P`` (but ``m`` is not) then ``n`` has no inverse - modulo ``P`` and the rule above doesn't apply; in this case define - ``hash(x)`` to be the constant value ``sys.hash_info.inf``. +- If ``x = m / n`` is a nonnegative rational number and ``n`` is + divisible by ``P`` (but ``m`` is not) then ``n`` has no inverse + modulo ``P`` and the rule above doesn't apply; in this case define + ``hash(x)`` to be the constant value ``sys.hash_info.inf``. - - If ``x = m / n`` is a negative rational number define ``hash(x)`` - as ``-hash(-x)``. If the resulting hash is ``-1``, replace it with - ``-2``. +- If ``x = m / n`` is a negative rational number define ``hash(x)`` + as ``-hash(-x)``. If the resulting hash is ``-1``, replace it with + ``-2``. - - The particular values ``sys.hash_info.inf``, ``-sys.hash_info.inf`` - and ``sys.hash_info.nan`` are used as hash values for positive - infinity, negative infinity, or nans (respectively). (All hashable - nans have the same hash value.) +- The particular values ``sys.hash_info.inf``, ``-sys.hash_info.inf`` + and ``sys.hash_info.nan`` are used as hash values for positive + infinity, negative infinity, or nans (respectively). (All hashable + nans have the same hash value.) - - For a :class:`complex` number ``z``, the hash values of the real - and imaginary parts are combined by computing ``hash(z.real) + - sys.hash_info.imag * hash(z.imag)``, reduced modulo - ``2**sys.hash_info.width`` so that it lies in - ``range(-2**(sys.hash_info.width - 1), 2**(sys.hash_info.width - - 1))``. Again, if the result is ``-1``, it's replaced with ``-2``. +- For a :class:`complex` number ``z``, the hash values of the real + and imaginary parts are combined by computing ``hash(z.real) + + sys.hash_info.imag * hash(z.imag)``, reduced modulo + ``2**sys.hash_info.width`` so that it lies in + ``range(-2**(sys.hash_info.width - 1), 2**(sys.hash_info.width - + 1))``. Again, if the result is ``-1``, it's replaced with ``-2``. To clarify the above rules, here's some example Python code, @@ -999,6 +1003,23 @@ functions based on regular expressions. rest lowercased. +.. method:: str.casefold() + + Return a casefolded copy of the string. Casefolded strings may be used for + caseless matching. + + Casefolding is similar to lowercasing but more aggressive because it is + intended to remove all case distinctions in a string. For example, the German + lowercase letter ``'ß'`` is equivalent to ``"ss"``. Since it is already + lowercase, :meth:`lower` would do nothing to ``'ß'``; :meth:`casefold` + converts it to ``"ss"``. + + The casefolding algorithm is described in section 3.13 of the Unicode + Standard. + + .. versionadded:: 3.3 + + .. method:: str.center(width[, fillchar]) Return centered in a string of length *width*. Padding is done using the @@ -1206,6 +1227,9 @@ functions based on regular expressions. Return a copy of the string with all the cased characters [4]_ converted to lowercase. + The lowercasing algorithm used is described in section 3.13 of the Unicode + Standard. + .. method:: str.lstrip([chars]) @@ -1278,7 +1302,7 @@ functions based on regular expressions. two empty strings, followed by the string itself. -.. method:: str.rsplit([sep[, maxsplit]]) +.. method:: str.rsplit(sep=None, maxsplit=-1) Return a list of the words in the string, using *sep* as the delimiter string. If *maxsplit* is given, at most *maxsplit* splits are done, the *rightmost* @@ -1300,7 +1324,7 @@ functions based on regular expressions. 'mississ' -.. method:: str.split([sep[, maxsplit]]) +.. method:: str.split(sep=None, maxsplit=-1) Return a list of the words in the string, using *sep* as the delimiter string. If *maxsplit* is given, at most *maxsplit* splits are done (thus, @@ -1357,7 +1381,8 @@ functions based on regular expressions. .. method:: str.swapcase() Return a copy of the string with uppercase characters converted to lowercase and - vice versa. + vice versa. Note that it is not necessarily true that + ``s.swapcase().swapcase() == s``. .. method:: str.title() @@ -1408,7 +1433,11 @@ functions based on regular expressions. Return a copy of the string with all the cased characters [4]_ converted to uppercase. Note that ``str.upper().isupper()`` might be ``False`` if ``s`` contains uncased characters or if the Unicode category of the resulting - character(s) is not "Lu" (Letter, uppercase), but e.g. "Lt" (Letter, titlecase). + character(s) is not "Lu" (Letter, uppercase), but e.g. "Lt" (Letter, + titlecase). + + The uppercasing algorithm used is described in section 3.13 of the Unicode + Standard. .. method:: str.zfill(width) @@ -1635,14 +1664,15 @@ iteration, the :func:`len` function, and the following methods: Return the number of *i*'s for which ``s[i] == x``. - .. versionadded:: 3.2 + .. versionadded:: 3.2 .. method:: range.index(x) Return the smallest *i* such that ``s[i] == x``. Raises :exc:`ValueError` when *x* is not in the range. - .. versionadded:: 3.2 + .. versionadded:: 3.2 + .. _typesseq-mutable: @@ -1673,6 +1703,8 @@ Note that while lists allow their items to be of any type, bytearray object single: append() (sequence method) single: extend() (sequence method) single: count() (sequence method) + single: clear() (sequence method) + single: copy() (sequence method) single: index() (sequence method) single: insert() (sequence method) single: pop() (sequence method) @@ -1704,6 +1736,12 @@ Note that while lists allow their items to be of any type, bytearray object | ``s.extend(x)`` | same as ``s[len(s):len(s)] = | \(2) | | | x`` | | +------------------------------+--------------------------------+---------------------+ +| ``s.clear()`` | remove all items from ``s`` | | +| | | | ++------------------------------+--------------------------------+---------------------+ +| ``s.copy()`` | return a shallow copy of ``s`` | | +| | | | ++------------------------------+--------------------------------+---------------------+ | ``s.count(x)`` | return number of *i*'s for | | | | which ``s[i] == x`` | | +------------------------------+--------------------------------+---------------------+ @@ -1782,6 +1820,9 @@ Notes: (8) :meth:`sort` is not supported by :class:`bytearray` objects. + .. versionadded:: 3.3 + :meth:`clear` and :meth:`!copy` methods. + .. _bytes-methods: @@ -1799,6 +1840,12 @@ the objects to strings, they have a :func:`decode` method. Wherever one of these methods needs to interpret the bytes as characters (e.g. the :func:`is...` methods), the ASCII character set is assumed. +.. versionadded:: 3.3 + The functions :func:`count`, :func:`find`, :func:`index`, + :func:`rfind` and :func:`rindex` have additional semantics compared to + the corresponding string functions: They also accept an integer in + range 0 to 255 (a byte) as their first argument. + .. note:: The methods on bytes and bytearray objects don't accept strings as their @@ -2208,13 +2255,13 @@ pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: .. method:: items() - Return a new view of the dictionary's items (``(key, value)`` pairs). See - below for documentation of view objects. + Return a new view of the dictionary's items (``(key, value)`` pairs). + See the :ref:`documentation of view objects <dict-views>`. .. method:: keys() - Return a new view of the dictionary's keys. See below for documentation of - view objects. + Return a new view of the dictionary's keys. See the :ref:`documentation + of view objects <dict-views>`. .. method:: pop(key[, default]) @@ -2248,8 +2295,12 @@ pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: .. method:: values() - Return a new view of the dictionary's values. See below for documentation of - view objects. + Return a new view of the dictionary's values. See the + :ref:`documentation of view objects <dict-views>`. + +.. seealso:: + :class:`types.MappingProxyType` can be used to create a read-only view + of a :class:`dict`. .. _dict-views: @@ -2337,7 +2388,7 @@ memoryview type :class:`memoryview` objects allow Python code to access the internal data of an object that supports the :ref:`buffer protocol <bufferobjects>` without -copying. Memory is generally interpreted as simple bytes. +copying. .. class:: memoryview(obj) @@ -2351,43 +2402,92 @@ copying. Memory is generally interpreted as simple bytes. is a single byte, but other types such as :class:`array.array` may have bigger elements. - ``len(view)`` returns the total number of elements in the memoryview, - *view*. The :class:`~memoryview.itemsize` attribute will give you the + ``len(view)`` is equal to the length of :class:`~memoryview.tolist`. + If ``view.ndim = 0``, the length is 1. If ``view.ndim = 1``, the length + is equal to the number of elements in the view. For higher dimensions, + the length is equal to the length of the nested list representation of + the view. The :class:`~memoryview.itemsize` attribute will give you the number of bytes in a single element. - A :class:`memoryview` supports slicing to expose its data. Taking a single - index will return a single element as a :class:`bytes` object. Full - slicing will result in a subview:: - - >>> v = memoryview(b'abcefg') - >>> v[1] - b'b' - >>> v[-1] - b'g' - >>> v[1:4] - <memory at 0x77ab28> - >>> bytes(v[1:4]) - b'bce' - - If the object the memoryview is over supports changing its data, the - memoryview supports slice assignment:: + A :class:`memoryview` supports slicing to expose its data. If + :class:`~memoryview.format` is one of the native format specifiers + from the :mod:`struct` module, indexing will return a single element + with the correct type. Full slicing will result in a subview:: + + >>> v = memoryview(b'abcefg') + >>> v[1] + 98 + >>> v[-1] + 103 + >>> v[1:4] + <memory at 0x7f3ddc9f4350> + >>> bytes(v[1:4]) + b'bce' + + Other native formats:: + + >>> import array + >>> a = array.array('l', [-11111111, 22222222, -33333333, 44444444]) + >>> a[0] + -11111111 + >>> a[-1] + 44444444 + >>> a[2:3].tolist() + [-33333333] + >>> a[::2].tolist() + [-11111111, -33333333] + >>> a[::-1].tolist() + [44444444, -33333333, 22222222, -11111111] + + .. versionadded:: 3.3 + + If the underlying object is writable, the memoryview supports slice + assignment. Resizing is not allowed:: >>> data = bytearray(b'abcefg') >>> v = memoryview(data) >>> v.readonly False - >>> v[0] = b'z' + >>> v[0] = ord(b'z') >>> data bytearray(b'zbcefg') >>> v[1:4] = b'123' >>> data bytearray(b'z123fg') - >>> v[2] = b'spam' + >>> v[2:3] = b'spam' Traceback (most recent call last): File "<stdin>", line 1, in <module> - ValueError: cannot modify size of memoryview object + ValueError: memoryview assignment: lvalue and rvalue have different structures + >>> v[2:6] = b'spam' + >>> data + bytearray(b'z1spam') + + Memoryviews of hashable (read-only) types are also hashable. The hash + is defined as ``hash(m) == hash(m.tobytes())``:: + + >>> v = memoryview(b'abcefg') + >>> hash(v) == hash(b'abcefg') + True + >>> hash(v[2:4]) == hash(b'ce') + True + >>> hash(v[::-2]) == hash(b'abcefg'[::-2]) + True + + Hashing of multi-dimensional objects is supported:: + + >>> buf = bytes(list(range(12))) + >>> x = memoryview(buf) + >>> y = x.cast('B', shape=[2,2,3]) + >>> x.tolist() + [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] + >>> y.tolist() + [[[0, 1, 2], [3, 4, 5]], [[6, 7, 8], [9, 10, 11]]] + >>> hash(x) == hash(y) == hash(y.tobytes()) + True + + .. versionchanged:: 3.3 + Memoryview objects are now hashable. - Notice how the size of the memoryview object cannot be changed. :class:`memoryview` has several methods: @@ -2402,12 +2502,20 @@ copying. Memory is generally interpreted as simple bytes. >>> bytes(m) b'abc' + For non-contiguous arrays the result is equal to the flattened list + representation with all elements converted to bytes. + .. method:: tolist() - Return the data in the buffer as a list of integers. :: + Return the data in the buffer as a list of elements. :: >>> memoryview(b'abc').tolist() [97, 98, 99] + >>> import array + >>> a = array.array('d', [1.1, 2.2, 3.3]) + >>> m = memoryview(a) + >>> m.tolist() + [1.1, 2.2, 3.3] .. method:: release() @@ -2434,7 +2542,7 @@ copying. Memory is generally interpreted as simple bytes. >>> with memoryview(b'abc') as m: ... m[0] ... - b'a' + 97 >>> m[0] Traceback (most recent call last): File "<stdin>", line 1, in <module> @@ -2442,45 +2550,219 @@ copying. Memory is generally interpreted as simple bytes. .. versionadded:: 3.2 + .. method:: cast(format[, shape]) + + Cast a memoryview to a new format or shape. *shape* defaults to + ``[byte_length//new_itemsize]``, which means that the result view + will be one-dimensional. The return value is a new memoryview, but + the buffer itself is not copied. Supported casts are 1D -> C-contiguous + and C-contiguous -> 1D. One of the formats must be a byte format + ('B', 'b' or 'c'). The byte length of the result must be the same + as the original length. + + Cast 1D/long to 1D/unsigned bytes:: + + >>> import array + >>> a = array.array('l', [1,2,3]) + >>> x = memoryview(a) + >>> x.format + 'l' + >>> x.itemsize + 8 + >>> len(x) + 3 + >>> x.nbytes + 24 + >>> y = x.cast('B') + >>> y.format + 'B' + >>> y.itemsize + 1 + >>> len(y) + 24 + >>> y.nbytes + 24 + + Cast 1D/unsigned bytes to 1D/char:: + + >>> b = bytearray(b'zyz') + >>> x = memoryview(b) + >>> x[0] = b'a' + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ValueError: memoryview: invalid value for format "B" + >>> y = x.cast('c') + >>> y[0] = b'a' + >>> b + bytearray(b'ayz') + + Cast 1D/bytes to 3D/ints to 1D/signed char:: + + >>> import struct + >>> buf = struct.pack("i"*12, *list(range(12))) + >>> x = memoryview(buf) + >>> y = x.cast('i', shape=[2,2,3]) + >>> y.tolist() + [[[0, 1, 2], [3, 4, 5]], [[6, 7, 8], [9, 10, 11]]] + >>> y.format + 'i' + >>> y.itemsize + 4 + >>> len(y) + 2 + >>> y.nbytes + 48 + >>> z = y.cast('b') + >>> z.format + 'b' + >>> z.itemsize + 1 + >>> len(z) + 48 + >>> z.nbytes + 48 + + Cast 1D/unsigned char to to 2D/unsigned long:: + + >>> buf = struct.pack("L"*6, *list(range(6))) + >>> x = memoryview(buf) + >>> y = x.cast('L', shape=[2,3]) + >>> len(y) + 2 + >>> y.nbytes + 48 + >>> y.tolist() + [[0, 1, 2], [3, 4, 5]] + + .. versionadded:: 3.3 + There are also several readonly attributes available: + .. attribute:: obj + + The underlying object of the memoryview:: + + >>> b = bytearray(b'xyz') + >>> m = memoryview(b) + >>> m.obj is b + True + + .. versionadded:: 3.3 + + .. attribute:: nbytes + + ``nbytes == product(shape) * itemsize == len(m.tobytes())``. This is + the amount of space in bytes that the array would use in a contiguous + representation. It is not necessarily equal to len(m):: + + >>> import array + >>> a = array.array('i', [1,2,3,4,5]) + >>> m = memoryview(a) + >>> len(m) + 5 + >>> m.nbytes + 20 + >>> y = m[::2] + >>> len(y) + 3 + >>> y.nbytes + 12 + >>> len(y.tobytes()) + 12 + + Multi-dimensional arrays:: + + >>> import struct + >>> buf = struct.pack("d"*12, *[1.5*x for x in range(12)]) + >>> x = memoryview(buf) + >>> y = x.cast('d', shape=[3,4]) + >>> y.tolist() + [[0.0, 1.5, 3.0, 4.5], [6.0, 7.5, 9.0, 10.5], [12.0, 13.5, 15.0, 16.5]] + >>> len(y) + 3 + >>> y.nbytes + 96 + + .. versionadded:: 3.3 + + .. attribute:: readonly + + A bool indicating whether the memory is read only. + .. attribute:: format A string containing the format (in :mod:`struct` module style) for each - element in the view. This defaults to ``'B'``, a simple bytestring. + element in the view. A memoryview can be created from exporters with + arbitrary format strings, but some methods (e.g. :meth:`tolist`) are + restricted to native single element formats. Special care must be taken + when comparing memoryviews. Since comparisons are required to return a + value for ``==`` and ``!=``, two memoryviews referencing the same + exporter can compare as not-equal if the exporter's format is not + understood:: + + >>> from ctypes import BigEndianStructure, c_long + >>> class BEPoint(BigEndianStructure): + ... _fields_ = [("x", c_long), ("y", c_long)] + ... + >>> point = BEPoint(100, 200) + >>> a = memoryview(point) + >>> b = memoryview(point) + >>> a == b + False + >>> a.tolist() + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + NotImplementedError: memoryview: unsupported format T{>l:x:>l:y:} .. attribute:: itemsize The size in bytes of each element of the memoryview:: - >>> m = memoryview(array.array('H', [1,2,3])) + >>> import array, struct + >>> m = memoryview(array.array('H', [32000, 32001, 32002])) >>> m.itemsize 2 >>> m[0] - b'\x01\x00' - >>> len(m[0]) == m.itemsize + 32000 + >>> struct.calcsize('H') == m.itemsize True - .. attribute:: shape - - A tuple of integers the length of :attr:`ndim` giving the shape of the - memory as a N-dimensional array. - .. attribute:: ndim An integer indicating how many dimensions of a multi-dimensional array the memory represents. + .. attribute:: shape + + A tuple of integers the length of :attr:`ndim` giving the shape of the + memory as a N-dimensional array. + .. attribute:: strides A tuple of integers the length of :attr:`ndim` giving the size in bytes to access each element for each dimension of the array. - .. attribute:: readonly + .. attribute:: suboffsets - A bool indicating whether the memory is read only. + Used internally for PIL-style arrays. The value is informational only. + + .. attribute:: c_contiguous + + A bool indicating whether the memory is C-contiguous. + + .. versionadded:: 3.3 - .. memoryview.suboffsets isn't documented because it only seems useful for C + .. attribute:: f_contiguous + + A bool indicating whether the memory is Fortran contiguous. + + .. versionadded:: 3.3 + + .. attribute:: contiguous + + A bool indicating whether the memory is contiguous. + + .. versionadded:: 3.3 .. _typecontextmanager: @@ -2704,7 +2986,7 @@ The Null Object This object is returned by functions that don't explicitly return a value. It supports no special operations. There is exactly one null object, named -``None`` (a built-in name). +``None`` (a built-in name). ``type(None)()`` produces the same singleton. It is written as ``None``. @@ -2716,7 +2998,8 @@ The Ellipsis Object This object is commonly used by slicing (see :ref:`slicings`). It supports no special operations. There is exactly one ellipsis object, named -:const:`Ellipsis` (a built-in name). +:const:`Ellipsis` (a built-in name). ``type(Ellipsis)()`` produces the +:const:`Ellipsis` singleton. It is written as ``Ellipsis`` or ``...``. @@ -2728,7 +3011,8 @@ The NotImplemented Object This object is returned from comparisons and binary operations when they are asked to operate on types they don't support. See :ref:`comparisons` for more -information. +information. There is exactly one ``NotImplemented`` object. +``type(NotImplemented)()`` produces the singleton instance. It is written as ``NotImplemented``. @@ -2794,6 +3078,13 @@ types, where they are relevant. Some of these are not reported by the The name of the class or type. +.. attribute:: class.__qualname__ + + The :term:`qualified name` of the class or type. + + .. versionadded:: 3.3 + + .. attribute:: class.__mro__ This attribute is a tuple of classes that are considered when looking for diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst index 12820e0..994506c 100644 --- a/Doc/library/struct.rst +++ b/Doc/library/struct.rst @@ -187,17 +187,24 @@ platform-dependent. | ``Q`` | :c:type:`unsigned long | integer | 8 | \(2), \(3) | | | long` | | | | +--------+--------------------------+--------------------+----------------+------------+ -| ``f`` | :c:type:`float` | float | 4 | \(4) | +| ``n`` | :c:type:`ssize_t` | integer | | \(4) | +--------+--------------------------+--------------------+----------------+------------+ -| ``d`` | :c:type:`double` | float | 8 | \(4) | +| ``N`` | :c:type:`size_t` | integer | | \(4) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``f`` | :c:type:`float` | float | 4 | \(5) | ++--------+--------------------------+--------------------+----------------+------------+ +| ``d`` | :c:type:`double` | float | 8 | \(5) | +--------+--------------------------+--------------------+----------------+------------+ | ``s`` | :c:type:`char[]` | bytes | | | +--------+--------------------------+--------------------+----------------+------------+ | ``p`` | :c:type:`char[]` | bytes | | | +--------+--------------------------+--------------------+----------------+------------+ -| ``P`` | :c:type:`void \*` | integer | | \(5) | +| ``P`` | :c:type:`void \*` | integer | | \(6) | +--------+--------------------------+--------------------+----------------+------------+ +.. versionchanged:: 3.3 + Added support for the ``'n'`` and ``'N'`` formats. + Notes: (1) @@ -219,11 +226,17 @@ Notes: Use of the :meth:`__index__` method for non-integers is new in 3.2. (4) + The ``'n'`` and ``'N'`` conversion codes are only available for the native + size (selected as the default or with the ``'@'`` byte order character). + For the standard size, you can use whichever of the other integer formats + fits your application. + +(5) For the ``'f'`` and ``'d'`` conversion codes, the packed representation uses the IEEE 754 binary32 (for ``'f'``) or binary64 (for ``'d'``) format, regardless of the floating-point format used by the platform. -(5) +(6) The ``'P'`` format character is only available for the native byte ordering (selected as the default or with the ``'@'`` byte order character). The byte order character ``'='`` chooses to use little- or big-endian ordering based diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 90a01d0..a8d65b2 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -30,16 +30,21 @@ convenience functions for all use cases they can handle. For more advanced use cases, the underlying :class:`Popen` interface can be used directly. -.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False) +.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) Run the command described by *args*. Wait for command to complete, then return the :attr:`returncode` attribute. The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the slightly odd notation in - the abbreviated signature). The full function signature is the same as - that of the :class:`Popen` constructor - this functions passes all - supplied arguments directly through to that interface. + in :ref:`frequently-used-arguments` (hence the use of keyword-only notation + in the abbreviated signature). The full function signature is largely the + same as that of the :class:`Popen` constructor - this function passes all + supplied arguments other than *timeout* directly through to that interface. + + The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout + expires, the child process will be killed and then waited for again. The + :exc:`TimeoutExpired` exception will be re-raised after the child process + has terminated. Examples:: @@ -62,8 +67,11 @@ use cases, the underlying :class:`Popen` interface can be used directly. process may block if it generates enough output to a pipe to fill up the OS pipe buffer. + .. versionchanged:: 3.3 + *timeout* was added. + -.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False) +.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) Run command with arguments. Wait for command to complete. If the return code was zero then return, otherwise raise :exc:`CalledProcessError`. The @@ -71,10 +79,15 @@ use cases, the underlying :class:`Popen` interface can be used directly. :attr:`returncode` attribute. The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the slightly odd notation in - the abbreviated signature). The full function signature is the same as - that of the :class:`Popen` constructor - this functions passes all - supplied arguments directly through to that interface. + in :ref:`frequently-used-arguments` (hence the use of keyword-only notation + in the abbreviated signature). The full function signature is largely the + same as that of the :class:`Popen` constructor - this function passes all + supplied arguments other than *timeout* directly through to that interface. + + The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout + expires, the child process will be killed and then waited for again. The + :exc:`TimeoutExpired` exception will be re-raised after the child process + has terminated. Examples:: @@ -86,8 +99,6 @@ use cases, the underlying :class:`Popen` interface can be used directly. ... subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 - .. versionadded:: 2.5 - .. warning:: Invoking the system shell with ``shell=True`` can be a security hazard @@ -101,8 +112,11 @@ use cases, the underlying :class:`Popen` interface can be used directly. process may block if it generates enough output to a pipe to fill up the OS pipe buffer. + .. versionchanged:: 3.3 + *timeout* was added. + -.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False) +.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) Run command with arguments and return its output as a byte string. @@ -112,11 +126,17 @@ use cases, the underlying :class:`Popen` interface can be used directly. attribute. The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the slightly odd notation in - the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor, except that *stdout* is - not permitted as it is used internally. All other supplied arguments are - passed directly through to the :class:`Popen` constructor. + in :ref:`frequently-used-arguments` (hence the use of keyword-only notation + in the abbreviated signature). The full function signature is largely the + same as that of the :class:`Popen` constructor - this functions passes all + supplied arguments other than *timeout* directly through to that interface. + In addition, *stdout* is not permitted as an argument, as it is used + internally to collect the output from the subprocess. + + The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout + expires, the child process will be killed and then waited for again. The + :exc:`TimeoutExpired` exception will be re-raised after the child process + has terminated. Examples:: @@ -147,7 +167,7 @@ use cases, the underlying :class:`Popen` interface can be used directly. ... shell=True) 'ls: non_existent_file: No such file or directory\n' - .. versionadded:: 2.7 + .. versionadded:: 3.1 .. warning:: @@ -161,6 +181,18 @@ use cases, the underlying :class:`Popen` interface can be used directly. read in the current process, the child process may block if it generates enough output to the pipe to fill up the OS pipe buffer. + .. versionchanged:: 3.3 + *timeout* was added. + + +.. data:: DEVNULL + + Special value that can be used as the *stdin*, *stdout* or *stderr* argument + to :class:`Popen` and indicates that the special file :data:`os.devnull` + will be used. + + .. versionadded:: 3.3 + .. data:: PIPE @@ -196,13 +228,14 @@ default values. The arguments that are most commonly needed are: *stdin*, *stdout* and *stderr* specify the executed program's standard input, standard output and standard error file handles, respectively. Valid values - are :data:`PIPE`, an existing file descriptor (a positive integer), an - existing file object, and ``None``. :data:`PIPE` indicates that a new pipe - to the child should be created. With the default settings of ``None``, no - redirection will occur; the child's file handles will be inherited from the - parent. Additionally, *stderr* can be :data:`STDOUT`, which indicates that - the stderr data from the child process should be captured into the same file - handle as for stdout. + are :data:`PIPE`, :data:`DEVNULL`, an existing file descriptor (a positive + integer), an existing file object, and ``None``. :data:`PIPE` indicates + that a new pipe to the child should be created. :data:`DEVNULL` indicates + that the special file :data:`os.devnull` will be used. With the default + settings of ``None``, no redirection will occur; the child's file handles + will be inherited from the parent. Additionally, *stderr* can be + :data:`STDOUT`, which indicates that the stderr data from the child + process should be captured into the same file handle as for *stdout*. When *stdout* or *stderr* are pipes and *universal_newlines* is :const:`True` then the output data is assumed to be encoded as UTF-8 and @@ -331,13 +364,14 @@ functions. *stdin*, *stdout* and *stderr* specify the executed program's standard input, standard output and standard error file handles, respectively. Valid values - are :data:`PIPE`, an existing file descriptor (a positive integer), an - existing :term:`file object`, and ``None``. :data:`PIPE` indicates that a - new pipe to the child should be created. With the default settings of - ``None``, no redirection will occur; the child's file handles will be - inherited from the parent. Additionally, *stderr* can be :data:`STDOUT`, - which indicates that the stderr data from the applications should be - captured into the same file handle as for stdout. + are :data:`PIPE`, :data:`DEVNULL`, an existing file descriptor (a positive + integer), an existing :term:`file object`, and ``None``. :data:`PIPE` + indicates that a new pipe to the child should be created. :data:`DEVNULL` + indicates that the special file :data:`os.devnull` will be used. With the + default settings of ``None``, no redirection will occur; the child's file + handles will be inherited from the parent. Additionally, *stderr* can be + :data:`STDOUT`, which indicates that the stderr data from the applications + should be captured into the same file handle as for stdout. If *preexec_fn* is set to a callable object, this object will be called in the child process just before the child is executed. @@ -456,6 +490,15 @@ arguments. :exc:`CalledProcessError` if the called process returns a non-zero return code. +All of the functions and methods that accept a *timeout* parameter, such as +:func:`call` and :meth:`Popen.communicate` will raise :exc:`TimeoutExpired` if +the timeout expires before the process exits. + +Exceptions defined in this module all inherit from :exc:`SubprocessError`. + + .. versionadded:: 3.3 + The :exc:`SubprocessError` base class was added. + Security ^^^^^^^^ @@ -479,11 +522,15 @@ Instances of the :class:`Popen` class have the following methods: attribute. -.. method:: Popen.wait() +.. method:: Popen.wait(timeout=None) Wait for child process to terminate. Set and return :attr:`returncode` attribute. + If the process does not terminate after *timeout* seconds, raise a + :exc:`TimeoutExpired` exception. It is safe to catch this exception and + retry the wait. + .. warning:: This will deadlock when using ``stdout=PIPE`` and/or @@ -491,13 +538,17 @@ Instances of the :class:`Popen` class have the following methods: a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use :meth:`communicate` to avoid that. + .. versionchanged:: 3.3 + *timeout* was added. -.. method:: Popen.communicate(input=None) + +.. method:: Popen.communicate(input=None, timeout=None) Interact with process: Send data to stdin. Read data from stdout and stderr, - until end-of-file is reached. Wait for process to terminate. The optional - *input* argument should be a byte string to be sent to the child process, or - ``None``, if no data should be sent to the child. + until end-of-file is reached. Wait for process to terminate. The optional + *input* argument should be data to be sent to the child process, or + ``None``, if no data should be sent to the child. The type of *input* + must be bytes or, if *universal_newlines* was ``True``, a string. :meth:`communicate` returns a tuple ``(stdoutdata, stderrdata)``. @@ -506,11 +557,29 @@ Instances of the :class:`Popen` class have the following methods: ``None`` in the result tuple, you need to give ``stdout=PIPE`` and/or ``stderr=PIPE`` too. + If the process does not terminate after *timeout* seconds, a + :exc:`TimeoutExpired` exception will be raised. Catching this exception and + retrying communication will not lose any output. + + The child process is not killed if the timeout expires, so in order to + cleanup properly a well-behaved application should kill the child process and + finish communication:: + + proc = subprocess.Popen(...) + try: + outs, errs = proc.communicate(timeout=15) + except TimeoutExpired: + proc.kill() + outs, errs = proc.communicate() + .. note:: The data read is buffered in memory, so do not use this method if the data size is large or unlimited. + .. versionchanged:: 3.3 + *timeout* was added. + .. method:: Popen.send_signal(signal) @@ -952,3 +1021,9 @@ runtime): backslash. If the number of backslashes is odd, the last backslash escapes the next double quotation mark as described in rule 3. + + +.. seealso:: + + :mod:`shlex` + Module which provides function to parse and escape command lines. diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 0e4adec..96450c5 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -235,14 +235,13 @@ always available. .. data:: flags - The struct sequence *flags* exposes the status of command line flags. The - attributes are read only. + The :term:`struct sequence` *flags* exposes the status of command line + flags. The attributes are read only. ============================= ============================= attribute flag ============================= ============================= :const:`debug` :option:`-d` - :const:`division_warning` :option:`-Q` :const:`inspect` :option:`-i` :const:`interactive` :option:`-i` :const:`optimize` :option:`-O` or :option:`-OO` @@ -262,15 +261,18 @@ always available. .. versionadded:: 3.2.3 The ``hash_randomization`` attribute. + .. versionchanged:: 3.3 + Removed obsolete ``division_warning`` attribute. + .. data:: float_info - A structseq holding information about the float type. It contains low level - information about the precision and internal representation. The values - correspond to the various floating-point constants defined in the standard - header file :file:`float.h` for the 'C' programming language; see section - 5.2.4.2.2 of the 1999 ISO/IEC C standard [C99]_, 'Characteristics of - floating types', for details. + A :term:`struct sequence` holding information about the float type. It + contains low level information about the precision and internal + representation. The values correspond to the various floating-point + constants defined in the standard header file :file:`float.h` for the 'C' + programming language; see section 5.2.4.2.2 of the 1999 ISO/IEC C standard + [C99]_, 'Characteristics of floating types', for details. +---------------------+----------------+--------------------------------------------------+ | attribute | float.h macro | explanation | @@ -520,8 +522,9 @@ always available. .. data:: hash_info - A structseq giving parameters of the numeric hash implementation. For - more details about hashing of numeric types, see :ref:`numeric-hash`. + A :term:`struct sequence` giving parameters of the numeric hash + implementation. For more details about hashing of numeric types, see + :ref:`numeric-hash`. +---------------------+--------------------------------------------------+ | attribute | explanation | @@ -556,8 +559,8 @@ always available. This is called ``hexversion`` since it only really looks meaningful when viewed as the result of passing it to the built-in :func:`hex` function. The - struct sequence :data:`sys.version_info` may be used for a more human-friendly - encoding of the same information. + :term:`struct sequence` :data:`sys.version_info` may be used for a more + human-friendly encoding of the same information. The ``hexversion`` is a 32-bit number with the following layout: @@ -585,8 +588,8 @@ always available. .. data:: int_info - A struct sequence that holds information about Python's - internal representation of integers. The attributes are read only. + A :term:`struct sequence` that holds information about Python's internal + representation of integers. The attributes are read only. +-------------------------+----------------------------------------------+ | Attribute | Explanation | @@ -641,9 +644,13 @@ always available. .. data:: maxunicode - An integer giving the largest supported code point for a Unicode character. The - value of this depends on the configuration option that specifies whether Unicode - characters are stored as UCS-2 or UCS-4. + An integer giving the value of the largest Unicode code point, + i.e. ``1114111`` (``0x10FFFF`` in hexadecimal). + + .. versionchanged:: 3.3 + Before :pep:`393`, ``sys.maxunicode`` used to be either ``0xFFFF`` + or ``0x10FFFF``, depending on the configuration option that specified + whether Unicode characters were stored as UCS-2 or UCS-4. .. data:: meta_path @@ -718,36 +725,35 @@ always available. This string contains a platform identifier that can be used to append platform-specific components to :data:`sys.path`, for instance. - For most Unix systems, this is the lowercased OS name as returned by ``uname - -s`` with the first part of the version as returned by ``uname -r`` appended, - e.g. ``'sunos5'``, *at the time when Python was built*. Unless you want to - test for a specific system version, it is therefore recommended to use the - following idiom:: + For Unix systems, except on Linux, this is the lowercased OS name as + returned by ``uname -s`` with the first part of the version as returned by + ``uname -r`` appended, e.g. ``'sunos5'`` or ``'freebsd8'``, *at the time + when Python was built*. Unless you want to test for a specific system + version, it is therefore recommended to use the following idiom:: if sys.platform.startswith('freebsd'): # FreeBSD-specific code here... elif sys.platform.startswith('linux'): # Linux-specific code here... - .. versionchanged:: 3.2.2 - Since lots of code check for ``sys.platform == 'linux2'``, and there is - no essential change between Linux 2.x and 3.x, ``sys.platform`` is always - set to ``'linux2'``, even on Linux 3.x. In Python 3.3 and later, the - value will always be set to ``'linux'``, so it is recommended to always - use the ``startswith`` idiom presented above. - For other systems, the values are: - ====================== =========================== - System :data:`platform` value - ====================== =========================== - Linux (2.x *and* 3.x) ``'linux2'`` - Windows ``'win32'`` - Windows/Cygwin ``'cygwin'`` - Mac OS X ``'darwin'`` - OS/2 ``'os2'`` - OS/2 EMX ``'os2emx'`` - ====================== =========================== + ================ =========================== + System :data:`platform` value + ================ =========================== + Linux ``'linux'`` + Windows ``'win32'`` + Windows/Cygwin ``'cygwin'`` + Mac OS X ``'darwin'`` + OS/2 ``'os2'`` + OS/2 EMX ``'os2emx'`` + ================ =========================== + + .. versionchanged:: 3.3 + On Linux, :attr:`sys.platform` doesn't contain the major version anymore. + It is always ``'linux'``, instead of ``'linux2'`` or ``'linux3'``. Since + older Python versions include the version number, it is recommended to + always use the ``startswith`` idiom presented above. .. seealso:: @@ -764,7 +770,7 @@ always available. independent Python files are installed; by default, this is the string ``'/usr/local'``. This can be set at build time with the ``--prefix`` argument to the :program:`configure` script. The main collection of Python - library modules is installed in the directory :file:`{prefix}/lib/python{X.Y}`` + library modules is installed in the directory :file:`{prefix}/lib/python{X.Y}` while the platform independent header files (all except :file:`pyconfig.h`) are stored in :file:`{prefix}/include/python{X.Y}`, where *X.Y* is the version number of Python, for example ``3.2``. @@ -806,11 +812,11 @@ always available. the interpreter loads extension modules. Among other things, this will enable a lazy resolving of symbols when importing a module, if called as ``sys.setdlopenflags(0)``. To share symbols across extension modules, call as - ``sys.setdlopenflags(ctypes.RTLD_GLOBAL)``. Symbolic names for the - flag modules can be either found in the :mod:`ctypes` module, or in the :mod:`DLFCN` - module. If :mod:`DLFCN` is not available, it can be generated from - :file:`/usr/include/dlfcn.h` using the :program:`h2py` script. Availability: - Unix. + ``sys.setdlopenflags(os.RTLD_GLOBAL)``. Symbolic names for the flag modules + can be found in the :mod:`os` module (``RTLD_xxx`` constants, e.g. + :data:`os.RTLD_LAZY`). + + Availability: Unix. .. function:: setprofile(profilefunc) @@ -1003,22 +1009,33 @@ always available. to a console and Python apps started with :program:`pythonw`. -.. data:: subversion +.. data:: thread_info - A triple (repo, branch, version) representing the Subversion information of the - Python interpreter. *repo* is the name of the repository, ``'CPython'``. - *branch* is a string of one of the forms ``'trunk'``, ``'branches/name'`` or - ``'tags/name'``. *version* is the output of ``svnversion``, if the interpreter - was built from a Subversion checkout; it contains the revision number (range) - and possibly a trailing 'M' if there were local modifications. If the tree was - exported (or svnversion was not available), it is the revision of - ``Include/patchlevel.h`` if the branch is a tag. Otherwise, it is ``None``. + A :term:`struct sequence` holding information about the thread + implementation. - .. deprecated:: 3.2.1 - Python is now `developed <http://docs.python.org/devguide/>`_ using - Mercurial. In recent Python 3.2 bugfix releases, :data:`subversion` - therefore contains placeholder information. It is removed in Python - 3.3. + +------------------+---------------------------------------------------------+ + | Attribute | Explanation | + +==================+=========================================================+ + | :const:`name` | Name of the thread implementation: | + | | | + | | * ``'nt'``: Windows threads | + | | * ``'os2'``: OS/2 threads | + | | * ``'pthread'``: POSIX threads | + | | * ``'solaris'``: Solaris threads | + +------------------+---------------------------------------------------------+ + | :const:`lock` | Name of the lock implementation: | + | | | + | | * ``'semaphore'``: a lock uses a semaphore | + | | * ``'mutex+cond'``: a lock uses a mutex | + | | and a condition variable | + | | * ``None`` if this information is unknown | + +------------------+---------------------------------------------------------+ + | :const:`version` | Name and version of the thread library. It is a string, | + | | or ``None`` if these informations are unknown. | + +------------------+---------------------------------------------------------+ + + .. versionadded:: 3.3 .. data:: tracebacklimit diff --git a/Doc/library/syslog.rst b/Doc/library/syslog.rst index 645c326..aa0f5a1 100644 --- a/Doc/library/syslog.rst +++ b/Doc/library/syslog.rst @@ -78,12 +78,14 @@ Priority levels (high to low): Facilities: :const:`LOG_KERN`, :const:`LOG_USER`, :const:`LOG_MAIL`, :const:`LOG_DAEMON`, :const:`LOG_AUTH`, :const:`LOG_LPR`, :const:`LOG_NEWS`, :const:`LOG_UUCP`, - :const:`LOG_CRON`, :const:`LOG_SYSLOG` and :const:`LOG_LOCAL0` to - :const:`LOG_LOCAL7`. + :const:`LOG_CRON`, :const:`LOG_SYSLOG`, :const:`LOG_LOCAL0` to + :const:`LOG_LOCAL7`, and, if defined in ``<syslog.h>``, + :const:`LOG_AUTHPRIV`. Log options: - :const:`LOG_PID`, :const:`LOG_CONS`, :const:`LOG_NDELAY`, :const:`LOG_NOWAIT` - and :const:`LOG_PERROR` if defined in ``<syslog.h>``. + :const:`LOG_PID`, :const:`LOG_CONS`, :const:`LOG_NDELAY`, and, if defined + in ``<syslog.h>``, :const:`LOG_ODELAY`, :const:`LOG_NOWAIT`, and + :const:`LOG_PERROR`. Examples diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index 46e4900..86dd33d 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -13,13 +13,13 @@ -------------- The :mod:`tarfile` module makes it possible to read and write tar -archives, including those using gzip or bz2 compression. +archives, including those using gzip, bz2 and lzma compression. Use the :mod:`zipfile` module to read or write :file:`.zip` files, or the higher-level functions in :ref:`shutil <archiving-operations>`. Some facts and figures: -* reads and writes :mod:`gzip` and :mod:`bz2` compressed archives. +* reads and writes :mod:`gzip`, :mod:`bz2` and :mod:`lzma` compressed archives. * read/write support for the POSIX.1-1988 (ustar) format. @@ -33,6 +33,9 @@ Some facts and figures: character devices and block devices and is able to acquire and restore file information like timestamp, access permissions and owner. +.. versionchanged:: 3.3 + Added support for :mod:`lzma` compression. + .. function:: open(name=None, mode='r', fileobj=None, bufsize=10240, \*\*kwargs) @@ -56,6 +59,8 @@ Some facts and figures: +------------------+---------------------------------------------+ | ``'r:bz2'`` | Open for reading with bzip2 compression. | +------------------+---------------------------------------------+ + | ``'r:xz'`` | Open for reading with lzma compression. | + +------------------+---------------------------------------------+ | ``'a' or 'a:'`` | Open for appending with no compression. The | | | file is created if it does not exist. | +------------------+---------------------------------------------+ @@ -65,11 +70,13 @@ Some facts and figures: +------------------+---------------------------------------------+ | ``'w:bz2'`` | Open for bzip2 compressed writing. | +------------------+---------------------------------------------+ + | ``'w:xz'`` | Open for lzma compressed writing. | + +------------------+---------------------------------------------+ - Note that ``'a:gz'`` or ``'a:bz2'`` is not possible. If *mode* is not suitable - to open a certain (compressed) file for reading, :exc:`ReadError` is raised. Use - *mode* ``'r'`` to avoid this. If a compression method is not supported, - :exc:`CompressionError` is raised. + Note that ``'a:gz'``, ``'a:bz2'`` or ``'a:xz'`` is not possible. If *mode* + is not suitable to open a certain (compressed) file for reading, + :exc:`ReadError` is raised. Use *mode* ``'r'`` to avoid this. If a + compression method is not supported, :exc:`CompressionError` is raised. If *fileobj* is specified, it is used as an alternative to a :term:`file object` opened in binary mode for *name*. It is supposed to be at position 0. @@ -100,6 +107,9 @@ Some facts and figures: | ``'r|bz2'`` | Open a bzip2 compressed *stream* for | | | reading. | +-------------+--------------------------------------------+ + | ``'r|xz'`` | Open a lzma compressed *stream* for | + | | reading. | + +-------------+--------------------------------------------+ | ``'w|'`` | Open an uncompressed *stream* for writing. | +-------------+--------------------------------------------+ | ``'w|gz'`` | Open a gzip compressed *stream* for | @@ -108,6 +118,9 @@ Some facts and figures: | ``'w|bz2'`` | Open a bzip2 compressed *stream* for | | | writing. | +-------------+--------------------------------------------+ + | ``'w|xz'`` | Open an lzma compressed *stream* for | + | | writing. | + +-------------+--------------------------------------------+ .. class:: TarFile @@ -263,9 +276,9 @@ be finalized; only the internally used file object will be closed. See the If *errorlevel* is ``0``, all errors are ignored when using :meth:`TarFile.extract`. Nevertheless, they appear as error messages in the debug output, when debugging - is enabled. If ``1``, all *fatal* errors are raised as :exc:`OSError` or - :exc:`IOError` exceptions. If ``2``, all *non-fatal* errors are raised as - :exc:`TarError` exceptions as well. + is enabled. If ``1``, all *fatal* errors are raised as :exc:`OSError` + exceptions. If ``2``, all *non-fatal* errors are raised as :exc:`TarError` + exceptions as well. The *encoding* and *errors* arguments define the character encoding to be used for reading or writing the archive and how conversion errors are going @@ -363,15 +376,12 @@ be finalized; only the internally used file object will be closed. See the .. method:: TarFile.extractfile(member) Extract a member from the archive as a file object. *member* may be a filename - or a :class:`TarInfo` object. If *member* is a regular file, a :term:`file-like - object` is returned. If *member* is a link, a file-like object is constructed from - the link's target. If *member* is none of the above, :const:`None` is returned. - - .. note:: + or a :class:`TarInfo` object. If *member* is a regular file or a link, an + :class:`io.BufferedReader` object is returned. Otherwise, :const:`None` is + returned. - The file-like object is read-only. It provides the methods - :meth:`read`, :meth:`readline`, :meth:`readlines`, :meth:`seek`, :meth:`tell`, - and :meth:`close`, and also supports iteration over its lines. + .. versionchanged:: 3.3 + Return an :class:`io.BufferedReader` object. .. method:: TarFile.add(name, arcname=None, recursive=True, exclude=None, *, filter=None) diff --git a/Doc/library/telnetlib.rst b/Doc/library/telnetlib.rst index 646634d..9bc79c5 100644 --- a/Doc/library/telnetlib.rst +++ b/Doc/library/telnetlib.rst @@ -162,9 +162,13 @@ Telnet Objects .. method:: Telnet.write(buffer) Write a byte string to the socket, doubling any IAC characters. This can - block if the connection is blocked. May raise :exc:`socket.error` if the + block if the connection is blocked. May raise :exc:`OSError` if the connection is closed. + .. versionchanged:: 3.3 + This method used to raise :exc:`socket.error`, which is now an alias + of :exc:`OSError`. + .. method:: Telnet.interact() diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index fff6c4e..dfeb250 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -25,7 +25,7 @@ instead a string of six random characters is used. Also, all the user-callable functions now take additional arguments which allow direct control over the location and name of temporary files. It is -no longer necessary to use the global *tempdir* and *template* variables. +no longer necessary to use the global *tempdir* variable. To maintain backward compatibility, the argument order is somewhat odd; it is recommended to use keyword arguments for clarity. diff --git a/Doc/library/test.rst b/Doc/library/test.rst index 40f4115..2e8ba32 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -80,17 +80,12 @@ A basic boilerplate is often used:: ... more test classes ... - def test_main(): - support.run_unittest(MyTestCase1, - MyTestCase2, - ... list other tests ... - ) - if __name__ == '__main__': - test_main() + unittest.main() -This boilerplate code allows the testing suite to be run by :mod:`test.regrtest` -as well as on its own as a script. +This code pattern allows the testing suite to be run by :mod:`test.regrtest`, +on its own as a script that supports the :mod:`unittest` CLI, or via the +`python -m unittest` CLI. The goal for regression testing is to try to break code. This leads to a few guidelines to be followed: @@ -129,22 +124,27 @@ guidelines to be followed: as what type of input is used. Minimize code duplication by subclassing a basic test class with a class that specifies the input:: - class TestFuncAcceptsSequences(unittest.TestCase): + class TestFuncAcceptsSequencesMixin: func = mySuperWhammyFunction def test_func(self): self.func(self.arg) - class AcceptLists(TestFuncAcceptsSequences): + class AcceptLists(TestFuncAcceptsSequencesMixin, unittest.TestCase): arg = [1, 2, 3] - class AcceptStrings(TestFuncAcceptsSequences): + class AcceptStrings(TestFuncAcceptsSequencesMixin, unittest.TestCase): arg = 'abc' - class AcceptTuples(TestFuncAcceptsSequences): + class AcceptTuples(TestFuncAcceptsSequencesMixin, unittest.TestCase): arg = (1, 2, 3) + When using this pattern, remember that all classes that inherit from + `unittest.TestCase` are run as tests. The `Mixin` class in the example above + does not have any data and so can't be run by itself, thus it does not + inherit from `unittest.TestCase`. + .. seealso:: @@ -160,14 +160,15 @@ Running tests using the command-line interface The :mod:`test` package can be run as a script to drive Python's regression test suite, thanks to the :option:`-m` option: :program:`python -m test`. Under the hood, it uses :mod:`test.regrtest`; the call :program:`python -m -test.regrtest` used in previous Python versions still works). -Running the script by itself automatically starts running all regression -tests in the :mod:`test` package. It does this by finding all modules in the -package whose name starts with ``test_``, importing them, and executing the -function :func:`test_main` if present. The names of tests to execute may also -be passed to the script. Specifying a single regression test (:program:`python --m test test_spam`) will minimize output and only print -whether the test passed or failed and thus minimize output. +test.regrtest` used in previous Python versions still works). Running the +script by itself automatically starts running all regression tests in the +:mod:`test` package. It does this by finding all modules in the package whose +name starts with ``test_``, importing them, and executing the function +:func:`test_main` if present or loading the tests via +unittest.TestLoader.loadTestsFromModule if ``test_main`` does not exist. The +names of tests to execute may also be passed to the script. Specifying a single +regression test (:program:`python -m test test_spam`) will minimize output and +only print whether the test passed or failed. Running :mod:`test` directly allows what resources are available for tests to use to be set. You do this by using the ``-u`` command-line @@ -223,14 +224,14 @@ The :mod:`test.support` module defines the following constants: .. data:: verbose - :const:`True` when verbose output is enabled. Should be checked when more + ``True`` when verbose output is enabled. Should be checked when more detailed information is desired about a running test. *verbose* is set by :mod:`test.regrtest`. .. data:: is_jython - :const:`True` if the running interpreter is Jython. + ``True`` if the running interpreter is Jython. .. data:: TESTFN @@ -249,7 +250,7 @@ The :mod:`test.support` module defines the following functions: .. function:: is_resource_enabled(resource) - Return :const:`True` if *resource* is enabled and available. The list of + Return ``True`` if *resource* is enabled and available. The list of available resources is only set when :mod:`test.regrtest` is executing the tests. @@ -258,7 +259,7 @@ The :mod:`test.support` module defines the following functions: Raise :exc:`ResourceDenied` if *resource* is not available. *msg* is the argument to :exc:`ResourceDenied` if it is raised. Always returns - :const:`True` if called by a function whose ``__name__`` is ``'__main__'``. + ``True`` if called by a function whose ``__name__`` is ``'__main__'``. Used when tests are executed by :mod:`test.regrtest`. @@ -286,6 +287,15 @@ The :mod:`test.support` module defines the following functions: This will run all tests defined in the named module. +.. function:: run_doctest(module, verbosity=None) + + Run :func:`doctest.testmod` on the given *module*. Return + ``(failure_count, test_count)``. + + If *verbosity* is ``None``, :func:`doctest.testmod` is run with verbosity + set to :data:`verbose`. Otherwise, it is run with verbosity set to + ``None``. + .. function:: check_warnings(\*filters, quiet=True) A convenience wrapper for :func:`warnings.catch_warnings()` that makes it @@ -296,12 +306,12 @@ The :mod:`test.support` module defines the following functions: ``check_warnings`` accepts 2-tuples of the form ``("message regexp", WarningCategory)`` as positional arguments. If one or more *filters* are - provided, or if the optional keyword argument *quiet* is :const:`False`, + provided, or if the optional keyword argument *quiet* is ``False``, it checks to make sure the warnings are as expected: each specified filter must match at least one of the warnings raised by the enclosed code or the test fails, and if any warnings are raised that do not match any of the specified filters the test fails. To disable the first of these checks, - set *quiet* to :const:`True`. + set *quiet* to ``True``. If no arguments are specified, it defaults to:: @@ -316,7 +326,7 @@ The :mod:`test.support` module defines the following functions: representing the most recent warning can also be accessed directly through the recorder object (see example below). If no warning has been raised, then any of the attributes that would otherwise be expected on an object - representing a warning will return :const:`None`. + representing a warning will return ``None``. The recorder object also has a :meth:`reset` method, which clears the warnings list. @@ -354,7 +364,7 @@ The :mod:`test.support` module defines the following functions: .. function:: captured_stdout() - This is a context manager that runs the :keyword:`with` statement body using + A context manager that runs the :keyword:`with` statement body using a :class:`StringIO.StringIO` object as sys.stdout. That object can be retrieved using the ``as`` clause of the :keyword:`with` statement. @@ -365,6 +375,57 @@ The :mod:`test.support` module defines the following functions: assert s.getvalue() == "hello\n" +.. function:: temp_cwd(name='tempcwd', quiet=False, path=None) + + A context manager that temporarily changes the current working + directory (CWD). + + An existing path may be provided as *path*, in which case this function + makes no changes to the file system. + + Otherwise, the new CWD is created in the current directory and it's named + *name*. If *quiet* is ``False`` and it's not possible to create or + change the CWD, an error is raised. If it's ``True``, only a warning + is raised and the original CWD is used. + + +.. function:: temp_umask(umask) + + A context manager that temporarily sets the process umask. + + +.. function:: can_symlink() + + Return ``True`` if the OS supports symbolic links, ``False`` + otherwise. + + +.. decorator:: skip_unless_symlink() + + A decorator for running tests that require support for symbolic links. + + +.. decorator:: anticipate_failure(condition) + + A decorator to conditionally mark tests with + :func:`unittest.expectedFailure`. Any use of this decorator should + have an associated comment identifying the relevant tracker issue. + + +.. decorator:: run_with_locale(catstr, *locales) + + A decorator for running a function in a different locale, correctly + resetting it after it has finished. *catstr* is the locale category as + a string (for example ``"LC_ALL"``). The *locales* passed will be tried + sequentially, and the first valid locale will be used. + + +.. function:: make_bad_fd() + + Create an invalid file descriptor by opening and closing a temporary file, + and returning its descripor. + + .. function:: import_module(name, deprecated=False) This function imports and returns the named module. Unlike a normal @@ -372,7 +433,7 @@ The :mod:`test.support` module defines the following functions: cannot be imported. Module and package deprecation messages are suppressed during this import - if *deprecated* is :const:`True`. + if *deprecated* is ``True``. .. versionadded:: 3.1 @@ -396,9 +457,9 @@ The :mod:`test.support` module defines the following functions: ``sys.modules`` when the fresh import is complete. Module and package deprecation messages are suppressed during this import - if *deprecated* is :const:`True`. + if *deprecated* is ``True``. - This function will raise :exc:`unittest.SkipTest` is the named module + This function will raise :exc:`unittest.SkipTest` if the named module cannot be imported. Example use:: @@ -413,6 +474,48 @@ The :mod:`test.support` module defines the following functions: .. versionadded:: 3.1 +.. function:: bind_port(sock, host=HOST) + + Bind the socket to a free port and return the port number. Relies on + ephemeral ports in order to ensure we are using an unbound port. This is + important as many tests may be running simultaneously, especially in a + buildbot environment. This method raises an exception if the + ``sock.family`` is :const:`~socket.AF_INET` and ``sock.type`` is + :const:`~socket.SOCK_STREAM`, and the socket has + :const:`~socket.SO_REUSEADDR` or :const:`~socket.SO_REUSEPORT` set on it. + Tests should never set these socket options for TCP/IP sockets. + The only case for setting these options is testing multicasting via + multiple UDP sockets. + + Additionally, if the :const:`~socket.SO_EXCLUSIVEADDRUSE` socket option is + available (i.e. on Windows), it will be set on the socket. This will + prevent anyone else from binding to our host/port for the duration of the + test. + + +.. function:: find_unused_port(family=socket.AF_INET, socktype=socket.SOCK_STREAM) + + Returns an unused port that should be suitable for binding. This is + achieved by creating a temporary socket with the same family and type as + the ``sock`` parameter (default is :const:`~socket.AF_INET`, + :const:`~socket.SOCK_STREAM`), + and binding it to the specified host address (defaults to ``0.0.0.0``) + with the port set to 0, eliciting an unused ephemeral port from the OS. + The temporary socket is then closed and deleted, and the ephemeral port is + returned. + + Either this method or :func:`bind_port` should be used for any tests + where a server socket needs to be bound to a particular port for the + duration of the test. + Which one to use depends on whether the calling code is creating a python + socket, or if an unused port needs to be provided in a constructor + or passed to an external program (i.e. the ``-accept`` argument to + openssl's s_server mode). Always prefer :func:`bind_port` over + :func:`find_unused_port` where possible. Using a hard coded port is + discouraged since it can makes multiple instances of the test impossible to + run simultaneously, which is a problem for buildbots. + + The :mod:`test.support` module defines the following classes: .. class:: TransientResource(exc, **kwargs) diff --git a/Doc/library/textwrap.rst b/Doc/library/textwrap.rst index a814962..a74789c 100644 --- a/Doc/library/textwrap.rst +++ b/Doc/library/textwrap.rst @@ -107,6 +107,15 @@ indentation from strings that have unwanted whitespace to the left of the text. expanded to spaces using the :meth:`expandtabs` method of *text*. + .. attribute:: tabsize + + (default: ``8``) If :attr:`expand_tabs` is true, then all tab characters + in *text* will be expanded to zero or more spaces, depending on the + current column and the given tab size. + + .. versionadded:: 3.3 + + .. attribute:: replace_whitespace (default: ``True``) If true, each whitespace character (as defined by diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index 43087ef..eaba38b 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -20,17 +20,6 @@ The :mod:`dummy_threading` module is provided for situations where methods and functions in this module in the Python 2.x series are still supported by this module. -.. impl-detail:: - - Due to the :term:`Global Interpreter Lock`, in CPython only one thread - can execute Python code at once (even though certain performance-oriented - libraries might overcome this limitation). - If you want your application to make better of use of the computational - resources of multi-core machines, you are advised to use - :mod:`multiprocessing` or :class:`concurrent.futures.ProcessPoolExecutor`. - However, threading is still an appropriate model if you want to run - multiple I/O-bound tasks simultaneously. - This module defines the following functions and objects: @@ -59,6 +48,17 @@ This module defines the following functions and objects: returned. +.. function:: get_ident() + + Return the 'thread identifier' of the current thread. This is a nonzero + integer. Its value has no direct meaning; it is intended as a magic cookie + to be used e.g. to index a dictionary of thread-specific data. Thread + identifiers may be recycled when a thread exits and another thread is + created. + + .. versionadded:: 3.3 + + .. function:: enumerate() Return a list of all :class:`Thread` objects currently alive. The list @@ -174,7 +174,7 @@ This module defines the following functions and objects: *size* argument specifies the stack size to be used for subsequently created threads, and must be 0 (use platform or configured default) or a positive integer value of at least 32,768 (32kB). If changing the thread stack size is - unsupported, a :exc:`ThreadError` is raised. If the specified stack size is + unsupported, a :exc:`RuntimeError` is raised. If the specified stack size is invalid, a :exc:`ValueError` is raised and the stack size is unmodified. 32kB is currently the minimum supported stack size value to guarantee sufficient stack space for the interpreter itself. Note that some platforms may have @@ -239,10 +239,11 @@ called is terminated. A thread has a name. The name can be passed to the constructor, and read or changed through the :attr:`~Thread.name` attribute. -A thread can be flagged as a "daemon thread". The significance of this flag -is that the entire Python program exits when only daemon threads are left. -The initial value is inherited from the creating thread. The flag can be -set through the :attr:`~Thread.daemon` property. +A thread can be flagged as a "daemon thread". The significance of this flag is +that the entire Python program exits when only daemon threads are left. The +initial value is inherited from the creating thread. The flag can be set +through the :attr:`~Thread.daemon` property or the *daemon* constructor +argument. There is a "main thread" object; this corresponds to the initial thread of control in the Python program. It is not a daemon thread. @@ -255,7 +256,8 @@ daemonic, and cannot be :meth:`~Thread.join`\ ed. They are never deleted, since it is impossible to detect the termination of alien threads. -.. class:: Thread(group=None, target=None, name=None, args=(), kwargs={}) +.. class:: Thread(group=None, target=None, name=None, args=(), kwargs={}, + verbose=None, *, daemon=None) This constructor should always be called with keyword arguments. Arguments are: @@ -274,10 +276,19 @@ since it is impossible to detect the termination of alien threads. *kwargs* is a dictionary of keyword arguments for the target invocation. Defaults to ``{}``. + *verbose* is a flag used for debugging messages. + + If not ``None``, *daemon* explicitly sets whether the thread is daemonic. + If ``None`` (the default), the daemonic property is inherited from the + current thread. + If the subclass overrides the constructor, it must make sure to invoke the base class constructor (``Thread.__init__()``) before doing anything else to the thread. + .. versionchanged:: 3.3 + Added the *daemon* argument. + .. method:: start() Start the thread's activity. @@ -368,6 +379,18 @@ since it is impossible to detect the termination of alien threads. property instead. +.. impl-detail:: + + Due to the :term:`Global Interpreter Lock`, in CPython only one thread + can execute Python code at once (even though certain performance-oriented + libraries might overcome this limitation). + If you want your application to make better of use of the computational + resources of multi-core machines, you are advised to use + :mod:`multiprocessing` or :class:`concurrent.futures.ProcessPoolExecutor`. + However, threading is still an appropriate model if you want to run + multiple I/O-bound tasks simultaneously. + + .. _lock-objects: Lock Objects @@ -435,7 +458,7 @@ All methods are executed atomically. are blocked waiting for the lock to become unlocked, allow exactly one of them to proceed. - When invoked on an unlocked lock, a :exc:`ThreadError` is raised. + When invoked on an unlocked lock, a :exc:`RuntimeError` is raised. There is no return value. @@ -973,27 +996,3 @@ is equivalent to:: Currently, :class:`Lock`, :class:`RLock`, :class:`Condition`, :class:`Semaphore`, and :class:`BoundedSemaphore` objects may be used as :keyword:`with` statement context managers. - - -.. _threaded-imports: - -Importing in threaded code --------------------------- - -While the import machinery is thread-safe, there are two key restrictions on -threaded imports due to inherent limitations in the way that thread-safety is -provided: - -* Firstly, other than in the main module, an import should not have the - side effect of spawning a new thread and then waiting for that thread in - any way. Failing to abide by this restriction can lead to a deadlock if - the spawned thread directly or indirectly attempts to import a module. -* Secondly, all import attempts must be completed before the interpreter - starts shutting itself down. This can be most easily achieved by only - performing imports from non-daemon threads created through the threading - module. Daemon threads and threads created directly with the thread - module will require some other form of synchronization to ensure they do - not attempt imports after system shutdown has commenced. Failure to - abide by this restriction will lead to intermittent exceptions and - crashes during interpreter shutdown (as the late imports attempt to - access machinery which is no longer in a valid state). diff --git a/Doc/library/time.rst b/Doc/library/time.rst index e0c7007..a8c70b3 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -41,25 +41,6 @@ An explanation of some terminology and conventions is in order. parsed, they are converted according to the POSIX and ISO C standards: values 69--99 are mapped to 1969--1999, and values 0--68 are mapped to 2000--2068. - For backward compatibility, years with less than 4 digits are treated - specially by :func:`asctime`, :func:`mktime`, and :func:`strftime` functions - that operate on a 9-tuple or :class:`struct_time` values. If year (the first - value in the 9-tuple) is specified with less than 4 digits, its interpretation - depends on the value of ``accept2dyear`` variable. - - If ``accept2dyear`` is true (default), a backward compatibility behavior is - invoked as follows: - - - for 2-digit year, century is guessed according to POSIX rules for - ``%y`` strptime format. A deprecation warning is issued when century - information is guessed in this way. - - - for 3-digit or negative year, a :exc:`ValueError` exception is raised. - - If ``accept2dyear`` is false (set by the program or as a result of a - non-empty value assigned to ``PYTHONY2K`` environment variable) all year - values are interpreted as given. - .. index:: single: UTC single: Coordinated Universal Time @@ -117,24 +98,6 @@ An explanation of some terminology and conventions is in order. The module defines the following functions and data items: - -.. data:: accept2dyear - - Boolean value indicating whether two-digit year values will be - mapped to 1969--2068 range by :func:`asctime`, :func:`mktime`, and - :func:`strftime` functions. This is true by default, but will be - set to false if the environment variable :envvar:`PYTHONY2K` has - been set to a non-empty string. It may also be modified at run - time. - - .. deprecated:: 3.2 - Mapping of 2-digit year values by :func:`asctime`, - :func:`mktime`, and :func:`strftime` functions to 1969--2068 - range is deprecated. Programs that need to process 2-digit - years should use ``%y`` code available in :func:`strptime` - function or convert 2-digit year values to 4-digit themselves. - - .. data:: altzone The offset of the local DST timezone, in seconds west of UTC, if one is defined. @@ -152,7 +115,8 @@ The module defines the following functions and data items: .. note:: - Unlike the C function of the same name, there is no trailing newline. + Unlike the C function of the same name, :func:`asctime` does not add a + trailing newline. .. function:: clock() @@ -172,6 +136,121 @@ The module defines the following functions and data items: :c:func:`QueryPerformanceCounter`. The resolution is typically better than one microsecond. + .. deprecated:: 3.3 + The behaviour of this function depends on the platform: use + :func:`perf_counter` or :func:`process_time` instead, depending on your + requirements, to have a well defined behaviour. + + +.. function:: clock_getres(clk_id) + + Return the resolution (precision) of the specified clock *clk_id*. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. function:: clock_gettime(clk_id) + + Return the time of the specified clock *clk_id*. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. class:: clock_info + + Clock information object returned by :func:`get_clock_info`. + + .. attribute:: implementation + + The name of the underlying C function used to get the clock value. + + .. attribute:: monotonic + + ``True`` if the clock cannot go backward, ``False`` otherwise. + + .. attribute:: adjusted + + ``True`` if the clock can be adjusted (e.g. by a NTP daemon), ``False`` + otherwise. + + .. attribute:: resolution + + The resolution of the clock in seconds (:class:`float`). + + .. versionadded:: 3.3 + + +.. function:: clock_settime(clk_id, time) + + Set the time of the specified clock *clk_id*. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. data:: CLOCK_HIGHRES + + The Solaris OS has a CLOCK_HIGHRES timer that attempts to use an optimal + hardware source, and may give close to nanosecond resolution. CLOCK_HIGHRES + is the nonadjustable, high-resolution clock. + + Availability: Solaris. + + .. versionadded:: 3.3 + + +.. data:: CLOCK_MONOTONIC + + Clock that cannot be set and represents monotonic time since some unspecified + starting point. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. data:: CLOCK_MONOTONIC_RAW + + Similar to :data:`CLOCK_MONOTONIC`, but provides access to a raw + hardware-based time that is not subject to NTP adjustments. + + Availability: Linux 2.6.28 or later. + + .. versionadded:: 3.3 + + +.. data:: CLOCK_PROCESS_CPUTIME_ID + + High-resolution per-process timer from the CPU. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. data:: CLOCK_REALTIME + + System-wide real-time clock. Setting this clock requires appropriate + privileges. + + Availability: Unix. + + .. versionadded:: 3.3 + + +.. data:: CLOCK_THREAD_CPUTIME_ID + + Thread-specific CPU-time clock. + + Availability: Unix. + + .. versionadded:: 3.3 + .. function:: ctime([secs]) @@ -186,6 +265,21 @@ The module defines the following functions and data items: Nonzero if a DST timezone is defined. +.. function:: get_clock_info(name) + + Get information on the specified clock as a :class:`clock_info` object. + Supported clock names and the corresponding functions to read their value + are: + + * ``'clock'``: :func:`time.clock` + * ``'monotonic'``: :func:`time.monotonic` + * ``'perf_counter'``: :func:`time.perf_counter` + * ``'process_time'``: :func:`time.process_time` + * ``'time'``: :func:`time.time` + + .. versionadded:: 3.3 + + .. function:: gmtime([secs]) Convert a time expressed in seconds since the epoch to a :class:`struct_time` in @@ -215,6 +309,48 @@ The module defines the following functions and data items: The earliest date for which it can generate a time is platform-dependent. +.. function:: monotonic() + + Return the value (in fractional seconds) of a monotonic clock, i.e. a clock + that cannot go backwards. The clock is not affected by system clock updates. + The reference point of the returned value is undefined, so that only the + difference between the results of consecutive calls is valid. + + On Windows versions older than Vista, :func:`monotonic` detects + :c:func:`GetTickCount` integer overflow (32 bits, roll-over after 49.7 days). + It increases an internal epoch (reference time by) 2\ :sup:`32` each time + that an overflow is detected. The epoch is stored in the process-local state + and so the value of :func:`monotonic` may be different in two Python + processes running for more than 49 days. On more recent versions of Windows + and on other operating systems, :func:`monotonic` is system-wide. + + Availability: Windows, Mac OS X, Linux, FreeBSD, OpenBSD, Solaris. + + .. versionadded:: 3.3 + + +.. function:: perf_counter() + + Return the value (in fractional seconds) of a performance counter, i.e. a + clock with the highest available resolution to measure a short duration. It + does include time elapsed during sleep and is system-wide. The reference + point of the returned value is undefined, so that only the difference between + the results of consecutive calls is valid. + + .. versionadded:: 3.3 + + +.. function:: process_time() + + Return the value (in fractional seconds) of the sum of the system and user + CPU time of the current process. It does not include time elapsed during + sleep. It is process-wide by definition. The reference point of the + returned value is undefined, so that only the difference between the results + of consecutive calls is valid. + + .. versionadded:: 3.3 + + .. function:: sleep(secs) Suspend execution for the given number of seconds. The argument may be a @@ -308,7 +444,7 @@ The module defines the following functions and data items: | ``%y`` | Year without century as a decimal number | | | | [00,99]. | | +-----------+------------------------------------------------+-------+ - | ``%Y`` | Year with century as a decimal number. | \(4) | + | ``%Y`` | Year with century as a decimal number. | | | | | | +-----------+------------------------------------------------+-------+ | ``%Z`` | Time zone name (no characters if no time zone | | @@ -332,12 +468,6 @@ The module defines the following functions and data items: When used with the :func:`strptime` function, ``%U`` and ``%W`` are only used in calculations when the day of the week and the year are specified. - (4) - Produces different results depending on the value of - ``time.accept2dyear`` variable. See :ref:`Year 2000 (Y2K) - issues <time-y2kissues>` for details. - - Here is an example, a format for dates compatible with that specified in the :rfc:`2822` Internet email standard. [#]_ :: @@ -418,8 +548,7 @@ The module defines the following functions and data items: +-------+-------------------+---------------------------------+ Note that unlike the C structure, the month value is a range of [1, 12], not - [0, 11]. A year value will be handled as described under :ref:`Year 2000 - (Y2K) issues <time-y2kissues>` above. A ``-1`` argument as the daylight + [0, 11]. A ``-1`` argument as the daylight savings flag, passed to :func:`mktime` will usually result in the correct daylight savings state to be filled in. diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst index cd2d205..c41e59d 100644 --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -104,13 +104,7 @@ The module also defines three convenience functions: .. function:: default_timer() - Define a default timer, in a platform specific manner. On Windows, - :func:`time.clock` has microsecond granularity but :func:`time.time`'s - granularity is 1/60th of a second; on Unix, :func:`time.clock` has 1/100th of - a second granularity and :func:`time.time` is much more precise. On either - platform, :func:`default_timer` measures wall clock time, not the CPU - time. This means that other processes running on the same computer may - interfere with the timing. + The default timer, which is always :func:`time.perf_counter`. .. function:: repeat(stmt='pass', setup='pass', timer=<default timer>, repeat=3, number=1000000) @@ -149,13 +143,20 @@ Where the following options are understood: statement to be executed once initially (default ``pass``) +.. cmdoption:: -p, --process + + measure process time, not wallclock time, using :func:`time.process_time` + instead of :func:`time.perf_counter`, which is the default + + .. versionadded:: 3.3 + .. cmdoption:: -t, --time - use :func:`time.time` (default on all platforms but Windows) + use :func:`time.time` (deprecated) .. cmdoption:: -c, --clock - use :func:`time.clock` (default on Windows) + use :func:`time.clock` (deprecated) .. cmdoption:: -v, --verbose @@ -173,12 +174,11 @@ similarly. If :option:`-n` is not given, a suitable number of loops is calculated by trying successive powers of 10 until the total time is at least 0.2 seconds. -:func:`default_timer` measurations can be affected by other programs running on -the same machine, so -the best thing to do when accurate timing is necessary is to repeat -the timing a few times and use the best time. The :option:`-r` option is good -for this; the default of 3 repetitions is probably enough in most cases. On -Unix, you can use :func:`time.clock` to measure CPU time. +:func:`default_timer` measurements can be affected by other programs running on +the same machine, so the best thing to do when accurate timing is necessary is +to repeat the timing a few times and use the best time. The :option:`-r` +option is good for this; the default of 3 repetitions is probably enough in +most cases. You can use :func:`time.process_time` to measure CPU time. .. note:: diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst index ae5635f..2e35836 100644 --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -179,35 +179,30 @@ A Simple Hello World Program :: - from tkinter import * - - class Application(Frame): - def say_hi(self): - print("hi there, everyone!") - - def createWidgets(self): - self.QUIT = Button(self) - self.QUIT["text"] = "QUIT" - self.QUIT["fg"] = "red" - self.QUIT["command"] = self.quit + import tkinter as tk - self.QUIT.pack({"side": "left"}) + class Application(tk.Frame): + def __init__(self, master=None): + tk.Frame.__init__(self, master) + self.pack() + self.createWidgets() - self.hi_there = Button(self) - self.hi_there["text"] = "Hello", - self.hi_there["command"] = self.say_hi + def createWidgets(self): + self.hi_there = tk.Button(self) + self.hi_there["text"] = "Hello World\n(click me)" + self.hi_there["command"] = self.say_hi + self.hi_there.pack(side="top") - self.hi_there.pack({"side": "left"}) + self.QUIT = tk.Button(self, text="QUIT", fg="red", + command=root.destroy) + self.QUIT.pack(side="bottom") - def __init__(self, master=None): - Frame.__init__(self, master) - self.pack() - self.createWidgets() + def say_hi(self): + print("hi there, everyone!") - root = Tk() - app = Application(master=root) - app.mainloop() - root.destroy() + root = tk.Tk() + app = Application(master=root) + app.mainloop() A (Very) Quick Look at Tcl/Tk @@ -740,22 +735,32 @@ Entry widget indexes (index, view index, etc.) displayed. You can use these :mod:`tkinter` functions to access these special points in text widgets: - AtEnd() +.. function:: AtEnd() refers to the last position in the text - AtInsert() + .. deprecated:: 3.3 + +.. function:: AtInsert() refers to the point where the text cursor is - AtSelFirst() + .. deprecated:: 3.3 + +.. function:: AtSelFirst() indicates the beginning point of the selected text - AtSelLast() + .. deprecated:: 3.3 + +.. function:: AtSelLast() denotes the last point of the selected text and finally - At(x[, y]) + .. deprecated:: 3.3 + +.. function:: At(x[, y]) refers to the character at pixel location *x*, *y* (with *y* not used in the case of a text entry widget, which contains a single line of text). + .. deprecated:: 3.3 + Text widget indexes The index notation for Text widgets is very rich and is best described in the Tk man pages. @@ -803,4 +808,3 @@ some widget (e.g. labels, buttons, menus). In these cases, Tk will not keep a reference to the image. When the last Python reference to the image object is deleted, the image data is deleted as well, and Tk will display an empty box wherever the image was used. - diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 70919ca..37d9f41 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -17,9 +17,11 @@ colorizers for on-screen displays. To simplify token stream handling, all :ref:`operators` and :ref:`delimiters` tokens are returned using the generic :data:`token.OP` token type. The exact -type can be determined by checking the token ``string`` field on the -:term:`named tuple` returned from :func:`tokenize.tokenize` for the character -sequence that identifies a specific operator token. +type can be determined by checking the ``exact_type`` property on the +:term:`named tuple` returned from :func:`tokenize.tokenize`. + +Tokenizing Input +---------------- The primary entry point is a :term:`generator`: @@ -39,9 +41,17 @@ The primary entry point is a :term:`generator`: returned as a :term:`named tuple` with the field names: ``type string start end line``. + The returned :term:`named tuple` has a additional property named + ``exact_type`` that contains the exact operator type for + :data:`token.OP` tokens. For all other token types ``exact_type`` + equals the named tuple ``type`` field. + .. versionchanged:: 3.1 Added support for named tuples. + .. versionchanged:: 3.3 + Added support for ``exact_type``. + :func:`tokenize` determines the source encoding of the file by looking for a UTF-8 BOM or encoding cookie, according to :pep:`263`. @@ -122,6 +132,38 @@ function it uses to do this is available: .. versionadded:: 3.2 +.. _tokenize-cli: + +Command-Line Usage +------------------ + +.. versionadded:: 3.3 + +The :mod:`tokenize` module can be executed as a script from the command line. +It is as simple as: + +.. code-block:: sh + + python -m tokenize [-e] [filename.py] + +The following options are accepted: + +.. program:: tokenize + +.. cmdoption:: -h, --help + + show this help message and exit + +.. cmdoption:: -e, --exact + + display token names using the exact type + +If :file:`filename.py` is specified its contents are tokenized to stdout. +Otherwise, tokenization is performed on stdin. + +Examples +------------------ + Example of a script rewriter that transforms float literals into Decimal objects:: @@ -164,3 +206,63 @@ objects:: result.append((toknum, tokval)) return untokenize(result).decode('utf-8') +Example of tokenizing from the command line. The script:: + + def say_hello(): + print("Hello, World!") + + say_hello() + +will be tokenized to the following output where the first column is the range +of the line/column coordinates where the token is found, the second column is +the name of the token, and the final column is the value of the token (if any) + +.. code-block:: sh + + $ python -m tokenize hello.py + 0,0-0,0: ENCODING 'utf-8' + 1,0-1,3: NAME 'def' + 1,4-1,13: NAME 'say_hello' + 1,13-1,14: OP '(' + 1,14-1,15: OP ')' + 1,15-1,16: OP ':' + 1,16-1,17: NEWLINE '\n' + 2,0-2,4: INDENT ' ' + 2,4-2,9: NAME 'print' + 2,9-2,10: OP '(' + 2,10-2,25: STRING '"Hello, World!"' + 2,25-2,26: OP ')' + 2,26-2,27: NEWLINE '\n' + 3,0-3,1: NL '\n' + 4,0-4,0: DEDENT '' + 4,0-4,9: NAME 'say_hello' + 4,9-4,10: OP '(' + 4,10-4,11: OP ')' + 4,11-4,12: NEWLINE '\n' + 5,0-5,0: ENDMARKER '' + +The exact token type names can be displayed using the ``-e`` option: + +.. code-block:: sh + + $ python -m tokenize -e hello.py + 0,0-0,0: ENCODING 'utf-8' + 1,0-1,3: NAME 'def' + 1,4-1,13: NAME 'say_hello' + 1,13-1,14: LPAR '(' + 1,14-1,15: RPAR ')' + 1,15-1,16: COLON ':' + 1,16-1,17: NEWLINE '\n' + 2,0-2,4: INDENT ' ' + 2,4-2,9: NAME 'print' + 2,9-2,10: LPAR '(' + 2,10-2,25: STRING '"Hello, World!"' + 2,25-2,26: RPAR ')' + 2,26-2,27: NEWLINE '\n' + 3,0-3,1: NL '\n' + 4,0-4,0: DEDENT '' + 4,0-4,9: NAME 'say_hello' + 4,9-4,10: LPAR '(' + 4,10-4,11: RPAR ')' + 4,11-4,12: NEWLINE '\n' + 5,0-5,0: ENDMARKER '' diff --git a/Doc/library/types.rst b/Doc/library/types.rst index d4a76b6..161d854 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -1,5 +1,5 @@ -:mod:`types` --- Names for built-in types -========================================= +:mod:`types` --- Dynamic type creation and names for built-in types +=================================================================== .. module:: types :synopsis: Names for built-in types. @@ -8,20 +8,72 @@ -------------- -This module defines names for some object types that are used by the standard +This module defines utility function to assist in dynamic creation of +new types. + +It also defines names for some object types that are used by the standard Python interpreter, but not exposed as builtins like :class:`int` or -:class:`str` are. Also, it does not include some of the types that arise -transparently during processing such as the ``listiterator`` type. +:class:`str` are. + + +Dynamic Type Creation +--------------------- + +.. function:: new_class(name, bases=(), kwds=None, exec_body=None) + + Creates a class object dynamically using the appropriate metaclass. + + The arguments are the components that make up a class definition: the + class name, the base classes (in order), the keyword arguments (such as + ``metaclass``) and the callback function to populate the class namespace. + + The *exec_body* callback should accept the class namespace as its sole + argument and update the namespace directly with the class contents. + + .. versionadded:: 3.3 + +.. function:: prepare_class(name, bases=(), kwds=None) + + Calculates the appropriate metaclass and creates the class namespace. + + The arguments are the components that make up a class definition: the + class name, the base classes (in order) and the keyword arguments (such as + ``metaclass``). -Typical use is for :func:`isinstance` or :func:`issubclass` checks. + The return value is a 3-tuple: ``metaclass, namespace, kwds`` -The module defines the following names: + *metaclass* is the appropriate metaclass + *namespace* is the prepared class namespace + *kwds* is an updated copy of the passed in *kwds* argument with any + ``'metaclass'`` entry removed. If no *kwds* argument is passed in, this + will be an empty dict. + + .. versionadded:: 3.3 + +.. seealso:: + + :pep:`3115` - Metaclasses in Python 3000 + Introduced the ``__prepare__`` namespace hook + + +Standard Interpreter Types +-------------------------- + +This module provides names for many of the types that are required to +implement a Python interpreter. It deliberately avoids including some of +the types that arise only incidentally during processing such as the +``listiterator`` type. + +Typical use is of these names is for :func:`isinstance` or +:func:`issubclass` checks. + +Standard names are defined for the following types: .. data:: FunctionType LambdaType - The type of user-defined functions and functions created by :keyword:`lambda` - expressions. + The type of user-defined functions and functions created by + :keyword:`lambda` expressions. .. data:: GeneratorType @@ -85,3 +137,55 @@ The module defines the following names: In other implementations of Python, this type may be identical to ``GetSetDescriptorType``. + +.. class:: MappingProxyType(mapping) + + Read-only proxy of a mapping. It provides a dynamic view on the mapping's + entries, which means that when the mapping changes, the view reflects these + changes. + + .. versionadded:: 3.3 + + .. describe:: key in proxy + + Return ``True`` if the underlying mapping has a key *key*, else + ``False``. + + .. describe:: proxy[key] + + Return the item of the underlying mapping with key *key*. Raises a + :exc:`KeyError` if *key* is not in the underlying mapping. + + .. describe:: iter(proxy) + + Return an iterator over the keys of the underlying mapping. This is a + shortcut for ``iter(proxy.keys())``. + + .. describe:: len(proxy) + + Return the number of items in the underlying mapping. + + .. method:: copy() + + Return a shallow copy of the underlying mapping. + + .. method:: get(key[, default]) + + Return the value for *key* if *key* is in the underlying mapping, else + *default*. If *default* is not given, it defaults to ``None``, so that + this method never raises a :exc:`KeyError`. + + .. method:: items() + + Return a new view of the underlying mapping's items (``(key, value)`` + pairs). + + .. method:: keys() + + Return a new view of the underlying mapping's keys. + + .. method:: values() + + Return a new view of the underlying mapping's values. + + diff --git a/Doc/library/unicodedata.rst b/Doc/library/unicodedata.rst index 42400df..b735a69 100644 --- a/Doc/library/unicodedata.rst +++ b/Doc/library/unicodedata.rst @@ -15,8 +15,8 @@ This module provides access to the Unicode Character Database (UCD) which defines character properties for all Unicode characters. The data contained in -this database is compiled from the `UCD version 6.0.0 -<http://www.unicode.org/Public/6.0.0/ucd>`_. +this database is compiled from the `UCD version 6.1.0 +<http://www.unicode.org/Public/6.1.0/ucd>`_. The module uses the same names and symbols as defined by Unicode Standard Annex #44, `"Unicode Character Database" @@ -29,6 +29,9 @@ following functions: Look up character by name. If a character with the given name is found, return the corresponding character. If not found, :exc:`KeyError` is raised. + .. versionchanged:: 3.3 + Support for name aliases [#]_ and named sequences [#]_ has been added. + .. function:: name(chr[, default]) @@ -160,3 +163,9 @@ Examples: >>> unicodedata.bidirectional('\u0660') # 'A'rabic, 'N'umber 'AN' + +.. rubric:: Footnotes + +.. [#] http://www.unicode.org/Public/6.1.0/ucd/NameAliases.txt + +.. [#] http://www.unicode.org/Public/6.1.0/ucd/NamedSequences.txt diff --git a/Doc/library/unittest.mock-examples.rst b/Doc/library/unittest.mock-examples.rst new file mode 100644 index 0000000..8e1e88a --- /dev/null +++ b/Doc/library/unittest.mock-examples.rst @@ -0,0 +1,1246 @@ +:mod:`unittest.mock` --- getting started +======================================== + +.. moduleauthor:: Michael Foord <michael@python.org> +.. currentmodule:: unittest.mock + +.. versionadded:: 3.3 + + +.. _getting-started: + +Using Mock +---------- + +Mock Patching Methods +~~~~~~~~~~~~~~~~~~~~~ + +Common uses for :class:`Mock` objects include: + +* Patching methods +* Recording method calls on objects + +You might want to replace a method on an object to check that +it is called with the correct arguments by another part of the system: + + >>> real = SomeClass() + >>> real.method = MagicMock(name='method') + >>> real.method(3, 4, 5, key='value') + <MagicMock name='method()' id='...'> + +Once our mock has been used (`real.method` in this example) it has methods +and attributes that allow you to make assertions about how it has been used. + +.. note:: + + In most of these examples the :class:`Mock` and :class:`MagicMock` classes + are interchangeable. As the `MagicMock` is the more capable class it makes + a sensible one to use by default. + +Once the mock has been called its :attr:`~Mock.called` attribute is set to +`True`. More importantly we can use the :meth:`~Mock.assert_called_with` or +:meth:`~Mock.assert_called_once_with` method to check that it was called with +the correct arguments. + +This example tests that calling `ProductionClass().method` results in a call to +the `something` method: + + >>> class ProductionClass(object): + ... def method(self): + ... self.something(1, 2, 3) + ... def something(self, a, b, c): + ... pass + ... + >>> real = ProductionClass() + >>> real.something = MagicMock() + >>> real.method() + >>> real.something.assert_called_once_with(1, 2, 3) + + + +Mock for Method Calls on an Object +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the last example we patched a method directly on an object to check that it +was called correctly. Another common use case is to pass an object into a +method (or some part of the system under test) and then check that it is used +in the correct way. + +The simple `ProductionClass` below has a `closer` method. If it is called with +an object then it calls `close` on it. + + >>> class ProductionClass(object): + ... def closer(self, something): + ... something.close() + ... + +So to test it we need to pass in an object with a `close` method and check +that it was called correctly. + + >>> real = ProductionClass() + >>> mock = Mock() + >>> real.closer(mock) + >>> mock.close.assert_called_with() + +We don't have to do any work to provide the 'close' method on our mock. +Accessing close creates it. So, if 'close' hasn't already been called then +accessing it in the test will create it, but :meth:`~Mock.assert_called_with` +will raise a failure exception. + + +Mocking Classes +~~~~~~~~~~~~~~~ + +A common use case is to mock out classes instantiated by your code under test. +When you patch a class, then that class is replaced with a mock. Instances +are created by *calling the class*. This means you access the "mock instance" +by looking at the return value of the mocked class. + +In the example below we have a function `some_function` that instantiates `Foo` +and calls a method on it. The call to `patch` replaces the class `Foo` with a +mock. The `Foo` instance is the result of calling the mock, so it is configured +by modifying the mock :attr:`~Mock.return_value`. + + >>> def some_function(): + ... instance = module.Foo() + ... return instance.method() + ... + >>> with patch('module.Foo') as mock: + ... instance = mock.return_value + ... instance.method.return_value = 'the result' + ... result = some_function() + ... assert result == 'the result' + + +Naming your mocks +~~~~~~~~~~~~~~~~~ + +It can be useful to give your mocks a name. The name is shown in the repr of +the mock and can be helpful when the mock appears in test failure messages. The +name is also propagated to attributes or methods of the mock: + + >>> mock = MagicMock(name='foo') + >>> mock + <MagicMock name='foo' id='...'> + >>> mock.method + <MagicMock name='foo.method' id='...'> + + +Tracking all Calls +~~~~~~~~~~~~~~~~~~ + +Often you want to track more than a single call to a method. The +:attr:`~Mock.mock_calls` attribute records all calls +to child attributes of the mock - and also to their children. + + >>> mock = MagicMock() + >>> mock.method() + <MagicMock name='mock.method()' id='...'> + >>> mock.attribute.method(10, x=53) + <MagicMock name='mock.attribute.method()' id='...'> + >>> mock.mock_calls + [call.method(), call.attribute.method(10, x=53)] + +If you make an assertion about `mock_calls` and any unexpected methods +have been called, then the assertion will fail. This is useful because as well +as asserting that the calls you expected have been made, you are also checking +that they were made in the right order and with no additional calls: + +You use the :data:`call` object to construct lists for comparing with +`mock_calls`: + + >>> expected = [call.method(), call.attribute.method(10, x=53)] + >>> mock.mock_calls == expected + True + + +Setting Return Values and Attributes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Setting the return values on a mock object is trivially easy: + + >>> mock = Mock() + >>> mock.return_value = 3 + >>> mock() + 3 + +Of course you can do the same for methods on the mock: + + >>> mock = Mock() + >>> mock.method.return_value = 3 + >>> mock.method() + 3 + +The return value can also be set in the constructor: + + >>> mock = Mock(return_value=3) + >>> mock() + 3 + +If you need an attribute setting on your mock, just do it: + + >>> mock = Mock() + >>> mock.x = 3 + >>> mock.x + 3 + +Sometimes you want to mock up a more complex situation, like for example +`mock.connection.cursor().execute("SELECT 1")`. If we wanted this call to +return a list, then we have to configure the result of the nested call. + +We can use :data:`call` to construct the set of calls in a "chained call" like +this for easy assertion afterwards: + + >>> mock = Mock() + >>> cursor = mock.connection.cursor.return_value + >>> cursor.execute.return_value = ['foo'] + >>> mock.connection.cursor().execute("SELECT 1") + ['foo'] + >>> expected = call.connection.cursor().execute("SELECT 1").call_list() + >>> mock.mock_calls + [call.connection.cursor(), call.connection.cursor().execute('SELECT 1')] + >>> mock.mock_calls == expected + True + +It is the call to `.call_list()` that turns our call object into a list of +calls representing the chained calls. + + +Raising exceptions with mocks +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A useful attribute is :attr:`~Mock.side_effect`. If you set this to an +exception class or instance then the exception will be raised when the mock +is called. + + >>> mock = Mock(side_effect=Exception('Boom!')) + >>> mock() + Traceback (most recent call last): + ... + Exception: Boom! + + +Side effect functions and iterables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`side_effect` can also be set to a function or an iterable. The use case for +`side_effect` as an iterable is where your mock is going to be called several +times, and you want each call to return a different value. When you set +`side_effect` to an iterable every call to the mock returns the next value +from the iterable: + + >>> mock = MagicMock(side_effect=[4, 5, 6]) + >>> mock() + 4 + >>> mock() + 5 + >>> mock() + 6 + + +For more advanced use cases, like dynamically varying the return values +depending on what the mock is called with, `side_effect` can be a function. +The function will be called with the same arguments as the mock. Whatever the +function returns is what the call returns: + + >>> vals = {(1, 2): 1, (2, 3): 2} + >>> def side_effect(*args): + ... return vals[args] + ... + >>> mock = MagicMock(side_effect=side_effect) + >>> mock(1, 2) + 1 + >>> mock(2, 3) + 2 + + +Creating a Mock from an Existing Object +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +One problem with over use of mocking is that it couples your tests to the +implementation of your mocks rather than your real code. Suppose you have a +class that implements `some_method`. In a test for another class, you +provide a mock of this object that *also* provides `some_method`. If later +you refactor the first class, so that it no longer has `some_method` - then +your tests will continue to pass even though your code is now broken! + +`Mock` allows you to provide an object as a specification for the mock, +using the `spec` keyword argument. Accessing methods / attributes on the +mock that don't exist on your specification object will immediately raise an +attribute error. If you change the implementation of your specification, then +tests that use that class will start failing immediately without you having to +instantiate the class in those tests. + + >>> mock = Mock(spec=SomeClass) + >>> mock.old_method() + Traceback (most recent call last): + ... + AttributeError: object has no attribute 'old_method' + +If you want a stronger form of specification that prevents the setting +of arbitrary attributes as well as the getting of them then you can use +`spec_set` instead of `spec`. + + + +Patch Decorators +---------------- + +.. note:: + + With `patch` it matters that you patch objects in the namespace where they + are looked up. This is normally straightforward, but for a quick guide + read :ref:`where to patch <where-to-patch>`. + + +A common need in tests is to patch a class attribute or a module attribute, +for example patching a builtin or patching a class in a module to test that it +is instantiated. Modules and classes are effectively global, so patching on +them has to be undone after the test or the patch will persist into other +tests and cause hard to diagnose problems. + +mock provides three convenient decorators for this: `patch`, `patch.object` and +`patch.dict`. `patch` takes a single string, of the form +`package.module.Class.attribute` to specify the attribute you are patching. It +also optionally takes a value that you want the attribute (or class or +whatever) to be replaced with. 'patch.object' takes an object and the name of +the attribute you would like patched, plus optionally the value to patch it +with. + +`patch.object`: + + >>> original = SomeClass.attribute + >>> @patch.object(SomeClass, 'attribute', sentinel.attribute) + ... def test(): + ... assert SomeClass.attribute == sentinel.attribute + ... + >>> test() + >>> assert SomeClass.attribute == original + + >>> @patch('package.module.attribute', sentinel.attribute) + ... def test(): + ... from package.module import attribute + ... assert attribute is sentinel.attribute + ... + >>> test() + +If you are patching a module (including `__builtin__`) then use `patch` +instead of `patch.object`: + + >>> mock = MagicMock(return_value = sentinel.file_handle) + >>> with patch('__builtin__.open', mock): + ... handle = open('filename', 'r') + ... + >>> mock.assert_called_with('filename', 'r') + >>> assert handle == sentinel.file_handle, "incorrect file handle returned" + +The module name can be 'dotted', in the form `package.module` if needed: + + >>> @patch('package.module.ClassName.attribute', sentinel.attribute) + ... def test(): + ... from package.module import ClassName + ... assert ClassName.attribute == sentinel.attribute + ... + >>> test() + +A nice pattern is to actually decorate test methods themselves: + + >>> class MyTest(unittest2.TestCase): + ... @patch.object(SomeClass, 'attribute', sentinel.attribute) + ... def test_something(self): + ... self.assertEqual(SomeClass.attribute, sentinel.attribute) + ... + >>> original = SomeClass.attribute + >>> MyTest('test_something').test_something() + >>> assert SomeClass.attribute == original + +If you want to patch with a Mock, you can use `patch` with only one argument +(or `patch.object` with two arguments). The mock will be created for you and +passed into the test function / method: + + >>> class MyTest(unittest2.TestCase): + ... @patch.object(SomeClass, 'static_method') + ... def test_something(self, mock_method): + ... SomeClass.static_method() + ... mock_method.assert_called_with() + ... + >>> MyTest('test_something').test_something() + +You can stack up multiple patch decorators using this pattern: + + >>> class MyTest(unittest2.TestCase): + ... @patch('package.module.ClassName1') + ... @patch('package.module.ClassName2') + ... def test_something(self, MockClass2, MockClass1): + ... self.assertTrue(package.module.ClassName1 is MockClass1) + ... self.assertTrue(package.module.ClassName2 is MockClass2) + ... + >>> MyTest('test_something').test_something() + +When you nest patch decorators the mocks are passed in to the decorated +function in the same order they applied (the normal *python* order that +decorators are applied). This means from the bottom up, so in the example +above the mock for `test_module.ClassName2` is passed in first. + +There is also :func:`patch.dict` for setting values in a dictionary just +during a scope and restoring the dictionary to its original state when the test +ends: + + >>> foo = {'key': 'value'} + >>> original = foo.copy() + >>> with patch.dict(foo, {'newkey': 'newvalue'}, clear=True): + ... assert foo == {'newkey': 'newvalue'} + ... + >>> assert foo == original + +`patch`, `patch.object` and `patch.dict` can all be used as context managers. + +Where you use `patch` to create a mock for you, you can get a reference to the +mock using the "as" form of the with statement: + + >>> class ProductionClass(object): + ... def method(self): + ... pass + ... + >>> with patch.object(ProductionClass, 'method') as mock_method: + ... mock_method.return_value = None + ... real = ProductionClass() + ... real.method(1, 2, 3) + ... + >>> mock_method.assert_called_with(1, 2, 3) + + +As an alternative `patch`, `patch.object` and `patch.dict` can be used as +class decorators. When used in this way it is the same as applying the +decorator indvidually to every method whose name starts with "test". + + +.. _further-examples: + +Further Examples +================ + + +Here are some more examples for some slightly more advanced scenarios. + + +Mocking chained calls +--------------------- + +Mocking chained calls is actually straightforward with mock once you +understand the :attr:`~Mock.return_value` attribute. When a mock is called for +the first time, or you fetch its `return_value` before it has been called, a +new `Mock` is created. + +This means that you can see how the object returned from a call to a mocked +object has been used by interrogating the `return_value` mock: + + >>> mock = Mock() + >>> mock().foo(a=2, b=3) + <Mock name='mock().foo()' id='...'> + >>> mock.return_value.foo.assert_called_with(a=2, b=3) + +From here it is a simple step to configure and then make assertions about +chained calls. Of course another alternative is writing your code in a more +testable way in the first place... + +So, suppose we have some code that looks a little bit like this: + + >>> class Something(object): + ... def __init__(self): + ... self.backend = BackendProvider() + ... def method(self): + ... response = self.backend.get_endpoint('foobar').create_call('spam', 'eggs').start_call() + ... # more code + +Assuming that `BackendProvider` is already well tested, how do we test +`method()`? Specifically, we want to test that the code section `# more +code` uses the response object in the correct way. + +As this chain of calls is made from an instance attribute we can monkey patch +the `backend` attribute on a `Something` instance. In this particular case +we are only interested in the return value from the final call to +`start_call` so we don't have much configuration to do. Let's assume the +object it returns is 'file-like', so we'll ensure that our response object +uses the builtin `file` as its `spec`. + +To do this we create a mock instance as our mock backend and create a mock +response object for it. To set the response as the return value for that final +`start_call` we could do this: + + `mock_backend.get_endpoint.return_value.create_call.return_value.start_call.return_value = mock_response`. + +We can do that in a slightly nicer way using the :meth:`~Mock.configure_mock` +method to directly set the return value for us: + + >>> something = Something() + >>> mock_response = Mock(spec=file) + >>> mock_backend = Mock() + >>> config = {'get_endpoint.return_value.create_call.return_value.start_call.return_value': mock_response} + >>> mock_backend.configure_mock(**config) + +With these we monkey patch the "mock backend" in place and can make the real +call: + + >>> something.backend = mock_backend + >>> something.method() + +Using :attr:`~Mock.mock_calls` we can check the chained call with a single +assert. A chained call is several calls in one line of code, so there will be +several entries in `mock_calls`. We can use :meth:`call.call_list` to create +this list of calls for us: + + >>> chained = call.get_endpoint('foobar').create_call('spam', 'eggs').start_call() + >>> call_list = chained.call_list() + >>> assert mock_backend.mock_calls == call_list + + +Partial mocking +--------------- + +In some tests I wanted to mock out a call to `datetime.date.today() +<http://docs.python.org/library/datetime.html#datetime.date.today>`_ to return +a known date, but I didn't want to prevent the code under test from +creating new date objects. Unfortunately `datetime.date` is written in C, and +so I couldn't just monkey-patch out the static `date.today` method. + +I found a simple way of doing this that involved effectively wrapping the date +class with a mock, but passing through calls to the constructor to the real +class (and returning real instances). + +The :func:`patch decorator <patch>` is used here to +mock out the `date` class in the module under test. The :attr:`side_effect` +attribute on the mock date class is then set to a lambda function that returns +a real date. When the mock date class is called a real date will be +constructed and returned by `side_effect`. + + >>> from datetime import date + >>> with patch('mymodule.date') as mock_date: + ... mock_date.today.return_value = date(2010, 10, 8) + ... mock_date.side_effect = lambda *args, **kw: date(*args, **kw) + ... + ... assert mymodule.date.today() == date(2010, 10, 8) + ... assert mymodule.date(2009, 6, 8) == date(2009, 6, 8) + ... + +Note that we don't patch `datetime.date` globally, we patch `date` in the +module that *uses* it. See :ref:`where to patch <where-to-patch>`. + +When `date.today()` is called a known date is returned, but calls to the +`date(...)` constructor still return normal dates. Without this you can find +yourself having to calculate an expected result using exactly the same +algorithm as the code under test, which is a classic testing anti-pattern. + +Calls to the date constructor are recorded in the `mock_date` attributes +(`call_count` and friends) which may also be useful for your tests. + +An alternative way of dealing with mocking dates, or other builtin classes, +is discussed in `this blog entry +<http://williamjohnbert.com/2011/07/how-to-unit-testing-in-django-with-mocking-and-patching/>`_. + + +Mocking a Generator Method +-------------------------- + +A Python generator is a function or method that uses the `yield statement +<http://docs.python.org/reference/simple_stmts.html#the-yield-statement>`_ to +return a series of values when iterated over [#]_. + +A generator method / function is called to return the generator object. It is +the generator object that is then iterated over. The protocol method for +iteration is `__iter__ +<http://docs.python.org/library/stdtypes.html#container.__iter__>`_, so we can +mock this using a `MagicMock`. + +Here's an example class with an "iter" method implemented as a generator: + + >>> class Foo(object): + ... def iter(self): + ... for i in [1, 2, 3]: + ... yield i + ... + >>> foo = Foo() + >>> list(foo.iter()) + [1, 2, 3] + + +How would we mock this class, and in particular its "iter" method? + +To configure the values returned from the iteration (implicit in the call to +`list`), we need to configure the object returned by the call to `foo.iter()`. + + >>> mock_foo = MagicMock() + >>> mock_foo.iter.return_value = iter([1, 2, 3]) + >>> list(mock_foo.iter()) + [1, 2, 3] + +.. [#] There are also generator expressions and more `advanced uses + <http://www.dabeaz.com/coroutines/index.html>`_ of generators, but we aren't + concerned about them here. A very good introduction to generators and how + powerful they are is: `Generator Tricks for Systems Programmers + <http://www.dabeaz.com/generators/>`_. + + +Applying the same patch to every test method +-------------------------------------------- + +If you want several patches in place for multiple test methods the obvious way +is to apply the patch decorators to every method. This can feel like unnecessary +repetition. For Python 2.6 or more recent you can use `patch` (in all its +various forms) as a class decorator. This applies the patches to all test +methods on the class. A test method is identified by methods whose names start +with `test`: + + >>> @patch('mymodule.SomeClass') + ... class MyTest(TestCase): + ... + ... def test_one(self, MockSomeClass): + ... self.assertTrue(mymodule.SomeClass is MockSomeClass) + ... + ... def test_two(self, MockSomeClass): + ... self.assertTrue(mymodule.SomeClass is MockSomeClass) + ... + ... def not_a_test(self): + ... return 'something' + ... + >>> MyTest('test_one').test_one() + >>> MyTest('test_two').test_two() + >>> MyTest('test_two').not_a_test() + 'something' + +An alternative way of managing patches is to use the :ref:`start-and-stop`. +These allow you to move the patching into your `setUp` and `tearDown` methods. + + >>> class MyTest(TestCase): + ... def setUp(self): + ... self.patcher = patch('mymodule.foo') + ... self.mock_foo = self.patcher.start() + ... + ... def test_foo(self): + ... self.assertTrue(mymodule.foo is self.mock_foo) + ... + ... def tearDown(self): + ... self.patcher.stop() + ... + >>> MyTest('test_foo').run() + +If you use this technique you must ensure that the patching is "undone" by +calling `stop`. This can be fiddlier than you might think, because if an +exception is raised in the setUp then tearDown is not called. +:meth:`unittest.TestCase.addCleanup` makes this easier: + + >>> class MyTest(TestCase): + ... def setUp(self): + ... patcher = patch('mymodule.foo') + ... self.addCleanup(patcher.stop) + ... self.mock_foo = patcher.start() + ... + ... def test_foo(self): + ... self.assertTrue(mymodule.foo is self.mock_foo) + ... + >>> MyTest('test_foo').run() + + +Mocking Unbound Methods +----------------------- + +Whilst writing tests today I needed to patch an *unbound method* (patching the +method on the class rather than on the instance). I needed self to be passed +in as the first argument because I want to make asserts about which objects +were calling this particular method. The issue is that you can't patch with a +mock for this, because if you replace an unbound method with a mock it doesn't +become a bound method when fetched from the instance, and so it doesn't get +self passed in. The workaround is to patch the unbound method with a real +function instead. The :func:`patch` decorator makes it so simple to +patch out methods with a mock that having to create a real function becomes a +nuisance. + +If you pass `autospec=True` to patch then it does the patching with a +*real* function object. This function object has the same signature as the one +it is replacing, but delegates to a mock under the hood. You still get your +mock auto-created in exactly the same way as before. What it means though, is +that if you use it to patch out an unbound method on a class the mocked +function will be turned into a bound method if it is fetched from an instance. +It will have `self` passed in as the first argument, which is exactly what I +wanted: + + >>> class Foo(object): + ... def foo(self): + ... pass + ... + >>> with patch.object(Foo, 'foo', autospec=True) as mock_foo: + ... mock_foo.return_value = 'foo' + ... foo = Foo() + ... foo.foo() + ... + 'foo' + >>> mock_foo.assert_called_once_with(foo) + +If we don't use `autospec=True` then the unbound method is patched out +with a Mock instance instead, and isn't called with `self`. + + +Checking multiple calls with mock +--------------------------------- + +mock has a nice API for making assertions about how your mock objects are used. + + >>> mock = Mock() + >>> mock.foo_bar.return_value = None + >>> mock.foo_bar('baz', spam='eggs') + >>> mock.foo_bar.assert_called_with('baz', spam='eggs') + +If your mock is only being called once you can use the +:meth:`assert_called_once_with` method that also asserts that the +:attr:`call_count` is one. + + >>> mock.foo_bar.assert_called_once_with('baz', spam='eggs') + >>> mock.foo_bar() + >>> mock.foo_bar.assert_called_once_with('baz', spam='eggs') + Traceback (most recent call last): + ... + AssertionError: Expected to be called once. Called 2 times. + +Both `assert_called_with` and `assert_called_once_with` make assertions about +the *most recent* call. If your mock is going to be called several times, and +you want to make assertions about *all* those calls you can use +:attr:`~Mock.call_args_list`: + + >>> mock = Mock(return_value=None) + >>> mock(1, 2, 3) + >>> mock(4, 5, 6) + >>> mock() + >>> mock.call_args_list + [call(1, 2, 3), call(4, 5, 6), call()] + +The :data:`call` helper makes it easy to make assertions about these calls. You +can build up a list of expected calls and compare it to `call_args_list`. This +looks remarkably similar to the repr of the `call_args_list`: + + >>> expected = [call(1, 2, 3), call(4, 5, 6), call()] + >>> mock.call_args_list == expected + True + + +Coping with mutable arguments +----------------------------- + +Another situation is rare, but can bite you, is when your mock is called with +mutable arguments. `call_args` and `call_args_list` store *references* to the +arguments. If the arguments are mutated by the code under test then you can no +longer make assertions about what the values were when the mock was called. + +Here's some example code that shows the problem. Imagine the following functions +defined in 'mymodule':: + + def frob(val): + pass + + def grob(val): + "First frob and then clear val" + frob(val) + val.clear() + +When we try to test that `grob` calls `frob` with the correct argument look +what happens: + + >>> with patch('mymodule.frob') as mock_frob: + ... val = set([6]) + ... mymodule.grob(val) + ... + >>> val + set([]) + >>> mock_frob.assert_called_with(set([6])) + Traceback (most recent call last): + ... + AssertionError: Expected: ((set([6]),), {}) + Called with: ((set([]),), {}) + +One possibility would be for mock to copy the arguments you pass in. This +could then cause problems if you do assertions that rely on object identity +for equality. + +Here's one solution that uses the :attr:`side_effect` +functionality. If you provide a `side_effect` function for a mock then +`side_effect` will be called with the same args as the mock. This gives us an +opportunity to copy the arguments and store them for later assertions. In this +example I'm using *another* mock to store the arguments so that I can use the +mock methods for doing the assertion. Again a helper function sets this up for +me. + + >>> from copy import deepcopy + >>> from unittest.mock import Mock, patch, DEFAULT + >>> def copy_call_args(mock): + ... new_mock = Mock() + ... def side_effect(*args, **kwargs): + ... args = deepcopy(args) + ... kwargs = deepcopy(kwargs) + ... new_mock(*args, **kwargs) + ... return DEFAULT + ... mock.side_effect = side_effect + ... return new_mock + ... + >>> with patch('mymodule.frob') as mock_frob: + ... new_mock = copy_call_args(mock_frob) + ... val = set([6]) + ... mymodule.grob(val) + ... + >>> new_mock.assert_called_with(set([6])) + >>> new_mock.call_args + call(set([6])) + +`copy_call_args` is called with the mock that will be called. It returns a new +mock that we do the assertion on. The `side_effect` function makes a copy of +the args and calls our `new_mock` with the copy. + +.. note:: + + If your mock is only going to be used once there is an easier way of + checking arguments at the point they are called. You can simply do the + checking inside a `side_effect` function. + + >>> def side_effect(arg): + ... assert arg == set([6]) + ... + >>> mock = Mock(side_effect=side_effect) + >>> mock(set([6])) + >>> mock(set()) + Traceback (most recent call last): + ... + AssertionError + +An alternative approach is to create a subclass of `Mock` or `MagicMock` that +copies (using :func:`copy.deepcopy`) the arguments. +Here's an example implementation: + + >>> from copy import deepcopy + >>> class CopyingMock(MagicMock): + ... def __call__(self, *args, **kwargs): + ... args = deepcopy(args) + ... kwargs = deepcopy(kwargs) + ... return super(CopyingMock, self).__call__(*args, **kwargs) + ... + >>> c = CopyingMock(return_value=None) + >>> arg = set() + >>> c(arg) + >>> arg.add(1) + >>> c.assert_called_with(set()) + >>> c.assert_called_with(arg) + Traceback (most recent call last): + ... + AssertionError: Expected call: mock(set([1])) + Actual call: mock(set([])) + >>> c.foo + <CopyingMock name='mock.foo' id='...'> + +When you subclass `Mock` or `MagicMock` all dynamically created attributes, +and the `return_value` will use your subclass automatically. That means all +children of a `CopyingMock` will also have the type `CopyingMock`. + + +Nesting Patches +--------------- + +Using patch as a context manager is nice, but if you do multiple patches you +can end up with nested with statements indenting further and further to the +right: + + >>> class MyTest(TestCase): + ... + ... def test_foo(self): + ... with patch('mymodule.Foo') as mock_foo: + ... with patch('mymodule.Bar') as mock_bar: + ... with patch('mymodule.Spam') as mock_spam: + ... assert mymodule.Foo is mock_foo + ... assert mymodule.Bar is mock_bar + ... assert mymodule.Spam is mock_spam + ... + >>> original = mymodule.Foo + >>> MyTest('test_foo').test_foo() + >>> assert mymodule.Foo is original + +With unittest `cleanup` functions and the :ref:`start-and-stop` we can +achieve the same effect without the nested indentation. A simple helper +method, `create_patch`, puts the patch in place and returns the created mock +for us: + + >>> class MyTest(TestCase): + ... + ... def create_patch(self, name): + ... patcher = patch(name) + ... thing = patcher.start() + ... self.addCleanup(patcher.stop) + ... return thing + ... + ... def test_foo(self): + ... mock_foo = self.create_patch('mymodule.Foo') + ... mock_bar = self.create_patch('mymodule.Bar') + ... mock_spam = self.create_patch('mymodule.Spam') + ... + ... assert mymodule.Foo is mock_foo + ... assert mymodule.Bar is mock_bar + ... assert mymodule.Spam is mock_spam + ... + >>> original = mymodule.Foo + >>> MyTest('test_foo').run() + >>> assert mymodule.Foo is original + + +Mocking a dictionary with MagicMock +----------------------------------- + +You may want to mock a dictionary, or other container object, recording all +access to it whilst having it still behave like a dictionary. + +We can do this with :class:`MagicMock`, which will behave like a dictionary, +and using :data:`~Mock.side_effect` to delegate dictionary access to a real +underlying dictionary that is under our control. + +When the `__getitem__` and `__setitem__` methods of our `MagicMock` are called +(normal dictionary access) then `side_effect` is called with the key (and in +the case of `__setitem__` the value too). We can also control what is returned. + +After the `MagicMock` has been used we can use attributes like +:data:`~Mock.call_args_list` to assert about how the dictionary was used: + + >>> my_dict = {'a': 1, 'b': 2, 'c': 3} + >>> def getitem(name): + ... return my_dict[name] + ... + >>> def setitem(name, val): + ... my_dict[name] = val + ... + >>> mock = MagicMock() + >>> mock.__getitem__.side_effect = getitem + >>> mock.__setitem__.side_effect = setitem + +.. note:: + + An alternative to using `MagicMock` is to use `Mock` and *only* provide + the magic methods you specifically want: + + >>> mock = Mock() + >>> mock.__setitem__ = Mock(side_effect=getitem) + >>> mock.__getitem__ = Mock(side_effect=setitem) + + A *third* option is to use `MagicMock` but passing in `dict` as the `spec` + (or `spec_set`) argument so that the `MagicMock` created only has + dictionary magic methods available: + + >>> mock = MagicMock(spec_set=dict) + >>> mock.__getitem__.side_effect = getitem + >>> mock.__setitem__.side_effect = setitem + +With these side effect functions in place, the `mock` will behave like a normal +dictionary but recording the access. It even raises a `KeyError` if you try +to access a key that doesn't exist. + + >>> mock['a'] + 1 + >>> mock['c'] + 3 + >>> mock['d'] + Traceback (most recent call last): + ... + KeyError: 'd' + >>> mock['b'] = 'fish' + >>> mock['d'] = 'eggs' + >>> mock['b'] + 'fish' + >>> mock['d'] + 'eggs' + +After it has been used you can make assertions about the access using the normal +mock methods and attributes: + + >>> mock.__getitem__.call_args_list + [call('a'), call('c'), call('d'), call('b'), call('d')] + >>> mock.__setitem__.call_args_list + [call('b', 'fish'), call('d', 'eggs')] + >>> my_dict + {'a': 1, 'c': 3, 'b': 'fish', 'd': 'eggs'} + + +Mock subclasses and their attributes +------------------------------------ + +There are various reasons why you might want to subclass `Mock`. One reason +might be to add helper methods. Here's a silly example: + + >>> class MyMock(MagicMock): + ... def has_been_called(self): + ... return self.called + ... + >>> mymock = MyMock(return_value=None) + >>> mymock + <MyMock id='...'> + >>> mymock.has_been_called() + False + >>> mymock() + >>> mymock.has_been_called() + True + +The standard behaviour for `Mock` instances is that attributes and the return +value mocks are of the same type as the mock they are accessed on. This ensures +that `Mock` attributes are `Mocks` and `MagicMock` attributes are `MagicMocks` +[#]_. So if you're subclassing to add helper methods then they'll also be +available on the attributes and return value mock of instances of your +subclass. + + >>> mymock.foo + <MyMock name='mock.foo' id='...'> + >>> mymock.foo.has_been_called() + False + >>> mymock.foo() + <MyMock name='mock.foo()' id='...'> + >>> mymock.foo.has_been_called() + True + +Sometimes this is inconvenient. For example, `one user +<https://code.google.com/p/mock/issues/detail?id=105>`_ is subclassing mock to +created a `Twisted adaptor +<http://twistedmatrix.com/documents/11.0.0/api/twisted.python.components.html>`_. +Having this applied to attributes too actually causes errors. + +`Mock` (in all its flavours) uses a method called `_get_child_mock` to create +these "sub-mocks" for attributes and return values. You can prevent your +subclass being used for attributes by overriding this method. The signature is +that it takes arbitrary keyword arguments (`**kwargs`) which are then passed +onto the mock constructor: + + >>> class Subclass(MagicMock): + ... def _get_child_mock(self, **kwargs): + ... return MagicMock(**kwargs) + ... + >>> mymock = Subclass() + >>> mymock.foo + <MagicMock name='mock.foo' id='...'> + >>> assert isinstance(mymock, Subclass) + >>> assert not isinstance(mymock.foo, Subclass) + >>> assert not isinstance(mymock(), Subclass) + +.. [#] An exception to this rule are the non-callable mocks. Attributes use the + callable variant because otherwise non-callable mocks couldn't have callable + methods. + + +Mocking imports with patch.dict +------------------------------- + +One situation where mocking can be hard is where you have a local import inside +a function. These are harder to mock because they aren't using an object from +the module namespace that we can patch out. + +Generally local imports are to be avoided. They are sometimes done to prevent +circular dependencies, for which there is *usually* a much better way to solve +the problem (refactor the code) or to prevent "up front costs" by delaying the +import. This can also be solved in better ways than an unconditional local +import (store the module as a class or module attribute and only do the import +on first use). + +That aside there is a way to use `mock` to affect the results of an import. +Importing fetches an *object* from the `sys.modules` dictionary. Note that it +fetches an *object*, which need not be a module. Importing a module for the +first time results in a module object being put in `sys.modules`, so usually +when you import something you get a module back. This need not be the case +however. + +This means you can use :func:`patch.dict` to *temporarily* put a mock in place +in `sys.modules`. Any imports whilst this patch is active will fetch the mock. +When the patch is complete (the decorated function exits, the with statement +body is complete or `patcher.stop()` is called) then whatever was there +previously will be restored safely. + +Here's an example that mocks out the 'fooble' module. + + >>> mock = Mock() + >>> with patch.dict('sys.modules', {'fooble': mock}): + ... import fooble + ... fooble.blob() + ... + <Mock name='mock.blob()' id='...'> + >>> assert 'fooble' not in sys.modules + >>> mock.blob.assert_called_once_with() + +As you can see the `import fooble` succeeds, but on exit there is no 'fooble' +left in `sys.modules`. + +This also works for the `from module import name` form: + + >>> mock = Mock() + >>> with patch.dict('sys.modules', {'fooble': mock}): + ... from fooble import blob + ... blob.blip() + ... + <Mock name='mock.blob.blip()' id='...'> + >>> mock.blob.blip.assert_called_once_with() + +With slightly more work you can also mock package imports: + + >>> mock = Mock() + >>> modules = {'package': mock, 'package.module': mock.module} + >>> with patch.dict('sys.modules', modules): + ... from package.module import fooble + ... fooble() + ... + <Mock name='mock.module.fooble()' id='...'> + >>> mock.module.fooble.assert_called_once_with() + + +Tracking order of calls and less verbose call assertions +-------------------------------------------------------- + +The :class:`Mock` class allows you to track the *order* of method calls on +your mock objects through the :attr:`~Mock.method_calls` attribute. This +doesn't allow you to track the order of calls between separate mock objects, +however we can use :attr:`~Mock.mock_calls` to achieve the same effect. + +Because mocks track calls to child mocks in `mock_calls`, and accessing an +arbitrary attribute of a mock creates a child mock, we can create our separate +mocks from a parent one. Calls to those child mock will then all be recorded, +in order, in the `mock_calls` of the parent: + + >>> manager = Mock() + >>> mock_foo = manager.foo + >>> mock_bar = manager.bar + + >>> mock_foo.something() + <Mock name='mock.foo.something()' id='...'> + >>> mock_bar.other.thing() + <Mock name='mock.bar.other.thing()' id='...'> + + >>> manager.mock_calls + [call.foo.something(), call.bar.other.thing()] + +We can then assert about the calls, including the order, by comparing with +the `mock_calls` attribute on the manager mock: + + >>> expected_calls = [call.foo.something(), call.bar.other.thing()] + >>> manager.mock_calls == expected_calls + True + +If `patch` is creating, and putting in place, your mocks then you can attach +them to a manager mock using the :meth:`~Mock.attach_mock` method. After +attaching calls will be recorded in `mock_calls` of the manager. + + >>> manager = MagicMock() + >>> with patch('mymodule.Class1') as MockClass1: + ... with patch('mymodule.Class2') as MockClass2: + ... manager.attach_mock(MockClass1, 'MockClass1') + ... manager.attach_mock(MockClass2, 'MockClass2') + ... MockClass1().foo() + ... MockClass2().bar() + ... + <MagicMock name='mock.MockClass1().foo()' id='...'> + <MagicMock name='mock.MockClass2().bar()' id='...'> + >>> manager.mock_calls + [call.MockClass1(), + call.MockClass1().foo(), + call.MockClass2(), + call.MockClass2().bar()] + +If many calls have been made, but you're only interested in a particular +sequence of them then an alternative is to use the +:meth:`~Mock.assert_has_calls` method. This takes a list of calls (constructed +with the :data:`call` object). If that sequence of calls are in +:attr:`~Mock.mock_calls` then the assert succeeds. + + >>> m = MagicMock() + >>> m().foo().bar().baz() + <MagicMock name='mock().foo().bar().baz()' id='...'> + >>> m.one().two().three() + <MagicMock name='mock.one().two().three()' id='...'> + >>> calls = call.one().two().three().call_list() + >>> m.assert_has_calls(calls) + +Even though the chained call `m.one().two().three()` aren't the only calls that +have been made to the mock, the assert still succeeds. + +Sometimes a mock may have several calls made to it, and you are only interested +in asserting about *some* of those calls. You may not even care about the +order. In this case you can pass `any_order=True` to `assert_has_calls`: + + >>> m = MagicMock() + >>> m(1), m.two(2, 3), m.seven(7), m.fifty('50') + (...) + >>> calls = [call.fifty('50'), call(1), call.seven(7)] + >>> m.assert_has_calls(calls, any_order=True) + + +More complex argument matching +------------------------------ + +Using the same basic concept as :data:`ANY` we can implement matchers to do more +complex assertions on objects used as arguments to mocks. + +Suppose we expect some object to be passed to a mock that by default +compares equal based on object identity (which is the Python default for user +defined classes). To use :meth:`~Mock.assert_called_with` we would need to pass +in the exact same object. If we are only interested in some of the attributes +of this object then we can create a matcher that will check these attributes +for us. + +You can see in this example how a 'standard' call to `assert_called_with` isn't +sufficient: + + >>> class Foo(object): + ... def __init__(self, a, b): + ... self.a, self.b = a, b + ... + >>> mock = Mock(return_value=None) + >>> mock(Foo(1, 2)) + >>> mock.assert_called_with(Foo(1, 2)) + Traceback (most recent call last): + ... + AssertionError: Expected: call(<__main__.Foo object at 0x...>) + Actual call: call(<__main__.Foo object at 0x...>) + +A comparison function for our `Foo` class might look something like this: + + >>> def compare(self, other): + ... if not type(self) == type(other): + ... return False + ... if self.a != other.a: + ... return False + ... if self.b != other.b: + ... return False + ... return True + ... + +And a matcher object that can use comparison functions like this for its +equality operation would look something like this: + + >>> class Matcher(object): + ... def __init__(self, compare, some_obj): + ... self.compare = compare + ... self.some_obj = some_obj + ... def __eq__(self, other): + ... return self.compare(self.some_obj, other) + ... + +Putting all this together: + + >>> match_foo = Matcher(compare, Foo(1, 2)) + >>> mock.assert_called_with(match_foo) + +The `Matcher` is instantiated with our compare function and the `Foo` object +we want to compare against. In `assert_called_with` the `Matcher` equality +method will be called, which compares the object the mock was called with +against the one we created our matcher with. If they match then +`assert_called_with` passes, and if they don't an `AssertionError` is raised: + + >>> match_wrong = Matcher(compare, Foo(3, 4)) + >>> mock.assert_called_with(match_wrong) + Traceback (most recent call last): + ... + AssertionError: Expected: ((<Matcher object at 0x...>,), {}) + Called with: ((<Foo object at 0x...>,), {}) + +With a bit of tweaking you could have the comparison function raise the +`AssertionError` directly and provide a more useful failure message. + +As of version 1.5, the Python testing library `PyHamcrest +<http://pypi.python.org/pypi/PyHamcrest>`_ provides similar functionality, +that may be useful here, in the form of its equality matcher +(`hamcrest.library.integration.match_equality +<http://packages.python.org/PyHamcrest/integration.html#hamcrest.library.integration.match_equality>`_). diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst new file mode 100644 index 0000000..12b0275 --- /dev/null +++ b/Doc/library/unittest.mock.rst @@ -0,0 +1,2221 @@ +:mod:`unittest.mock` --- mock object library +============================================ + +.. module:: unittest.mock + :synopsis: Mock object library. +.. moduleauthor:: Michael Foord <michael@python.org> +.. currentmodule:: unittest.mock + +.. versionadded:: 3.3 + +:mod:`unittest.mock` is a library for testing in Python. It allows you to +replace parts of your system under test with mock objects and make assertions +about how they have been used. + +`unittest.mock` provides a core :class:`Mock` class removing the need to +create a host of stubs throughout your test suite. After performing an +action, you can make assertions about which methods / attributes were used +and arguments they were called with. You can also specify return values and +set needed attributes in the normal way. + +Additionally, mock provides a :func:`patch` decorator that handles patching +module and class level attributes within the scope of a test, along with +:const:`sentinel` for creating unique objects. See the `quick guide`_ for +some examples of how to use :class:`Mock`, :class:`MagicMock` and +:func:`patch`. + +Mock is very easy to use and is designed for use with :mod:`unittest`. Mock +is based on the 'action -> assertion' pattern instead of `'record -> replay'` +used by many mocking frameworks. + +There is a backport of `unittest.mock` for earlier versions of Python, +available as `mock on PyPI <http://pypi.python.org/pypi/mock>`_. + +**Source code:** :source:`Lib/unittest/mock.py` + + +Quick Guide +----------- + +:class:`Mock` and :class:`MagicMock` objects create all attributes and +methods as you access them and store details of how they have been used. You +can configure them, to specify return values or limit what attributes are +available, and then make assertions about how they have been used: + + >>> from unittest.mock import MagicMock + >>> thing = ProductionClass() + >>> thing.method = MagicMock(return_value=3) + >>> thing.method(3, 4, 5, key='value') + 3 + >>> thing.method.assert_called_with(3, 4, 5, key='value') + +:attr:`side_effect` allows you to perform side effects, including raising an +exception when a mock is called: + + >>> mock = Mock(side_effect=KeyError('foo')) + >>> mock() + Traceback (most recent call last): + ... + KeyError: 'foo' + + >>> values = {'a': 1, 'b': 2, 'c': 3} + >>> def side_effect(arg): + ... return values[arg] + ... + >>> mock.side_effect = side_effect + >>> mock('a'), mock('b'), mock('c') + (1, 2, 3) + >>> mock.side_effect = [5, 4, 3, 2, 1] + >>> mock(), mock(), mock() + (5, 4, 3) + +Mock has many other ways you can configure it and control its behaviour. For +example the `spec` argument configures the mock to take its specification +from another object. Attempting to access attributes or methods on the mock +that don't exist on the spec will fail with an `AttributeError`. + +The :func:`patch` decorator / context manager makes it easy to mock classes or +objects in a module under test. The object you specify will be replaced with a +mock (or other object) during the test and restored when the test ends: + + >>> from unittest.mock import patch + >>> @patch('module.ClassName2') + ... @patch('module.ClassName1') + ... def test(MockClass1, MockClass2): + ... module.ClassName1() + ... module.ClassName2() + ... assert MockClass1 is module.ClassName1 + ... assert MockClass2 is module.ClassName2 + ... assert MockClass1.called + ... assert MockClass2.called + ... + >>> test() + +.. note:: + + When you nest patch decorators the mocks are passed in to the decorated + function in the same order they applied (the normal *python* order that + decorators are applied). This means from the bottom up, so in the example + above the mock for `module.ClassName1` is passed in first. + + With `patch` it matters that you patch objects in the namespace where they + are looked up. This is normally straightforward, but for a quick guide + read :ref:`where to patch <where-to-patch>`. + +As well as a decorator `patch` can be used as a context manager in a with +statement: + + >>> with patch.object(ProductionClass, 'method', return_value=None) as mock_method: + ... thing = ProductionClass() + ... thing.method(1, 2, 3) + ... + >>> mock_method.assert_called_once_with(1, 2, 3) + + +There is also :func:`patch.dict` for setting values in a dictionary just +during a scope and restoring the dictionary to its original state when the test +ends: + + >>> foo = {'key': 'value'} + >>> original = foo.copy() + >>> with patch.dict(foo, {'newkey': 'newvalue'}, clear=True): + ... assert foo == {'newkey': 'newvalue'} + ... + >>> assert foo == original + +Mock supports the mocking of Python :ref:`magic methods <magic-methods>`. The +easiest way of using magic methods is with the :class:`MagicMock` class. It +allows you to do things like: + + >>> mock = MagicMock() + >>> mock.__str__.return_value = 'foobarbaz' + >>> str(mock) + 'foobarbaz' + >>> mock.__str__.assert_called_with() + +Mock allows you to assign functions (or other Mock instances) to magic methods +and they will be called appropriately. The `MagicMock` class is just a Mock +variant that has all of the magic methods pre-created for you (well, all the +useful ones anyway). + +The following is an example of using magic methods with the ordinary Mock +class: + + >>> mock = Mock() + >>> mock.__str__ = Mock(return_value='wheeeeee') + >>> str(mock) + 'wheeeeee' + +For ensuring that the mock objects in your tests have the same api as the +objects they are replacing, you can use :ref:`auto-speccing <auto-speccing>`. +Auto-speccing can be done through the `autospec` argument to patch, or the +:func:`create_autospec` function. Auto-speccing creates mock objects that +have the same attributes and methods as the objects they are replacing, and +any functions and methods (including constructors) have the same call +signature as the real object. + +This ensures that your mocks will fail in the same way as your production +code if they are used incorrectly: + + >>> from unittest.mock import create_autospec + >>> def function(a, b, c): + ... pass + ... + >>> mock_function = create_autospec(function, return_value='fishy') + >>> mock_function(1, 2, 3) + 'fishy' + >>> mock_function.assert_called_once_with(1, 2, 3) + >>> mock_function('wrong arguments') + Traceback (most recent call last): + ... + TypeError: <lambda>() takes exactly 3 arguments (1 given) + +`create_autospec` can also be used on classes, where it copies the signature of +the `__init__` method, and on callable objects where it copies the signature of +the `__call__` method. + + + +The Mock Class +-------------- + + +`Mock` is a flexible mock object intended to replace the use of stubs and +test doubles throughout your code. Mocks are callable and create attributes as +new mocks when you access them [#]_. Accessing the same attribute will always +return the same mock. Mocks record how you use them, allowing you to make +assertions about what your code has done to them. + +:class:`MagicMock` is a subclass of `Mock` with all the magic methods +pre-created and ready to use. There are also non-callable variants, useful +when you are mocking out objects that aren't callable: +:class:`NonCallableMock` and :class:`NonCallableMagicMock` + +The :func:`patch` decorators makes it easy to temporarily replace classes +in a particular module with a `Mock` object. By default `patch` will create +a `MagicMock` for you. You can specify an alternative class of `Mock` using +the `new_callable` argument to `patch`. + + +.. class:: Mock(spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, **kwargs) + + Create a new `Mock` object. `Mock` takes several optional arguments + that specify the behaviour of the Mock object: + + * `spec`: This can be either a list of strings or an existing object (a + class or instance) that acts as the specification for the mock object. If + you pass in an object then a list of strings is formed by calling dir on + the object (excluding unsupported magic attributes and methods). + Accessing any attribute not in this list will raise an `AttributeError`. + + If `spec` is an object (rather than a list of strings) then + :attr:`__class__` returns the class of the spec object. This allows mocks + to pass `isinstance` tests. + + * `spec_set`: A stricter variant of `spec`. If used, attempting to *set* + or get an attribute on the mock that isn't on the object passed as + `spec_set` will raise an `AttributeError`. + + * `side_effect`: A function to be called whenever the Mock is called. See + the :attr:`~Mock.side_effect` attribute. Useful for raising exceptions or + dynamically changing return values. The function is called with the same + arguments as the mock, and unless it returns :data:`DEFAULT`, the return + value of this function is used as the return value. + + Alternatively `side_effect` can be an exception class or instance. In + this case the exception will be raised when the mock is called. + + If `side_effect` is an iterable then each call to the mock will return + the next value from the iterable. + + A `side_effect` can be cleared by setting it to `None`. + + * `return_value`: The value returned when the mock is called. By default + this is a new Mock (created on first access). See the + :attr:`return_value` attribute. + + * `wraps`: Item for the mock object to wrap. If `wraps` is not None then + calling the Mock will pass the call through to the wrapped object + (returning the real result). Attribute access on the mock will return a + Mock object that wraps the corresponding attribute of the wrapped + object (so attempting to access an attribute that doesn't exist will + raise an `AttributeError`). + + If the mock has an explicit `return_value` set then calls are not passed + to the wrapped object and the `return_value` is returned instead. + + * `name`: If the mock has a name then it will be used in the repr of the + mock. This can be useful for debugging. The name is propagated to child + mocks. + + Mocks can also be called with arbitrary keyword arguments. These will be + used to set attributes on the mock after it is created. See the + :meth:`configure_mock` method for details. + + + .. method:: assert_called_with(*args, **kwargs) + + This method is a convenient way of asserting that calls are made in a + particular way: + + >>> mock = Mock() + >>> mock.method(1, 2, 3, test='wow') + <Mock name='mock.method()' id='...'> + >>> mock.method.assert_called_with(1, 2, 3, test='wow') + + + .. method:: assert_called_once_with(*args, **kwargs) + + Assert that the mock was called exactly once and with the specified + arguments. + + >>> mock = Mock(return_value=None) + >>> mock('foo', bar='baz') + >>> mock.assert_called_once_with('foo', bar='baz') + >>> mock('foo', bar='baz') + >>> mock.assert_called_once_with('foo', bar='baz') + Traceback (most recent call last): + ... + AssertionError: Expected to be called once. Called 2 times. + + + .. method:: assert_any_call(*args, **kwargs) + + assert the mock has been called with the specified arguments. + + The assert passes if the mock has *ever* been called, unlike + :meth:`assert_called_with` and :meth:`assert_called_once_with` that + only pass if the call is the most recent one. + + >>> mock = Mock(return_value=None) + >>> mock(1, 2, arg='thing') + >>> mock('some', 'thing', 'else') + >>> mock.assert_any_call(1, 2, arg='thing') + + + .. method:: assert_has_calls(calls, any_order=False) + + assert the mock has been called with the specified calls. + The `mock_calls` list is checked for the calls. + + If `any_order` is False (the default) then the calls must be + sequential. There can be extra calls before or after the + specified calls. + + If `any_order` is True then the calls can be in any order, but + they must all appear in :attr:`mock_calls`. + + >>> mock = Mock(return_value=None) + >>> mock(1) + >>> mock(2) + >>> mock(3) + >>> mock(4) + >>> calls = [call(2), call(3)] + >>> mock.assert_has_calls(calls) + >>> calls = [call(4), call(2), call(3)] + >>> mock.assert_has_calls(calls, any_order=True) + + + .. method:: reset_mock() + + The reset_mock method resets all the call attributes on a mock object: + + >>> mock = Mock(return_value=None) + >>> mock('hello') + >>> mock.called + True + >>> mock.reset_mock() + >>> mock.called + False + + This can be useful where you want to make a series of assertions that + reuse the same object. Note that `reset_mock` *doesn't* clear the + return value, :attr:`side_effect` or any child attributes you have + set using normal assignment. Child mocks and the return value mock + (if any) are reset as well. + + + .. method:: mock_add_spec(spec, spec_set=False) + + Add a spec to a mock. `spec` can either be an object or a + list of strings. Only attributes on the `spec` can be fetched as + attributes from the mock. + + If `spec_set` is `True` then only attributes on the spec can be set. + + + .. method:: attach_mock(mock, attribute) + + Attach a mock as an attribute of this one, replacing its name and + parent. Calls to the attached mock will be recorded in the + :attr:`method_calls` and :attr:`mock_calls` attributes of this one. + + + .. method:: configure_mock(**kwargs) + + Set attributes on the mock through keyword arguments. + + Attributes plus return values and side effects can be set on child + mocks using standard dot notation and unpacking a dictionary in the + method call: + + >>> mock = Mock() + >>> attrs = {'method.return_value': 3, 'other.side_effect': KeyError} + >>> mock.configure_mock(**attrs) + >>> mock.method() + 3 + >>> mock.other() + Traceback (most recent call last): + ... + KeyError + + The same thing can be achieved in the constructor call to mocks: + + >>> attrs = {'method.return_value': 3, 'other.side_effect': KeyError} + >>> mock = Mock(some_attribute='eggs', **attrs) + >>> mock.some_attribute + 'eggs' + >>> mock.method() + 3 + >>> mock.other() + Traceback (most recent call last): + ... + KeyError + + `configure_mock` exists to make it easier to do configuration + after the mock has been created. + + + .. method:: __dir__() + + `Mock` objects limit the results of `dir(some_mock)` to useful results. + For mocks with a `spec` this includes all the permitted attributes + for the mock. + + See :data:`FILTER_DIR` for what this filtering does, and how to + switch it off. + + + .. method:: _get_child_mock(**kw) + + Create the child mocks for attributes and return value. + By default child mocks will be the same type as the parent. + Subclasses of Mock may want to override this to customize the way + child mocks are made. + + For non-callable mocks the callable variant will be used (rather than + any custom subclass). + + + .. attribute:: called + + A boolean representing whether or not the mock object has been called: + + >>> mock = Mock(return_value=None) + >>> mock.called + False + >>> mock() + >>> mock.called + True + + .. attribute:: call_count + + An integer telling you how many times the mock object has been called: + + >>> mock = Mock(return_value=None) + >>> mock.call_count + 0 + >>> mock() + >>> mock() + >>> mock.call_count + 2 + + + .. attribute:: return_value + + Set this to configure the value returned by calling the mock: + + >>> mock = Mock() + >>> mock.return_value = 'fish' + >>> mock() + 'fish' + + The default return value is a mock object and you can configure it in + the normal way: + + >>> mock = Mock() + >>> mock.return_value.attribute = sentinel.Attribute + >>> mock.return_value() + <Mock name='mock()()' id='...'> + >>> mock.return_value.assert_called_with() + + `return_value` can also be set in the constructor: + + >>> mock = Mock(return_value=3) + >>> mock.return_value + 3 + >>> mock() + 3 + + + .. attribute:: side_effect + + This can either be a function to be called when the mock is called, + or an exception (class or instance) to be raised. + + If you pass in a function it will be called with same arguments as the + mock and unless the function returns the :data:`DEFAULT` singleton the + call to the mock will then return whatever the function returns. If the + function returns :data:`DEFAULT` then the mock will return its normal + value (from the :attr:`return_value`. + + An example of a mock that raises an exception (to test exception + handling of an API): + + >>> mock = Mock() + >>> mock.side_effect = Exception('Boom!') + >>> mock() + Traceback (most recent call last): + ... + Exception: Boom! + + Using `side_effect` to return a sequence of values: + + >>> mock = Mock() + >>> mock.side_effect = [3, 2, 1] + >>> mock(), mock(), mock() + (3, 2, 1) + + The `side_effect` function is called with the same arguments as the + mock (so it is wise for it to take arbitrary args and keyword + arguments) and whatever it returns is used as the return value for + the call. The exception is if `side_effect` returns :data:`DEFAULT`, + in which case the normal :attr:`return_value` is used. + + >>> mock = Mock(return_value=3) + >>> def side_effect(*args, **kwargs): + ... return DEFAULT + ... + >>> mock.side_effect = side_effect + >>> mock() + 3 + + `side_effect` can be set in the constructor. Here's an example that + adds one to the value the mock is called with and returns it: + + >>> side_effect = lambda value: value + 1 + >>> mock = Mock(side_effect=side_effect) + >>> mock(3) + 4 + >>> mock(-8) + -7 + + Setting `side_effect` to `None` clears it: + + >>> m = Mock(side_effect=KeyError, return_value=3) + >>> m() + Traceback (most recent call last): + ... + KeyError + >>> m.side_effect = None + >>> m() + 3 + + + .. attribute:: call_args + + This is either `None` (if the mock hasn't been called), or the + arguments that the mock was last called with. This will be in the + form of a tuple: the first member is any ordered arguments the mock + was called with (or an empty tuple) and the second member is any + keyword arguments (or an empty dictionary). + + >>> mock = Mock(return_value=None) + >>> print mock.call_args + None + >>> mock() + >>> mock.call_args + call() + >>> mock.call_args == () + True + >>> mock(3, 4) + >>> mock.call_args + call(3, 4) + >>> mock.call_args == ((3, 4),) + True + >>> mock(3, 4, 5, key='fish', next='w00t!') + >>> mock.call_args + call(3, 4, 5, key='fish', next='w00t!') + + `call_args`, along with members of the lists :attr:`call_args_list`, + :attr:`method_calls` and :attr:`mock_calls` are :data:`call` objects. + These are tuples, so they can be unpacked to get at the individual + arguments and make more complex assertions. See + :ref:`calls as tuples <calls-as-tuples>`. + + + .. attribute:: call_args_list + + This is a list of all the calls made to the mock object in sequence + (so the length of the list is the number of times it has been + called). Before any calls have been made it is an empty list. The + :data:`call` object can be used for conveniently constructing lists of + calls to compare with `call_args_list`. + + >>> mock = Mock(return_value=None) + >>> mock() + >>> mock(3, 4) + >>> mock(key='fish', next='w00t!') + >>> mock.call_args_list + [call(), call(3, 4), call(key='fish', next='w00t!')] + >>> expected = [(), ((3, 4),), ({'key': 'fish', 'next': 'w00t!'},)] + >>> mock.call_args_list == expected + True + + Members of `call_args_list` are :data:`call` objects. These can be + unpacked as tuples to get at the individual arguments. See + :ref:`calls as tuples <calls-as-tuples>`. + + + .. attribute:: method_calls + + As well as tracking calls to themselves, mocks also track calls to + methods and attributes, and *their* methods and attributes: + + >>> mock = Mock() + >>> mock.method() + <Mock name='mock.method()' id='...'> + >>> mock.property.method.attribute() + <Mock name='mock.property.method.attribute()' id='...'> + >>> mock.method_calls + [call.method(), call.property.method.attribute()] + + Members of `method_calls` are :data:`call` objects. These can be + unpacked as tuples to get at the individual arguments. See + :ref:`calls as tuples <calls-as-tuples>`. + + + .. attribute:: mock_calls + + `mock_calls` records *all* calls to the mock object, its methods, magic + methods *and* return value mocks. + + >>> mock = MagicMock() + >>> result = mock(1, 2, 3) + >>> mock.first(a=3) + <MagicMock name='mock.first()' id='...'> + >>> mock.second() + <MagicMock name='mock.second()' id='...'> + >>> int(mock) + 1 + >>> result(1) + <MagicMock name='mock()()' id='...'> + >>> expected = [call(1, 2, 3), call.first(a=3), call.second(), + ... call.__int__(), call()(1)] + >>> mock.mock_calls == expected + True + + Members of `mock_calls` are :data:`call` objects. These can be + unpacked as tuples to get at the individual arguments. See + :ref:`calls as tuples <calls-as-tuples>`. + + + .. attribute:: __class__ + + Normally the `__class__` attribute of an object will return its type. + For a mock object with a `spec` `__class__` returns the spec class + instead. This allows mock objects to pass `isinstance` tests for the + object they are replacing / masquerading as: + + >>> mock = Mock(spec=3) + >>> isinstance(mock, int) + True + + `__class__` is assignable to, this allows a mock to pass an + `isinstance` check without forcing you to use a spec: + + >>> mock = Mock() + >>> mock.__class__ = dict + >>> isinstance(mock, dict) + True + +.. class:: NonCallableMock(spec=None, wraps=None, name=None, spec_set=None, **kwargs) + + A non-callable version of `Mock`. The constructor parameters have the same + meaning of `Mock`, with the exception of `return_value` and `side_effect` + which have no meaning on a non-callable mock. + +Mock objects that use a class or an instance as a `spec` or `spec_set` are able +to pass `isintance` tests: + + >>> mock = Mock(spec=SomeClass) + >>> isinstance(mock, SomeClass) + True + >>> mock = Mock(spec_set=SomeClass()) + >>> isinstance(mock, SomeClass) + True + +The `Mock` classes have support for mocking magic methods. See :ref:`magic +methods <magic-methods>` for the full details. + +The mock classes and the :func:`patch` decorators all take arbitrary keyword +arguments for configuration. For the `patch` decorators the keywords are +passed to the constructor of the mock being created. The keyword arguments +are for configuring attributes of the mock: + + >>> m = MagicMock(attribute=3, other='fish') + >>> m.attribute + 3 + >>> m.other + 'fish' + +The return value and side effect of child mocks can be set in the same way, +using dotted notation. As you can't use dotted names directly in a call you +have to create a dictionary and unpack it using `**`: + + >>> attrs = {'method.return_value': 3, 'other.side_effect': KeyError} + >>> mock = Mock(some_attribute='eggs', **attrs) + >>> mock.some_attribute + 'eggs' + >>> mock.method() + 3 + >>> mock.other() + Traceback (most recent call last): + ... + KeyError + + +.. class:: PropertyMock(*args, **kwargs) + + A mock intended to be used as a property, or other descriptor, on a class. + `PropertyMock` provides `__get__` and `__set__` methods so you can specify + a return value when it is fetched. + + Fetching a `PropertyMock` instance from an object calls the mock, with + no args. Setting it calls the mock with the value being set. + + >>> class Foo(object): + ... @property + ... def foo(self): + ... return 'something' + ... @foo.setter + ... def foo(self, value): + ... pass + ... + >>> with patch('__main__.Foo.foo', new_callable=PropertyMock) as mock_foo: + ... mock_foo.return_value = 'mockity-mock' + ... this_foo = Foo() + ... print this_foo.foo + ... this_foo.foo = 6 + ... + mockity-mock + >>> mock_foo.mock_calls + [call(), call(6)] + +Because of the way mock attributes are stored you can't directly attach a +`PropertyMock` to a mock object. Instead you can attach it to the mock type +object:: + + >>> m = MagicMock() + >>> p = PropertyMock(return_value=3) + >>> type(m).foo = p + >>> m.foo + 3 + >>> p.assert_called_once_with() + + +Calling +~~~~~~~ + +Mock objects are callable. The call will return the value set as the +:attr:`~Mock.return_value` attribute. The default return value is a new Mock +object; it is created the first time the return value is accessed (either +explicitly or by calling the Mock) - but it is stored and the same one +returned each time. + +Calls made to the object will be recorded in the attributes +like :attr:`~Mock.call_args` and :attr:`~Mock.call_args_list`. + +If :attr:`~Mock.side_effect` is set then it will be called after the call has +been recorded, so if `side_effect` raises an exception the call is still +recorded. + +The simplest way to make a mock raise an exception when called is to make +:attr:`~Mock.side_effect` an exception class or instance: + + >>> m = MagicMock(side_effect=IndexError) + >>> m(1, 2, 3) + Traceback (most recent call last): + ... + IndexError + >>> m.mock_calls + [call(1, 2, 3)] + >>> m.side_effect = KeyError('Bang!') + >>> m('two', 'three', 'four') + Traceback (most recent call last): + ... + KeyError: 'Bang!' + >>> m.mock_calls + [call(1, 2, 3), call('two', 'three', 'four')] + +If `side_effect` is a function then whatever that function returns is what +calls to the mock return. The `side_effect` function is called with the +same arguments as the mock. This allows you to vary the return value of the +call dynamically, based on the input: + + >>> def side_effect(value): + ... return value + 1 + ... + >>> m = MagicMock(side_effect=side_effect) + >>> m(1) + 2 + >>> m(2) + 3 + >>> m.mock_calls + [call(1), call(2)] + +If you want the mock to still return the default return value (a new mock), or +any set return value, then there are two ways of doing this. Either return +`mock.return_value` from inside `side_effect`, or return :data:`DEFAULT`: + + >>> m = MagicMock() + >>> def side_effect(*args, **kwargs): + ... return m.return_value + ... + >>> m.side_effect = side_effect + >>> m.return_value = 3 + >>> m() + 3 + >>> def side_effect(*args, **kwargs): + ... return DEFAULT + ... + >>> m.side_effect = side_effect + >>> m() + 3 + +To remove a `side_effect`, and return to the default behaviour, set the +`side_effect` to `None`: + + >>> m = MagicMock(return_value=6) + >>> def side_effect(*args, **kwargs): + ... return 3 + ... + >>> m.side_effect = side_effect + >>> m() + 3 + >>> m.side_effect = None + >>> m() + 6 + +The `side_effect` can also be any iterable object. Repeated calls to the mock +will return values from the iterable (until the iterable is exhausted and +a `StopIteration` is raised): + + >>> m = MagicMock(side_effect=[1, 2, 3]) + >>> m() + 1 + >>> m() + 2 + >>> m() + 3 + >>> m() + Traceback (most recent call last): + ... + StopIteration + +If any members of the iterable are exceptions they will be raised instead of +returned:: + + >>> iterable = (33, ValueError, 66) + >>> m = MagicMock(side_effect=iterable) + >>> m() + 33 + >>> m() + Traceback (most recent call last): + ... + ValueError + >>> m() + 66 + + +.. _deleting-attributes: + +Deleting Attributes +~~~~~~~~~~~~~~~~~~~ + +Mock objects create attributes on demand. This allows them to pretend to be +objects of any type. + +You may want a mock object to return `False` to a `hasattr` call, or raise an +`AttributeError` when an attribute is fetched. You can do this by providing +an object as a `spec` for a mock, but that isn't always convenient. + +You "block" attributes by deleting them. Once deleted, accessing an attribute +will raise an `AttributeError`. + + >>> mock = MagicMock() + >>> hasattr(mock, 'm') + True + >>> del mock.m + >>> hasattr(mock, 'm') + False + >>> del mock.f + >>> mock.f + Traceback (most recent call last): + ... + AttributeError: f + + +Attaching Mocks as Attributes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When you attach a mock as an attribute of another mock (or as the return +value) it becomes a "child" of that mock. Calls to the child are recorded in +the :attr:`~Mock.method_calls` and :attr:`~Mock.mock_calls` attributes of the +parent. This is useful for configuring child mocks and then attaching them to +the parent, or for attaching mocks to a parent that records all calls to the +children and allows you to make assertions about the order of calls between +mocks: + + >>> parent = MagicMock() + >>> child1 = MagicMock(return_value=None) + >>> child2 = MagicMock(return_value=None) + >>> parent.child1 = child1 + >>> parent.child2 = child2 + >>> child1(1) + >>> child2(2) + >>> parent.mock_calls + [call.child1(1), call.child2(2)] + +The exception to this is if the mock has a name. This allows you to prevent +the "parenting" if for some reason you don't want it to happen. + + >>> mock = MagicMock() + >>> not_a_child = MagicMock(name='not-a-child') + >>> mock.attribute = not_a_child + >>> mock.attribute() + <MagicMock name='not-a-child()' id='...'> + >>> mock.mock_calls + [] + +Mocks created for you by :func:`patch` are automatically given names. To +attach mocks that have names to a parent you use the :meth:`~Mock.attach_mock` +method: + + >>> thing1 = object() + >>> thing2 = object() + >>> parent = MagicMock() + >>> with patch('__main__.thing1', return_value=None) as child1: + ... with patch('__main__.thing2', return_value=None) as child2: + ... parent.attach_mock(child1, 'child1') + ... parent.attach_mock(child2, 'child2') + ... child1('one') + ... child2('two') + ... + >>> parent.mock_calls + [call.child1('one'), call.child2('two')] + + +.. [#] The only exceptions are magic methods and attributes (those that have + leading and trailing double underscores). Mock doesn't create these but + instead of raises an ``AttributeError``. This is because the interpreter + will often implicitly request these methods, and gets *very* confused to + get a new Mock object when it expects a magic method. If you need magic + method support see :ref:`magic methods <magic-methods>`. + + +The patchers +============ + +The patch decorators are used for patching objects only within the scope of +the function they decorate. They automatically handle the unpatching for you, +even if exceptions are raised. All of these functions can also be used in with +statements or as class decorators. + + +patch +----- + +.. note:: + + `patch` is straightforward to use. The key is to do the patching in the + right namespace. See the section `where to patch`_. + +.. function:: patch(target, new=DEFAULT, spec=None, create=False, spec_set=None, autospec=None, new_callable=None, **kwargs) + + `patch` acts as a function decorator, class decorator or a context + manager. Inside the body of the function or with statement, the `target` + is patched with a `new` object. When the function/with statement exits + the patch is undone. + + If `new` is omitted, then the target is replaced with a + :class:`MagicMock`. If `patch` is used as a decorator and `new` is + omitted, the created mock is passed in as an extra argument to the + decorated function. If `patch` is used as a context manager the created + mock is returned by the context manager. + + `target` should be a string in the form `'package.module.ClassName'`. The + `target` is imported and the specified object replaced with the `new` + object, so the `target` must be importable from the environment you are + calling `patch` from. The target is imported when the decorated function + is executed, not at decoration time. + + The `spec` and `spec_set` keyword arguments are passed to the `MagicMock` + if patch is creating one for you. + + In addition you can pass `spec=True` or `spec_set=True`, which causes + patch to pass in the object being mocked as the spec/spec_set object. + + `new_callable` allows you to specify a different class, or callable object, + that will be called to create the `new` object. By default `MagicMock` is + used. + + A more powerful form of `spec` is `autospec`. If you set `autospec=True` + then the mock with be created with a spec from the object being replaced. + All attributes of the mock will also have the spec of the corresponding + attribute of the object being replaced. Methods and functions being mocked + will have their arguments checked and will raise a `TypeError` if they are + called with the wrong signature. For mocks + replacing a class, their return value (the 'instance') will have the same + spec as the class. See the :func:`create_autospec` function and + :ref:`auto-speccing`. + + Instead of `autospec=True` you can pass `autospec=some_object` to use an + arbitrary object as the spec instead of the one being replaced. + + By default `patch` will fail to replace attributes that don't exist. If + you pass in `create=True`, and the attribute doesn't exist, patch will + create the attribute for you when the patched function is called, and + delete it again afterwards. This is useful for writing tests against + attributes that your production code creates at runtime. It is off by by + default because it can be dangerous. With it switched on you can write + passing tests against APIs that don't actually exist! + + Patch can be used as a `TestCase` class decorator. It works by + decorating each test method in the class. This reduces the boilerplate + code when your test methods share a common patchings set. `patch` finds + tests by looking for method names that start with `patch.TEST_PREFIX`. + By default this is `test`, which matches the way `unittest` finds tests. + You can specify an alternative prefix by setting `patch.TEST_PREFIX`. + + Patch can be used as a context manager, with the with statement. Here the + patching applies to the indented block after the with statement. If you + use "as" then the patched object will be bound to the name after the + "as"; very useful if `patch` is creating a mock object for you. + + `patch` takes arbitrary keyword arguments. These will be passed to + the `Mock` (or `new_callable`) on construction. + + `patch.dict(...)`, `patch.multiple(...)` and `patch.object(...)` are + available for alternate use-cases. + +`patch` as function decorator, creating the mock for you and passing it into +the decorated function: + + >>> @patch('__main__.SomeClass') + ... def function(normal_argument, mock_class): + ... print(mock_class is SomeClass) + ... + >>> function(None) + True + +Patching a class replaces the class with a `MagicMock` *instance*. If the +class is instantiated in the code under test then it will be the +:attr:`~Mock.return_value` of the mock that will be used. + +If the class is instantiated multiple times you could use +:attr:`~Mock.side_effect` to return a new mock each time. Alternatively you +can set the `return_value` to be anything you want. + +To configure return values on methods of *instances* on the patched class +you must do this on the `return_value`. For example: + + >>> class Class(object): + ... def method(self): + ... pass + ... + >>> with patch('__main__.Class') as MockClass: + ... instance = MockClass.return_value + ... instance.method.return_value = 'foo' + ... assert Class() is instance + ... assert Class().method() == 'foo' + ... + +If you use `spec` or `spec_set` and `patch` is replacing a *class*, then the +return value of the created mock will have the same spec. + + >>> Original = Class + >>> patcher = patch('__main__.Class', spec=True) + >>> MockClass = patcher.start() + >>> instance = MockClass() + >>> assert isinstance(instance, Original) + >>> patcher.stop() + +The `new_callable` argument is useful where you want to use an alternative +class to the default :class:`MagicMock` for the created mock. For example, if +you wanted a :class:`NonCallableMock` to be used: + + >>> thing = object() + >>> with patch('__main__.thing', new_callable=NonCallableMock) as mock_thing: + ... assert thing is mock_thing + ... thing() + ... + Traceback (most recent call last): + ... + TypeError: 'NonCallableMock' object is not callable + +Another use case might be to replace an object with a `StringIO` instance: + + >>> from StringIO import StringIO + >>> def foo(): + ... print 'Something' + ... + >>> @patch('sys.stdout', new_callable=StringIO) + ... def test(mock_stdout): + ... foo() + ... assert mock_stdout.getvalue() == 'Something\n' + ... + >>> test() + +When `patch` is creating a mock for you, it is common that the first thing +you need to do is to configure the mock. Some of that configuration can be done +in the call to patch. Any arbitrary keywords you pass into the call will be +used to set attributes on the created mock: + + >>> patcher = patch('__main__.thing', first='one', second='two') + >>> mock_thing = patcher.start() + >>> mock_thing.first + 'one' + >>> mock_thing.second + 'two' + +As well as attributes on the created mock attributes, like the +:attr:`~Mock.return_value` and :attr:`~Mock.side_effect`, of child mocks can +also be configured. These aren't syntactically valid to pass in directly as +keyword arguments, but a dictionary with these as keys can still be expanded +into a `patch` call using `**`: + + >>> config = {'method.return_value': 3, 'other.side_effect': KeyError} + >>> patcher = patch('__main__.thing', **config) + >>> mock_thing = patcher.start() + >>> mock_thing.method() + 3 + >>> mock_thing.other() + Traceback (most recent call last): + ... + KeyError + + +patch.object +------------ + +.. function:: patch.object(target, attribute, new=DEFAULT, spec=None, create=False, spec_set=None, autospec=None, new_callable=None, **kwargs) + + patch the named member (`attribute`) on an object (`target`) with a mock + object. + + `patch.object` can be used as a decorator, class decorator or a context + manager. Arguments `new`, `spec`, `create`, `spec_set`, `autospec` and + `new_callable` have the same meaning as for `patch`. Like `patch`, + `patch.object` takes arbitrary keyword arguments for configuring the mock + object it creates. + + When used as a class decorator `patch.object` honours `patch.TEST_PREFIX` + for choosing which methods to wrap. + +You can either call `patch.object` with three arguments or two arguments. The +three argument form takes the object to be patched, the attribute name and the +object to replace the attribute with. + +When calling with the two argument form you omit the replacement object, and a +mock is created for you and passed in as an extra argument to the decorated +function: + + >>> @patch.object(SomeClass, 'class_method') + ... def test(mock_method): + ... SomeClass.class_method(3) + ... mock_method.assert_called_with(3) + ... + >>> test() + +`spec`, `create` and the other arguments to `patch.object` have the same +meaning as they do for `patch`. + + +patch.dict +---------- + +.. function:: patch.dict(in_dict, values=(), clear=False, **kwargs) + + Patch a dictionary, or dictionary like object, and restore the dictionary + to its original state after the test. + + `in_dict` can be a dictionary or a mapping like container. If it is a + mapping then it must at least support getting, setting and deleting items + plus iterating over keys. + + `in_dict` can also be a string specifying the name of the dictionary, which + will then be fetched by importing it. + + `values` can be a dictionary of values to set in the dictionary. `values` + can also be an iterable of `(key, value)` pairs. + + If `clear` is True then the dictionary will be cleared before the new + values are set. + + `patch.dict` can also be called with arbitrary keyword arguments to set + values in the dictionary. + + `patch.dict` can be used as a context manager, decorator or class + decorator. When used as a class decorator `patch.dict` honours + `patch.TEST_PREFIX` for choosing which methods to wrap. + +`patch.dict` can be used to add members to a dictionary, or simply let a test +change a dictionary, and ensure the dictionary is restored when the test +ends. + + >>> foo = {} + >>> with patch.dict(foo, {'newkey': 'newvalue'}): + ... assert foo == {'newkey': 'newvalue'} + ... + >>> assert foo == {} + + >>> import os + >>> with patch.dict('os.environ', {'newkey': 'newvalue'}): + ... print os.environ['newkey'] + ... + newvalue + >>> assert 'newkey' not in os.environ + +Keywords can be used in the `patch.dict` call to set values in the dictionary: + + >>> mymodule = MagicMock() + >>> mymodule.function.return_value = 'fish' + >>> with patch.dict('sys.modules', mymodule=mymodule): + ... import mymodule + ... mymodule.function('some', 'args') + ... + 'fish' + +`patch.dict` can be used with dictionary like objects that aren't actually +dictionaries. At the very minimum they must support item getting, setting, +deleting and either iteration or membership test. This corresponds to the +magic methods `__getitem__`, `__setitem__`, `__delitem__` and either +`__iter__` or `__contains__`. + + >>> class Container(object): + ... def __init__(self): + ... self.values = {} + ... def __getitem__(self, name): + ... return self.values[name] + ... def __setitem__(self, name, value): + ... self.values[name] = value + ... def __delitem__(self, name): + ... del self.values[name] + ... def __iter__(self): + ... return iter(self.values) + ... + >>> thing = Container() + >>> thing['one'] = 1 + >>> with patch.dict(thing, one=2, two=3): + ... assert thing['one'] == 2 + ... assert thing['two'] == 3 + ... + >>> assert thing['one'] == 1 + >>> assert list(thing) == ['one'] + + +patch.multiple +-------------- + +.. function:: patch.multiple(target, spec=None, create=False, spec_set=None, autospec=None, new_callable=None, **kwargs) + + Perform multiple patches in a single call. It takes the object to be + patched (either as an object or a string to fetch the object by importing) + and keyword arguments for the patches:: + + with patch.multiple(settings, FIRST_PATCH='one', SECOND_PATCH='two'): + ... + + Use :data:`DEFAULT` as the value if you want `patch.multiple` to create + mocks for you. In this case the created mocks are passed into a decorated + function by keyword, and a dictionary is returned when `patch.multiple` is + used as a context manager. + + `patch.multiple` can be used as a decorator, class decorator or a context + manager. The arguments `spec`, `spec_set`, `create`, `autospec` and + `new_callable` have the same meaning as for `patch`. These arguments will + be applied to *all* patches done by `patch.multiple`. + + When used as a class decorator `patch.multiple` honours `patch.TEST_PREFIX` + for choosing which methods to wrap. + +If you want `patch.multiple` to create mocks for you, then you can use +:data:`DEFAULT` as the value. If you use `patch.multiple` as a decorator +then the created mocks are passed into the decorated function by keyword. + + >>> thing = object() + >>> other = object() + + >>> @patch.multiple('__main__', thing=DEFAULT, other=DEFAULT) + ... def test_function(thing, other): + ... assert isinstance(thing, MagicMock) + ... assert isinstance(other, MagicMock) + ... + >>> test_function() + +`patch.multiple` can be nested with other `patch` decorators, but put arguments +passed by keyword *after* any of the standard arguments created by `patch`: + + >>> @patch('sys.exit') + ... @patch.multiple('__main__', thing=DEFAULT, other=DEFAULT) + ... def test_function(mock_exit, other, thing): + ... assert 'other' in repr(other) + ... assert 'thing' in repr(thing) + ... assert 'exit' in repr(mock_exit) + ... + >>> test_function() + +If `patch.multiple` is used as a context manager, the value returned by the +context manger is a dictionary where created mocks are keyed by name: + + >>> with patch.multiple('__main__', thing=DEFAULT, other=DEFAULT) as values: + ... assert 'other' in repr(values['other']) + ... assert 'thing' in repr(values['thing']) + ... assert values['thing'] is thing + ... assert values['other'] is other + ... + + +.. _start-and-stop: + +patch methods: start and stop +----------------------------- + +All the patchers have `start` and `stop` methods. These make it simpler to do +patching in `setUp` methods or where you want to do multiple patches without +nesting decorators or with statements. + +To use them call `patch`, `patch.object` or `patch.dict` as normal and keep a +reference to the returned `patcher` object. You can then call `start` to put +the patch in place and `stop` to undo it. + +If you are using `patch` to create a mock for you then it will be returned by +the call to `patcher.start`. + + >>> patcher = patch('package.module.ClassName') + >>> from package import module + >>> original = module.ClassName + >>> new_mock = patcher.start() + >>> assert module.ClassName is not original + >>> assert module.ClassName is new_mock + >>> patcher.stop() + >>> assert module.ClassName is original + >>> assert module.ClassName is not new_mock + + +A typical use case for this might be for doing multiple patches in the `setUp` +method of a `TestCase`: + + >>> class MyTest(TestCase): + ... def setUp(self): + ... self.patcher1 = patch('package.module.Class1') + ... self.patcher2 = patch('package.module.Class2') + ... self.MockClass1 = self.patcher1.start() + ... self.MockClass2 = self.patcher2.start() + ... + ... def tearDown(self): + ... self.patcher1.stop() + ... self.patcher2.stop() + ... + ... def test_something(self): + ... assert package.module.Class1 is self.MockClass1 + ... assert package.module.Class2 is self.MockClass2 + ... + >>> MyTest('test_something').run() + +.. caution:: + + If you use this technique you must ensure that the patching is "undone" by + calling `stop`. This can be fiddlier than you might think, because if an + exception is raised in the ``setUp`` then ``tearDown`` is not called. + :meth:`unittest.TestCase.addCleanup` makes this easier: + + >>> class MyTest(TestCase): + ... def setUp(self): + ... patcher = patch('package.module.Class') + ... self.MockClass = patcher.start() + ... self.addCleanup(patcher.stop) + ... + ... def test_something(self): + ... assert package.module.Class is self.MockClass + ... + + As an added bonus you no longer need to keep a reference to the `patcher` + object. + +In fact `start` and `stop` are just aliases for the context manager +`__enter__` and `__exit__` methods. + + +TEST_PREFIX +----------- + +All of the patchers can be used as class decorators. When used in this way +they wrap every test method on the class. The patchers recognise methods that +start with `test` as being test methods. This is the same way that the +:class:`unittest.TestLoader` finds test methods by default. + +It is possible that you want to use a different prefix for your tests. You can +inform the patchers of the different prefix by setting `patch.TEST_PREFIX`: + + >>> patch.TEST_PREFIX = 'foo' + >>> value = 3 + >>> + >>> @patch('__main__.value', 'not three') + ... class Thing(object): + ... def foo_one(self): + ... print value + ... def foo_two(self): + ... print value + ... + >>> + >>> Thing().foo_one() + not three + >>> Thing().foo_two() + not three + >>> value + 3 + + +Nesting Patch Decorators +------------------------ + +If you want to perform multiple patches then you can simply stack up the +decorators. + +You can stack up multiple patch decorators using this pattern: + + >>> @patch.object(SomeClass, 'class_method') + ... @patch.object(SomeClass, 'static_method') + ... def test(mock1, mock2): + ... assert SomeClass.static_method is mock1 + ... assert SomeClass.class_method is mock2 + ... SomeClass.static_method('foo') + ... SomeClass.class_method('bar') + ... return mock1, mock2 + ... + >>> mock1, mock2 = test() + >>> mock1.assert_called_once_with('foo') + >>> mock2.assert_called_once_with('bar') + + +Note that the decorators are applied from the bottom upwards. This is the +standard way that Python applies decorators. The order of the created mocks +passed into your test function matches this order. + + +.. _where-to-patch: + +Where to patch +-------------- + +`patch` works by (temporarily) changing the object that a *name* points to with +another one. There can be many names pointing to any individual object, so +for patching to work you must ensure that you patch the name used by the system +under test. + +The basic principle is that you patch where an object is *looked up*, which +is not necessarily the same place as where it is defined. A couple of +examples will help to clarify this. + +Imagine we have a project that we want to test with the following structure:: + + a.py + -> Defines SomeClass + + b.py + -> from a import SomeClass + -> some_function instantiates SomeClass + +Now we want to test `some_function` but we want to mock out `SomeClass` using +`patch`. The problem is that when we import module b, which we will have to +do then it imports `SomeClass` from module a. If we use `patch` to mock out +`a.SomeClass` then it will have no effect on our test; module b already has a +reference to the *real* `SomeClass` and it looks like our patching had no +effect. + +The key is to patch out `SomeClass` where it is used (or where it is looked up +). In this case `some_function` will actually look up `SomeClass` in module b, +where we have imported it. The patching should look like:: + + @patch('b.SomeClass') + +However, consider the alternative scenario where instead of `from a import +SomeClass` module b does `import a` and `some_function` uses `a.SomeClass`. Both +of these import forms are common. In this case the class we want to patch is +being looked up on the a module and so we have to patch `a.SomeClass` instead:: + + @patch('a.SomeClass') + + +Patching Descriptors and Proxy Objects +-------------------------------------- + +Both patch_ and patch.object_ correctly patch and restore descriptors: class +methods, static methods and properties. You should patch these on the *class* +rather than an instance. They also work with *some* objects +that proxy attribute access, like the `django setttings object +<http://www.voidspace.org.uk/python/weblog/arch_d7_2010_12_04.shtml#e1198>`_. + + +MagicMock and magic method support +================================== + +.. _magic-methods: + +Mocking Magic Methods +--------------------- + +:class:`Mock` supports mocking the Python protocol methods, also known as +"magic methods". This allows mock objects to replace containers or other +objects that implement Python protocols. + +Because magic methods are looked up differently from normal methods [#]_, this +support has been specially implemented. This means that only specific magic +methods are supported. The supported list includes *almost* all of them. If +there are any missing that you need please let us know. + +You mock magic methods by setting the method you are interested in to a function +or a mock instance. If you are using a function then it *must* take ``self`` as +the first argument [#]_. + + >>> def __str__(self): + ... return 'fooble' + ... + >>> mock = Mock() + >>> mock.__str__ = __str__ + >>> str(mock) + 'fooble' + + >>> mock = Mock() + >>> mock.__str__ = Mock() + >>> mock.__str__.return_value = 'fooble' + >>> str(mock) + 'fooble' + + >>> mock = Mock() + >>> mock.__iter__ = Mock(return_value=iter([])) + >>> list(mock) + [] + +One use case for this is for mocking objects used as context managers in a +`with` statement: + + >>> mock = Mock() + >>> mock.__enter__ = Mock(return_value='foo') + >>> mock.__exit__ = Mock(return_value=False) + >>> with mock as m: + ... assert m == 'foo' + ... + >>> mock.__enter__.assert_called_with() + >>> mock.__exit__.assert_called_with(None, None, None) + +Calls to magic methods do not appear in :attr:`~Mock.method_calls`, but they +are recorded in :attr:`~Mock.mock_calls`. + +.. note:: + + If you use the `spec` keyword argument to create a mock then attempting to + set a magic method that isn't in the spec will raise an `AttributeError`. + +The full list of supported magic methods is: + +* ``__hash__``, ``__sizeof__``, ``__repr__`` and ``__str__`` +* ``__dir__``, ``__format__`` and ``__subclasses__`` +* ``__floor__``, ``__trunc__`` and ``__ceil__`` +* Comparisons: ``__cmp__``, ``__lt__``, ``__gt__``, ``__le__``, ``__ge__``, + ``__eq__`` and ``__ne__`` +* Container methods: ``__getitem__``, ``__setitem__``, ``__delitem__``, + ``__contains__``, ``__len__``, ``__iter__``, ``__getslice__``, + ``__setslice__``, ``__reversed__`` and ``__missing__`` +* Context manager: ``__enter__`` and ``__exit__`` +* Unary numeric methods: ``__neg__``, ``__pos__`` and ``__invert__`` +* The numeric methods (including right hand and in-place variants): + ``__add__``, ``__sub__``, ``__mul__``, ``__div__``, + ``__floordiv__``, ``__mod__``, ``__divmod__``, ``__lshift__``, + ``__rshift__``, ``__and__``, ``__xor__``, ``__or__``, and ``__pow__`` +* Numeric conversion methods: ``__complex__``, ``__int__``, ``__float__``, + ``__index__`` and ``__coerce__`` +* Descriptor methods: ``__get__``, ``__set__`` and ``__delete__`` +* Pickling: ``__reduce__``, ``__reduce_ex__``, ``__getinitargs__``, + ``__getnewargs__``, ``__getstate__`` and ``__setstate__`` + + +The following methods exist but are *not* supported as they are either in use +by mock, can't be set dynamically, or can cause problems: + +* ``__getattr__``, ``__setattr__``, ``__init__`` and ``__new__`` +* ``__prepare__``, ``__instancecheck__``, ``__subclasscheck__``, ``__del__`` + + + +Magic Mock +---------- + +There are two `MagicMock` variants: `MagicMock` and `NonCallableMagicMock`. + + +.. class:: MagicMock(*args, **kw) + + ``MagicMock`` is a subclass of :class:`Mock` with default implementations + of most of the magic methods. You can use ``MagicMock`` without having to + configure the magic methods yourself. + + The constructor parameters have the same meaning as for :class:`Mock`. + + If you use the `spec` or `spec_set` arguments then *only* magic methods + that exist in the spec will be created. + + +.. class:: NonCallableMagicMock(*args, **kw) + + A non-callable version of `MagicMock`. + + The constructor parameters have the same meaning as for + :class:`MagicMock`, with the exception of `return_value` and + `side_effect` which have no meaning on a non-callable mock. + +The magic methods are setup with `MagicMock` objects, so you can configure them +and use them in the usual way: + + >>> mock = MagicMock() + >>> mock[3] = 'fish' + >>> mock.__setitem__.assert_called_with(3, 'fish') + >>> mock.__getitem__.return_value = 'result' + >>> mock[2] + 'result' + +By default many of the protocol methods are required to return objects of a +specific type. These methods are preconfigured with a default return value, so +that they can be used without you having to do anything if you aren't interested +in the return value. You can still *set* the return value manually if you want +to change the default. + +Methods and their defaults: + +* ``__lt__``: NotImplemented +* ``__gt__``: NotImplemented +* ``__le__``: NotImplemented +* ``__ge__``: NotImplemented +* ``__int__`` : 1 +* ``__contains__`` : False +* ``__len__`` : 1 +* ``__iter__`` : iter([]) +* ``__exit__`` : False +* ``__complex__`` : 1j +* ``__float__`` : 1.0 +* ``__bool__`` : True +* ``__index__`` : 1 +* ``__hash__`` : default hash for the mock +* ``__str__`` : default str for the mock +* ``__sizeof__``: default sizeof for the mock + +For example: + + >>> mock = MagicMock() + >>> int(mock) + 1 + >>> len(mock) + 0 + >>> list(mock) + [] + >>> object() in mock + False + +The two equality method, `__eq__` and `__ne__`, are special. +They do the default equality comparison on identity, using a side +effect, unless you change their return value to return something else: + + >>> MagicMock() == 3 + False + >>> MagicMock() != 3 + True + >>> mock = MagicMock() + >>> mock.__eq__.return_value = True + >>> mock == 3 + True + +The return value of `MagicMock.__iter__` can be any iterable object and isn't +required to be an iterator: + + >>> mock = MagicMock() + >>> mock.__iter__.return_value = ['a', 'b', 'c'] + >>> list(mock) + ['a', 'b', 'c'] + >>> list(mock) + ['a', 'b', 'c'] + +If the return value *is* an iterator, then iterating over it once will consume +it and subsequent iterations will result in an empty list: + + >>> mock.__iter__.return_value = iter(['a', 'b', 'c']) + >>> list(mock) + ['a', 'b', 'c'] + >>> list(mock) + [] + +``MagicMock`` has all of the supported magic methods configured except for some +of the obscure and obsolete ones. You can still set these up if you want. + +Magic methods that are supported but not setup by default in ``MagicMock`` are: + +* ``__subclasses__`` +* ``__dir__`` +* ``__format__`` +* ``__get__``, ``__set__`` and ``__delete__`` +* ``__reversed__`` and ``__missing__`` +* ``__reduce__``, ``__reduce_ex__``, ``__getinitargs__``, ``__getnewargs__``, + ``__getstate__`` and ``__setstate__`` +* ``__getformat__`` and ``__setformat__`` + + + +.. [#] Magic methods *should* be looked up on the class rather than the + instance. Different versions of Python are inconsistent about applying this + rule. The supported protocol methods should work with all supported versions + of Python. +.. [#] The function is basically hooked up to the class, but each ``Mock`` + instance is kept isolated from the others. + + +Helpers +======= + +sentinel +-------- + +.. data:: sentinel + + The ``sentinel`` object provides a convenient way of providing unique + objects for your tests. + + Attributes are created on demand when you access them by name. Accessing + the same attribute will always return the same object. The objects + returned have a sensible repr so that test failure messages are readable. + +Sometimes when testing you need to test that a specific object is passed as an +argument to another method, or returned. It can be common to create named +sentinel objects to test this. `sentinel` provides a convenient way of +creating and testing the identity of objects like this. + +In this example we monkey patch `method` to return `sentinel.some_object`: + + >>> real = ProductionClass() + >>> real.method = Mock(name="method") + >>> real.method.return_value = sentinel.some_object + >>> result = real.method() + >>> assert result is sentinel.some_object + >>> sentinel.some_object + sentinel.some_object + + +DEFAULT +------- + + +.. data:: DEFAULT + + The `DEFAULT` object is a pre-created sentinel (actually + `sentinel.DEFAULT`). It can be used by :attr:`~Mock.side_effect` + functions to indicate that the normal return value should be used. + + + +call +---- + +.. function:: call(*args, **kwargs) + + `call` is a helper object for making simpler assertions, for comparing with + :attr:`~Mock.call_args`, :attr:`~Mock.call_args_list`, + :attr:`~Mock.mock_calls` and :attr:`~Mock.method_calls`. `call` can also be + used with :meth:`~Mock.assert_has_calls`. + + >>> m = MagicMock(return_value=None) + >>> m(1, 2, a='foo', b='bar') + >>> m() + >>> m.call_args_list == [call(1, 2, a='foo', b='bar'), call()] + True + +.. method:: call.call_list() + + For a call object that represents multiple calls, `call_list` + returns a list of all the intermediate calls as well as the + final call. + +`call_list` is particularly useful for making assertions on "chained calls". A +chained call is multiple calls on a single line of code. This results in +multiple entries in :attr:`~Mock.mock_calls` on a mock. Manually constructing +the sequence of calls can be tedious. + +:meth:`~call.call_list` can construct the sequence of calls from the same +chained call: + + >>> m = MagicMock() + >>> m(1).method(arg='foo').other('bar')(2.0) + <MagicMock name='mock().method().other()()' id='...'> + >>> kall = call(1).method(arg='foo').other('bar')(2.0) + >>> kall.call_list() + [call(1), + call().method(arg='foo'), + call().method().other('bar'), + call().method().other()(2.0)] + >>> m.mock_calls == kall.call_list() + True + +.. _calls-as-tuples: + +A `call` object is either a tuple of (positional args, keyword args) or +(name, positional args, keyword args) depending on how it was constructed. When +you construct them yourself this isn't particularly interesting, but the `call` +objects that are in the :attr:`Mock.call_args`, :attr:`Mock.call_args_list` and +:attr:`Mock.mock_calls` attributes can be introspected to get at the individual +arguments they contain. + +The `call` objects in :attr:`Mock.call_args` and :attr:`Mock.call_args_list` +are two-tuples of (positional args, keyword args) whereas the `call` objects +in :attr:`Mock.mock_calls`, along with ones you construct yourself, are +three-tuples of (name, positional args, keyword args). + +You can use their "tupleness" to pull out the individual arguments for more +complex introspection and assertions. The positional arguments are a tuple +(an empty tuple if there are no positional arguments) and the keyword +arguments are a dictionary: + + >>> m = MagicMock(return_value=None) + >>> m(1, 2, 3, arg='one', arg2='two') + >>> kall = m.call_args + >>> args, kwargs = kall + >>> args + (1, 2, 3) + >>> kwargs + {'arg2': 'two', 'arg': 'one'} + >>> args is kall[0] + True + >>> kwargs is kall[1] + True + + >>> m = MagicMock() + >>> m.foo(4, 5, 6, arg='two', arg2='three') + <MagicMock name='mock.foo()' id='...'> + >>> kall = m.mock_calls[0] + >>> name, args, kwargs = kall + >>> name + 'foo' + >>> args + (4, 5, 6) + >>> kwargs + {'arg2': 'three', 'arg': 'two'} + >>> name is m.mock_calls[0][0] + True + + +create_autospec +--------------- + +.. function:: create_autospec(spec, spec_set=False, instance=False, **kwargs) + + Create a mock object using another object as a spec. Attributes on the + mock will use the corresponding attribute on the `spec` object as their + spec. + + Functions or methods being mocked will have their arguments checked to + ensure that they are called with the correct signature. + + If `spec_set` is `True` then attempting to set attributes that don't exist + on the spec object will raise an `AttributeError`. + + If a class is used as a spec then the return value of the mock (the + instance of the class) will have the same spec. You can use a class as the + spec for an instance object by passing `instance=True`. The returned mock + will only be callable if instances of the mock are callable. + + `create_autospec` also takes arbitrary keyword arguments that are passed to + the constructor of the created mock. + +See :ref:`auto-speccing` for examples of how to use auto-speccing with +`create_autospec` and the `autospec` argument to :func:`patch`. + + +ANY +--- + +.. data:: ANY + +Sometimes you may need to make assertions about *some* of the arguments in a +call to mock, but either not care about some of the arguments or want to pull +them individually out of :attr:`~Mock.call_args` and make more complex +assertions on them. + +To ignore certain arguments you can pass in objects that compare equal to +*everything*. Calls to :meth:`~Mock.assert_called_with` and +:meth:`~Mock.assert_called_once_with` will then succeed no matter what was +passed in. + + >>> mock = Mock(return_value=None) + >>> mock('foo', bar=object()) + >>> mock.assert_called_once_with('foo', bar=ANY) + +`ANY` can also be used in comparisons with call lists like +:attr:`~Mock.mock_calls`: + + >>> m = MagicMock(return_value=None) + >>> m(1) + >>> m(1, 2) + >>> m(object()) + >>> m.mock_calls == [call(1), call(1, 2), ANY] + True + + + +FILTER_DIR +---------- + +.. data:: FILTER_DIR + +`FILTER_DIR` is a module level variable that controls the way mock objects +respond to `dir` (only for Python 2.6 or more recent). The default is `True`, +which uses the filtering described below, to only show useful members. If you +dislike this filtering, or need to switch it off for diagnostic purposes, then +set `mock.FILTER_DIR = False`. + +With filtering on, `dir(some_mock)` shows only useful attributes and will +include any dynamically created attributes that wouldn't normally be shown. +If the mock was created with a `spec` (or `autospec` of course) then all the +attributes from the original are shown, even if they haven't been accessed +yet: + + >>> dir(Mock()) + ['assert_any_call', + 'assert_called_once_with', + 'assert_called_with', + 'assert_has_calls', + 'attach_mock', + ... + >>> from urllib import request + >>> dir(Mock(spec=request)) + ['AbstractBasicAuthHandler', + 'AbstractDigestAuthHandler', + 'AbstractHTTPHandler', + 'BaseHandler', + ... + +Many of the not-very-useful (private to `Mock` rather than the thing being +mocked) underscore and double underscore prefixed attributes have been +filtered from the result of calling `dir` on a `Mock`. If you dislike this +behaviour you can switch it off by setting the module level switch +`FILTER_DIR`: + + >>> from unittest import mock + >>> mock.FILTER_DIR = False + >>> dir(mock.Mock()) + ['_NonCallableMock__get_return_value', + '_NonCallableMock__get_side_effect', + '_NonCallableMock__return_value_doc', + '_NonCallableMock__set_return_value', + '_NonCallableMock__set_side_effect', + '__call__', + '__class__', + ... + +Alternatively you can just use `vars(my_mock)` (instance members) and +`dir(type(my_mock))` (type members) to bypass the filtering irrespective of +`mock.FILTER_DIR`. + + +mock_open +--------- + +.. function:: mock_open(mock=None, read_data=None) + + A helper function to create a mock to replace the use of `open`. It works + for `open` called directly or used as a context manager. + + The `mock` argument is the mock object to configure. If `None` (the + default) then a `MagicMock` will be created for you, with the API limited + to methods or attributes available on standard file handles. + + `read_data` is a string for the `read` method of the file handle to return. + This is an empty string by default. + +Using `open` as a context manager is a great way to ensure your file handles +are closed properly and is becoming common:: + + with open('/some/path', 'w') as f: + f.write('something') + +The issue is that even if you mock out the call to `open` it is the +*returned object* that is used as a context manager (and has `__enter__` and +`__exit__` called). + +Mocking context managers with a :class:`MagicMock` is common enough and fiddly +enough that a helper function is useful. + + >>> m = mock_open() + >>> with patch('__main__.open', m, create=True): + ... with open('foo', 'w') as h: + ... h.write('some stuff') + ... + >>> m.mock_calls + [call('foo', 'w'), + call().__enter__(), + call().write('some stuff'), + call().__exit__(None, None, None)] + >>> m.assert_called_once_with('foo', 'w') + >>> handle = m() + >>> handle.write.assert_called_once_with('some stuff') + +And for reading files: + + >>> with patch('__main__.open', mock_open(read_data='bibble'), create=True) as m: + ... with open('foo') as h: + ... result = h.read() + ... + >>> m.assert_called_once_with('foo') + >>> assert result == 'bibble' + + +.. _auto-speccing: + +Autospeccing +------------ + +Autospeccing is based on the existing `spec` feature of mock. It limits the +api of mocks to the api of an original object (the spec), but it is recursive +(implemented lazily) so that attributes of mocks only have the same api as +the attributes of the spec. In addition mocked functions / methods have the +same call signature as the original so they raise a `TypeError` if they are +called incorrectly. + +Before I explain how auto-speccing works, here's why it is needed. + +`Mock` is a very powerful and flexible object, but it suffers from two flaws +when used to mock out objects from a system under test. One of these flaws is +specific to the `Mock` api and the other is a more general problem with using +mock objects. + +First the problem specific to `Mock`. `Mock` has two assert methods that are +extremely handy: :meth:`~Mock.assert_called_with` and +:meth:`~Mock.assert_called_once_with`. + + >>> mock = Mock(name='Thing', return_value=None) + >>> mock(1, 2, 3) + >>> mock.assert_called_once_with(1, 2, 3) + >>> mock(1, 2, 3) + >>> mock.assert_called_once_with(1, 2, 3) + Traceback (most recent call last): + ... + AssertionError: Expected to be called once. Called 2 times. + +Because mocks auto-create attributes on demand, and allow you to call them +with arbitrary arguments, if you misspell one of these assert methods then +your assertion is gone: + +.. code-block:: pycon + + >>> mock = Mock(name='Thing', return_value=None) + >>> mock(1, 2, 3) + >>> mock.assret_called_once_with(4, 5, 6) + +Your tests can pass silently and incorrectly because of the typo. + +The second issue is more general to mocking. If you refactor some of your +code, rename members and so on, any tests for code that is still using the +*old api* but uses mocks instead of the real objects will still pass. This +means your tests can all pass even though your code is broken. + +Note that this is another reason why you need integration tests as well as +unit tests. Testing everything in isolation is all fine and dandy, but if you +don't test how your units are "wired together" there is still lots of room +for bugs that tests might have caught. + +`mock` already provides a feature to help with this, called speccing. If you +use a class or instance as the `spec` for a mock then you can only access +attributes on the mock that exist on the real class: + + >>> from urllib import request + >>> mock = Mock(spec=request.Request) + >>> mock.assret_called_with + Traceback (most recent call last): + ... + AttributeError: Mock object has no attribute 'assret_called_with' + +The spec only applies to the mock itself, so we still have the same issue +with any methods on the mock: + +.. code-block:: pycon + + >>> mock.has_data() + <mock.Mock object at 0x...> + >>> mock.has_data.assret_called_with() + +Auto-speccing solves this problem. You can either pass `autospec=True` to +`patch` / `patch.object` or use the `create_autospec` function to create a +mock with a spec. If you use the `autospec=True` argument to `patch` then the +object that is being replaced will be used as the spec object. Because the +speccing is done "lazily" (the spec is created as attributes on the mock are +accessed) you can use it with very complex or deeply nested objects (like +modules that import modules that import modules) without a big performance +hit. + +Here's an example of it in use: + + >>> from urllib import request + >>> patcher = patch('__main__.request', autospec=True) + >>> mock_request = patcher.start() + >>> request is mock_request + True + >>> mock_request.Request + <MagicMock name='request.Request' spec='Request' id='...'> + +You can see that `request.Request` has a spec. `request.Request` takes two +arguments in the constructor (one of which is `self`). Here's what happens if +we try to call it incorrectly: + + >>> req = request.Request() + Traceback (most recent call last): + ... + TypeError: <lambda>() takes at least 2 arguments (1 given) + +The spec also applies to instantiated classes (i.e. the return value of +specced mocks): + + >>> req = request.Request('foo') + >>> req + <NonCallableMagicMock name='request.Request()' spec='Request' id='...'> + +`Request` objects are not callable, so the return value of instantiating our +mocked out `request.Request` is a non-callable mock. With the spec in place +any typos in our asserts will raise the correct error: + + >>> req.add_header('spam', 'eggs') + <MagicMock name='request.Request().add_header()' id='...'> + >>> req.add_header.assret_called_with + Traceback (most recent call last): + ... + AttributeError: Mock object has no attribute 'assret_called_with' + >>> req.add_header.assert_called_with('spam', 'eggs') + +In many cases you will just be able to add `autospec=True` to your existing +`patch` calls and then be protected against bugs due to typos and api +changes. + +As well as using `autospec` through `patch` there is a +:func:`create_autospec` for creating autospecced mocks directly: + + >>> from urllib import request + >>> mock_request = create_autospec(request) + >>> mock_request.Request('foo', 'bar') + <NonCallableMagicMock name='mock.Request()' spec='Request' id='...'> + +This isn't without caveats and limitations however, which is why it is not +the default behaviour. In order to know what attributes are available on the +spec object, autospec has to introspect (access attributes) the spec. As you +traverse attributes on the mock a corresponding traversal of the original +object is happening under the hood. If any of your specced objects have +properties or descriptors that can trigger code execution then you may not be +able to use autospec. On the other hand it is much better to design your +objects so that introspection is safe [#]_. + +A more serious problem is that it is common for instance attributes to be +created in the `__init__` method and not to exist on the class at all. +`autospec` can't know about any dynamically created attributes and restricts +the api to visible attributes. + + >>> class Something(object): + ... def __init__(self): + ... self.a = 33 + ... + >>> with patch('__main__.Something', autospec=True): + ... thing = Something() + ... thing.a + ... + Traceback (most recent call last): + ... + AttributeError: Mock object has no attribute 'a' + +There are a few different ways of resolving this problem. The easiest, but +not necessarily the least annoying, way is to simply set the required +attributes on the mock after creation. Just because `autospec` doesn't allow +you to fetch attributes that don't exist on the spec it doesn't prevent you +setting them: + + >>> with patch('__main__.Something', autospec=True): + ... thing = Something() + ... thing.a = 33 + ... + +There is a more aggressive version of both `spec` and `autospec` that *does* +prevent you setting non-existent attributes. This is useful if you want to +ensure your code only *sets* valid attributes too, but obviously it prevents +this particular scenario: + + >>> with patch('__main__.Something', autospec=True, spec_set=True): + ... thing = Something() + ... thing.a = 33 + ... + Traceback (most recent call last): + ... + AttributeError: Mock object has no attribute 'a' + +Probably the best way of solving the problem is to add class attributes as +default values for instance members initialised in `__init__`. Note that if +you are only setting default attributes in `__init__` then providing them via +class attributes (shared between instances of course) is faster too. e.g. + +.. code-block:: python + + class Something(object): + a = 33 + +This brings up another issue. It is relatively common to provide a default +value of `None` for members that will later be an object of a different type. +`None` would be useless as a spec because it wouldn't let you access *any* +attributes or methods on it. As `None` is *never* going to be useful as a +spec, and probably indicates a member that will normally of some other type, +`autospec` doesn't use a spec for members that are set to `None`. These will +just be ordinary mocks (well - `MagicMocks`): + + >>> class Something(object): + ... member = None + ... + >>> mock = create_autospec(Something) + >>> mock.member.foo.bar.baz() + <MagicMock name='mock.member.foo.bar.baz()' id='...'> + +If modifying your production classes to add defaults isn't to your liking +then there are more options. One of these is simply to use an instance as the +spec rather than the class. The other is to create a subclass of the +production class and add the defaults to the subclass without affecting the +production class. Both of these require you to use an alternative object as +the spec. Thankfully `patch` supports this - you can simply pass the +alternative object as the `autospec` argument: + + >>> class Something(object): + ... def __init__(self): + ... self.a = 33 + ... + >>> class SomethingForTest(Something): + ... a = 33 + ... + >>> p = patch('__main__.Something', autospec=SomethingForTest) + >>> mock = p.start() + >>> mock.a + <NonCallableMagicMock name='Something.a' spec='int' id='...'> + + +.. [#] This only applies to classes or already instantiated objects. Calling + a mocked class to create a mock instance *does not* create a real instance. + It is only attribute lookups - along with calls to `dir` - that are done. + diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 4d3216b..0af9038 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -792,11 +792,14 @@ Test cases Run the test, collecting the result into the test result object passed as *result*. If *result* is omitted or ``None``, a temporary result object is created (by calling the :meth:`defaultTestResult` method) and - used. The result object is not returned to :meth:`run`'s caller. + used. The result object is returned to :meth:`run`'s caller. The same effect may be had by simply calling the :class:`TestCase` instance. + .. versionchanged:: 3.3 + Previous versions of ``run`` did not return the result. Neither did + calling an instance. .. method:: skipTest(reason) @@ -857,10 +860,11 @@ Test cases | <TestCase.assertNotIsInstance>` | | | +-----------------------------------------+-----------------------------+---------------+ - All the assert methods (except :meth:`assertRaises`, - :meth:`assertRaisesRegex`, :meth:`assertWarns`, :meth:`assertWarnsRegex`) - accept a *msg* argument that, if specified, is used as the error message on - failure (see also :data:`longMessage`). + All the assert methods accept a *msg* argument that, if specified, is used + as the error message on failure (see also :data:`longMessage`). + Note that the *msg* keyword argument can be passed to :meth:`assertRaises`, + :meth:`assertRaisesRegex`, :meth:`assertWarns`, :meth:`assertWarnsRegex` + only when they are used as a context manager. .. method:: assertEqual(first, second, msg=None) @@ -955,7 +959,7 @@ Test cases +---------------------------------------------------------+--------------------------------------+------------+ .. method:: assertRaises(exception, callable, *args, **kwds) - assertRaises(exception) + assertRaises(exception, msg=None) Test that an exception is raised when *callable* is called with any positional or keyword arguments that are also passed to @@ -964,12 +968,16 @@ Test cases To catch any of a group of exceptions, a tuple containing the exception classes may be passed as *exception*. - If only the *exception* argument is given, returns a context manager so - that the code under test can be written inline rather than as a function:: + If only the *exception* and possibly the *msg* arguments are given, + return a context manager so that the code under test can be written + inline rather than as a function:: with self.assertRaises(SomeException): do_something() + When used as a context manager, :meth:`assertRaises` accepts the + additional keyword argument *msg*. + The context manager will store the caught exception object in its :attr:`exception` attribute. This can be useful if the intention is to perform additional checks on the exception raised:: @@ -986,9 +994,12 @@ Test cases .. versionchanged:: 3.2 Added the :attr:`exception` attribute. + .. versionchanged:: 3.3 + Added the *msg* keyword argument when used as a context manager. + .. method:: assertRaisesRegex(exception, regex, callable, *args, **kwds) - assertRaisesRegex(exception, regex) + assertRaisesRegex(exception, regex, msg=None) Like :meth:`assertRaises` but also tests that *regex* matches on the string representation of the raised exception. *regex* may be @@ -1005,12 +1016,16 @@ Test cases .. versionadded:: 3.1 under the name ``assertRaisesRegexp``. + .. versionchanged:: 3.2 Renamed to :meth:`assertRaisesRegex`. + .. versionchanged:: 3.3 + Added the *msg* keyword argument when used as a context manager. + .. method:: assertWarns(warning, callable, *args, **kwds) - assertWarns(warning) + assertWarns(warning, msg=None) Test that a warning is triggered when *callable* is called with any positional or keyword arguments that are also passed to @@ -1019,12 +1034,16 @@ Test cases To catch any of a group of warnings, a tuple containing the warning classes may be passed as *warnings*. - If only the *warning* argument is given, returns a context manager so - that the code under test can be written inline rather than as a function:: + If only the *warning* and possibly the *msg* arguments are given, + returns a context manager so that the code under test can be written + inline rather than as a function:: with self.assertWarns(SomeWarning): do_something() + When used as a context manager, :meth:`assertRaises` accepts the + additional keyword argument *msg*. + The context manager will store the caught warning object in its :attr:`warning` attribute, and the source line which triggered the warnings in the :attr:`filename` and :attr:`lineno` attributes. @@ -1042,9 +1061,12 @@ Test cases .. versionadded:: 3.2 + .. versionchanged:: 3.3 + Added the *msg* keyword argument when used as a context manager. + .. method:: assertWarnsRegex(warning, regex, callable, *args, **kwds) - assertWarnsRegex(warning, regex) + assertWarnsRegex(warning, regex, msg=None) Like :meth:`assertWarns` but also tests that *regex* matches on the message of the triggered warning. *regex* may be a regular expression @@ -1062,6 +1084,8 @@ Test cases .. versionadded:: 3.2 + .. versionchanged:: 3.3 + Added the *msg* keyword argument when used as a context manager. There are also other methods used to perform more specific checks, such as: @@ -1151,21 +1175,6 @@ Test cases :meth:`.assertNotRegex`. - .. method:: assertDictContainsSubset(subset, dictionary, msg=None) - - Tests whether the key/value pairs in *dictionary* are a superset of - those in *subset*. If not, an error message listing the missing keys - and mismatched values is generated. - - Note, the arguments are in the opposite order of what the method name - dictates. Instead, consider using the set-methods on :ref:`dictionary - views <dict-views>`, for example: ``d.keys() <= e.keys()`` or - ``d.items() <= d.items()``. - - .. versionadded:: 3.1 - .. deprecated:: 3.2 - - .. method:: assertCountEqual(first, second, msg=None) Test that sequence *first* contains the same elements as *second*, @@ -1180,21 +1189,6 @@ Test cases .. versionadded:: 3.2 - .. method:: assertSameElements(first, second, msg=None) - - Test that sequence *first* contains the same elements as *second*, - regardless of their order. When they don't, an error message listing - the differences between the sequences will be generated. - - Duplicate elements are ignored when comparing *first* and *second*. - It is the equivalent of ``assertEqual(set(first), set(second))`` - but it works with sequences of unhashable objects as well. Because - duplicates are ignored, this method has been deprecated in favour of - :meth:`assertCountEqual`. - - .. versionadded:: 3.1 - .. deprecated:: 3.2 - .. _type-specific-methods: diff --git a/Doc/library/urllib.error.rst b/Doc/library/urllib.error.rst index 282329f..e20db27 100644 --- a/Doc/library/urllib.error.rst +++ b/Doc/library/urllib.error.rst @@ -8,21 +8,23 @@ The :mod:`urllib.error` module defines the exception classes for exceptions -raised by :mod:`urllib.request`. The base exception class is :exc:`URLError`, -which inherits from :exc:`IOError`. +raised by :mod:`urllib.request`. The base exception class is :exc:`URLError`. The following exceptions are raised by :mod:`urllib.error` as appropriate: .. exception:: URLError The handlers raise this exception (or derived exceptions) when they run into - a problem. It is a subclass of :exc:`IOError`. + a problem. It is a subclass of :exc:`OSError`. .. attribute:: reason The reason for this error. It can be a message string or another - exception instance (:exc:`socket.error` for remote URLs, :exc:`OSError` - for local URLs). + exception instance. + + .. versionchanged:: 3.3 + :exc:`URLError` has been made a subclass of :exc:`OSError` instead + of :exc:`IOError`. .. exception:: HTTPError diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index db26b27..cd90a80 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -16,7 +16,7 @@ authentication, redirections, cookies and more. The :mod:`urllib.request` module defines the following functions: -.. function:: urlopen(url, data=None[, timeout], *, cafile=None, capath=None) +.. function:: urlopen(url, data=None[, timeout], *, cafile=None, capath=None, cadefault=True) Open the URL *url*, which can be either a string or a :class:`Request` object. @@ -53,9 +53,15 @@ The :mod:`urllib.request` module defines the following functions: point to a directory of hashed certificate files. More information can be found in :meth:`ssl.SSLContext.load_verify_locations`. + The *cadefault* parameter specifies whether to fall back to loading a + default certificate store defined by the underlying OpenSSL library if the + *cafile* and *capath* parameters are omitted. This will only work on + some non-Windows platforms. + .. warning:: - If neither *cafile* nor *capath* is specified, an HTTPS request - will not do any verification of the server's certificate. + If neither *cafile* nor *capath* is specified, and *cadefault* is False, + an HTTPS request will not do any verification of the server's + certificate. This function returns a file-like object that works as a :term:`context manager`, with two additional methods from the :mod:`urllib.response` module @@ -92,6 +98,9 @@ The :mod:`urllib.request` module defines the following functions: .. versionadded:: 3.2 *data* can be an iterable object. + .. versionchanged:: 3.3 + *cadefault* was added. + .. function:: install_opener(opener) Install an :class:`OpenerDirector` instance as the default global opener. @@ -145,7 +154,7 @@ The :mod:`urllib.request` module defines the following functions: The following classes are provided: -.. class:: Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False) +.. class:: Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None) This class is an abstraction of a URL request. @@ -198,6 +207,13 @@ The following classes are provided: document, and the user had no option to approve the automatic fetching of the image, this should be true. + *method* should be a string that indicates the HTTP request method that + will be used (e.g. ``'HEAD'``). Its value is stored in the + :attr:`~Request.method` attribute and is used by :meth:`get_method()`. + + .. versionchanged:: 3.3 + :attr:`Request.method` argument is added to the Request class. + .. class:: OpenerDirector() @@ -263,10 +279,11 @@ The following classes are provided: .. class:: HTTPBasicAuthHandler(password_mgr=None) - Handle authentication with the remote host. *password_mgr*, if given, should be - something that is compatible with :class:`HTTPPasswordMgr`; refer to section - :ref:`http-password-mgr` for information on the interface that must be - supported. + Handle authentication with the remote host. *password_mgr*, if given, should + be something that is compatible with :class:`HTTPPasswordMgr`; refer to + section :ref:`http-password-mgr` for information on the interface that must + be supported. HTTPBasicAuthHandler will raise a :exc:`ValueError` when + presented with a wrong Authentication scheme. .. class:: ProxyBasicAuthHandler(password_mgr=None) @@ -288,10 +305,19 @@ The following classes are provided: .. class:: HTTPDigestAuthHandler(password_mgr=None) - Handle authentication with the remote host. *password_mgr*, if given, should be - something that is compatible with :class:`HTTPPasswordMgr`; refer to section - :ref:`http-password-mgr` for information on the interface that must be - supported. + Handle authentication with the remote host. *password_mgr*, if given, should + be something that is compatible with :class:`HTTPPasswordMgr`; refer to + section :ref:`http-password-mgr` for information on the interface that must + be supported. When both Digest Authentication Handler and Basic + Authentication Handler are both added, Digest Authentication is always tried + first. If the Digest Authentication returns a 40x response again, it is sent + to Basic Authentication handler to Handle. This Handler method will raise a + :exc:`ValueError` when presented with an authentication scheme other than + Digest or Basic. + + .. versionchanged:: 3.3 + Raise :exc:`ValueError` on unsupported Authentication Scheme. + .. class:: ProxyDigestAuthHandler(password_mgr=None) @@ -382,27 +408,25 @@ request. boolean, indicates whether the request is unverifiable as defined by RFC 2965. -.. method:: Request.add_data(data) - - Set the :class:`Request` data to *data*. This is ignored by all handlers except - HTTP handlers --- and there it should be a byte string, and will change the - request to be ``POST`` rather than ``GET``. - - -.. method:: Request.get_method() - - Return a string indicating the HTTP request method. This is only meaningful for - HTTP requests, and currently always returns ``'GET'`` or ``'POST'``. +.. attribute:: Request.method + The HTTP request method to use. This value is used by + :meth:`~Request.get_method` to override the computed HTTP request + method that would otherwise be returned. This attribute is initialized with + the value of the *method* argument passed to the constructor. -.. method:: Request.has_data() + .. versionadded:: 3.3 - Return whether the instance has a non-\ ``None`` data. +.. method:: Request.get_method() -.. method:: Request.get_data() + Return a string indicating the HTTP request method. If + :attr:`Request.method` is not ``None``, return its value, otherwise return + ``'GET'`` if :attr:`Request.data` is ``None``, or ``'POST'`` if it's not. + This is only meaningful for HTTP requests. - Return the instance's data. + .. versionchanged:: 3.3 + get_method now looks at the value of :attr:`Request.method`. .. method:: Request.add_header(key, val) @@ -432,20 +456,60 @@ request. Return the URL given in the constructor. +.. method:: Request.set_proxy(host, type) + + Prepare the request by connecting to a proxy server. The *host* and *type* will + replace those of the instance, and the instance's selector will be the original + URL given in the constructor. + + +.. method:: Request.add_data(data) + + Set the :class:`Request` data to *data*. This is ignored by all handlers except + HTTP handlers --- and there it should be a byte string, and will change the + request to be ``POST`` rather than ``GET``. Deprecated in 3.3, use + :attr:`Request.data`. + + .. deprecated:: 3.3 + + +.. method:: Request.has_data() + + Return whether the instance has a non-\ ``None`` data. Deprecated in 3.3, + use :attr:`Request.data`. + + .. deprecated:: 3.3 + + +.. method:: Request.get_data() + + Return the instance's data. Deprecated in 3.3, use :attr:`Request.data`. + + .. deprecated:: 3.3 + + .. method:: Request.get_type() - Return the type of the URL --- also known as the scheme. + Return the type of the URL --- also known as the scheme. Deprecated in 3.3, + use :attr:`Request.type`. + + .. deprecated:: 3.3 .. method:: Request.get_host() - Return the host to which a connection will be made. + Return the host to which a connection will be made. Deprecated in 3.3, use + :attr:`Request.host`. + + .. deprecated:: 3.3 .. method:: Request.get_selector() Return the selector --- the part of the URL that is sent to the server. + Deprecated in 3.3, use :attr:`Request.selector`. + .. deprecated:: 3.3 .. method:: Request.get_header(header_name, default=None) @@ -460,21 +524,22 @@ request. .. method:: Request.set_proxy(host, type) - Prepare the request by connecting to a proxy server. The *host* and *type* will - replace those of the instance, and the instance's selector will be the original - URL given in the constructor. - - .. method:: Request.get_origin_req_host() - Return the request-host of the origin transaction, as defined by :rfc:`2965`. - See the documentation for the :class:`Request` constructor. + Return the request-host of the origin transaction, as defined by + :rfc:`2965`. See the documentation for the :class:`Request` constructor. + Deprecated in 3.3, use :attr:`Request.origin_req_host`. + + .. deprecated:: 3.3 .. method:: Request.is_unverifiable() Return whether the request is unverifiable, as defined by RFC 2965. See the - documentation for the :class:`Request` constructor. + documentation for the :class:`Request` constructor. Deprecated in 3.3, use + :attr:`Request.is_unverifiable`. + + .. deprecated:: 3.3 .. _opener-director-objects: @@ -1139,16 +1204,14 @@ The following functions and classes are ported from the Python 2 module ``urllib`` (as opposed to ``urllib2``). They might become deprecated at some point in the future. - .. function:: urlretrieve(url, filename=None, reporthook=None, data=None) - Copy a network object denoted by a URL to a local file, if necessary. If the URL - points to a local file, or a valid cached copy of the object exists, the object - is not copied. Return a tuple ``(filename, headers)`` where *filename* is the + Copy a network object denoted by a URL to a local file. If the URL + points to a local file, the object will not be copied unless filename is supplied. + Return a tuple ``(filename, headers)`` where *filename* is the local file name under which the object can be found, and *headers* is whatever the :meth:`info` method of the object returned by :func:`urlopen` returned (for - a remote object, possibly cached). Exceptions are the same as for - :func:`urlopen`. + a remote object). Exceptions are the same as for :func:`urlopen`. The second argument, if present, specifies the file location to copy to (if absent, the location will be a tempfile with a generated name). The third @@ -1159,6 +1222,13 @@ some point in the future. third argument may be ``-1`` on older FTP servers which do not return a file size in response to a retrieval request. + The following example illustrates the most common usage scenario:: + + >>> import urllib.request + >>> local_filename, headers = urllib.request.urlretrieve('http://python.org/') + >>> html = open(local_filename) + >>> html.close() + If the *url* uses the :file:`http:` scheme identifier, the optional *data* argument may be given to specify a ``POST`` request (normally the request type is ``GET``). The *data* argument must be a bytes object in standard @@ -1171,20 +1241,20 @@ some point in the future. the download is interrupted. The *Content-Length* is treated as a lower bound: if there's more data to read, - :func:`urlretrieve` reads more data, but if less data is available, it raises - the exception. + urlretrieve reads more data, but if less data is available, it raises the + exception. You can still retrieve the downloaded data in this case, it is stored in the :attr:`content` attribute of the exception instance. - If no *Content-Length* header was supplied, :func:`urlretrieve` can not check - the size of the data it has downloaded, and just returns it. In this case - you just have to assume that the download was successful. + If no *Content-Length* header was supplied, urlretrieve can not check the size + of the data it has downloaded, and just returns it. In this case you just have + to assume that the download was successful. .. function:: urlcleanup() - Clear the cache that may have been built up by previous calls to - :func:`urlretrieve`. + Cleans up temporary files that may have been left behind by previous + calls to :func:`urlretrieve`. .. class:: URLopener(proxies=None, **x509) @@ -1208,7 +1278,7 @@ some point in the future. *key_file* and *cert_file* are supported to provide an SSL key and certificate; both are needed to support client authentication. - :class:`URLopener` objects will raise an :exc:`IOError` exception if the server + :class:`URLopener` objects will raise an :exc:`OSError` exception if the server returns an error code. .. method:: open(fullurl, data=None) diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst index 8af19a2..8387f5a 100644 --- a/Doc/library/warnings.rst +++ b/Doc/library/warnings.rst @@ -339,8 +339,7 @@ Available Functions Write a warning to a file. The default implementation calls ``formatwarning(message, category, filename, lineno, line)`` and writes the resulting string to *file*, which defaults to ``sys.stderr``. You may replace - this function with an alternative implementation by assigning to - ``warnings.showwarning``. + this function with any callable by assigning to ``warnings.showwarning``. *line* is a line of source code to be included in the warning message; if *line* is not supplied, :func:`showwarning` will try to read the line specified by *filename* and *lineno*. diff --git a/Doc/library/webbrowser.rst b/Doc/library/webbrowser.rst index b1a8d73..1a4cb69 100644 --- a/Doc/library/webbrowser.rst +++ b/Doc/library/webbrowser.rst @@ -96,47 +96,55 @@ A number of browser types are predefined. This table gives the type names that may be passed to the :func:`get` function and the corresponding instantiations for the controller classes, all defined in this module. -+-----------------------+-----------------------------------------+-------+ -| Type Name | Class Name | Notes | -+=======================+=========================================+=======+ -| ``'mozilla'`` | :class:`Mozilla('mozilla')` | | -+-----------------------+-----------------------------------------+-------+ -| ``'firefox'`` | :class:`Mozilla('mozilla')` | | -+-----------------------+-----------------------------------------+-------+ -| ``'netscape'`` | :class:`Mozilla('netscape')` | | -+-----------------------+-----------------------------------------+-------+ -| ``'galeon'`` | :class:`Galeon('galeon')` | | -+-----------------------+-----------------------------------------+-------+ -| ``'epiphany'`` | :class:`Galeon('epiphany')` | | -+-----------------------+-----------------------------------------+-------+ -| ``'skipstone'`` | :class:`BackgroundBrowser('skipstone')` | | -+-----------------------+-----------------------------------------+-------+ -| ``'kfmclient'`` | :class:`Konqueror()` | \(1) | -+-----------------------+-----------------------------------------+-------+ -| ``'konqueror'`` | :class:`Konqueror()` | \(1) | -+-----------------------+-----------------------------------------+-------+ -| ``'kfm'`` | :class:`Konqueror()` | \(1) | -+-----------------------+-----------------------------------------+-------+ -| ``'mosaic'`` | :class:`BackgroundBrowser('mosaic')` | | -+-----------------------+-----------------------------------------+-------+ -| ``'opera'`` | :class:`Opera()` | | -+-----------------------+-----------------------------------------+-------+ -| ``'grail'`` | :class:`Grail()` | | -+-----------------------+-----------------------------------------+-------+ -| ``'links'`` | :class:`GenericBrowser('links')` | | -+-----------------------+-----------------------------------------+-------+ -| ``'elinks'`` | :class:`Elinks('elinks')` | | -+-----------------------+-----------------------------------------+-------+ -| ``'lynx'`` | :class:`GenericBrowser('lynx')` | | -+-----------------------+-----------------------------------------+-------+ -| ``'w3m'`` | :class:`GenericBrowser('w3m')` | | -+-----------------------+-----------------------------------------+-------+ -| ``'windows-default'`` | :class:`WindowsDefault` | \(2) | -+-----------------------+-----------------------------------------+-------+ -| ``'macosx'`` | :class:`MacOSX('default')` | \(3) | -+-----------------------+-----------------------------------------+-------+ -| ``'safari'`` | :class:`MacOSX('safari')` | \(3) | -+-----------------------+-----------------------------------------+-------+ ++------------------------+-----------------------------------------+-------+ +| Type Name | Class Name | Notes | ++========================+=========================================+=======+ +| ``'mozilla'`` | :class:`Mozilla('mozilla')` | | ++------------------------+-----------------------------------------+-------+ +| ``'firefox'`` | :class:`Mozilla('mozilla')` | | ++------------------------+-----------------------------------------+-------+ +| ``'netscape'`` | :class:`Mozilla('netscape')` | | ++------------------------+-----------------------------------------+-------+ +| ``'galeon'`` | :class:`Galeon('galeon')` | | ++------------------------+-----------------------------------------+-------+ +| ``'epiphany'`` | :class:`Galeon('epiphany')` | | ++------------------------+-----------------------------------------+-------+ +| ``'skipstone'`` | :class:`BackgroundBrowser('skipstone')` | | ++------------------------+-----------------------------------------+-------+ +| ``'kfmclient'`` | :class:`Konqueror()` | \(1) | ++------------------------+-----------------------------------------+-------+ +| ``'konqueror'`` | :class:`Konqueror()` | \(1) | ++------------------------+-----------------------------------------+-------+ +| ``'kfm'`` | :class:`Konqueror()` | \(1) | ++------------------------+-----------------------------------------+-------+ +| ``'mosaic'`` | :class:`BackgroundBrowser('mosaic')` | | ++------------------------+-----------------------------------------+-------+ +| ``'opera'`` | :class:`Opera()` | | ++------------------------+-----------------------------------------+-------+ +| ``'grail'`` | :class:`Grail()` | | ++------------------------+-----------------------------------------+-------+ +| ``'links'`` | :class:`GenericBrowser('links')` | | ++------------------------+-----------------------------------------+-------+ +| ``'elinks'`` | :class:`Elinks('elinks')` | | ++------------------------+-----------------------------------------+-------+ +| ``'lynx'`` | :class:`GenericBrowser('lynx')` | | ++------------------------+-----------------------------------------+-------+ +| ``'w3m'`` | :class:`GenericBrowser('w3m')` | | ++------------------------+-----------------------------------------+-------+ +| ``'windows-default'`` | :class:`WindowsDefault` | \(2) | ++------------------------+-----------------------------------------+-------+ +| ``'macosx'`` | :class:`MacOSX('default')` | \(3) | ++------------------------+-----------------------------------------+-------+ +| ``'safari'`` | :class:`MacOSX('safari')` | \(3) | ++------------------------+-----------------------------------------+-------+ +| ``'google-chrome'`` | :class:`Chrome('google-chrome')` | | ++------------------------+-----------------------------------------+-------+ +| ``'chrome'`` | :class:`Chrome('chrome')` | | ++------------------------+-----------------------------------------+-------+ +| ``'chromium'`` | :class:`Chromium('chromium')` | | ++------------------------+-----------------------------------------+-------+ +| ``'chromium-browser'`` | :class:`Chromium('chromium-browser')` | | ++------------------------+-----------------------------------------+-------+ Notes: @@ -153,12 +161,15 @@ Notes: (3) Only on Mac OS X platform. +.. versionadded:: 3.3 + Support for Chrome/Chromium has been added. + Here are some simple examples:: - url = 'http://www.python.org/' + url = 'http://docs.python.org/' # Open URL in a new tab, if a browser window is already open. - webbrowser.open_new_tab(url + 'doc/') + webbrowser.open_new_tab(url) # Open URL in new window, raising the window if possible. webbrowser.open_new(url) diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst index 5cf30ee..376752e 100644 --- a/Doc/library/winreg.rst +++ b/Doc/library/winreg.rst @@ -38,7 +38,11 @@ This module offers the following functions: *key* is the predefined handle to connect to. The return value is the handle of the opened key. If the function fails, a - :exc:`WindowsError` exception is raised. + :exc:`OSError` exception is raised. + + .. versionchanged:: 3.3 + This function used to raise a :exc:`WindowsError`, which is now an + alias of :exc:`OSError`. .. function:: CreateKey(key, sub_key) @@ -57,7 +61,11 @@ This module offers the following functions: If the key already exists, this function opens the existing key. The return value is the handle of the opened key. If the function fails, a - :exc:`WindowsError` exception is raised. + :exc:`OSError` exception is raised. + + .. versionchanged:: 3.3 + This function used to raise a :exc:`WindowsError`, which is now an + alias of :exc:`OSError`. .. function:: CreateKeyEx(key, sub_key, reserved=0, access=KEY_ALL_ACCESS) @@ -82,10 +90,14 @@ This module offers the following functions: If the key already exists, this function opens the existing key. The return value is the handle of the opened key. If the function fails, a - :exc:`WindowsError` exception is raised. + :exc:`OSError` exception is raised. .. versionadded:: 3.2 + .. versionchanged:: 3.3 + This function used to raise a :exc:`WindowsError`, which is now an + alias of :exc:`OSError`. + .. function:: DeleteKey(key, sub_key) @@ -100,7 +112,11 @@ This module offers the following functions: *This method can not delete keys with subkeys.* If the method succeeds, the entire key, including all of its values, is removed. - If the method fails, a :exc:`WindowsError` exception is raised. + If the method fails, a :exc:`OSError` exception is raised. + + .. versionchanged:: 3.3 + This function used to raise a :exc:`WindowsError`, which is now an + alias of :exc:`OSError`. .. function:: DeleteKeyEx(key, sub_key, access=KEY_ALL_ACCESS, reserved=0) @@ -129,12 +145,16 @@ This module offers the following functions: *This method can not delete keys with subkeys.* If the method succeeds, the entire key, including all of its values, is - removed. If the method fails, a :exc:`WindowsError` exception is raised. + removed. If the method fails, a :exc:`OSError` exception is raised. On unsupported Windows versions, :exc:`NotImplementedError` is raised. .. versionadded:: 3.2 + .. versionchanged:: 3.3 + This function used to raise a :exc:`WindowsError`, which is now an + alias of :exc:`OSError`. + .. function:: DeleteValue(key, value) @@ -156,9 +176,13 @@ This module offers the following functions: *index* is an integer that identifies the index of the key to retrieve. The function retrieves the name of one subkey each time it is called. It is - typically called repeatedly until a :exc:`WindowsError` exception is + typically called repeatedly until a :exc:`OSError` exception is raised, indicating, no more values are available. + .. versionchanged:: 3.3 + This function used to raise a :exc:`WindowsError`, which is now an + alias of :exc:`OSError`. + .. function:: EnumValue(key, index) @@ -170,7 +194,7 @@ This module offers the following functions: *index* is an integer that identifies the index of the value to retrieve. The function retrieves the name of one subkey each time it is called. It is - typically called repeatedly, until a :exc:`WindowsError` exception is + typically called repeatedly, until a :exc:`OSError` exception is raised, indicating no more values. The result is a tuple of 3 items: @@ -189,6 +213,10 @@ This module offers the following functions: | | :meth:`SetValueEx`) | +-------+--------------------------------------------+ + .. versionchanged:: 3.3 + This function used to raise a :exc:`WindowsError`, which is now an + alias of :exc:`OSError`. + .. function:: ExpandEnvironmentStrings(str) @@ -260,10 +288,14 @@ This module offers the following functions: The result is a new handle to the specified key. - If the function fails, :exc:`WindowsError` is raised. + If the function fails, :exc:`OSError` is raised. .. versionchanged:: 3.2 Allow the use of named arguments. + .. versionchanged:: 3.3 + This function used to raise a :exc:`WindowsError`, which is now an + alias of :exc:`OSError`. + .. function:: OpenKeyEx() diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst index c5c8802..d00781c 100644 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -5,49 +5,232 @@ :synopsis: Implementation of the ElementTree API. .. moduleauthor:: Fredrik Lundh <fredrik@pythonware.com> -**Source code:** :source:`Lib/xml/etree/ElementTree.py` +The :mod:`xml.etree.ElementTree` module implements a simple and efficient API +for parsing and creating XML data. + +.. versionchanged:: 3.3 + This module will use a fast implementation whenever available. + The :mod:`xml.etree.cElementTree` module is deprecated. + +Tutorial +-------- + +This is a short tutorial for using :mod:`xml.etree.ElementTree` (``ET`` in +short). The goal is to demonstrate some of the building blocks and basic +concepts of the module. + +XML tree and elements +^^^^^^^^^^^^^^^^^^^^^ + +XML is an inherently hierarchical data format, and the most natural way to +represent it is with a tree. ``ET`` has two classes for this purpose - +:class:`ElementTree` represents the whole XML document as a tree, and +:class:`Element` represents a single node in this tree. Interactions with +the whole document (reading and writing to/from files) are usually done +on the :class:`ElementTree` level. Interactions with a single XML element +and its sub-elements are done on the :class:`Element` level. + +.. _elementtree-parsing-xml: + +Parsing XML +^^^^^^^^^^^ + +We'll be using the following XML document contained in a Python string as the +sample data for this section:: + + countrydata = r'''<?xml version="1.0"?> + <data> + <country name="Liechtenshtein"> + <rank>1</rank> + <year>2008</year> + <gdppc>141100</gdppc> + <neighbor name="Austria" direction="E"/> + <neighbor name="Switzerland" direction="W"/> + </country> + <country name="Singapore"> + <rank>4</rank> + <year>2011</year> + <gdppc>59900</gdppc> + <neighbor name="Malaysia" direction="N"/> + </country> + <country name="Panama"> + <rank>68</rank> + <year>2011</year> + <gdppc>13600</gdppc> + <neighbor name="Costa Rica" direction="W"/> + <neighbor name="Colombia" direction="E"/> + </country> + </data> + ''' + +First, import the module and parse the data:: + + import xml.etree.ElementTree as ET + + root = ET.fromstring(countrydata) + +:func:`fromstring` parses XML from a string directly into an :class:`Element`, +which is the root element of the parsed tree. Other parsing functions may +create an :class:`ElementTree`. Make sure to check the documentation to be +sure. + +As an :class:`Element`, ``root`` has a tag and a dictionary of attributes:: + + >>> root.tag + 'data' + >>> root.attrib + {} + +It also has children nodes over which we can iterate:: + + >>> for child in root: + ... print(child.tag, child.attrib) + ... + country {'name': 'Liechtenshtein'} + country {'name': 'Singapore'} + country {'name': 'Panama'} + +Children are nested, and we can access specific child nodes by index:: + + >>> root[0][1].text + '2008' + +Finding interesting elements +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +:class:`Element` has some useful methods that help iterate recursively over all +the sub-tree below it (its children, their children, and so on). For example, +:meth:`Element.iter`:: + + >>> for neighbor in root.iter('neighbor'): + ... print(neighbor.attrib) + ... + {'name': 'Austria', 'direction': 'E'} + {'name': 'Switzerland', 'direction': 'W'} + {'name': 'Malaysia', 'direction': 'N'} + {'name': 'Costa Rica', 'direction': 'W'} + {'name': 'Colombia', 'direction': 'E'} + +More sophisticated specification of which elements to look for is possible by +using :ref:`XPath <elementtree-xpath>`. + +Building XML documents +^^^^^^^^^^^^^^^^^^^^^^ + +``ET`` provides a simple way to build XML documents and write them to files. +The :meth:`ElementTree.write` method serves this purpose. + +Once created, an :class:`Element` object may be manipulated by directly changing +its fields (such as :attr:`Element.text`), adding and modifying attributes +(:meth:`Element.set` method), as well as adding new children (for example +with :meth:`Element.append`). + +The :func:`SubElement` function also provides a convenient way to create new +sub-elements for a given element:: + + >>> a = ET.Element('a') + >>> b = ET.SubElement(a, 'b') + >>> c = ET.SubElement(a, 'c') + >>> d = ET.SubElement(c, 'd') + >>> ET.dump(a) + <a><b /><c><d /></c></a> + +Additional resources +^^^^^^^^^^^^^^^^^^^^ --------------- - -The :class:`Element` type is a flexible container object, designed to store -hierarchical data structures in memory. The type can be described as a cross -between a list and a dictionary. - -Each element has a number of properties associated with it: - -* a tag which is a string identifying what kind of data this element represents - (the element type, in other words). - -* a number of attributes, stored in a Python dictionary. - -* a text string. - -* an optional tail string. - -* a number of child elements, stored in a Python sequence - -To create an element instance, use the :class:`Element` constructor or the -:func:`SubElement` factory function. - -The :class:`ElementTree` class can be used to wrap an element structure, and -convert it from and to XML. +See http://effbot.org/zone/element-index.htm for tutorials and links to other +docs. -A C implementation of this API is available as :mod:`xml.etree.cElementTree`. -See http://effbot.org/zone/element-index.htm for tutorials and links to other -docs. Fredrik Lundh's page is also the location of the development version of -the xml.etree.ElementTree. +.. _elementtree-xpath: -.. versionchanged:: 3.2 - The ElementTree API is updated to 1.3. For more information, see - `Introducing ElementTree 1.3 - <http://effbot.org/zone/elementtree-13-intro.htm>`_. +XPath support +------------- +This module provides limited support for +`XPath expressions <http://www.w3.org/TR/xpath>`_ for locating elements in a +tree. The goal is to support a small subset of the abbreviated syntax; a full +XPath engine is outside the scope of the module. + +Example +^^^^^^^ + +Here's an example that demonstrates some of the XPath capabilities of the +module. We'll be using the ``countrydata`` XML document from the +:ref:`Parsing XML <elementtree-parsing-xml>` section:: + + import xml.etree.ElementTree as ET + + root = ET.fromstring(countrydata) + + # Top-level elements + root.findall(".") + + # All 'neighbor' grand-children of 'country' children of the top-level + # elements + root.findall("./country/neighbor") + + # Nodes with name='Singapore' that have a 'year' child + root.findall(".//year/..[@name='Singapore']") + + # 'year' nodes that are children of nodes with name='Singapore' + root.findall(".//*[@name='Singapore']/year") + + # All 'neighbor' nodes that are the second child of their parent + root.findall(".//neighbor[2]") + +Supported XPath syntax +^^^^^^^^^^^^^^^^^^^^^^ + ++-----------------------+------------------------------------------------------+ +| Syntax | Meaning | ++=======================+======================================================+ +| ``tag`` | Selects all child elements with the given tag. | +| | For example, ``spam`` selects all child elements | +| | named ``spam``, ``spam/egg`` selects all | +| | grandchildren named ``egg`` in all children named | +| | ``spam``. | ++-----------------------+------------------------------------------------------+ +| ``*`` | Selects all child elements. For example, ``*/egg`` | +| | selects all grandchildren named ``egg``. | ++-----------------------+------------------------------------------------------+ +| ``.`` | Selects the current node. This is mostly useful | +| | at the beginning of the path, to indicate that it's | +| | a relative path. | ++-----------------------+------------------------------------------------------+ +| ``//`` | Selects all subelements, on all levels beneath the | +| | current element. For example, ``.//egg`` selects | +| | all ``egg`` elements in the entire tree. | ++-----------------------+------------------------------------------------------+ +| ``..`` | Selects the parent element. | ++-----------------------+------------------------------------------------------+ +| ``[@attrib]`` | Selects all elements that have the given attribute. | ++-----------------------+------------------------------------------------------+ +| ``[@attrib='value']`` | Selects all elements for which the given attribute | +| | has the given value. The value cannot contain | +| | quotes. | ++-----------------------+------------------------------------------------------+ +| ``[tag]`` | Selects all elements that have a child named | +| | ``tag``. Only immediate children are supported. | ++-----------------------+------------------------------------------------------+ +| ``[position]`` | Selects all elements that are located at the given | +| | position. The position can be either an integer | +| | (1 is the first position), the expression ``last()`` | +| | (for the last position), or a position relative to | +| | the last position (e.g. ``last()-1``). | ++-----------------------+------------------------------------------------------+ + +Predicates (expressions within square brackets) must be preceded by a tag +name, an asterisk, or another predicate. ``position`` predicates must be +preceded by a tag name. + +Reference +--------- .. _elementtree-functions: Functions ---------- +^^^^^^^^^ .. function:: Comment(text=None) @@ -198,8 +381,7 @@ Functions .. _elementtree-element-objects: Element Objects ---------------- - +^^^^^^^^^^^^^^^ .. class:: Element(tag, attrib={}, **extra) @@ -281,14 +463,15 @@ Element Objects .. method:: append(subelement) - Adds the element *subelement* to the end of this elements internal list - of subelements. + Adds the element *subelement* to the end of this element's internal list + of subelements. Raises :exc:`TypeError` if *subelement* is not an + :class:`Element`. .. method:: extend(subelements) Appends *subelements* from a sequence object with zero or more elements. - Raises :exc:`AssertionError` if a subelement is not a valid object. + Raises :exc:`TypeError` if a subelement is not an :class:`Element`. .. versionadded:: 3.2 @@ -296,21 +479,24 @@ Element Objects .. method:: find(match) Finds the first subelement matching *match*. *match* may be a tag name - or path. Returns an element instance or ``None``. + or a :ref:`path <elementtree-xpath>`. Returns an element instance + or ``None``. .. method:: findall(match) - Finds all matching subelements, by tag name or path. Returns a list - containing all matching elements in document order. + Finds all matching subelements, by tag name or + :ref:`path <elementtree-xpath>`. Returns a list containing all matching + elements in document order. .. method:: findtext(match, default=None) Finds text for the first subelement matching *match*. *match* may be - a tag name or path. Returns the text content of the first matching - element, or *default* if no element was found. Note that if the matching - element has no text content an empty string is returned. + a tag name or a :ref:`path <elementtree-xpath>`. Returns the text content + of the first matching element, or *default* if no element was found. + Note that if the matching element has no text content an empty string + is returned. .. method:: getchildren() @@ -325,9 +511,10 @@ Element Objects Use method :meth:`Element.iter` instead. - .. method:: insert(index, element) + .. method:: insert(index, subelement) - Inserts a subelement at the given position in this element. + Inserts *subelement* at the given position in this element. Raises + :exc:`TypeError` if *subelement* is not an :class:`Element`. .. method:: iter(tag=None) @@ -343,8 +530,9 @@ Element Objects .. method:: iterfind(match) - Finds all matching subelements, by tag name or path. Returns an iterable - yielding all matching elements in document order. + Finds all matching subelements, by tag name or + :ref:`path <elementtree-xpath>`. Returns an iterable yielding all + matching elements in document order. .. versionadded:: 3.2 @@ -389,7 +577,7 @@ Element Objects .. _elementtree-elementtree-objects: ElementTree Objects -------------------- +^^^^^^^^^^^^^^^^^^^ .. class:: ElementTree(element=None, file=None) @@ -411,26 +599,17 @@ ElementTree Objects .. method:: find(match) - Finds the first toplevel element matching *match*. *match* may be a tag - name or path. Same as getroot().find(match). Returns the first matching - element, or ``None`` if no element was found. + Same as :meth:`Element.find`, starting at the root of the tree. .. method:: findall(match) - Finds all matching subelements, by tag name or path. Same as - getroot().findall(match). *match* may be a tag name or path. Returns a - list containing all matching elements, in document order. + Same as :meth:`Element.findall`, starting at the root of the tree. .. method:: findtext(match, default=None) - Finds the element text for the first toplevel element with given tag. - Same as getroot().findtext(match). *match* may be a tag name or path. - *default* is the value to return if the element was not found. Returns - the text content of the first matching element, or the default value no - element was found. Note that if the element is found, but has no text - content, this method returns an empty string. + Same as :meth:`Element.findtext`, starting at the root of the tree. .. method:: getiterator(tag=None) @@ -453,9 +632,7 @@ ElementTree Objects .. method:: iterfind(match) - Finds all matching subelements, by tag name or path. Same as - getroot().iterfind(match). Returns an iterable yielding all matching - elements in document order. + Same as :meth:`Element.iterfind`, starting at the root of the tree. .. versionadded:: 3.2 @@ -510,7 +687,7 @@ Example of changing the attribute "target" of every link in first paragraph:: .. _elementtree-qname-objects: QName Objects -------------- +^^^^^^^^^^^^^ .. class:: QName(text_or_uri, tag=None) @@ -526,7 +703,7 @@ QName Objects .. _elementtree-treebuilder-objects: TreeBuilder Objects -------------------- +^^^^^^^^^^^^^^^^^^^ .. class:: TreeBuilder(element_factory=None) @@ -577,7 +754,7 @@ TreeBuilder Objects .. _elementtree-xmlparser-objects: XMLParser Objects ------------------ +^^^^^^^^^^^^^^^^^ .. class:: XMLParser(html=0, target=None, encoding=None) @@ -645,6 +822,24 @@ This is an example of counting the maximum depth of an XML file:: >>> parser.close() 4 +Exceptions +^^^^^^^^^^ + +.. class:: ParseError + + XML parse error, raised by the various parsing methods in this module when + parsing fails. The string representation of an instance of this exception + will contain a user-friendly error message. In addition, it will have + the following attributes available: + + .. attribute:: code + + A numeric error code from the expat parser. See the documentation of + :mod:`xml.parsers.expat` for the list of error codes and their meanings. + + .. attribute:: position + + A tuple of *line*, *column* numbers, specifying where the error occurred. .. rubric:: Footnotes diff --git a/Doc/library/xmlrpc.client.rst b/Doc/library/xmlrpc.client.rst index e72770a..1871c99 100644 --- a/Doc/library/xmlrpc.client.rst +++ b/Doc/library/xmlrpc.client.rst @@ -8,7 +8,7 @@ .. XXX Not everything is documented yet. It might be good to describe - Marshaller, Unmarshaller, getparser, dumps, loads, and Transport. + Marshaller, Unmarshaller, getparser and Transport. **Source code:** :source:`Lib/xmlrpc/client.py` @@ -21,7 +21,12 @@ supports writing XML-RPC client code; it handles all the details of translating between conformable Python objects and XML on the wire. -.. class:: ServerProxy(uri, transport=None, encoding=None, verbose=False, allow_none=False, use_datetime=False) +.. class:: ServerProxy(uri, transport=None, encoding=None, verbose=False, \ + allow_none=False, use_datetime=False, \ + use_builtin_types=False) + + .. versionchanged:: 3.3 + The *use_builtin_types* flag was added. A :class:`ServerProxy` instance is an object that manages communication with a remote XML-RPC server. The required first argument is a URI (Uniform Resource @@ -34,9 +39,13 @@ between conformable Python objects and XML on the wire. XML; the default behaviour is for ``None`` to raise a :exc:`TypeError`. This is a commonly-used extension to the XML-RPC specification, but isn't supported by all clients and servers; see http://ontosys.com/xml-rpc/extensions.php for a - description. The *use_datetime* flag can be used to cause date/time values to - be presented as :class:`datetime.datetime` objects; this is false by default. - :class:`datetime.datetime` objects may be passed to calls. + description. The *use_builtin_types* flag can be used to cause date/time values + to be presented as :class:`datetime.datetime` objects and binary data to be + presented as :class:`bytes` objects; this flag is false by default. + :class:`datetime.datetime` and :class:`bytes` objects may be passed to calls. + + The obsolete *use_datetime* flag is similar to *use_builtin_types* but it + applies only to date/time values. Both the HTTP and HTTPS transports support the URL syntax extension for HTTP Basic Authentication: ``http://user:pass@host:port/path``. The ``user:pass`` @@ -78,12 +87,12 @@ between conformable Python objects and XML on the wire. | | only their *__dict__* attribute is | | | transmitted. | +---------------------------------+---------------------------------------------+ - | :const:`dates` | in seconds since the epoch (pass in an | - | | instance of the :class:`DateTime` class) or | + | :const:`dates` | In seconds since the epoch. Pass in an | + | | instance of the :class:`DateTime` class or | | | a :class:`datetime.datetime` instance. | +---------------------------------+---------------------------------------------+ - | :const:`binary data` | pass in an instance of the :class:`Binary` | - | | wrapper class | + | :const:`binary data` | Pass in an instance of the :class:`Binary` | + | | wrapper class or a :class:`bytes` instance. | +---------------------------------+---------------------------------------------+ This is the full set of data types supported by XML-RPC. Method calls may also @@ -98,8 +107,9 @@ between conformable Python objects and XML on the wire. ensure that the string is free of characters that aren't allowed in XML, such as the control characters with ASCII values between 0 and 31 (except, of course, tab, newline and carriage return); failing to do this will result in an XML-RPC - request that isn't well-formed XML. If you have to pass arbitrary strings via - XML-RPC, use the :class:`Binary` wrapper class described below. + request that isn't well-formed XML. If you have to pass arbitrary bytes + via XML-RPC, use the :class:`bytes` class or the class:`Binary` wrapper class + described below. :class:`Server` is retained as an alias for :class:`ServerProxy` for backwards compatibility. New code should use :class:`ServerProxy`. @@ -249,7 +259,7 @@ The client code for the preceding server:: Binary Objects -------------- -This class may be initialized from string data (which may include NULs). The +This class may be initialized from bytes data (which may include NULs). The primary access to the content of a :class:`Binary` object is provided by an attribute: @@ -257,15 +267,15 @@ attribute: .. attribute:: Binary.data The binary data encapsulated by the :class:`Binary` instance. The data is - provided as an 8-bit string. + provided as a :class:`bytes` object. :class:`Binary` objects have the following methods, supported mainly for internal use by the marshalling/unmarshalling code: -.. method:: Binary.decode(string) +.. method:: Binary.decode(bytes) - Accept a base64 string and decode it as the instance's new data. + Accept a base64 :class:`bytes` object and decode it as the instance's new data. .. method:: Binary.encode(out) @@ -471,14 +481,21 @@ Convenience Functions it via an extension, provide a true value for *allow_none*. -.. function:: loads(data, use_datetime=False) +.. function:: loads(data, use_datetime=False, use_builtin_types=False) Convert an XML-RPC request or response into Python objects, a ``(params, methodname)``. *params* is a tuple of argument; *methodname* is a string, or ``None`` if no method name is present in the packet. If the XML-RPC packet represents a fault condition, this function will raise a :exc:`Fault` exception. - The *use_datetime* flag can be used to cause date/time values to be presented as - :class:`datetime.datetime` objects; this is false by default. + The *use_builtin_types* flag can be used to cause date/time values to be + presented as :class:`datetime.datetime` objects and binary data to be + presented as :class:`bytes` objects; this flag is false by default. + + The obsolete *use_datetime* flag is similar to *use_builtin_types* but it + applies only to date/time values. + + .. versionchanged:: 3.3 + The *use_builtin_types* flag was added. .. _xmlrpc-client-example: diff --git a/Doc/library/xmlrpc.server.rst b/Doc/library/xmlrpc.server.rst index 67feba6..6493fd4 100644 --- a/Doc/library/xmlrpc.server.rst +++ b/Doc/library/xmlrpc.server.rst @@ -16,7 +16,9 @@ servers written in Python. Servers can either be free standing, using :class:`CGIXMLRPCRequestHandler`. -.. class:: SimpleXMLRPCServer(addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True) +.. class:: SimpleXMLRPCServer(addr, requestHandler=SimpleXMLRPCRequestHandler,\ + logRequests=True, allow_none=False, encoding=None,\ + bind_and_activate=True, use_builtin_types=False) Create a new server instance. This class provides methods for registration of functions that can be called by the XML-RPC protocol. The *requestHandler* @@ -25,18 +27,31 @@ servers written in Python. Servers can either be free standing, using are passed to the :class:`socketserver.TCPServer` constructor. If *logRequests* is true (the default), requests will be logged; setting this parameter to false will turn off logging. The *allow_none* and *encoding* parameters are passed - on to :mod:`xmlrpc.client` and control the XML-RPC responses that will be returned + on to :mod:`xmlrpc.client` and control the XML-RPC responses that will be returned from the server. The *bind_and_activate* parameter controls whether :meth:`server_bind` and :meth:`server_activate` are called immediately by the constructor; it defaults to true. Setting it to false allows code to manipulate the *allow_reuse_address* class variable before the address is bound. + The *use_builtin_types* parameter is passed to the + :func:`~xmlrpc.client.loads` function and controls which types are processed + when date/times values or binary data are received; it defaults to false. + .. versionchanged:: 3.3 + The *use_builtin_types* flag was added. -.. class:: CGIXMLRPCRequestHandler(allow_none=False, encoding=None) + +.. class:: CGIXMLRPCRequestHandler(allow_none=False, encoding=None,\ + use_builtin_types=False) Create a new instance to handle XML-RPC requests in a CGI environment. The *allow_none* and *encoding* parameters are passed on to :mod:`xmlrpc.client` and control the XML-RPC responses that will be returned from the server. + The *use_builtin_types* parameter is passed to the + :func:`~xmlrpc.client.loads` function and controls which types are processed + when date/times values or binary data are received; it defaults to false. + + .. versionchanged:: 3.3 + The *use_builtin_types* flag was added. .. class:: SimpleXMLRPCRequestHandler() @@ -233,12 +248,17 @@ to HTTP GET requests. Servers can either be free standing, using :class:`DocCGIXMLRPCRequestHandler`. -.. class:: DocXMLRPCServer(addr, requestHandler=DocXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True) +.. class:: DocXMLRPCServer(addr, requestHandler=DocXMLRPCRequestHandler,\ + logRequests=True, allow_none=False, encoding=None,\ + bind_and_activate=True, use_builtin_types=True) Create a new server instance. All parameters have the same meaning as for :class:`SimpleXMLRPCServer`; *requestHandler* defaults to :class:`DocXMLRPCRequestHandler`. + .. versionchanged:: 3.3 + The *use_builtin_types* flag was added. + .. class:: DocCGIXMLRPCRequestHandler() diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index bcec134..409de44 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -87,7 +87,30 @@ The module defines the following items: .. data:: ZIP_DEFLATED The numeric constant for the usual ZIP compression method. This requires the - zlib module. No other compression methods are currently supported. + zlib module. + + +.. data:: ZIP_BZIP2 + + The numeric constant for the BZIP2 compression method. This requires the + bz2 module. + + .. versionadded:: 3.3 + +.. data:: ZIP_LZMA + + The numeric constant for the LZMA compression method. This requires the + lzma module. + + .. versionadded:: 3.3 + + .. note:: + + The ZIP file format specification has included support for bzip2 compression + since 2001, and for LZMA compression since 2006. However, some tools + (including older Python releases) do not support these compression + methods, and may either refuse to process the ZIP file altogether, + or fail to extract individual files. .. seealso:: @@ -118,9 +141,11 @@ ZipFile Objects adding a ZIP archive to another file (such as :file:`python.exe`). If *mode* is ``a`` and the file does not exist at all, it is created. *compression* is the ZIP compression method to use when writing the archive, - and should be :const:`ZIP_STORED` or :const:`ZIP_DEFLATED`; unrecognized - values will cause :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED` - is specified but the :mod:`zlib` module is not available, :exc:`RuntimeError` + and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, + :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized + values will cause :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED`, + :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` is specified but the corresponded module + (:mod:`zlib`, :mod:`bz2` or :mod:`lzma`) is not available, :exc:`RuntimeError` is also raised. The default is :const:`ZIP_STORED`. If *allowZip64* is ``True`` zipfile will create ZIP files that use the ZIP64 extensions when the zipfile is larger than 2 GB. If it is false (the default) :mod:`zipfile` @@ -143,6 +168,9 @@ ZipFile Objects .. versionadded:: 3.2 Added the ability to use :class:`ZipFile` as a context manager. + .. versionchanged:: 3.3 + Added support for :mod:`bzip2` and :mod:`lzma` compression. + .. method:: ZipFile.close() diff --git a/Doc/library/zipimport.rst b/Doc/library/zipimport.rst index 4f17092..b47c35b 100644 --- a/Doc/library/zipimport.rst +++ b/Doc/library/zipimport.rst @@ -85,9 +85,12 @@ zipimporter Objects .. method:: get_data(pathname) - Return the data associated with *pathname*. Raise :exc:`IOError` if the + Return the data associated with *pathname*. Raise :exc:`OSError` if the file wasn't found. + .. versionchanged:: 3.3 + :exc:`IOError` used to be raised instead of :exc:`OSError`. + .. method:: get_filename(fullname) diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst index 897d919..1e9a2bc 100644 --- a/Doc/library/zlib.rst +++ b/Doc/library/zlib.rst @@ -120,6 +120,7 @@ The available exception and functions in this module are: won't fit into memory at once. The *wbits* parameter controls the size of the window buffer. + Compression objects support the following methods: @@ -150,7 +151,7 @@ Compression objects support the following methods: compress a set of data that share a common initial prefix. -Decompression objects support the following methods, and two attributes: +Decompression objects support the following methods and attributes: .. attribute:: Decompress.unused_data @@ -160,13 +161,6 @@ Decompression objects support the following methods, and two attributes: available. If the whole bytestring turned out to contain compressed data, this is ``b""``, an empty bytes object. - The only way to determine where a bytestring of compressed data ends is by actually - decompressing it. This means that when compressed data is contained part of a - larger file, you can only find the end of it by reading data and feeding it - followed by some non-empty bytestring into a decompression object's - :meth:`decompress` method until the :attr:`unused_data` attribute is no longer - empty. - .. attribute:: Decompress.unconsumed_tail @@ -177,6 +171,17 @@ Decompression objects support the following methods, and two attributes: :meth:`decompress` method call in order to get correct output. +.. attribute:: Decompress.eof + + A boolean indicating whether the end of the compressed data stream has been + reached. + + This makes it possible to distinguish between a properly-formed compressed + stream, and an incomplete or truncated one. + + .. versionadded:: 3.3 + + .. method:: Decompress.decompress(data[, max_length]) Decompress *data*, returning a bytes object containing the uncompressed data @@ -211,6 +216,24 @@ Decompression objects support the following methods, and two attributes: seeks into the stream at a future point. +Information about the version of the zlib library in use is available through +the following constants: + + +.. data:: ZLIB_VERSION + + The version string of the zlib library that was used for building the module. + This may be different from the zlib library actually used at runtime, which + is available as :const:`ZLIB_RUNTIME_VERSION`. + + +.. data:: ZLIB_RUNTIME_VERSION + + The version string of the zlib library actually loaded by the interpreter. + + .. versionadded:: 3.3 + + .. seealso:: Module :mod:`gzip` |
