diff options
Diffstat (limited to 'Doc')
186 files changed, 7333 insertions, 1830 deletions
diff --git a/Doc/Makefile b/Doc/Makefile index ea30231..a42e98b 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -15,11 +15,12 @@ ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees -D latex_paper_size=$(PAPER) \ .PHONY: help build html htmlhelp latex text changes linkcheck \ suspicious coverage doctest pydoc-topics htmlview clean dist check serve \ - autobuild-dev autobuild-stable + autobuild-dev autobuild-stable venv help: @echo "Please use \`make <target>' where <target> is one of" @echo " clean to remove build files" + @echo " venv to create a venv with necessary tools" @echo " html to make standalone HTML files" @echo " htmlview to open the index page built by the html target in your browser" @echo " htmlhelp to make HTML files and a HTML help project" @@ -95,14 +96,18 @@ doctest: pydoc-topics: BUILDER = pydoc-topics pydoc-topics: build - @echo "Building finished; now copy build/pydoc-topics/topics.py" \ - "to ../Lib/pydoc_data/topics.py" + @echo "Building finished; now run this:" \ + "cp build/pydoc-topics/topics.py ../Lib/pydoc_data/topics.py" htmlview: html $(PYTHON) -c "import webbrowser; webbrowser.open('build/html/index.html')" clean: - -rm -rf build/* + -rm -rf build/* venv/* + +venv: + $(PYTHON) -m venv venv + ./venv/bin/python3 -m pip install -U Sphinx dist: rm -rf dist @@ -172,4 +177,3 @@ autobuild-stable: exit 1;; \ esac @make autobuild-dev - diff --git a/Doc/README.txt b/Doc/README.txt index 58bda62..580d102 100644 --- a/Doc/README.txt +++ b/Doc/README.txt @@ -3,7 +3,7 @@ Python Documentation README This directory contains the reStructuredText (reST) sources to the Python documentation. You don't need to build them yourself, prebuilt versions are -available at <https://docs.python.org/3.4/download.html>. +available at <https://docs.python.org/dev/download.html>. Documentation on authoring Python documentation, including information about both style and markup, is available in the "Documenting Python" chapter of the diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index 3c0f4b9..ed62dea 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -152,7 +152,7 @@ Unless otherwise stated, buffers are not NUL-terminated. any conversion. Raises :exc:`TypeError` if the object is not a Unicode object. The C variable may also be declared as :c:type:`PyObject\*`. -``w*`` (:class:`bytearray` or read-write byte-oriented buffer) [Py_buffer] +``w*`` (read-write :term:`bytes-like object`) [Py_buffer] This format accepts any object which implements the read-write buffer interface. It fills a :c:type:`Py_buffer` structure provided by the caller. The buffer may contain embedded null bytes. The caller have to call diff --git a/Doc/c-api/codec.rst b/Doc/c-api/codec.rst index 83252af..dfe3d43 100644 --- a/Doc/c-api/codec.rst +++ b/Doc/c-api/codec.rst @@ -116,3 +116,8 @@ Registry API for Unicode encoding error handlers Replace the unicode encode error with backslash escapes (``\x``, ``\u`` and ``\U``). +.. c:function:: PyObject* PyCodec_NameReplaceErrors(PyObject *exc) + + Replace the unicode encode error with ``\N{...}`` escapes. + + .. versionadded:: 3.5 diff --git a/Doc/c-api/concrete.rst b/Doc/c-api/concrete.rst index 2d56386..47dab81 100644 --- a/Doc/c-api/concrete.rst +++ b/Doc/c-api/concrete.rst @@ -112,5 +112,6 @@ Other Objects weakref.rst capsule.rst gen.rst + coro.rst datetime.rst diff --git a/Doc/c-api/coro.rst b/Doc/c-api/coro.rst new file mode 100644 index 0000000..2fe50b5 --- /dev/null +++ b/Doc/c-api/coro.rst @@ -0,0 +1,34 @@ +.. highlightlang:: c + +.. _coro-objects: + +Coroutine Objects +----------------- + +.. versionadded:: 3.5 + +Coroutine objects are what functions declared with an ``async`` keyword +return. + + +.. c:type:: PyCoroObject + + The C structure used for coroutine objects. + + +.. c:var:: PyTypeObject PyCoro_Type + + The type object corresponding to coroutine objects. + + +.. c:function:: int PyCoro_CheckExact(PyObject *ob) + + Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be *NULL*. + + +.. c:function:: PyObject* PyCoro_New(PyFrameObject *frame, PyObject *name, PyObject *qualname) + + Create and return a new coroutine object based on the *frame* object, + with ``__name__`` and ``__qualname__`` set to *name* and *qualname*. + A reference to *frame* is stolen by this function. The *frame* argument + must not be *NULL*. diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index c2df767..6ae723b 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -9,13 +9,19 @@ Exception Handling The functions described in this chapter will let you handle and raise Python exceptions. It is important to understand some of the basics of Python -exception handling. It works somewhat like the Unix :c:data:`errno` variable: +exception handling. It works somewhat like the POSIX :c:data:`errno` variable: there is a global indicator (per thread) of the last error that occurred. Most -functions don't clear this on success, but will set it to indicate the cause of -the error on failure. Most functions also return an error indicator, usually -*NULL* if they are supposed to return a pointer, or ``-1`` if they return an -integer (exception: the :c:func:`PyArg_\*` functions return ``1`` for success and -``0`` for failure). +C API functions don't clear this on success, but will set it to indicate the +cause of the error on failure. Most C API functions also return an error +indicator, usually *NULL* if they are supposed to return a pointer, or ``-1`` +if they return an integer (exception: the :c:func:`PyArg_\*` functions +return ``1`` for success and ``0`` for failure). + +Concretely, the error indicator consists of three object pointers: the +exception's type, the exception's value, and the traceback object. Any +of those pointers can be NULL if non-set (although some combinations are +forbidden, for example you can't have a non-NULL traceback if the exception +type is NULL). When a function must fail because some function it called failed, it generally doesn't set the error indicator; the function it called already set it. It is @@ -27,12 +33,21 @@ the caller that an error has been set. If the error is not handled or carefully propagated, additional calls into the Python/C API may not behave as intended and may fail in mysterious ways. -The error indicator consists of three Python objects corresponding to the result -of ``sys.exc_info()``. API functions exist to interact with the error indicator -in various ways. There is a separate error indicator for each thread. +.. note:: + The error indicator is **not** the result of :func:`sys.exc_info()`. + The former corresponds to an exception that is not yet caught (and is + therefore still propagating), while the latter returns an exception after + it is caught (and has therefore stopped propagating). -.. XXX Order of these should be more thoughtful. - Either alphabetical or some kind of structure. + +Printing and clearing +===================== + + +.. c:function:: void PyErr_Clear() + + Clear the error indicator. If the error indicator is not set, there is no + effect. .. c:function:: void PyErr_PrintEx(int set_sys_last_vars) @@ -51,127 +66,24 @@ in various ways. There is a separate error indicator for each thread. Alias for ``PyErr_PrintEx(1)``. -.. c:function:: PyObject* PyErr_Occurred() - - Test whether the error indicator is set. If set, return the exception *type* - (the first argument to the last call to one of the :c:func:`PyErr_Set\*` - functions or to :c:func:`PyErr_Restore`). If not set, return *NULL*. You do not - own a reference to the return value, so you do not need to :c:func:`Py_DECREF` - it. - - .. note:: - - Do not compare the return value to a specific exception; use - :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could - easily fail since the exception may be an instance instead of a class, in the - case of a class exception, or it may be a subclass of the expected exception.) - - -.. c:function:: int PyErr_ExceptionMatches(PyObject *exc) - - Equivalent to ``PyErr_GivenExceptionMatches(PyErr_Occurred(), exc)``. This - should only be called when an exception is actually set; a memory access - violation will occur if no exception has been raised. - - -.. c:function:: int PyErr_GivenExceptionMatches(PyObject *given, PyObject *exc) - - Return true if the *given* exception matches the exception in *exc*. If - *exc* is a class object, this also returns true when *given* is an instance - of a subclass. If *exc* is a tuple, all exceptions in the tuple (and - recursively in subtuples) are searched for a match. - - -.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) - - Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below - can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is - not an instance of the same class. This function can be used to instantiate - the class in that case. If the values are already normalized, nothing happens. - The delayed normalization is implemented to improve performance. - - .. note:: - - This function *does not* implicitly set the ``__traceback__`` - attribute on the exception value. If setting the traceback - appropriately is desired, the following additional snippet is needed:: - - if (tb != NULL) { - PyException_SetTraceback(val, tb); - } - - -.. c:function:: void PyErr_Clear() - - Clear the error indicator. If the error indicator is not set, there is no - effect. - - -.. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) - - Retrieve the error indicator into three variables whose addresses are passed. - If the error indicator is not set, set all three variables to *NULL*. If it is - set, it will be cleared and you own a reference to each object retrieved. The - value and traceback object may be *NULL* even when the type object is not. - - .. note:: - - This function is normally only used by code that needs to handle exceptions or - by code that needs to save and restore the error indicator temporarily. - - -.. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback) - - Set the error indicator from the three objects. If the error indicator is - already set, it is cleared first. If the objects are *NULL*, the error - indicator is cleared. Do not pass a *NULL* type and non-*NULL* value or - traceback. The exception type should be a class. Do not pass an invalid - exception type or value. (Violating these rules will cause subtle problems - later.) This call takes away a reference to each object: you must own a - reference to each object before the call and after the call you no longer own - these references. (If you don't understand this, don't use this function. I - warned you.) - - .. note:: - - This function is normally only used by code that needs to save and restore the - error indicator temporarily; use :c:func:`PyErr_Fetch` to save the current - exception state. - - -.. c:function:: void PyErr_GetExcInfo(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) - - Retrieve the exception info, as known from ``sys.exc_info()``. This refers - to an exception that was already caught, not to an exception that was - freshly raised. Returns new references for the three objects, any of which - may be *NULL*. Does not modify the exception info state. - - .. note:: - - This function is not normally used by code that wants to handle exceptions. - Rather, it can be used when code needs to save and restore the exception - state temporarily. Use :c:func:`PyErr_SetExcInfo` to restore or clear the - exception state. - - .. versionadded:: 3.3 - +.. c:function:: void PyErr_WriteUnraisable(PyObject *obj) -.. c:function:: void PyErr_SetExcInfo(PyObject *type, PyObject *value, PyObject *traceback) + This utility function prints a warning message to ``sys.stderr`` when an + exception has been set but it is impossible for the interpreter to actually + raise the exception. It is used, for example, when an exception occurs in an + :meth:`__del__` method. - Set the exception info, as known from ``sys.exc_info()``. This refers - to an exception that was already caught, not to an exception that was - freshly raised. This function steals the references of the arguments. - To clear the exception state, pass *NULL* for all three arguments. - For general rules about the three arguments, see :c:func:`PyErr_Restore`. + The function is called with a single argument *obj* that identifies the context + in which the unraisable exception occurred. The repr of *obj* will be printed in + the warning message. - .. note:: - This function is not normally used by code that wants to handle exceptions. - Rather, it can be used when code needs to save and restore the exception - state temporarily. Use :c:func:`PyErr_GetExcInfo` to read the exception - state. +Raising exceptions +================== - .. versionadded:: 3.3 +These functions help you set the current thread's error indicator. +For convenience, some of these functions will always return a +NULL pointer for use in a ``return`` statement. .. c:function:: void PyErr_SetString(PyObject *type, const char *message) @@ -197,6 +109,14 @@ in various ways. There is a separate error indicator for each thread. string. +.. c:function:: PyObject* PyErr_FormatV(PyObject *exception, const char *format, va_list vargs) + + Same as :c:func:`PyErr_Format`, but taking a :c:type:`va_list` argument rather + than a variable number of arguments. + + .. versionadded:: 3.5 + + .. c:function:: void PyErr_SetNone(PyObject *type) This is a shorthand for ``PyErr_SetObject(type, Py_None)``. @@ -346,6 +266,22 @@ in various ways. There is a separate error indicator for each thread. use. +Issuing warnings +================ + +Use these functions to issue warnings from C code. They mirror similar +functions exported by the Python :mod:`warnings` module. They normally +print a warning message to *sys.stderr*; however, it is +also possible that the user has specified that warnings are to be turned into +errors, and in that case they will raise an exception. It is also possible that +the functions raise an exception because of a problem with the warning machinery. +The return value is ``0`` if no exception is raised, or ``-1`` if an exception +is raised. (It is not possible to determine whether a warning message is +actually printed, nor what the reason is for the exception; this is +intentional.) If an exception is raised, the caller should do its normal +exception handling (for example, :c:func:`Py_DECREF` owned references and return +an error value). + .. c:function:: int PyErr_WarnEx(PyObject *category, const char *message, Py_ssize_t stack_level) Issue a warning message. The *category* argument is a warning category (see @@ -355,18 +291,6 @@ in various ways. There is a separate error indicator for each thread. is the function calling :c:func:`PyErr_WarnEx`, 2 is the function above that, and so forth. - This function normally prints a warning message to *sys.stderr*; however, it is - also possible that the user has specified that warnings are to be turned into - errors, and in that case this will raise an exception. It is also possible that - the function raises an exception because of a problem with the warning machinery - (the implementation imports the :mod:`warnings` module to do the heavy lifting). - The return value is ``0`` if no exception is raised, or ``-1`` if an exception - is raised. (It is not possible to determine whether a warning message is - actually printed, nor what the reason is for the exception; this is - intentional.) If an exception is raised, the caller should do its normal - exception handling (for example, :c:func:`Py_DECREF` owned references and return - an error value). - Warning categories must be subclasses of :c:data:`Warning`; the default warning category is :c:data:`RuntimeWarning`. The standard Python warning categories are available as global variables whose names are ``PyExc_`` followed by the Python @@ -410,6 +334,139 @@ in various ways. There is a separate error indicator for each thread. .. versionadded:: 3.2 +Querying the error indicator +============================ + +.. c:function:: PyObject* PyErr_Occurred() + + Test whether the error indicator is set. If set, return the exception *type* + (the first argument to the last call to one of the :c:func:`PyErr_Set\*` + functions or to :c:func:`PyErr_Restore`). If not set, return *NULL*. You do not + own a reference to the return value, so you do not need to :c:func:`Py_DECREF` + it. + + .. note:: + + Do not compare the return value to a specific exception; use + :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could + easily fail since the exception may be an instance instead of a class, in the + case of a class exception, or it may be a subclass of the expected exception.) + + +.. c:function:: int PyErr_ExceptionMatches(PyObject *exc) + + Equivalent to ``PyErr_GivenExceptionMatches(PyErr_Occurred(), exc)``. This + should only be called when an exception is actually set; a memory access + violation will occur if no exception has been raised. + + +.. c:function:: int PyErr_GivenExceptionMatches(PyObject *given, PyObject *exc) + + Return true if the *given* exception matches the exception type in *exc*. If + *exc* is a class object, this also returns true when *given* is an instance + of a subclass. If *exc* is a tuple, all exception types in the tuple (and + recursively in subtuples) are searched for a match. + + +.. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) + + Retrieve the error indicator into three variables whose addresses are passed. + If the error indicator is not set, set all three variables to *NULL*. If it is + set, it will be cleared and you own a reference to each object retrieved. The + value and traceback object may be *NULL* even when the type object is not. + + .. note:: + + This function is normally only used by code that needs to catch exceptions or + by code that needs to save and restore the error indicator temporarily, e.g.:: + + { + PyObject **type, **value, **traceback; + PyErr_Fetch(&type, &value, &traceback); + + /* ... code that might produce other errors ... */ + + PyErr_Restore(type, value, traceback); + } + + +.. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback) + + Set the error indicator from the three objects. If the error indicator is + already set, it is cleared first. If the objects are *NULL*, the error + indicator is cleared. Do not pass a *NULL* type and non-*NULL* value or + traceback. The exception type should be a class. Do not pass an invalid + exception type or value. (Violating these rules will cause subtle problems + later.) This call takes away a reference to each object: you must own a + reference to each object before the call and after the call you no longer own + these references. (If you don't understand this, don't use this function. I + warned you.) + + .. note:: + + This function is normally only used by code that needs to save and restore the + error indicator temporarily. Use :c:func:`PyErr_Fetch` to save the current + error indicator. + + +.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) + + Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below + can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is + not an instance of the same class. This function can be used to instantiate + the class in that case. If the values are already normalized, nothing happens. + The delayed normalization is implemented to improve performance. + + .. note:: + + This function *does not* implicitly set the ``__traceback__`` + attribute on the exception value. If setting the traceback + appropriately is desired, the following additional snippet is needed:: + + if (tb != NULL) { + PyException_SetTraceback(val, tb); + } + + +.. c:function:: void PyErr_GetExcInfo(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) + + Retrieve the exception info, as known from ``sys.exc_info()``. This refers + to an exception that was *already caught*, not to an exception that was + freshly raised. Returns new references for the three objects, any of which + may be *NULL*. Does not modify the exception info state. + + .. note:: + + This function is not normally used by code that wants to handle exceptions. + Rather, it can be used when code needs to save and restore the exception + state temporarily. Use :c:func:`PyErr_SetExcInfo` to restore or clear the + exception state. + + .. versionadded:: 3.3 + + +.. c:function:: void PyErr_SetExcInfo(PyObject *type, PyObject *value, PyObject *traceback) + + Set the exception info, as known from ``sys.exc_info()``. This refers + to an exception that was *already caught*, not to an exception that was + freshly raised. This function steals the references of the arguments. + To clear the exception state, pass *NULL* for all three arguments. + For general rules about the three arguments, see :c:func:`PyErr_Restore`. + + .. note:: + + This function is not normally used by code that wants to handle exceptions. + Rather, it can be used when code needs to save and restore the exception + state temporarily. Use :c:func:`PyErr_GetExcInfo` to read the exception + state. + + .. versionadded:: 3.3 + + +Signal Handling +=============== + + .. c:function:: int PyErr_CheckSignals() .. index:: @@ -443,13 +500,21 @@ in various ways. There is a separate error indicator for each thread. .. c:function:: int PySignal_SetWakeupFd(int fd) - This utility function specifies a file descriptor to which a ``'\0'`` byte will - be written whenever a signal is received. It returns the previous such file - descriptor. The value ``-1`` disables the feature; this is the initial state. + This utility function specifies a file descriptor to which the signal number + is written as a single byte whenever a signal is received. *fd* must be + non-blocking. It returns the previous such file descriptor. + + The value ``-1`` disables the feature; this is the initial state. This is equivalent to :func:`signal.set_wakeup_fd` in Python, but without any error checking. *fd* should be a valid file descriptor. The function should only be called from the main thread. + .. versionchanged:: 3.5 + On Windows, the function now also supports socket handles. + + +Exception Classes +================= .. c:function:: PyObject* PyErr_NewException(const char *name, PyObject *base, PyObject *dict) @@ -475,18 +540,6 @@ in various ways. There is a separate error indicator for each thread. .. versionadded:: 3.2 -.. c:function:: void PyErr_WriteUnraisable(PyObject *obj) - - This utility function prints a warning message to ``sys.stderr`` when an - exception has been set but it is impossible for the interpreter to actually - raise the exception. It is used, for example, when an exception occurs in an - :meth:`__del__` method. - - The function is called with a single argument *obj* that identifies the context - in which the unraisable exception occurred. The repr of *obj* will be printed in - the warning message. - - Exception Objects ================= diff --git a/Doc/c-api/gen.rst b/Doc/c-api/gen.rst index 33cd27a..3ab073b 100644 --- a/Doc/c-api/gen.rst +++ b/Doc/c-api/gen.rst @@ -7,7 +7,7 @@ Generator Objects Generator objects are what Python uses to implement generator iterators. They are normally created by iterating over a function that yields values, rather -than explicitly calling :c:func:`PyGen_New`. +than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`. .. c:type:: PyGenObject @@ -20,19 +20,25 @@ than explicitly calling :c:func:`PyGen_New`. The type object corresponding to generator objects -.. c:function:: int PyGen_Check(ob) +.. c:function:: int PyGen_Check(PyObject *ob) Return true if *ob* is a generator object; *ob* must not be *NULL*. -.. c:function:: int PyGen_CheckExact(ob) +.. c:function:: int PyGen_CheckExact(PyObject *ob) - Return true if *ob*'s type is *PyGen_Type* is a generator object; *ob* must not - be *NULL*. + Return true if *ob*'s type is *PyGen_Type*; *ob* must not be *NULL*. .. c:function:: PyObject* PyGen_New(PyFrameObject *frame) - Create and return a new generator object based on the *frame* object. A - reference to *frame* is stolen by this function. The parameter must not be + Create and return a new generator object based on the *frame* object. + A reference to *frame* is stolen by this function. The argument must not be *NULL*. + +.. c:function:: PyObject* PyGen_NewWithQualName(PyFrameObject *frame, PyObject *name, PyObject *qualname) + + Create and return a new generator object based on the *frame* object, + with ``__name__`` and ``__qualname__`` set to *name* and *qualname*. + A reference to *frame* is stolen by this function. The *frame* argument + must not be *NULL*. diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst index 60865f4..bf1b495 100644 --- a/Doc/c-api/import.rst +++ b/Doc/c-api/import.rst @@ -183,9 +183,9 @@ Importing Modules .. c:function:: long PyImport_GetMagicNumber() - Return the magic number for Python bytecode files (a.k.a. :file:`.pyc` and - :file:`.pyo` files). The magic number should be present in the first four bytes - of the bytecode file, in little-endian byte order. Returns -1 on error. + Return the magic number for Python bytecode files (a.k.a. :file:`.pyc` file). + The magic number should be present in the first four bytes of the bytecode + file, in little-endian byte order. Returns -1 on error. .. versionchanged:: 3.3 Return value of -1 upon failure. diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 4bb5064..e6d6d9f 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -134,6 +134,9 @@ Process-wide parameters change for the duration of the program's execution. No code in the Python interpreter will change the contents of this storage. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. c:function:: wchar* Py_GetProgramName() @@ -245,6 +248,9 @@ Process-wide parameters :data:`sys.exec_prefix` to be empty. It is up to the caller to modify these if required after calling :c:func:`Py_Initialize`. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + The path argument is copied internally, so the caller may free it after the call completes. @@ -344,6 +350,9 @@ Process-wide parameters :data:`sys.path`, which is the same as prepending the current working directory (``"."``). + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. note:: It is recommended that applications embedding the Python interpreter for purposes other than executing a single script pass 0 as *updatepath*, @@ -368,6 +377,9 @@ Process-wide parameters to 1 unless the :program:`python` interpreter was started with the :option:`-I`. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. versionchanged:: 3.4 The *updatepath* value depends on :option:`-I`. @@ -382,6 +394,9 @@ Process-wide parameters execution. No code in the Python interpreter will change the contents of this storage. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. c:function:: w_char* Py_GetPythonHome() diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 7908622..7339006 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -92,8 +92,8 @@ functions are thread-safe, the :term:`GIL <global interpreter lock>` does not need to be held. The default raw memory block allocator uses the following functions: -:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when -requesting zero bytes. +:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call +``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes. .. versionadded:: 3.4 @@ -106,6 +106,17 @@ requesting zero bytes. been initialized in any way. +.. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize) + + Allocates *nelem* elements each whose size in bytes is *elsize* and returns + a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the + request fails. The memory is initialized to zeros. Requesting zero elements + or elements of size zero bytes returns a distinct non-*NULL* pointer if + possible, as if ``PyMem_RawCalloc(1, 1)`` had been called instead. + + .. versionadded:: 3.5 + + .. c:function:: void* PyMem_RawRealloc(void *p, size_t n) Resizes the memory block pointed to by *p* to *n* bytes. The contents will @@ -136,8 +147,8 @@ behavior when requesting zero bytes, are available for allocating and releasing memory from the Python heap. The default memory block allocator uses the following functions: -:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when -requesting zero bytes. +:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call +``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes. .. warning:: @@ -152,6 +163,17 @@ requesting zero bytes. been called instead. The memory will not have been initialized in any way. +.. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize) + + Allocates *nelem* elements each whose size in bytes is *elsize* and returns + a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the + request fails. The memory is initialized to zeros. Requesting zero elements + or elements of size zero bytes returns a distinct non-*NULL* pointer if + possible, as if ``PyMem_Calloc(1, 1)`` had been called instead. + + .. versionadded:: 3.5 + + .. c:function:: void* PyMem_Realloc(void *p, size_t n) Resizes the memory block pointed to by *p* to *n* bytes. The contents will be @@ -210,7 +232,7 @@ Customize Memory Allocators .. versionadded:: 3.4 -.. c:type:: PyMemAllocator +.. c:type:: PyMemAllocatorEx Structure used to describe a memory block allocator. The structure has four fields: @@ -222,11 +244,19 @@ Customize Memory Allocators +----------------------------------------------------------+---------------------------------------+ | ``void* malloc(void *ctx, size_t size)`` | allocate a memory block | +----------------------------------------------------------+---------------------------------------+ + | ``void* calloc(void *ctx, size_t nelem, size_t elsize)`` | allocate a memory block initialized | + | | with zeros | + +----------------------------------------------------------+---------------------------------------+ | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block | +----------------------------------------------------------+---------------------------------------+ | ``void free(void *ctx, void *ptr)`` | free a memory block | +----------------------------------------------------------+---------------------------------------+ + .. versionchanged:: 3.5 + The :c:type:`PyMemAllocator` structure was renamed to + :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. + + .. c:type:: PyMemAllocatorDomain Enum used to identify an allocator domain. Domains: @@ -239,12 +269,12 @@ Customize Memory Allocators :c:func:`PyObject_Realloc` and :c:func:`PyObject_Free` -.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) +.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) Get the memory block allocator of the specified domain. -.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) +.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) Set the memory block allocator of the specified domain. diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index 985a347..df9301f 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -7,8 +7,6 @@ Module Objects .. index:: object: module -There are only a few functions special to module objects. - .. c:var:: PyTypeObject PyModule_Type @@ -109,6 +107,14 @@ There are only a few functions special to module objects. unencodable filenames, use :c:func:`PyModule_GetFilenameObject` instead. +Per-interpreter module state +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Single-phase initialization creates singleton modules that can store additional +information as part of the interpreter, allow that state to be retrieved later +with only a reference to the module definition, rather than to the module +itself. + .. c:function:: void* PyModule_GetState(PyObject *module) Return the "state" of the module, that is, a pointer to the block of memory @@ -146,27 +152,6 @@ There are only a few functions special to module objects. Initializing C modules ^^^^^^^^^^^^^^^^^^^^^^ -These functions are usually used in the module initialization function. - -.. c:function:: PyObject* PyModule_Create(PyModuleDef *module) - - Create a new module object, given the definition in *module*. This behaves - like :c:func:`PyModule_Create2` with *module_api_version* set to - :const:`PYTHON_API_VERSION`. - - -.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version) - - Create a new module object, given the definition in *module*, assuming the - API version *module_api_version*. If that version does not match the version - of the running interpreter, a :exc:`RuntimeWarning` is emitted. - - .. note:: - - Most uses of this function should be using :c:func:`PyModule_Create` - instead; only use this if you are sure you need it. - - .. c:type:: PyModuleDef This struct holds all information that is needed to create a module object. @@ -210,9 +195,10 @@ These functions are usually used in the module initialization function. A pointer to a table of module-level functions, described by :c:type:`PyMethodDef` values. Can be *NULL* if no functions are present. - .. c:member:: inquiry m_reload + .. c:member:: PyModuleDef_Slot* m_slots - Currently unused, should be *NULL*. + An array of slot definitions for multi-phase initialization, terminated by + a *NULL* entry. .. c:member:: traverseproc m_traverse @@ -229,6 +215,61 @@ These functions are usually used in the module initialization function. A function to call during deallocation of the module object, or *NULL* if not needed. +The module initialization function may create and return the module object +directly. This is referred to as "single-phase initialization", and uses one +of the following two module creation functions: + +.. c:function:: PyObject* PyModule_Create(PyModuleDef *module) + + Create a new module object, given the definition in *module*. This behaves + like :c:func:`PyModule_Create2` with *module_api_version* set to + :const:`PYTHON_API_VERSION`. + + +.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version) + + Create a new module object, given the definition in *module*, assuming the + API version *module_api_version*. If that version does not match the version + of the running interpreter, a :exc:`RuntimeWarning` is emitted. + + .. note:: + + Most uses of this function should be using :c:func:`PyModule_Create` + instead; only use this if you are sure you need it. + + +Alternatively, the module initialization function may instead return a +:c:type:`PyModuleDef` instance with a non-empty ``m_slots`` array. This is +referred to as "multi-phase initialization", and ``PyModuleDef`` instance +should be initialized with the following function: + +.. c:function:: PyObject* PyModuleDef_Init(PyModuleDef *module) + + Ensures a module definition is a properly initialized Python object that + correctly reports its type and reference count. + +.. XXX (ncoghlan): It's not clear if it makes sense to document PyModule_ExecDef + PyModule_FromDefAndSpec or PyModule_FromDefAndSpec2 here, as end user code + generally shouldn't be calling those. + +The module initialization function (if using single phase initialization) or +a function called from a module execution slot (if using multiphase +initialization), can use the following functions to help initialize the module +state: + +.. c:function:: int PyModule_SetDocString(PyObject *module, const char *docstring) + + Set the docstring for *module* to *docstring*. Return ``-1`` on error, ``0`` + on success. + +.. c:function:: int PyModule_AddFunctions(PyObject *module, PyMethodDef *functions) + + Add the functions from the ``NULL`` terminated *functions* array to *module*. + Refer to the :c:type:`PyMethodDef` documentation for details on individual + entries (due to the lack of a shared module namespace, module level + "functions" implemented in C typically receive the module as their first + parameter, making them similar to instance methods on Python classes). + .. c:function:: int PyModule_AddObject(PyObject *module, const char *name, PyObject *value) @@ -236,7 +277,6 @@ These functions are usually used in the module initialization function. be used from the module's initialization function. This steals a reference to *value*. Return ``-1`` on error, ``0`` on success. - .. c:function:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value) Add an integer constant to *module* as *name*. This convenience function can be diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst index 21951c3..9bcb649 100644 --- a/Doc/c-api/number.rst +++ b/Doc/c-api/number.rst @@ -30,6 +30,14 @@ Number Protocol the equivalent of the Python expression ``o1 * o2``. +.. c:function:: PyObject* PyNumber_MatrixMultiply(PyObject *o1, PyObject *o2) + + Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on + failure. This is the equivalent of the Python expression ``o1 @ o2``. + + .. versionadded:: 3.5 + + .. c:function:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2) Return the floor of *o1* divided by *o2*, or *NULL* on failure. This is @@ -146,6 +154,15 @@ Number Protocol the Python statement ``o1 *= o2``. +.. c:function:: PyObject* PyNumber_InPlaceMatrixMultiply(PyObject *o1, PyObject *o2) + + Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on + failure. The operation is done *in-place* when *o1* supports it. This is + the equivalent of the Python statement ``o1 @= o2``. + + .. versionadded:: 3.5 + + .. c:function:: PyObject* PyNumber_InPlaceFloorDivide(PyObject *o1, PyObject *o2) Returns the mathematical floor of dividing *o1* by *o2*, or *NULL* on failure. diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index 7f1cc69..b925a61 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -44,7 +44,7 @@ the definition of all other Python objects. PyObject ob_base; - See documentation of :c:type::`PyObject` above. + See documentation of :c:type:`PyObject` above. .. c:macro:: PyObject_VAR_HEAD diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index 7cead07..3d83b27 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -47,6 +47,60 @@ Operating System Utilities not call those functions directly! :c:type:`PyOS_sighandler_t` is a typedef alias for :c:type:`void (\*)(int)`. +.. c:function:: wchar_t* Py_DecodeLocale(const char* arg, size_t *size) + + Decode a byte string from the locale encoding with the :ref:`surrogateescape + error handler <surrogateescape>`: undecodable bytes are decoded as + characters in range U+DC80..U+DCFF. If a byte sequence can be decoded as a + surrogate character, escape the bytes using the surrogateescape error + handler instead of decoding them. + + Return a pointer to a newly allocated wide character string, use + :c:func:`PyMem_RawFree` to free the memory. If size is not ``NULL``, write + the number of wide characters excluding the null character into ``*size`` + + Return ``NULL`` on decoding error or memory allocation error. If *size* is + not ``NULL``, ``*size`` is set to ``(size_t)-1`` on memory error or set to + ``(size_t)-2`` on decoding error. + + Decoding errors should never happen, unless there is a bug in the C + library. + + Use the :c:func:`Py_EncodeLocale` function to encode the character string + back to a byte string. + + .. seealso:: + + The :c:func:`PyUnicode_DecodeFSDefaultAndSize` and + :c:func:`PyUnicode_DecodeLocaleAndSize` functions. + + .. versionadded:: 3.5 + + +.. c:function:: char* Py_EncodeLocale(const wchar_t *text, size_t *error_pos) + + Encode a wide character string to the locale encoding with the + :ref:`surrogateescape error handler <surrogateescape>`: surrogate characters + in the range U+DC80..U+DCFF are converted to bytes 0x80..0xFF. + + Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free` + to free the memory. Return ``NULL`` on encoding error or memory allocation + error + + If error_pos is not ``NULL``, ``*error_pos`` is set to the index of the + invalid character on encoding error, or set to ``(size_t)-1`` otherwise. + + Use the :c:func:`Py_DecodeLocale` function to decode the bytes string back + to a wide character string. + + .. seealso:: + + The :c:func:`PyUnicode_EncodeFSDefault` and + :c:func:`PyUnicode_EncodeLocale` functions. + + .. versionadded:: 3.5 + + .. _systemfunctions: System Functions diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index b43622a..ed83e82 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -220,9 +220,14 @@ type objects) *must* have the :attr:`ob_size` field. the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both *NULL*. -.. c:member:: void* PyTypeObject.tp_reserved +.. c:member:: PyAsyncMethods* tp_as_async - Reserved slot, formerly known as tp_compare. + Pointer to an additional structure that contains fields relevant only to + objects which implement :term:`awaitable` and :term:`asynchronous iterator` + protocols at the C-level. See :ref:`async-structs` for details. + + .. versionadded:: 3.5 + Formerly known as ``tp_compare`` and ``tp_reserved``. .. c:member:: reprfunc PyTypeObject.tp_repr @@ -1118,6 +1123,9 @@ Number Object Structures binaryfunc nb_inplace_true_divide; unaryfunc nb_index; + + binaryfunc nb_matrix_multiply; + binaryfunc nb_inplace_matrix_multiply; } PyNumberMethods; .. note:: @@ -1329,3 +1337,58 @@ Buffer Object Structures :c:func:`PyBuffer_Release` is the interface for the consumer that wraps this function. + + +.. _async-structs: + + +Async Object Structures +======================= + +.. sectionauthor:: Yury Selivanov <yselivanov@sprymix.com> + +.. versionadded:: 3.5 + +.. c:type:: PyAsyncMethods + + This structure holds pointers to the functions required to implement + :term:`awaitable` and :term:`asynchronous iterator` objects. + + Here is the structure definition:: + + typedef struct { + unaryfunc am_await; + unaryfunc am_aiter; + unaryfunc am_anext; + } PyAsyncMethods; + +.. c:member:: unaryfunc PyAsyncMethods.am_await + + The signature of this function is:: + + PyObject *am_await(PyObject *self) + + The returned object must be an iterator, i.e. :c:func:`PyIter_Check` must + return ``1`` for it. + + This slot may be set to *NULL* if an object is not an :term:`awaitable`. + +.. c:member:: unaryfunc PyAsyncMethods.am_aiter + + The signature of this function is:: + + PyObject *am_aiter(PyObject *self) + + Must return an :term:`awaitable` object. See :meth:`__anext__` for details. + + This slot may be set to *NULL* if an object does not implement + asynchronous iteration protocol. + +.. c:member:: unaryfunc PyAsyncMethods.am_anext + + The signature of this function is:: + + PyObject *am_anext(PyObject *self) + + Must return an :term:`awaitable` object. See :meth:`__anext__` for details. + This slot may be set to *NULL*. diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index f7e99d6..2eeadb5 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -765,11 +765,13 @@ system. *errors* is ``NULL``. *str* must end with a null character but cannot contain embedded null characters. + Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from + :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at + Python startup). + .. seealso:: - Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from - :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at - Python startup). + The :c:func:`Py_DecodeLocale` function. .. versionadded:: 3.3 @@ -790,11 +792,13 @@ system. *errors* is ``NULL``. Return a :class:`bytes` object. *str* cannot contain embedded null characters. + Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to + :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at + Python startup). + .. seealso:: - Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to - :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at - Python startup). + The :c:func:`Py_EncodeLocale` function. .. versionadded:: 3.3 @@ -839,12 +843,14 @@ used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. + :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the + locale encoding and cannot be modified later. If you need to decode a string + from the current locale encoding, use + :c:func:`PyUnicode_DecodeLocaleAndSize`. + .. seealso:: - :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the - locale encoding and cannot be modified later. If you need to decode a - string from the current locale encoding, use - :c:func:`PyUnicode_DecodeLocaleAndSize`. + The :c:func:`Py_DecodeLocale` function. .. versionchanged:: 3.2 Use ``"strict"`` error handler on Windows. @@ -874,12 +880,13 @@ used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. + :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the + locale encoding and cannot be modified later. If you need to encode a string + to the current locale encoding, use :c:func:`PyUnicode_EncodeLocale`. + .. seealso:: - :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the - locale encoding and cannot be modified later. If you need to encode a - string to the current locale encoding, use - :c:func:`PyUnicode_EncodeLocale`. + The :c:func:`Py_EncodeLocale` function. .. versionadded:: 3.2 diff --git a/Doc/conf.py b/Doc/conf.py index f803de2..28dd80f 100644 --- a/Doc/conf.py +++ b/Doc/conf.py @@ -36,6 +36,9 @@ highlight_language = 'python3' # Require Sphinx 1.2 for build. needs_sphinx = '1.2' +# Ignore any .rst files in the venv/ directory. +exclude_patterns = ['venv/*'] + # Options for HTML output # ----------------------- diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index 6025617..e388195 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -349,6 +349,11 @@ PyErr_Format:PyObject*:exception:+1: PyErr_Format:const char*:format:: PyErr_Format::...:: +PyErr_FormatV:PyObject*::null: +PyErr_FormatV:PyObject*:exception:+1: +PyErr_FormatV:const char*:format:: +PyErr_FormatV:va_list:vargs:: + PyErr_WarnEx:int::: PyErr_WarnEx:PyObject*:category:0: PyErr_WarnEx:const char*:message:: @@ -486,6 +491,12 @@ PyFunction_SetDefaults:PyObject*:defaults:+1: PyGen_New:PyObject*::+1: PyGen_New:PyFrameObject*:frame:0: +PyGen_NewWithQualName:PyObject*::+1: +PyGen_NewWithQualName:PyFrameObject*:frame:0: + +PyCoro_New:PyObject*::+1: +PyCoro_New:PyFrameObject*:frame:0: + Py_InitModule:PyObject*::0: Py_InitModule:const char*:name:: Py_InitModule:PyMethodDef[]:methods:: diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst index 53d7527..d0f8947 100644 --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -868,23 +868,31 @@ tarballs or zipfiles. Create an archive file (eg. ``zip`` or ``tar``). *base_name* is the name of the file to create, minus any format-specific extension; *format* is the - archive format: one of ``zip``, ``tar``, ``ztar``, or ``gztar``. *root_dir* is - a directory that will be the root directory of the archive; ie. we typically - ``chdir`` into *root_dir* before creating the archive. *base_dir* is the - directory where we start archiving from; ie. *base_dir* will be the common - prefix of all files and directories in the archive. *root_dir* and *base_dir* - both default to the current directory. Returns the name of the archive file. + archive format: one of ``zip``, ``tar``, ``gztar``, ``bztar``, ``xztar``, or + ``ztar``. *root_dir* is a directory that will be the root directory of the + archive; ie. we typically ``chdir`` into *root_dir* before creating the + archive. *base_dir* is the directory where we start archiving from; ie. + *base_dir* will be the common prefix of all files and directories in the + archive. *root_dir* and *base_dir* both default to the current directory. + Returns the name of the archive file. + + .. versionchanged: 3.5 + Added support for the ``xztar`` format. .. function:: make_tarball(base_name, base_dir[, compress='gzip', verbose=0, dry_run=0]) 'Create an (optional compressed) archive as a tar file from all files in and - under *base_dir*. *compress* must be ``'gzip'`` (the default), ``'compress'``, - ``'bzip2'``, or ``None``. Both :program:`tar` and the compression utility named - by *compress* must be on the default program search path, so this is probably - Unix-specific. The output tar file will be named :file:`base_dir.tar`, - possibly plus the appropriate compression extension (:file:`.gz`, :file:`.bz2` - or :file:`.Z`). Return the output filename. + under *base_dir*. *compress* must be ``'gzip'`` (the default), + ``'bzip2'``, ``'xz'``, ``'compress'``, or ``None``. For the ``'compress'`` + method the compression utility named by :program:`compress` must be on the + default program search path, so this is probably Unix-specific. The output + tar file will be named :file:`base_dir.tar`, possibly plus the appropriate + compression extension (``.gz``, ``.bz2``, ``.xz`` or ``.Z``). Return the + output filename. + + .. versionchanged: 3.5 + Added support for the ``xz`` compression. .. function:: make_zipfile(base_name, base_dir[, verbose=0, dry_run=0]) @@ -1193,12 +1201,12 @@ other utility module. .. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None]) - Byte-compile a collection of Python source files to either :file:`.pyc` or - :file:`.pyo` files in a :file:`__pycache__` subdirectory (see :pep:`3147`). + Byte-compile a collection of Python source files to :file:`.pyc` files in a + :file:`__pycache__` subdirectory (see :pep:`3147` and :pep:`488`). *py_files* is a list of files to compile; any files that don't end in :file:`.py` are silently skipped. *optimize* must be one of the following: - * ``0`` - don't optimize (generate :file:`.pyc`) + * ``0`` - don't optimize * ``1`` - normal optimization (like ``python -O``) * ``2`` - extra optimization (like ``python -OO``) @@ -1222,10 +1230,13 @@ other utility module. doing, leave it set to ``None``. .. versionchanged:: 3.2.3 - Create ``.pyc`` or ``.pyo`` files with an :func:`import magic tag + Create ``.pyc`` files with an :func:`import magic tag <imp.get_tag>` in their name, in a :file:`__pycache__` subdirectory instead of files without tag in the current directory. + .. versionchanged: 3.5 + Create ``.pyc`` files according to :pep:`488`. + .. function:: rfc822_escape(header) diff --git a/Doc/distutils/builtdist.rst b/Doc/distutils/builtdist.rst index c5827b6..144619b 100644 --- a/Doc/distutils/builtdist.rst +++ b/Doc/distutils/builtdist.rst @@ -72,13 +72,19 @@ The available formats for built distributions are: +-------------+------------------------------+---------+ | Format | Description | Notes | +=============+==============================+=========+ -| ``gztar`` | gzipped tar file | (1),(3) | +| ``gztar`` | gzipped tar file | \(1) | | | (:file:`.tar.gz`) | | +-------------+------------------------------+---------+ +| ``bztar`` | bzipped tar file | | +| | (:file:`.tar.bz2`) | | ++-------------+------------------------------+---------+ +| ``xztar`` | xzipped tar file | | +| | (:file:`.tar.xz`) | | ++-------------+------------------------------+---------+ | ``ztar`` | compressed tar file | \(3) | | | (:file:`.tar.Z`) | | +-------------+------------------------------+---------+ -| ``tar`` | tar file (:file:`.tar`) | \(3) | +| ``tar`` | tar file (:file:`.tar`) | | +-------------+------------------------------+---------+ | ``zip`` | zip file (:file:`.zip`) | (2),(4) | +-------------+------------------------------+---------+ @@ -94,6 +100,9 @@ The available formats for built distributions are: | ``msi`` | Microsoft Installer. | | +-------------+------------------------------+---------+ +.. versionchanged: 3.5 + Added support for the ``xztar`` format. + Notes: @@ -104,8 +113,7 @@ Notes: default on Windows (3) - requires external utilities: :program:`tar` and possibly one of :program:`gzip`, - :program:`bzip2`, or :program:`compress` + requires external :program:`compress` utility. (4) requires either external :program:`zip` utility or :mod:`zipfile` module (part @@ -119,21 +127,22 @@ You don't have to use the :command:`bdist` command with the :option:`--formats` option; you can also use the command that directly implements the format you're interested in. Some of these :command:`bdist` "sub-commands" actually generate several similar formats; for instance, the :command:`bdist_dumb` command -generates all the "dumb" archive formats (``tar``, ``ztar``, ``gztar``, and -``zip``), and :command:`bdist_rpm` generates both binary and source RPMs. The -:command:`bdist` sub-commands, and the formats generated by each, are: - -+--------------------------+-----------------------+ -| Command | Formats | -+==========================+=======================+ -| :command:`bdist_dumb` | tar, ztar, gztar, zip | -+--------------------------+-----------------------+ -| :command:`bdist_rpm` | rpm, srpm | -+--------------------------+-----------------------+ -| :command:`bdist_wininst` | wininst | -+--------------------------+-----------------------+ -| :command:`bdist_msi` | msi | -+--------------------------+-----------------------+ +generates all the "dumb" archive formats (``tar``, ``gztar``, ``bztar``, +``xztar``, ``ztar``, and ``zip``), and :command:`bdist_rpm` generates both +binary and source RPMs. The :command:`bdist` sub-commands, and the formats +generated by each, are: + ++--------------------------+-------------------------------------+ +| Command | Formats | ++==========================+=====================================+ +| :command:`bdist_dumb` | tar, gztar, bztar, xztar, ztar, zip | ++--------------------------+-------------------------------------+ +| :command:`bdist_rpm` | rpm, srpm | ++--------------------------+-------------------------------------+ +| :command:`bdist_wininst` | wininst | ++--------------------------+-------------------------------------+ +| :command:`bdist_msi` | msi | ++--------------------------+-------------------------------------+ The following sections give details on the individual :command:`bdist_\*` commands. diff --git a/Doc/distutils/introduction.rst b/Doc/distutils/introduction.rst index 0ece646..8f46bd7 100644 --- a/Doc/distutils/introduction.rst +++ b/Doc/distutils/introduction.rst @@ -156,8 +156,8 @@ module pure Python module a module written in Python and contained in a single :file:`.py` file (and - possibly associated :file:`.pyc` and/or :file:`.pyo` files). Sometimes referred - to as a "pure module." + possibly associated :file:`.pyc` files). Sometimes referred to as a + "pure module." extension module a module written in the low-level language of the Python implementation: C/C++ @@ -210,5 +210,3 @@ distribution root the top-level directory of your source tree (or source distribution); the directory where :file:`setup.py` exists. Generally :file:`setup.py` will be run from this directory. - - diff --git a/Doc/distutils/sourcedist.rst b/Doc/distutils/sourcedist.rst index b9f0cc8..d79f00d 100644 --- a/Doc/distutils/sourcedist.rst +++ b/Doc/distutils/sourcedist.rst @@ -32,12 +32,18 @@ to create a gzipped tarball and a zip file. The available formats are: | ``bztar`` | bzip2'ed tar file | | | | (:file:`.tar.bz2`) | | +-----------+-------------------------+---------+ +| ``xztar`` | xz'ed tar file | | +| | (:file:`.tar.xz`) | | ++-----------+-------------------------+---------+ | ``ztar`` | compressed tar file | \(4) | | | (:file:`.tar.Z`) | | +-----------+-------------------------+---------+ | ``tar`` | tar file (:file:`.tar`) | | +-----------+-------------------------+---------+ +.. versionchanged: 3.5 + Added support for the ``xztar`` format. + Notes: (1) @@ -54,7 +60,7 @@ Notes: requires the :program:`compress` program. Notice that this format is now pending for deprecation and will be removed in the future versions of Python. -When using any ``tar`` format (``gztar``, ``bztar``, ``ztar`` or +When using any ``tar`` format (``gztar``, ``bztar``, ``xztar``, ``ztar`` or ``tar``), under Unix you can specify the ``owner`` and ``group`` names that will be set for each member of the archive. diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst index 6cb686a..acd60ae 100644 --- a/Doc/extending/embedding.rst +++ b/Doc/extending/embedding.rst @@ -58,12 +58,18 @@ perform some operation on a file. :: int main(int argc, char *argv[]) { - Py_SetProgramName(argv[0]); /* optional but recommended */ - Py_Initialize(); - PyRun_SimpleString("from time import time,ctime\n" - "print('Today is', ctime(time()))\n"); - Py_Finalize(); - return 0; + wchar_t *program = Py_DecodeLocale(argv[0], NULL); + if (program == NULL) { + fprintf(stderr, "Fatal error: cannot decode argv[0]\n"); + exit(1); + } + Py_SetProgramName(program); /* optional but recommended */ + Py_Initialize(); + PyRun_SimpleString("from time import time,ctime\n" + "print('Today is', ctime(time()))\n"); + Py_Finalize(); + PyMem_RawFree(program); + return 0; } The :c:func:`Py_SetProgramName` function should be called before @@ -160,7 +166,7 @@ for data conversion between Python and C, and for error reporting. The interesting part with respect to embedding Python starts with :: Py_Initialize(); - pName = PyUnicode_FromString(argv[1]); + pName = PyUnicode_DecodeFSDefault(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst index ba6bfa7..a83fb6e 100644 --- a/Doc/extending/extending.rst +++ b/Doc/extending/extending.rst @@ -375,11 +375,17 @@ optionally followed by an import of the module:: int main(int argc, char *argv[]) { + wchar_t *program = Py_DecodeLocale(argv[0], NULL); + if (program == NULL) { + fprintf(stderr, "Fatal error: cannot decode argv[0]\n"); + exit(1); + } + /* Add a built-in module, before Py_Initialize */ PyImport_AppendInittab("spam", PyInit_spam); /* Pass argv[0] to the Python interpreter */ - Py_SetProgramName(argv[0]); + Py_SetProgramName(program); /* Initialize the Python interpreter. Required. */ Py_Initialize(); @@ -391,6 +397,10 @@ optionally followed by an import of the module:: ... + PyMem_RawFree(program); + return 0; + } + .. note:: Removing entries from ``sys.modules`` or importing compiled modules into diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst index aaa37b8..b7e35f4 100644 --- a/Doc/extending/newtypes.rst +++ b/Doc/extending/newtypes.rst @@ -80,7 +80,7 @@ Moving on, we come to the crunch --- the type object. :: 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ - 0, /* tp_reserved */ + 0, /* tp_as_async */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 1de86ef..d00185e 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -69,11 +69,34 @@ Glossary :ref:`the difference between arguments and parameters <faq-argument-vs-parameter>`, and :pep:`362`. + asynchronous context manager + An object which controls the environment seen in an + :keyword:`async with` statement by defining :meth:`__aenter__` and + :meth:`__aexit__` methods. Introduced by :pep:`492`. + + asynchronous iterable + An object, that can be used in an :keyword:`async for` statement. + Must return an :term:`awaitable` from its :meth:`__aiter__` method, + which should in turn be resolved in an :term:`asynchronous iterator` + object. Introduced by :pep:`492`. + + asynchronous iterator + An object that implements :meth:`__aiter__` and :meth:`__anext__` + methods, that must return :term:`awaitable` objects. + :keyword:`async for` resolves awaitable returned from asynchronous + iterator's :meth:`__anext__` method until it raises + :exc:`StopAsyncIteration` exception. Introduced by :pep:`492`. + attribute A value associated with an object which is referenced by name using dotted expressions. For example, if an object *o* has an attribute *a* it would be referenced as *o.a*. + awaitable + An object that can be used in an :keyword:`await` expression. Can be + a :term:`coroutine` or an object with an :meth:`__await__` method. + See also :pep:`492`. + BDFL Benevolent Dictator For Life, a.k.a. `Guido van Rossum <https://www.python.org/~guido/>`_, Python's creator. @@ -88,10 +111,17 @@ Glossary bytes-like object An object that supports the :ref:`bufferobjects`, like :class:`bytes`, :class:`bytearray` or :class:`memoryview`. Bytes-like objects can - be used for various operations that expect binary data, such as - compression, saving to a binary file or sending over a socket. - Some operations need the binary data to be mutable, in which case - not all bytes-like objects can apply. + be used for various operations that work with binary data; these include + compression, saving to a binary file, and sending over a socket. + + Some operations need the binary data to be mutable. The documentation + often refers to these as "read-write bytes-like objects". Example + mutable buffer objects include :class:`bytearray` and a + :class:`memoryview` of a :class:`bytearray`. + Other operations require the binary data to be stored in + immutable objects ("read-only bytes-like objects"); examples + of these include :class:`bytes` and a :class:`memoryview` + of a :class:`bytes` object. bytecode Python source code is compiled into bytecode, the internal representation @@ -139,6 +169,20 @@ Glossary statement by defining :meth:`__enter__` and :meth:`__exit__` methods. See :pep:`343`. + coroutine + Coroutines is a more generalized form of subroutines. Subroutines are + entered at one point and exited at another point. Coroutines can be + entered, exited, and resumed at many different points. They can be + implemented with the :keyword:`async def` statement. See also + :pep:`492`. + + coroutine function + A function which returns a :term:`coroutine` object. A coroutine + function may be defined with the :keyword:`async def` statement, + and may contain :keyword:`await`, :keyword:`async for`, and + :keyword:`async with` keywords. These were introduced + by :pep:`492`. + CPython The canonical implementation of the Python programming language, as distributed on `python.org <https://www.python.org>`_. The term "CPython" @@ -290,14 +334,23 @@ Glossary .. index:: single: generator generator - A function which returns an iterator. It looks like a normal function - except that it contains :keyword:`yield` statements for producing a series - of values usable in a for-loop or that can be retrieved one at a time with - the :func:`next` function. Each :keyword:`yield` temporarily suspends - processing, remembering the location execution state (including local - variables and pending try-statements). When the generator resumes, it - picks-up where it left-off (in contrast to functions which start fresh on - every invocation). + A function which returns a :term:`generator iterator`. It looks like a + normal function except that it contains :keyword:`yield` expressions + for producing a series of values usable in a for-loop or that can be + retrieved one at a time with the :func:`next` function. + + Usually refers to a generator function, but may refer to a + *generator iterator* in some contexts. In cases where the intended + meaning isn't clear, using the full terms avoids ambiguity. + + generator iterator + An object created by a :term:`generator` function. + + Each :keyword:`yield` temporarily suspends processing, remembering the + location execution state (including local variables and pending + try-statements). When the *generator iterator* resumes, it picks-up where + it left-off (in contrast to functions which start fresh on every + invocation). .. index:: single: generator expression @@ -402,6 +455,19 @@ Glossary than compiled ones, though their programs generally also run more slowly. See also :term:`interactive`. + interpreter shutdown + When asked to shut down, the Python interpreter enters a special phase + where it gradually releases all allocated resources, such as modules + and various critical internal structures. It also makes several calls + to the :term:`garbage collector <garbage collection>`. This can trigger + the execution of code in user-defined destructors or weakref callbacks. + Code executed during the shutdown phase can encounter various + exceptions as the resources it relies on may not function anymore + (common examples are library modules or the warnings machinery). + + The main reason for interpreter shutdown is that the ``__main__`` module + or the script being run has finished executing. + iterable An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as :class:`list`, :class:`str`, @@ -444,12 +510,13 @@ Glossary A number of tools in Python accept key functions to control how elements are ordered or grouped. They include :func:`min`, :func:`max`, - :func:`sorted`, :meth:`list.sort`, :func:`heapq.nsmallest`, - :func:`heapq.nlargest`, and :func:`itertools.groupby`. + :func:`sorted`, :meth:`list.sort`, :func:`heapq.merge`, + :func:`heapq.nsmallest`, :func:`heapq.nlargest`, and + :func:`itertools.groupby`. There are several ways to create a key function. For example. the :meth:`str.lower` method can serve as a key function for case insensitive - sorts. Alternatively, an ad-hoc key function can be built from a + sorts. Alternatively, a key function can be built from a :keyword:`lambda` expression such as ``lambda r: (r[0], r[2])``. Also, the :mod:`operator` module provides three key function constructors: :func:`~operator.attrgetter`, :func:`~operator.itemgetter`, and diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index e362631..7524c4a 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -758,6 +758,14 @@ All Argument Clinic converters accept the following arguments: In addition, some converters accept additional arguments. Here is a list of these arguments, along with their meanings: + ``accept`` + A set of Python types (and possibly pseudo-types); + this restricts the allowable Python argument to values of these types. + (This is not a general-purpose facility; as a rule it only supports + specific lists of types as shown in the legacy converter table.) + + To accept ``None``, add ``NoneType`` to this set. + ``bitwise`` Only supported for unsigned integers. The native integer value of this Python argument will be written to the parameter without any range checking, @@ -772,39 +780,27 @@ of these arguments, along with their meanings: Only supported for strings. Specifies the encoding to use when converting this string from a Python str (Unicode) value into a C ``char *`` value. - ``length`` - Only supported for strings. If true, requests that the length of the - string be passed in to the impl function, just after the string parameter, - in a parameter named ``<parameter_name>_length``. - - ``nullable`` - Only supported for strings. If true, this parameter may also be set to - ``None``, in which case the C parameter will be set to ``NULL``. ``subclass_of`` Only supported for the ``object`` converter. Requires that the Python value be a subclass of a Python type, as expressed in C. - ``types`` - Only supported for the ``object`` (and ``self``) converter. Specifies + ``type`` + Only supported for the ``object`` and ``self`` converters. Specifies the C type that will be used to declare the variable. Default value is ``"PyObject *"``. - ``types`` - A string containing a list of Python types (and possibly pseudo-types); - this restricts the allowable Python argument to values of these types. - (This is not a general-purpose facility; as a rule it only supports - specific lists of types as shown in the legacy converter table.) - ``zeroes`` Only supported for strings. If true, embedded NUL bytes (``'\\0'``) are - permitted inside the value. + permitted inside the value. The length of the string will be passed in + to the impl function, just after the string parameter, as a parameter named + ``<parameter_name>_length``. Please note, not every possible combination of arguments will work. -Often these arguments are implemented internally by specific ``PyArg_ParseTuple`` +Usually these arguments are implemented by specific ``PyArg_ParseTuple`` *format units*, with specific behavior. For example, currently you cannot -call ``str`` and pass in ``zeroes=True`` without also specifying an ``encoding``; -although it's perfectly reasonable to think this would work, these semantics don't +call ``unsigned_short`` without also specifying ``bitwise=True``. +Although it's perfectly reasonable to think this would work, these semantics don't map to any existing format unit. So Argument Clinic doesn't support it. (Or, at least, not yet.) @@ -816,13 +812,13 @@ on the right is the text you'd replace it with. ``'B'`` ``unsigned_char(bitwise=True)`` ``'b'`` ``unsigned_char`` ``'c'`` ``char`` -``'C'`` ``int(types='str')`` +``'C'`` ``int(accept={str})`` ``'d'`` ``double`` ``'D'`` ``Py_complex`` -``'es#'`` ``str(encoding='name_of_encoding', length=True, zeroes=True)`` ``'es'`` ``str(encoding='name_of_encoding')`` -``'et#'`` ``str(encoding='name_of_encoding', types='bytes bytearray str', length=True)`` -``'et'`` ``str(encoding='name_of_encoding', types='bytes bytearray str')`` +``'es#'`` ``str(encoding='name_of_encoding', zeroes=True)`` +``'et'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str})`` +``'et#'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str}, zeroes=True)`` ``'f'`` ``float`` ``'h'`` ``short`` ``'H'`` ``unsigned_short(bitwise=True)`` @@ -830,29 +826,30 @@ on the right is the text you'd replace it with. ``'I'`` ``unsigned_int(bitwise=True)`` ``'k'`` ``unsigned_long(bitwise=True)`` ``'K'`` ``unsigned_PY_LONG_LONG(bitwise=True)`` +``'l'`` ``long`` ``'L'`` ``PY_LONG_LONG`` ``'n'`` ``Py_ssize_t`` +``'O'`` ``object`` ``'O!'`` ``object(subclass_of='&PySomething_Type')`` ``'O&'`` ``object(converter='name_of_c_function')`` -``'O'`` ``object`` ``'p'`` ``bool`` -``'s#'`` ``str(length=True)`` ``'S'`` ``PyBytesObject`` ``'s'`` ``str`` -``'s*'`` ``Py_buffer(types='str bytes bytearray buffer')`` -``'u#'`` ``Py_UNICODE(length=True)`` -``'u'`` ``Py_UNICODE`` +``'s#'`` ``str(zeroes=True)`` +``'s*'`` ``Py_buffer(accept={buffer, str})`` ``'U'`` ``unicode`` -``'w*'`` ``Py_buffer(types='bytearray rwbuffer')`` -``'y#'`` ``str(types='bytes', length=True)`` +``'u'`` ``Py_UNICODE`` +``'u#'`` ``Py_UNICODE(zeroes=True)`` +``'w*'`` ``Py_buffer(accept={rwbuffer})`` ``'Y'`` ``PyByteArrayObject`` -``'y'`` ``str(types='bytes')`` +``'y'`` ``str(accept={bytes})`` +``'y#'`` ``str(accept={robuffer}, zeroes=True)`` ``'y*'`` ``Py_buffer`` -``'Z#'`` ``Py_UNICODE(nullable=True, length=True)`` -``'z#'`` ``str(nullable=True, length=True)`` -``'Z'`` ``Py_UNICODE(nullable=True)`` -``'z'`` ``str(nullable=True)`` -``'z*'`` ``Py_buffer(types='str bytes bytearray buffer', nullable=True)`` +``'Z'`` ``Py_UNICODE(accept={str, NoneType})`` +``'Z#'`` ``Py_UNICODE(accept={str, NoneType}, zeroes=True)`` +``'z'`` ``str(accept={str, NoneType})`` +``'z#'`` ``str(accept={str, NoneType}, zeroes=True)`` +``'z*'`` ``Py_buffer(accept={buffer, str, NoneType})`` ========= ================================================================================= As an example, here's our sample ``pickle.Pickler.dump`` using the proper diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index 1969b32..945a240 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -481,10 +481,10 @@ Here's a sample usage of the ``generate_ints()`` generator: You could equally write ``for i in generate_ints(5)``, or ``a,b,c = generate_ints(3)``. -Inside a generator function, ``return value`` is semantically equivalent to -``raise StopIteration(value)``. If no value is returned or the bottom of the -function is reached, the procession of values ends and the generator cannot -return any further values. +Inside a generator function, ``return value`` causes ``StopIteration(value)`` +to be raised from the :meth:`~generator.__next__` method. Once this happens, or +the bottom of the function is reached, the procession of values ends and the +generator cannot yield any further values. You could achieve the effect of generators manually by writing your own class and storing all the local variables of the generator as instance variables. For diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst index 6305d24..813bc0f 100644 --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -325,6 +325,15 @@ which, when run, will produce:: MainThread: Look out! +.. versionchanged:: 3.5 + Prior to Python 3.5, the :class:`QueueListener` always passed every message + received from the queue to every handler it was initialized with. (This was + because it was assumed that level filtering was all done on the other side, + where the queue is filled.) From 3.5 onwards, this behaviour can be changed + by passing a keyword argument ``respect_handler_level=True`` to the + listener's constructor. When this is done, the listener compares the level + of each message with the handler's level, and only passes a message to a + handler if it's appropriate to do so. .. _network-logging: @@ -1680,7 +1689,7 @@ as in the following complete example:: def main(): logging.basicConfig(level=logging.INFO, format='%(message)s') - logging.info(_('message 1', set_value=set([1, 2, 3]), snowman='\u2603')) + logging.info(_('message 1', set_value={1, 2, 3}, snowman='\u2603')) if __name__ == '__main__': main() diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst index 5e875cd..a2aaf36 100644 --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -207,13 +207,12 @@ that's ``str``/``bytes`` in Python 2 and ``bytes`` in Python 3). The following table lists the **unique** methods of each data type across Python 2 & 3 (e.g., the ``decode()`` method is usable on the equivalent binary data type in either Python 2 or 3, but it can't be used by the text data type consistently -between Python 2 and 3 because ``str`` in Python 3 doesn't have the method). +between Python 2 and 3 because ``str`` in Python 3 doesn't have the method). Do +note that as of Python 3.5 the ``__mod__`` method was added to the bytes type. ======================== ===================== **Text data** **Binary data** ------------------------ --------------------- -__mod__ (``%`` operator) ------------------------- --------------------- \ decode ------------------------ --------------------- encode @@ -348,10 +347,12 @@ tox with your continuous integration system so that you never accidentally break Python 2 or 3 support. You may also want to use use the ``-bb`` flag with the Python 3 interpreter to -trigger an exception when you are comparing bytes to strings. Usually it's -simply ``False``, but if you made a mistake in your separation of text/binary -data handling you may be accidentally comparing text and binary data. This flag -will raise an exception when that occurs to help track down such cases. +trigger an exception when you are comparing bytes to strings or bytes to an int +(the latter is available starting in Python 3.5). By default type-differing +comparisons simply return ``False``, but if you made a mistake in your +separation of text/binary data handling or indexing on bytes you wouldn't easily +find the mistake. This flag will raise an exception when these kinds of +comparisons occur, making the mistake much easier to track down. And that's mostly it! At this point your code base is compatible with both Python 2 and 3 simultaneously. Your testing will also be set up so that you diff --git a/Doc/howto/regex.rst b/Doc/howto/regex.rst index 9ae04d7..ad2c6ab 100644 --- a/Doc/howto/regex.rst +++ b/Doc/howto/regex.rst @@ -1138,7 +1138,7 @@ Empty matches are replaced only when they're not adjacent to a previous match. If *replacement* is a string, any backslash escapes in it are processed. That is, ``\n`` is converted to a single newline character, ``\r`` is converted to a -carriage return, and so forth. Unknown escapes such as ``\j`` are left alone. +carriage return, and so forth. Unknown escapes such as ``\&`` are left alone. Backreferences, such as ``\6``, are replaced with the substring matched by the corresponding group in the RE. This lets you incorporate portions of the original text in the resulting replacement string. diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst index b49ac39..ee31a9c 100644 --- a/Doc/howto/unicode.rst +++ b/Doc/howto/unicode.rst @@ -280,8 +280,9 @@ and optionally an *errors* argument. The *errors* argument specifies the response when the input string can't be converted according to the encoding's rules. Legal values for this argument are ``'strict'`` (raise a :exc:`UnicodeDecodeError` exception), ``'replace'`` (use -``U+FFFD``, ``REPLACEMENT CHARACTER``), or ``'ignore'`` (just leave the -character out of the Unicode result). +``U+FFFD``, ``REPLACEMENT CHARACTER``), ``'ignore'`` (just leave the +character out of the Unicode result), or ``'backslashreplace'`` (inserts a +``\xNN`` escape sequence). The following examples show the differences:: >>> b'\x80abc'.decode("utf-8", "strict") #doctest: +NORMALIZE_WHITESPACE @@ -291,6 +292,8 @@ The following examples show the differences:: invalid start byte >>> b'\x80abc'.decode("utf-8", "replace") '\ufffdabc' + >>> b'\x80abc'.decode("utf-8", "backslashreplace") + '\\x80abc' >>> b'\x80abc'.decode("utf-8", "ignore") 'abc' @@ -325,8 +328,9 @@ The *errors* parameter is the same as the parameter of the :meth:`~bytes.decode` method but supports a few more possible handlers. As well as ``'strict'``, ``'ignore'``, and ``'replace'`` (which in this case inserts a question mark instead of the unencodable character), there is -also ``'xmlcharrefreplace'`` (inserts an XML character reference) and -``backslashreplace`` (inserts a ``\uNNNN`` escape sequence). +also ``'xmlcharrefreplace'`` (inserts an XML character reference), +``backslashreplace`` (inserts a ``\uNNNN`` escape sequence) and +``namereplace`` (inserts a ``\N{...}`` escape sequence). The following example shows the different results:: @@ -346,6 +350,8 @@ The following example shows the different results:: b'ꀀabcd޴' >>> u.encode('ascii', 'backslashreplace') b'\\ua000abcd\\u07b4' + >>> u.encode('ascii', 'namereplace') + b'\\N{YI SYLLABLE IT}abcd\\u07b4' The low-level routines for registering and accessing the available encodings are found in the :mod:`codecs` module. Implementing new diff --git a/Doc/includes/noddy.c b/Doc/includes/noddy.c index 8f79fcf..19a27a8 100644 --- a/Doc/includes/noddy.c +++ b/Doc/includes/noddy.c @@ -38,7 +38,7 @@ static PyModuleDef noddymodule = { }; PyMODINIT_FUNC -PyInit_noddy(void) +PyInit_noddy(void) { PyObject* m; diff --git a/Doc/includes/run-func.c b/Doc/includes/run-func.c index 1c9860d..986d670 100644 --- a/Doc/includes/run-func.c +++ b/Doc/includes/run-func.c @@ -13,7 +13,7 @@ main(int argc, char *argv[]) } Py_Initialize(); - pName = PyUnicode_FromString(argv[1]); + pName = PyUnicode_DecodeFSDefault(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); diff --git a/Doc/includes/typestruct.h b/Doc/includes/typestruct.h index fcb846a..5c9f8f5 100644 --- a/Doc/includes/typestruct.h +++ b/Doc/includes/typestruct.h @@ -9,7 +9,7 @@ typedef struct _typeobject { printfunc tp_print; getattrfunc tp_getattr; setattrfunc tp_setattr; - void *tp_reserved; + PyAsyncMethods *tp_as_async; reprfunc tp_repr; /* Method suites for standard classes */ diff --git a/Doc/install/index.rst b/Doc/install/index.rst index 8f3ad72..876f350 100644 --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -361,7 +361,7 @@ And here are the values used on Windows: Type of file Installation directory =============== =========================================================== modules :file:`{userbase}\\Python{XY}\\site-packages` -scripts :file:`{userbase}\\Scripts` +scripts :file:`{userbase}\\Python{XY}\\Scripts` data :file:`{userbase}` C headers :file:`{userbase}\\Python{XY}\\Include\\{distname}` =============== =========================================================== diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst index 72f2963..73d8b6b 100644 --- a/Doc/library/__future__.rst +++ b/Doc/library/__future__.rst @@ -87,6 +87,9 @@ language using this mechanism: | unicode_literals | 2.6.0a2 | 3.0 | :pep:`3112`: | | | | | *Bytes literals in Python 3000* | +------------------+-------------+--------------+---------------------------------------------+ +| generator_stop | 3.5.0b1 | 3.7 | :pep:`479`: | +| | | | *StopIteration handling inside generators* | ++------------------+-------------+--------------+---------------------------------------------+ .. seealso:: diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 067fb8f..72095a8 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -135,7 +135,7 @@ ArgumentParser objects formatter_class=argparse.HelpFormatter, \ prefix_chars='-', fromfile_prefix_chars=None, \ argument_default=None, conflict_handler='error', \ - add_help=True) + add_help=True, allow_abbrev=True) Create a new :class:`ArgumentParser` object. All parameters should be passed as keyword arguments. Each parameter has its own more detailed description @@ -169,6 +169,12 @@ ArgumentParser objects * add_help_ - Add a -h/--help option to the parser (default: ``True``) + * allow_abbrev_ - Allows long options to be abbreviated if the + abbreviation is unambiguous. (default: ``True``) + + .. versionchanged:: 3.5 + *allow_abbrev* parameter was added. + The following sections describe how each of these are used. @@ -518,6 +524,26 @@ calls, we supply ``argument_default=SUPPRESS``:: >>> parser.parse_args([]) Namespace() +.. _allow_abbrev: + +allow_abbrev +^^^^^^^^^^^^ + +Normally, when you pass an argument list to the +:meth:`~ArgumentParser.parse_args` method of a :class:`ArgumentParser`, +it :ref:`recognizes abbreviations <prefix-matching>` of long options. + +This feature can be disabled by setting ``allow_abbrev`` to ``False``:: + + >>> parser = argparse.ArgumentParser(prog='PROG', allow_abbrev=False) + >>> parser.add_argument('--foobar', action='store_true') + >>> parser.add_argument('--foonley', action='store_false') + >>> parser.parse_args(['--foon']) + usage: PROG [-h] [--foobar] [--foonley] + PROG: error: unrecognized arguments: --foon + +.. versionadded:: 3.5 + conflict_handler ^^^^^^^^^^^^^^^^ @@ -1410,9 +1436,9 @@ argument:: Argument abbreviations (prefix matching) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The :meth:`~ArgumentParser.parse_args` method allows long options to be -abbreviated to a prefix, if the abbreviation is unambiguous (the prefix matches -a unique option):: +The :meth:`~ArgumentParser.parse_args` method :ref:`by default <allow_abbrev>` +allows long options to be abbreviated to a prefix, if the abbreviation is +unambiguous (the prefix matches a unique option):: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-bacon') @@ -1426,6 +1452,7 @@ a unique option):: PROG: error: ambiguous option: -ba could match -badger, -bacon An error is produced for arguments that could produce more than one options. +This feature can be disabled by setting :ref:`allow_abbrev` to ``False``. Beyond ``sys.argv`` diff --git a/Doc/library/asynchat.rst b/Doc/library/asynchat.rst index c6fa061..794da8c 100644 --- a/Doc/library/asynchat.rst +++ b/Doc/library/asynchat.rst @@ -147,40 +147,6 @@ connection requests. by the channel after :meth:`found_terminator` is called. -asynchat - Auxiliary Classes ------------------------------------------- - -.. class:: fifo(list=None) - - A :class:`fifo` holding data which has been pushed by the application but - not yet popped for writing to the channel. A :class:`fifo` is a list used - to hold data and/or producers until they are required. If the *list* - argument is provided then it should contain producers or data items to be - written to the channel. - - - .. method:: is_empty() - - Returns ``True`` if and only if the fifo is empty. - - - .. method:: first() - - Returns the least-recently :meth:`push`\ ed item from the fifo. - - - .. method:: push(data) - - Adds the given data (which may be a string or a producer object) to the - producer fifo. - - - .. method:: pop() - - If the fifo is not empty, returns ``True, first()``, deleting the popped - item. Returns ``False, None`` for an empty fifo. - - .. _asynchat-example: asynchat Example diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst index 3a196db..e27cef8 100644 --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -99,7 +99,7 @@ To schedule a callback from a different thread, the :meth:`BaseEventLoop.call_soon_threadsafe` method should be used. Example to schedule a coroutine from a different thread:: - loop.call_soon_threadsafe(asyncio.async, coro_func()) + loop.call_soon_threadsafe(asyncio.ensure_future, coro_func()) Most asyncio objects are not thread safe. You should only worry if you access objects outside the event loop. For example, to cancel a future, don't call @@ -162,10 +162,10 @@ Detect coroutine objects never scheduled ---------------------------------------- When a coroutine function is called and its result is not passed to -:func:`async` or to the :meth:`BaseEventLoop.create_task` method, the execution -of the coroutine object will never be scheduled which is probably a bug. -:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to :ref:`log a -warning <asyncio-logger>` to detect it. +:func:`ensure_future` or to the :meth:`BaseEventLoop.create_task` method, +the execution of the coroutine object will never be scheduled which is +probably a bug. :ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` +to :ref:`log a warning <asyncio-logger>` to detect it. Example with the bug:: @@ -184,7 +184,7 @@ Output in debug mode:: File "test.py", line 7, in <module> test() -The fix is to call the :func:`async` function or the +The fix is to call the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method with the coroutine object. .. seealso:: diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 33d3148..7d7caa8 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -36,7 +36,7 @@ Run an event loop Run until the :class:`Future` is done. If the argument is a :ref:`coroutine object <coroutine>`, it is wrapped by - :func:`async`. + :func:`ensure_future`. Return the Future's result, or raise its exception. diff --git a/Doc/library/asyncio-eventloops.rst b/Doc/library/asyncio-eventloops.rst index afb8b9f..ae3bf90 100644 --- a/Doc/library/asyncio-eventloops.rst +++ b/Doc/library/asyncio-eventloops.rst @@ -100,8 +100,6 @@ Common limits of Windows event loops: :class:`ProactorEventLoop` specific limits: -- SSL is not supported: :meth:`~BaseEventLoop.create_connection` and - :meth:`~BaseEventLoop.create_server` cannot be used with SSL for example - :meth:`~BaseEventLoop.create_datagram_endpoint` (UDP) is not supported - :meth:`~BaseEventLoop.add_reader` and :meth:`~BaseEventLoop.add_writer` are not supported @@ -112,6 +110,10 @@ The best resolution is 0.5 msec. The resolution depends on the hardware <http://fr.wikipedia.org/wiki/High_Precision_Event_Timer>`_) and on the Windows configuration. See :ref:`asyncio delayed calls <asyncio-delayed-calls>`. +.. versionchanged:: 3.5 + + :class:`ProactorEventLoop` now supports SSL. + Mac OS X ^^^^^^^^ diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst index 2e671e8..e68c234 100644 --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -448,9 +448,9 @@ buffer size reaches the low-water mark. Coroutines and protocols ------------------------ -Coroutines can be scheduled in a protocol method using :func:`async`, but there -is no guarantee made about the execution order. Protocols are not aware of -coroutines created in protocol methods and so will not wait for them. +Coroutines can be scheduled in a protocol method using :func:`ensure_future`, +but there is no guarantee made about the execution order. Protocols are not +aware of coroutines created in protocol methods and so will not wait for them. To have a reliable execution order, use :ref:`stream objects <asyncio-streams>` in a coroutine with ``yield from``. For example, the :meth:`StreamWriter.drain` diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index 3370672..f11c09a 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -8,7 +8,6 @@ Queues: * :class:`Queue` * :class:`PriorityQueue` * :class:`LifoQueue` -* :class:`JoinableQueue` asyncio queue API was designed to be close to classes of the :mod:`queue` module (:class:`~queue.Queue`, :class:`~queue.PriorityQueue`, @@ -144,16 +143,6 @@ LifoQueue first. -JoinableQueue -^^^^^^^^^^^^^ - -.. class:: JoinableQueue - - Deprecated alias for :class:`Queue`. - - .. deprecated:: 3.4.4 - - Exceptions ^^^^^^^^^^ diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst index e7ff7d2..166ab73 100644 --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -8,17 +8,23 @@ Tasks and coroutines Coroutines ---------- -A coroutine is a generator that follows certain conventions. For -documentation purposes, all coroutines should be decorated with -``@asyncio.coroutine``, but this cannot be strictly enforced. - -Coroutines use the ``yield from`` syntax introduced in :pep:`380`, +Coroutines used with :mod:`asyncio` may be implemented using the +:keyword:`async def` statement, or by using :term:`generators <generator>`. +The :keyword:`async def` type of coroutine was added in Python 3.5, and +is recommended if there is no need to support older Python versions. + +Generator-based coroutines should be decorated with :func:`@asyncio.coroutine +<asyncio.coroutine>`, although this is not strictly enforced. +The decorator enables compatibility with :keyword:`async def` coroutines, +and also serves as documentation. Generator-based +coroutines use the ``yield from`` syntax introduced in :pep:`380`, instead of the original ``yield`` syntax. The word "coroutine", like the word "generator", is used for two different (though related) concepts: -- The function that defines a coroutine (a function definition +- The function that defines a coroutine + (a function definition using :keyword:`async def` or decorated with ``@asyncio.coroutine``). If disambiguation is needed we will call this a *coroutine function* (:func:`iscoroutinefunction` returns ``True``). @@ -30,29 +36,30 @@ different (though related) concepts: Things a coroutine can do: -- ``result = yield from future`` -- suspends the coroutine until the +- ``result = await future`` or ``result = yield from future`` -- + suspends the coroutine until the future is done, then returns the future's result, or raises an exception, which will be propagated. (If the future is cancelled, it will raise a ``CancelledError`` exception.) Note that tasks are futures, and everything said about futures also applies to tasks. -- ``result = yield from coroutine`` -- wait for another coroutine to +- ``result = await coroutine`` or ``result = yield from coroutine`` -- + wait for another coroutine to produce a result (or raise an exception, which will be propagated). The ``coroutine`` expression must be a *call* to another coroutine. - ``return expression`` -- produce a result to the coroutine that is - waiting for this one using ``yield from``. + waiting for this one using :keyword:`await` or ``yield from``. - ``raise exception`` -- raise an exception in the coroutine that is - waiting for this one using ``yield from``. + waiting for this one using :keyword:`await` or ``yield from``. -Calling a coroutine does not start its code running -- it is just a -generator, and the coroutine object returned by the call is really a -generator object, which doesn't do anything until you iterate over it. -In the case of a coroutine object, there are two basic ways to start -it running: call ``yield from coroutine`` from another coroutine +Calling a coroutine does not start its code running -- +the coroutine object returned by the call doesn't do anything until you +schedule its execution. There are two basic ways to start it running: +call ``await coroutine`` or ``yield from coroutine`` from another coroutine (assuming the other coroutine is already running!), or schedule its execution -using the :func:`async` function or the :meth:`BaseEventLoop.create_task` +using the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method. @@ -60,9 +67,15 @@ Coroutines (and tasks) can only run when the event loop is running. .. decorator:: coroutine - Decorator to mark coroutines. + Decorator to mark generator-based coroutines. This enables + the generator use :keyword:`!yield from` to call :keyword:`async + def` coroutines, and also enables the generator to be called by + :keyword:`async def` coroutines, for instance using an + :keyword:`await` expression. + + There is no need to decorate :keyword:`async def` coroutines themselves. - If the coroutine is not yielded from before it is destroyed, an error + If the generator is not yielded from before it is destroyed, an error message is logged. See :ref:`Detect coroutines never scheduled <asyncio-coroutine-not-scheduled>`. @@ -72,7 +85,7 @@ Coroutines (and tasks) can only run when the event loop is running. even if they are plain Python functions returning a :class:`Future`. This is intentional to have a freedom of tweaking the implementation of these functions in the future. If such a function is needed to be - used in a callback-style code, wrap its result with :func:`async`. + used in a callback-style code, wrap its result with :func:`ensure_future`. .. _asyncio-hello-world-coroutine: @@ -84,8 +97,7 @@ Example of coroutine displaying ``"Hello World"``:: import asyncio - @asyncio.coroutine - def hello_world(): + async def hello_world(): print("Hello World!") loop = asyncio.get_event_loop() @@ -111,20 +123,30 @@ using the :meth:`sleep` function:: import asyncio import datetime - @asyncio.coroutine - def display_date(loop): + async def display_date(loop): end_time = loop.time() + 5.0 while True: print(datetime.datetime.now()) if (loop.time() + 1.0) >= end_time: break - yield from asyncio.sleep(1) + await asyncio.sleep(1) loop = asyncio.get_event_loop() # Blocking call which returns when the display_date() coroutine is done loop.run_until_complete(display_date(loop)) loop.close() +The same coroutine implemented using a generator:: + + @asyncio.coroutine + def display_date(loop): + end_time = loop.time() + 5.0 + while True: + print(datetime.datetime.now()) + if (loop.time() + 1.0) >= end_time: + break + yield from asyncio.sleep(1) + .. seealso:: The :ref:`display the current date with call_later() @@ -139,15 +161,13 @@ Example chaining coroutines:: import asyncio - @asyncio.coroutine - def compute(x, y): + async def compute(x, y): print("Compute %s + %s ..." % (x, y)) - yield from asyncio.sleep(1.0) + await asyncio.sleep(1.0) return x + y - @asyncio.coroutine - def print_sum(x, y): - result = yield from compute(x, y) + async def print_sum(x, y): + result = await compute(x, y) print("%s + %s = %s" % (x, y, result)) loop = asyncio.get_event_loop() @@ -374,7 +394,7 @@ Task <coroutine>` did not complete. It is probably a bug and a warning is logged: see :ref:`Pending task destroyed <asyncio-pending-task-destroyed>`. - Don't directly create :class:`Task` instances: use the :func:`async` + Don't directly create :class:`Task` instances: use the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method. This class is :ref:`not thread safe <asyncio-multithreading>`. @@ -550,12 +570,14 @@ Task functions .. function:: iscoroutine(obj) - Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`. + Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`, + which may be based on a generator or an :keyword:`async def` coroutine. -.. function:: iscoroutinefunction(obj) +.. function:: iscoroutinefunction(func) - Return ``True`` if *func* is a decorated :ref:`coroutine function - <coroutine>`. + Return ``True`` if *func* is determined to be a :ref:`coroutine function + <coroutine>`, which may be a decorated generator function or an + :keyword:`async def` function. .. coroutinefunction:: sleep(delay, result=None, \*, loop=None) diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index 488cda5..1b8d9cf 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -120,6 +120,10 @@ All of the classes in this module may safely be accessed from multiple threads. .. versionchanged:: 3.4 The ``'x'`` (exclusive creation) mode was added. + .. versionchanged:: 3.5 + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + Incremental (de)compression --------------------------- @@ -162,15 +166,32 @@ Incremental (de)compression you need to decompress a multi-stream input with :class:`BZ2Decompressor`, you must use a new decompressor for each stream. - .. method:: decompress(data) + .. method:: decompress(data, max_length=-1) + + Decompress *data* (a :term:`bytes-like object`), returning + uncompressed data as bytes. Some of *data* may be buffered + internally, for use in later calls to :meth:`decompress`. The + returned data should be concatenated with the output of any + previous calls to :meth:`decompress`. + + If *max_length* is nonnegative, returns at most *max_length* + bytes of decompressed data. If this limit is reached and further + output can be produced, the :attr:`~.needs_input` attribute will + be set to ``False``. In this case, the next call to + :meth:`~.decompress` may provide *data* as ``b''`` to obtain + more of the output. - Provide data to the decompressor object. Returns a chunk of decompressed - data if possible, or an empty byte string otherwise. + If all of the input data was decompressed and returned (either + because this was less than *max_length* bytes, or because + *max_length* was negative), the :attr:`~.needs_input` attribute + will be set to ``True``. - Attempting to decompress data after the end of the current stream is - reached raises an :exc:`EOFError`. If any data is found after the end of - the stream, it is ignored and saved in the :attr:`unused_data` attribute. + Attempting to decompress data after the end of stream is reached + raises an `EOFError`. Any data found after the end of the + stream is ignored and saved in the :attr:`~.unused_data` attribute. + .. versionchanged:: 3.5 + Added the *max_length* parameter. .. attribute:: eof @@ -186,6 +207,13 @@ Incremental (de)compression If this attribute is accessed before the end of the stream has been reached, its value will be ``b''``. + .. attribute:: needs_input + + ``False`` if the :meth:`.decompress` method can provide more + decompressed data before requiring new uncompressed input. + + .. versionadded:: 3.5 + One-shot (de)compression ------------------------ diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst index fa13145..0a9e884 100644 --- a/Doc/library/cgi.rst +++ b/Doc/library/cgi.rst @@ -157,6 +157,9 @@ return bytes):: if not line: break linecount = linecount + 1 +:class:`FieldStorage` objects also support being used in a :keyword:`with` +statement, which will automatically close them when done. + If an error is encountered when obtaining the contents of an uploaded file (for example, when the user interrupts the form submission by clicking on a Back or Cancel button) the :attr:`~FieldStorage.done` attribute of the @@ -182,6 +185,10 @@ A form submitted via POST that also has a query string will contain both The :attr:`~FieldStorage.file` attribute is automatically closed upon the garbage collection of the creating :class:`FieldStorage` instance. +.. versionchanged:: 3.5 + Added support for the context management protocol to the + :class:`FieldStorage` class. + Higher Level Interface ---------------------- diff --git a/Doc/library/cmath.rst b/Doc/library/cmath.rst index a981d94..ab619a0 100644 --- a/Doc/library/cmath.rst +++ b/Doc/library/cmath.rst @@ -207,6 +207,38 @@ Classification functions and ``False`` otherwise. +.. function:: isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0) + + Return ``True`` if the values *a* and *b* are close to each other and + ``False`` otherwise. + + Whether or not two values are considered close is determined according to + given absolute and relative tolerances. + + *rel_tol* is the relative tolerance -- it is the maximum allowed difference + between *a* and *b*, relative to the larger absolute value of *a* or *b*. + For example, to set a tolerance of 5%, pass ``rel_tol=0.05``. The default + tolerance is ``1e-09``, which assures that the two values are the same + within about 9 decimal digits. *rel_tol* must be greater than zero. + + *abs_tol* is the minimum absolute tolerance -- useful for comparisons near + zero. *abs_tol* must be at least zero. + + If no errors occur, the result will be: + ``abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)``. + + The IEEE 754 special values of ``NaN``, ``inf``, and ``-inf`` will be + handled according to IEEE rules. Specifically, ``NaN`` is not considered + close to any other value, including ``NaN``. ``inf`` and ``-inf`` are only + considered close to themselves. + + .. versionadded:: 3.5 + + .. seealso:: + + :pep:`485` -- A function for testing approximate equality + + Constants --------- diff --git a/Doc/library/code.rst b/Doc/library/code.rst index 5b5d7cc..275201c 100644 --- a/Doc/library/code.rst +++ b/Doc/library/code.rst @@ -4,6 +4,7 @@ .. module:: code :synopsis: Facilities to implement read-eval-print loops. +**Source code:** :source:`Lib/code.py` The ``code`` module provides facilities to implement read-eval-print loops in Python. Two classes and convenience functions are included which can be used to @@ -113,6 +114,9 @@ Interactive Interpreter Objects because it is within the interpreter object implementation. The output is written by the :meth:`write` method. + .. versionchanged:: 3.5 The full chained traceback is displayed instead + of just the primary traceback. + .. method:: InteractiveInterpreter.write(data) @@ -165,4 +169,3 @@ interpreter objects as well as the following additions. newline. When the user enters the EOF key sequence, :exc:`EOFError` is raised. The base implementation reads from ``sys.stdin``; a subclass may replace this with a different implementation. - diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 19d7192..743ccba 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -7,6 +7,7 @@ .. sectionauthor:: Marc-André Lemburg <mal@lemburg.com> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/codecs.py` .. index:: single: Unicode @@ -29,10 +30,9 @@ module features are restricted to use specifically with The module defines the following functions for encoding and decoding with any codec: -.. function:: encode(obj, [encoding[, errors]]) +.. function:: encode(obj, encoding='utf-8', errors='strict') - Encodes *obj* using the codec registered for *encoding*. The default - encoding is ``utf-8``. + Encodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The default error handler is ``'strict'`` meaning that encoding errors raise @@ -40,10 +40,9 @@ any codec: :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more information on codec error handling. -.. function:: decode(obj, [encoding[, errors]]) +.. function:: decode(obj, encoding='utf-8', errors='strict') - Decodes *obj* using the codec registered for *encoding*. The default - encoding is ``utf-8``. + Decodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The default error handler is ``'strict'`` meaning that decoding errors raise @@ -104,7 +103,6 @@ The full details for each codec can also be looked up directly: To simplify access to the various codec components, the module provides these additional functions which use :func:`lookup` for the codec lookup: - .. function:: getencoder(encoding) Look up the codec for the given encoding and return its encoder function. @@ -274,6 +272,7 @@ implement the file protocols. Codec authors also need to define how the codec will handle encoding and decoding errors. +.. _surrogateescape: .. _error-handlers: Error Handlers @@ -315,10 +314,14 @@ The following error handlers are only applicable to | | reference (only for encoding). Implemented | | | in :func:`xmlcharrefreplace_errors`. | +-------------------------+-----------------------------------------------+ -| ``'backslashreplace'`` | Replace with backslashed escape sequences | -| | (only for encoding). Implemented in | +| ``'backslashreplace'`` | Replace with backslashed escape sequences. | +| | Implemented in | | | :func:`backslashreplace_errors`. | +-------------------------+-----------------------------------------------+ +| ``'namereplace'`` | Replace with ``\N{...}`` escape sequences | +| | (only for encoding). Implemented in | +| | :func:`namereplace_errors`. | ++-------------------------+-----------------------------------------------+ | ``'surrogateescape'`` | On decoding, replace byte with individual | | | surrogate code ranging from ``U+DC80`` to | | | ``U+DCFF``. This code will then be turned | @@ -344,6 +347,13 @@ In addition, the following error handler is specific to the given codecs: .. versionchanged:: 3.4 The ``'surrogatepass'`` error handlers now works with utf-16\* and utf-32\* codecs. +.. versionadded:: 3.5 + The ``'namereplace'`` error handler. + +.. versionchanged:: 3.5 + The ``'backslashreplace'`` error handlers now works with decoding and + translating. + The set of allowed values can be extended by registering a new named error handler: @@ -411,9 +421,17 @@ functions: .. function:: backslashreplace_errors(exception) - Implements the ``'backslashreplace'`` error handling (for encoding with + Implements the ``'backslashreplace'`` error handling (for + :term:`text encodings <text encoding>` only): malformed data is + replaced by a backslashed escape sequence. + +.. function:: namereplace_errors(exception) + + Implements the ``'namereplace'`` error handling (for encoding with :term:`text encodings <text encoding>` only): the - unencodable character is replaced by a backslashed escape sequence. + unencodable character is replaced by a ``\N{...}`` escape sequence. + + .. versionadded:: 3.5 .. _codec-objects: @@ -1142,8 +1160,16 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | koi8_r | | Russian | +-----------------+--------------------------------+--------------------------------+ +| koi8_t | | Tajik | +| | | | +| | | .. versionadded:: 3.5 | ++-----------------+--------------------------------+--------------------------------+ | koi8_u | | Ukrainian | +-----------------+--------------------------------+--------------------------------+ +| kz1048 | kz_1048, strk1048_2002, rk1048 | Kazakh | +| | | | +| | | .. versionadded:: 3.5 | ++-----------------+--------------------------------+--------------------------------+ | mac_cyrillic | maccyrillic | Bulgarian, Byelorussian, | | | | Macedonian, Russian, Serbian | +-----------------+--------------------------------+--------------------------------+ @@ -1444,4 +1470,3 @@ This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this is only done once (on the first write to the byte stream). For decoding an optional UTF-8 encoded BOM at the start of the data will be skipped. - diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index 99c4311..dc2704e 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -33,13 +33,14 @@ The collections module offers the following :term:`ABCs <abstract base class>`: .. tabularcolumns:: |l|L|L|L| -========================= ===================== ====================== ==================================================== +========================== ====================== ======================= ==================================================== ABC Inherits from Abstract Methods Mixin Methods -========================= ===================== ====================== ==================================================== +========================== ====================== ======================= ==================================================== :class:`Container` ``__contains__`` :class:`Hashable` ``__hash__`` :class:`Iterable` ``__iter__`` :class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__`` +:class:`Generator` :class:`Iterator` ``send``, ``throw`` ``close``, ``__iter__``, ``__next__`` :class:`Sized` ``__len__`` :class:`Callable` ``__call__`` @@ -80,7 +81,11 @@ ABC Inherits from Abstract Methods Mixin :class:`KeysView` :class:`MappingView`, ``__contains__``, :class:`Set` ``__iter__`` :class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__`` -========================= ===================== ====================== ==================================================== +:class:`Awaitable` ``__await__`` +:class:`Coroutine` :class:`Awaitable` ``send``, ``throw`` ``close`` +:class:`AsyncIterable` ``__aiter__`` +:class:`AsyncIterator` :class:`AsyncIterable` ``__anext__`` ``__aiter__`` +========================== ====================== ======================= ==================================================== .. class:: Container @@ -102,11 +107,34 @@ ABC Inherits from Abstract Methods Mixin :meth:`~iterator.__next__` methods. See also the definition of :term:`iterator`. +.. class:: Generator + + ABC for generator classes that implement the protocol defined in + :pep:`342` that extends iterators with the :meth:`~generator.send`, + :meth:`~generator.throw` and :meth:`~generator.close` methods. + See also the definition of :term:`generator`. + + .. versionadded:: 3.5 + .. class:: Sequence MutableSequence ABCs for read-only and mutable :term:`sequences <sequence>`. + Implementation note: Some of the mixin methods, such as + :meth:`__iter__`, :meth:`__reversed__` and :meth:`index`, make + repeated calls to the underlying :meth:`__getitem__` method. + Consequently, if :meth:`__getitem__` is implemented with constant + access speed, the mixin methods will have linear performance; + however, if the underlying method is linear (as it would be with a + linked list), the mixins will have quadratic performance and will + likely need to be overridden. + + .. versionchanged:: 3.5 + The index() method added support for *stop* and *start* + arguments. + + .. class:: Set MutableSet @@ -124,6 +152,54 @@ ABC Inherits from Abstract Methods Mixin ABCs for mapping, items, keys, and values :term:`views <view>`. +.. class:: Awaitable + + ABC for :term:`awaitable` objects, which can be used in :keyword:`await` + expressions. Custom implementations must provide the :meth:`__await__` + method. + + :term:`Coroutine` objects and instances of the + :class:`~collections.abc.Coroutine` ABC are all instances of this ABC. + + .. note:: + In CPython, generator-based coroutines are *awaitables*, even though + they do not have an :meth:`__await__` method. This ABC + implements an :meth:`~class.__instancecheck__` method to make them + instances of itself. + + .. versionadded:: 3.5 + +.. class:: Coroutine + + ABC for coroutine compatible classes. These implement the + following methods, defined in :ref:`coroutine-objects`: + :meth:`~coroutine.send`, :meth:`~coroutine.throw`, and + :meth:`~coroutine.close`. Custom implementations must also implement + :meth:`__await__`. All :class:`Coroutine` instances are also instances of + :class:`Awaitable`. See also the definition of :term:`coroutine`. + + .. note:: + In CPython, generator-based coroutines are *awaitables* and *coroutines*, + even though they do not have an :meth:`__await__` method. This ABC + implements an :meth:`~class.__instancecheck__` method to make them + instances of itself. + + .. versionadded:: 3.5 + +.. class:: AsyncIterable + + ABC for classes that provide ``__aiter__`` method. See also the + definition of :term:`asynchronous iterable`. + + .. versionadded:: 3.5 + +.. class:: AsyncIterator + + ABC for classes that provide ``__aiter__`` and ``__anext__`` + methods. See also the definition of :term:`asynchronous iterator`. + + .. versionadded:: 3.5 + These ABCs allow us to ask classes or instances if they provide particular functionality, for example:: diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 766bd38..f2befe7 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -387,7 +387,7 @@ or subtracting from an empty counter. Section 4.6.3, Exercise 19*. * To enumerate all distinct multisets of a given size over a given set of - elements, see :func:`itertools.combinations_with_replacement`. + elements, see :func:`itertools.combinations_with_replacement`: map(Counter, combinations_with_replacement('ABC', 2)) --> AA AB AC BB BC CC @@ -437,6 +437,13 @@ or subtracting from an empty counter. Remove all elements from the deque leaving it with length 0. + .. method:: copy() + + Create a shallow copy of the deque. + + .. versionadded:: 3.5 + + .. method:: count(x) Count the number of deque elements equal to *x*. @@ -457,6 +464,22 @@ or subtracting from an empty counter. elements in the iterable argument. + .. method:: index(x[, start[, stop]]) + + Return the position of *x* in the deque (at or after index *start* + and before index *stop*). Returns the first match or raises + :exc:`ValueError` if not found. + + .. versionadded:: 3.5 + + + .. method:: insert(i, x) + + Insert *x* into the deque at position *i*. + + .. versionadded:: 3.5 + + .. method:: pop() Remove and return an element from the right side of the deque. If no @@ -471,7 +494,7 @@ or subtracting from an empty counter. .. method:: remove(value) - Removed the first occurrence of *value*. If not found, raises a + Remove the first occurrence of *value*. If not found, raises a :exc:`ValueError`. @@ -504,6 +527,9 @@ the :keyword:`in` operator, and subscript references such as ``d[-1]``. Indexed access is O(1) at both ends but slows to O(n) in the middle. For fast random access, use lists instead. +Starting in version 3.5, deques support ``__add__()``, ``__mul__()``, +and ``__imul__()``. + Example: .. doctest:: @@ -899,6 +925,15 @@ create a new named tuple type from the :attr:`_fields` attribute: >>> Point3D = namedtuple('Point3D', Point._fields + ('z',)) +Docstrings can be customized by making direct assignments to the ``__doc__`` +fields: + + >>> Book = namedtuple('Book', ['id', 'title', 'authors']) + >>> Book.__doc__ = 'Hardcover book in active collection' + >>> Book.id.__doc__ = '13-digit ISBN' + >>> Book.title.__doc__ = 'Title of first printing' + >>> Book.author.__doc__ = 'List of authors sorted by last name' + Default values can be implemented by using :meth:`_replace` to customize a prototype instance: @@ -982,6 +1017,9 @@ The :class:`OrderedDict` constructor and :meth:`update` method both accept keyword arguments, but their order is lost because Python's function call semantics pass-in keyword arguments using a regular unordered dictionary. +.. versionchanged:: 3.5 + The items, keys, and values :term:`views <view>` of :class:`OrderedDict` now + support reverse iteration using :func:`reversed`. :class:`OrderedDict` Examples and Recipes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 9712de2..0325f1a 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -42,7 +42,8 @@ compile Python sources. .. cmdoption:: -q - Do not print the list of files compiled, print only error messages. + Do not print the list of files compiled. If passed once, error messages will + still be printed. If passed twice (``-qq``), all output is suppressed. .. cmdoption:: -d destdir @@ -70,9 +71,33 @@ compile Python sources. is to write files to their :pep:`3147` locations and names, which allows byte-code files from multiple versions of Python to coexist. +.. cmdoption:: -r + + Control the maximum recursion level for subdirectories. + If this is given, then ``-l`` option will not be taken into account. + :program:`python -m compileall <directory> -r 0` is equivalent to + :program:`python -m compileall <directory> -l`. + +.. cmdoption:: -j N + + Use *N* workers to compile the files within the given directory. + If ``0`` is used, then the result of :func:`os.cpu_count()` + will be used. + .. versionchanged:: 3.2 Added the ``-i``, ``-b`` and ``-h`` options. +.. versionchanged:: 3.5 + Added the ``-j`` and ``-r`` options. + +.. versionchanged:: 3.5 + ``-q`` option was changed to a multilevel value. + +.. versionchanged:: 3.5 + ``-b`` will always produce a byte-code file ending in ``.pyc``, never + ``.pyo``. + + There is no command-line option to control the optimization level used by the :func:`compile` function, because the Python interpreter itself already provides the option: :program:`python -O -m compileall`. @@ -80,7 +105,7 @@ provides the option: :program:`python -O -m compileall`. Public functions ---------------- -.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) +.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1) Recursively descend the directory tree named by *dir*, compiling all :file:`.py` files along the way. @@ -101,8 +126,9 @@ Public functions file considered for compilation, and if it returns a true value, the file is skipped. - If *quiet* is true, nothing is printed to the standard output unless errors - occur. + If *quiet* is ``False`` or ``0`` (the default), the filenames and other + information are printed to standard out. Set to ``1``, only errors are + printed. Set to ``2``, all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of @@ -113,11 +139,26 @@ Public functions *optimize* specifies the optimization level for the compiler. It is passed to the built-in :func:`compile` function. + The argument *workers* specifies how many workers are used to + compile files in parallel. The default is to not use multiple workers. + If the platform can't use multiple workers and *workers* argument is given, + then sequential compilation will be used as a fallback. If *workers* is + lower than ``0``, a :exc:`ValueError` will be raised. + .. versionchanged:: 3.2 Added the *legacy* and *optimize* parameter. + .. versionchanged:: 3.5 + Added the *workers* parameter. + + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. -.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. + +.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1) Compile the file with path *fullname*. @@ -131,8 +172,9 @@ Public functions file being compiled, and if it returns a true value, the file is not compiled and ``True`` is returned. - If *quiet* is true, nothing is printed to the standard output unless errors - occur. + If *quiet* is ``False`` or ``0`` (the default), the filenames and other + information are printed to standard out. Set to ``1``, only errors are + printed. Set to ``2``, all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of @@ -145,8 +187,14 @@ Public functions .. versionadded:: 3.2 + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. + + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. -.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False, optimize=-1) +.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1) Byte-compile all the :file:`.py` files found along ``sys.path``. If *skip_curdir* is true (the default), the current directory is not included @@ -157,6 +205,12 @@ Public functions .. versionchanged:: 3.2 Added the *legacy* and *optimize* parameter. + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. + + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. To force a recompile of all the :file:`.py` files in the :file:`Lib/` subdirectory and all its subdirectories:: diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index 48b4362..11b3916 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -38,7 +38,7 @@ Executor Objects future = executor.submit(pow, 323, 1235) print(future.result()) - .. method:: map(func, *iterables, timeout=None) + .. method:: map(func, *iterables, timeout=None, chunksize=1) Equivalent to :func:`map(func, *iterables) <map>` except *func* is executed asynchronously and several calls to *func* may be made concurrently. The @@ -48,7 +48,16 @@ Executor Objects *timeout* can be an int or a float. If *timeout* is not specified or ``None``, there is no limit to the wait time. If a call raises an exception, then that exception will be raised when its value is - retrieved from the iterator. + retrieved from the iterator. When using :class:`ProcessPoolExecutor`, this + method chops *iterables* into a number of chunks which it submits to the + pool as separate tasks. The (approximate) size of these chunks can be + specified by setting *chunksize* to a positive integer. For very long + iterables, using a large value for *chunksize* can significantly improve + performance compared to the default size of 1. With :class:`ThreadPoolExecutor`, + *chunksize* has no effect. + + .. versionchanged:: 3.5 + Added the *chunksize* argument. .. method:: shutdown(wait=True) @@ -115,11 +124,19 @@ And:: executor.submit(wait_on_future) -.. class:: ThreadPoolExecutor(max_workers) +.. class:: ThreadPoolExecutor(max_workers=None) An :class:`Executor` subclass that uses a pool of at most *max_workers* threads to execute calls asynchronously. + .. versionchanged:: 3.5 + If *max_workers* is ``None`` or + not given, it will default to the number of processors on the machine, + multiplied by ``5``, assuming that :class:`ThreadPoolExecutor` is often + used to overlap I/O instead of CPU work and the number of workers + should be higher than the number of workers + for :class:`ProcessPoolExecutor`. + .. _threadpoolexecutor-example: @@ -175,6 +192,8 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. An :class:`Executor` subclass that executes calls asynchronously using a pool of at most *max_workers* processes. If *max_workers* is ``None`` or not given, it will default to the number of processors on the machine. + If *max_workers* is lower or equal to ``0``, then a :exc:`ValueError` + will be raised. .. versionchanged:: 3.3 When one of the worker processes terminates abruptly, a diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst index 92551bc..c9187a3 100644 --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -11,6 +11,8 @@ .. sectionauthor:: Christopher G. Petrilli <petrilli@amber.org> .. sectionauthor:: Łukasz Langa <lukasz@langa.pl> +**Source code:** :source:`Lib/configparser.py` + .. index:: pair: .ini; file pair: configuration; file @@ -142,12 +144,13 @@ datatypes, you should convert on your own: >>> float(topsecret['CompressionLevel']) 9.0 -Extracting Boolean values is not that simple, though. Passing the value -to ``bool()`` would do no good since ``bool('False')`` is still -``True``. This is why config parsers also provide :meth:`getboolean`. -This method is case-insensitive and recognizes Boolean values from -``'yes'``/``'no'``, ``'on'``/``'off'`` and ``'1'``/``'0'`` [1]_. -For example: +Since this task is so common, config parsers provide a range of handy getter +methods to handle integers, floats and booleans. The last one is the most +interesting because simply passing the value to ``bool()`` would do no good +since ``bool('False')`` is still ``True``. This is why config parsers also +provide :meth:`getboolean`. This method is case-insensitive and recognizes +Boolean values from ``'yes'``/``'no'``, ``'on'``/``'off'``, +``'true'``/``'false'`` and ``'1'``/``'0'`` [1]_. For example: .. doctest:: @@ -159,10 +162,8 @@ For example: True Apart from :meth:`getboolean`, config parsers also provide equivalent -:meth:`getint` and :meth:`getfloat` methods, but these are far less -useful since conversion using :func:`int` and :func:`float` is -sufficient for these types. - +:meth:`getint` and :meth:`getfloat` methods. You can register your own +converters and customize the provided ones. [1]_ Fallback Values --------------- @@ -317,11 +318,11 @@ from ``get()`` calls. .. class:: ExtendedInterpolation() An alternative handler for interpolation which implements a more advanced - syntax, used for instance in ``zc.buildout``. Extended interpolation is + syntax, used for instance in ``zc.buildout``. Extended interpolation is using ``${section:option}`` to denote a value from a foreign section. - Interpolation can span multiple levels. For convenience, if the ``section:`` - part is omitted, interpolation defaults to the current section (and possibly - the default values from the special section). + Interpolation can span multiple levels. For convenience, if the + ``section:`` part is omitted, interpolation defaults to the current section + (and possibly the default values from the special section). For example, the configuration specified above with basic interpolation, would look like this with extended interpolation: @@ -399,13 +400,13 @@ However, there are a few differences that should be taken into account: * ``parser.popitem()`` never returns it. * ``parser.get(section, option, **kwargs)`` - the second argument is **not** - a fallback value. Note however that the section-level ``get()`` methods are + a fallback value. Note however that the section-level ``get()`` methods are compatible both with the mapping protocol and the classic configparser API. * ``parser.items()`` is compatible with the mapping protocol (returns a list of *section_name*, *section_proxy* pairs including the DEFAULTSECT). However, this method can also be invoked with arguments: ``parser.items(section, raw, - vars)``. The latter call returns a list of *option*, *value* pairs for + vars)``. The latter call returns a list of *option*, *value* pairs for a specified ``section``, with all interpolations expanded (unless ``raw=True`` is provided). @@ -539,9 +540,9 @@ the :meth:`__init__` options: * *delimiters*, default value: ``('=', ':')`` - Delimiters are substrings that delimit keys from values within a section. The - first occurrence of a delimiting substring on a line is considered a delimiter. - This means values (but not keys) can contain the delimiters. + Delimiters are substrings that delimit keys from values within a section. + The first occurrence of a delimiting substring on a line is considered + a delimiter. This means values (but not keys) can contain the delimiters. See also the *space_around_delimiters* argument to :meth:`ConfigParser.write`. @@ -553,7 +554,7 @@ the :meth:`__init__` options: Comment prefixes are strings that indicate the start of a valid comment within a config file. *comment_prefixes* are used only on otherwise empty lines (optionally indented) whereas *inline_comment_prefixes* can be used after - every valid value (e.g. section names, options and empty lines as well). By + every valid value (e.g. section names, options and empty lines as well). By default inline comments are disabled and ``'#'`` and ``';'`` are used as prefixes for whole line comments. @@ -563,10 +564,10 @@ the :meth:`__init__` options: Please note that config parsers don't support escaping of comment prefixes so using *inline_comment_prefixes* may prevent users from specifying option - values with characters used as comment prefixes. When in doubt, avoid setting - *inline_comment_prefixes*. In any circumstances, the only way of storing - comment prefix characters at the beginning of a line in multiline values is to - interpolate the prefix, for example:: + values with characters used as comment prefixes. When in doubt, avoid + setting *inline_comment_prefixes*. In any circumstances, the only way of + storing comment prefix characters at the beginning of a line in multiline + values is to interpolate the prefix, for example:: >>> from configparser import ConfigParser, ExtendedInterpolation >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) @@ -611,7 +612,7 @@ the :meth:`__init__` options: When set to ``True``, the parser will not allow for any section or option duplicates while reading from a single source (using :meth:`read_file`, - :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict + :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict parsers in new applications. .. versionchanged:: 3.2 @@ -646,12 +647,12 @@ the :meth:`__init__` options: The convention of allowing a special section of default values for other sections or interpolation purposes is a powerful concept of this library, - letting users create complex declarative configurations. This section is + letting users create complex declarative configurations. This section is normally called ``"DEFAULT"`` but this can be customized to point to any - other valid section name. Some typical values include: ``"general"`` or - ``"common"``. The name provided is used for recognizing default sections when - reading from any source and is used when writing configuration back to - a file. Its current value can be retrieved using the + other valid section name. Some typical values include: ``"general"`` or + ``"common"``. The name provided is used for recognizing default sections + when reading from any source and is used when writing configuration back to + a file. Its current value can be retrieved using the ``parser_instance.default_section`` attribute and may be modified at runtime (i.e. to convert files from one format to another). @@ -660,14 +661,30 @@ the :meth:`__init__` options: Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. ``None`` can be used to turn off interpolation completely, ``ExtendedInterpolation()`` provides a more - advanced variant inspired by ``zc.buildout``. More on the subject in the + advanced variant inspired by ``zc.buildout``. More on the subject in the `dedicated documentation section <#interpolation-of-values>`_. :class:`RawConfigParser` has a default value of ``None``. +* *converters*, default value: not set + + Config parsers provide option value getters that perform type conversion. By + default :meth:`getint`, :meth:`getfloat`, and :meth:`getboolean` are + implemented. Should other getters be desirable, users may define them in + a subclass or pass a dictionary where each key is a name of the converter and + each value is a callable implementing said conversion. For instance, passing + ``{'decimal': decimal.Decimal}`` would add :meth:`getdecimal` on both the + parser object and all section proxies. In other words, it will be possible + to write both ``parser_instance.getdecimal('section', 'key', fallback=0)`` + and ``parser_instance['section'].getdecimal('key', 0)``. + + If the converter needs to access the state of the parser, it can be + implemented as a method on a config parser subclass. If the name of this + method starts with ``get``, it will be available on all section proxies, in + the dict-compatible form (see the ``getdecimal()`` example above). More advanced customization may be achieved by overriding default values of -these parser attributes. The defaults are defined on the classes, so they -may be overridden by subclasses or by attribute assignment. +these parser attributes. The defaults are defined on the classes, so they may +be overridden by subclasses or by attribute assignment. .. attribute:: BOOLEAN_STATES @@ -725,10 +742,11 @@ may be overridden by subclasses or by attribute assignment. .. attribute:: SECTCRE - A compiled regular expression used to parse section headers. The default - matches ``[section]`` to the name ``"section"``. Whitespace is considered part - of the section name, thus ``[ larch ]`` will be read as a section of name - ``" larch "``. Override this attribute if that's unsuitable. For example: + A compiled regular expression used to parse section headers. The default + matches ``[section]`` to the name ``"section"``. Whitespace is considered + part of the section name, thus ``[ larch ]`` will be read as a section of + name ``" larch "``. Override this attribute if that's unsuitable. For + example: .. doctest:: @@ -859,7 +877,7 @@ interpolation if an option used is not defined elsewhere. :: ConfigParser Objects -------------------- -.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation()) +.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation(), converters={}) The main configuration parser. When *defaults* is given, it is initialized into the dictionary of intrinsic defaults. When *dict_type* is given, it @@ -869,8 +887,8 @@ ConfigParser Objects When *delimiters* is given, it is used as the set of substrings that divide keys from values. When *comment_prefixes* is given, it will be used as the set of substrings that prefix comments in otherwise empty lines. - Comments can be indented. When *inline_comment_prefixes* is given, it will be - used as the set of substrings that prefix comments in non-empty lines. + Comments can be indented. When *inline_comment_prefixes* is given, it will + be used as the set of substrings that prefix comments in non-empty lines. When *strict* is ``True`` (the default), the parser won't allow for any section or option duplicates while reading from a single source (file, @@ -884,13 +902,13 @@ ConfigParser Objects When *default_section* is given, it specifies the name for the special section holding default values for other sections and interpolation purposes - (normally named ``"DEFAULT"``). This value can be retrieved and changed on + (normally named ``"DEFAULT"``). This value can be retrieved and changed on runtime using the ``default_section`` instance attribute. Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. ``None`` can be used to turn off interpolation completely, ``ExtendedInterpolation()`` provides a more - advanced variant inspired by ``zc.buildout``. More on the subject in the + advanced variant inspired by ``zc.buildout``. More on the subject in the `dedicated documentation section <#interpolation-of-values>`_. All option names used in interpolation will be passed through the @@ -899,6 +917,12 @@ ConfigParser Objects converts option names to lower case), the values ``foo %(bar)s`` and ``foo %(BAR)s`` are equivalent. + When *converters* is given, it should be a dictionary where each key + represents the name of a type converter and each value is a callable + implementing the conversion from string to the desired datatype. Every + converter gets its own corresponding :meth:`get*()` method on the parser + object and section proxies. + .. versionchanged:: 3.1 The default *dict_type* is :class:`collections.OrderedDict`. @@ -907,6 +931,9 @@ ConfigParser Objects *empty_lines_in_values*, *default_section* and *interpolation* were added. + .. versionchanged:: 3.5 + The *converters* argument was added. + .. method:: defaults() @@ -944,7 +971,7 @@ ConfigParser Objects .. method:: has_option(section, option) If the given *section* exists, and contains the given *option*, return - :const:`True`; otherwise return :const:`False`. If the specified + :const:`True`; otherwise return :const:`False`. If the specified *section* is :const:`None` or an empty string, DEFAULT is assumed. @@ -1069,7 +1096,7 @@ ConfigParser Objects :meth:`get` method. .. versionchanged:: 3.2 - Items present in *vars* no longer appear in the result. The previous + Items present in *vars* no longer appear in the result. The previous behaviour mixed actual parser options with variables provided for interpolation. @@ -1170,7 +1197,7 @@ RawConfigParser Objects .. note:: Consider using :class:`ConfigParser` instead which checks types of - the values to be stored internally. If you don't want interpolation, you + the values to be stored internally. If you don't want interpolation, you can use ``ConfigParser(interpolation=None)``. @@ -1181,7 +1208,7 @@ RawConfigParser Objects *default section* name is passed, :exc:`ValueError` is raised. Type of *section* is not checked which lets users create non-string named - sections. This behaviour is unsupported and may cause internal errors. + sections. This behaviour is unsupported and may cause internal errors. .. method:: set(section, option, value) @@ -1282,3 +1309,4 @@ Exceptions .. [1] Config parsers allow for heavy customization. If you are interested in changing the behaviour outlined by the footnote reference, consult the `Customizing Parser Behaviour`_ section. + diff --git a/Doc/library/constants.rst b/Doc/library/constants.rst index 42b5af2..d5a0f09 100644 --- a/Doc/library/constants.rst +++ b/Doc/library/constants.rst @@ -45,7 +45,6 @@ A small number of constants live in the built-in namespace. They are: for more details. - .. data:: Ellipsis The same as ``...``. Special value used mostly in conjunction with extended diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index 6f36864..550b347 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -172,6 +172,16 @@ Functions and classes provided: .. versionadded:: 3.4 +.. function:: redirect_stderr(new_target) + + Similar to :func:`~contextlib.redirect_stdout` but redirecting + :data:`sys.stderr` to another file or file-like object. + + This context manager is :ref:`reentrant <reentrant-cms>`. + + .. versionadded:: 3.5 + + .. class:: ContextDecorator() A base class that enables a context manager to also be used as a decorator. diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index 313dce0..602e909 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -5,6 +5,7 @@ :synopsis: Write and read tabular data to and from delimited files. .. sectionauthor:: Skip Montanaro <skip@pobox.com> +**Source code:** :source:`Lib/csv.py` .. index:: single: csv @@ -418,7 +419,7 @@ Writer Objects :class:`Writer` objects (:class:`DictWriter` instances and objects returned by the :func:`writer` function) have the following public methods. A *row* must be -a sequence of strings or numbers for :class:`Writer` objects and a dictionary +an iterable of strings or numbers for :class:`Writer` objects and a dictionary mapping fieldnames to strings or numbers (by passing them through :func:`str` first) for :class:`DictWriter` objects. Note that complex numbers are written out surrounded by parens. This may cause some problems for other programs which @@ -430,6 +431,8 @@ read CSV files (assuming they support complex numbers at all). Write the *row* parameter to the writer's file object, formatted according to the current dialect. + .. versionchanged:: 3.5 + Added support of arbitrary iterables. .. method:: csvwriter.writerows(rows) diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index f3e60b4..e8dfd83 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -599,6 +599,13 @@ The module :mod:`curses` defines the following functions: Only one *ch* can be pushed before :meth:`getch` is called. +.. function:: update_lines_cols() + + Update :envvar:`LINES` and :envvar:`COLS`. Useful for detecting manual screen resize. + + .. versionadded:: 3.5 + + .. function:: unget_wch(ch) Push *ch* so the next :meth:`get_wch` will return it. diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 88f3f6e..976cd49 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -7,6 +7,8 @@ .. sectionauthor:: Tim Peters <tim@zope.com> .. sectionauthor:: A.M. Kuchling <amk@amk.ca> +**Source code:** :source:`Lib/datetime.py` + .. XXX what order should the types be discussed in? The :mod:`datetime` module supplies classes for manipulating dates and times in @@ -757,13 +759,19 @@ Other constructors, all class methods: :attr:`tzinfo` ``None``. This may raise :exc:`OverflowError`, if the timestamp is out of the range of values supported by the platform C :c:func:`gmtime` function, and :exc:`OSError` on :c:func:`gmtime` failure. - It's common for this to be restricted to years in 1970 through 2038. See also - :meth:`fromtimestamp`. + It's common for this to be restricted to years in 1970 through 2038. + + To get an aware :class:`.datetime` object, call :meth:`fromtimestamp`:: + + datetime.fromtimestamp(timestamp, timezone.utc) + + On the POSIX compliant platforms, it is equivalent to the following + expression:: - On the POSIX compliant platforms, ``utcfromtimestamp(timestamp)`` - is equivalent to the following expression:: + datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(seconds=timestamp) - datetime(1970, 1, 1) + timedelta(seconds=timestamp) + except the latter formula always supports the full years range: between + :const:`MINYEAR` and :const:`MAXYEAR` inclusive. .. versionchanged:: 3.3 Raise :exc:`OverflowError` instead of :exc:`ValueError` if the timestamp @@ -1376,10 +1384,13 @@ Supported operations: * efficient pickling -* in Boolean contexts, a :class:`.time` object is considered to be true if and - only if, after converting it to minutes and subtracting :meth:`utcoffset` (or - ``0`` if that's ``None``), the result is non-zero. +In boolean contexts, a :class:`.time` object is always considered to be true. +.. versionchanged:: 3.5 + Before Python 3.5, a :class:`.time` object was considered to be false if it + represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. Instance methods: diff --git a/Doc/library/dbm.rst b/Doc/library/dbm.rst index e6a82d6..3f3c43d 100644 --- a/Doc/library/dbm.rst +++ b/Doc/library/dbm.rst @@ -325,13 +325,18 @@ The module defines the following: dumbdbm database is created, files with :file:`.dat` and :file:`.dir` extensions are created. - The optional *flag* argument is currently ignored; the database is always opened - for update, and will be created if it does not exist. + The optional *flag* argument supports only the semantics of ``'c'`` + and ``'n'`` values. Other values will default to database being always + opened for update, and will be created if it does not exist. The optional *mode* argument is the Unix mode of the file, used only when the database has to be created. It defaults to octal ``0o666`` (and will be modified by the prevailing umask). + .. versionchanged:: 3.5 + :func:`.open` always creates a new database when the flag has the value + ``'n'``. + In addition to the methods provided by the :class:`collections.abc.MutableMapping` class, :class:`dumbdbm` objects provide the following methods: diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index 759be70..5d7ffff 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -12,6 +12,8 @@ .. moduleauthor:: Stefan Krah <skrah at bytereef.org> .. sectionauthor:: Raymond D. Hettinger <python at rcn.com> +**Source code:** :source:`Lib/decimal.py` + .. import modules for testing inline doctests with the Sphinx doctest builder .. testsetup:: * @@ -742,7 +744,7 @@ Decimal objects * ``"NaN"``, indicating that the operand is a quiet NaN (Not a Number). * ``"sNaN"``, indicating that the operand is a signaling NaN. - .. method:: quantize(exp, rounding=None, context=None, watchexp=True) + .. method:: quantize(exp, rounding=None, context=None) Return a value equal to the first operand after rounding and having the exponent of the second operand. @@ -765,14 +767,8 @@ Decimal objects ``context`` argument; if neither argument is given the rounding mode of the current thread's context is used. - If *watchexp* is set (default), then an error is returned whenever the - resulting exponent is greater than :attr:`Emax` or less than - :attr:`Etiny`. - - .. deprecated:: 3.3 - *watchexp* is an implementation detail from the pure Python version - and is not present in the C version. It will be removed in version - 3.4, where it defaults to ``True``. + An error is returned whenever the resulting exponent is greater than + :attr:`Emax` or less than :attr:`Etiny`. .. method:: radix() @@ -2092,4 +2088,3 @@ Alternatively, inputs can be rounded upon creation using the >>> Context(prec=5, rounding=ROUND_DOWN).create_decimal('1.2345678') Decimal('1.2345') - diff --git a/Doc/library/development.rst b/Doc/library/development.rst index 06e7048..d2b5fa2 100644 --- a/Doc/library/development.rst +++ b/Doc/library/development.rst @@ -16,6 +16,7 @@ The list of modules described in this chapter is: .. toctree:: + typing.rst pydoc.rst doctest.rst unittest.rst diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index 5f72ea6..efaac7a 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -7,6 +7,8 @@ .. sectionauthor:: Tim Peters <tim_one@users.sourceforge.net> .. Markup by Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/difflib.py` + .. testsetup:: import sys @@ -25,7 +27,9 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. little fancier than, an algorithm published in the late 1980's by Ratcliff and Obershelp under the hyperbolic name "gestalt pattern matching." The idea is to find the longest contiguous matching subsequence that contains no "junk" - elements (the Ratcliff and Obershelp algorithm doesn't address junk). The same + elements; these "junk" elements are ones that are uninteresting in some + sense, such as blank lines or whitespace. (Handling junk is an + extension to the Ratcliff and Obershelp algorithm.) The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that "look right" to people. @@ -100,7 +104,8 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The following methods are public: - .. method:: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5) + .. method:: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, \ + numlines=5, *, charset='utf-8') Compares *fromlines* and *tolines* (lists of strings) and returns a string which is a complete HTML file containing a table showing line by line differences with @@ -119,6 +124,10 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. the next difference highlight at the top of the browser without any leading context). + .. versionchanged:: 3.5 + *charset* keyword-only argument was added. The default charset of + HTML document changed from ``'ISO-8859-1'`` to ``'utf-8'``. + .. method:: make_table(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5) Compares *fromlines* and *tolines* (lists of strings) and returns a string which @@ -208,7 +217,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. Compare *a* and *b* (lists of strings); return a :class:`Differ`\ -style delta (a :term:`generator` generating the delta lines). - Optional keyword parameters *linejunk* and *charjunk* are for filter functions + Optional keyword parameters *linejunk* and *charjunk* are filtering functions (or ``None``): *linejunk*: A function that accepts a single string argument, and returns @@ -222,7 +231,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. *charjunk*: A function that accepts a character (a string of length 1), and returns if the character is junk, or false if not. The default is module-level function :func:`IS_CHARACTER_JUNK`, which filters out whitespace characters (a - blank or tab; note: bad idea to include newline in this!). + blank or tab; it's a bad idea to include newline in this!). :file:`Tools/scripts/ndiff.py` is a command-line front-end to this function. @@ -306,6 +315,21 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. See :ref:`difflib-interface` for a more detailed example. +.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\\n') + + Compare *a* and *b* (lists of bytes objects) using *dfunc*; yield a + sequence of delta lines (also bytes) in the format returned by *dfunc*. + *dfunc* must be a callable, typically either :func:`unified_diff` or + :func:`context_diff`. + + Allows you to compare data with unknown or inconsistent encoding. All + inputs except *n* must be bytes objects, not str. Works by losslessly + converting all inputs (except *n*) to str, and calling ``dfunc(a, b, + fromfile, tofile, fromfiledate, tofiledate, n, lineterm)``. The output of + *dfunc* is then converted back to bytes, so the delta lines that you + receive have the same unknown/inconsistent encodings as *a* and *b*. + + .. versionadded:: 3.5 .. function:: IS_LINE_JUNK(line) @@ -622,6 +646,12 @@ The :class:`Differ` class has this constructor: length 1), and returns true if the character is junk. The default is ``None``, meaning that no character is considered junk. + These junk-filtering functions speed up matching to find + differences and do not cause any differing lines or characters to + be ignored. Read the description of the + :meth:`~SequenceMatcher.find_longest_match` method's *isjunk* + parameter for an explanation. + :class:`Differ` objects are used (deltas generated) via a single method: diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index b816dcc..7a214ed 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -48,8 +48,9 @@ code. .. class:: Bytecode(x, *, first_line=None, current_offset=None) - Analyse the bytecode corresponding to a function, method, string of source - code, or a code object (as returned by :func:`compile`). + + Analyse the bytecode corresponding to a function, generator, method, string + of source code, or a code object (as returned by :func:`compile`). This is a convenience wrapper around many of the functions listed below, most notably :func:`get_instructions`, as iterating over a :class:`Bytecode` @@ -109,7 +110,7 @@ operation is being performed, so the intermediate analysis object isn't useful: .. function:: code_info(x) Return a formatted multi-line string with detailed code object information - for the supplied function, method, source code string or code object. + for the supplied function, generator, method, source code string or code object. Note that the exact contents of code info strings are highly implementation dependent and they may change arbitrarily across Python VMs or Python @@ -136,11 +137,11 @@ operation is being performed, so the intermediate analysis object isn't useful: .. function:: dis(x=None, *, file=None) Disassemble the *x* object. *x* can denote either a module, a class, a - method, a function, a code object, a string of source code or a byte sequence - of raw bytecode. For a module, it disassembles all functions. For a class, - it disassembles all methods. For a code object or sequence of raw bytecode, - it prints one line per bytecode instruction. Strings are first compiled to - code objects with the :func:`compile` built-in function before being + method, a function, a generator, a code object, a string of source code or + a byte sequence of raw bytecode. For a module, it disassembles all functions. + For a class, it disassembles all methods. For a code object or sequence of + raw bytecode, it prints one line per bytecode instruction. Strings are first + compiled to code objects with the :func:`compile` built-in function before being disassembled. If no object is provided, this function disassembles the last traceback. @@ -345,6 +346,14 @@ result back on the stack. Implements ``TOS = iter(TOS)``. +.. opcode:: GET_YIELD_FROM_ITER + + If ``TOS`` is a :term:`generator iterator` or :term:`coroutine` object + it is left as is. Otherwise, implements ``TOS = iter(TOS)``. + + .. versionadded:: 3.5 + + **Binary operations** Binary operations remove the top of the stack (TOS) and the second top-most @@ -361,6 +370,13 @@ result back on the stack. Implements ``TOS = TOS1 * TOS``. +.. opcode:: BINARY_MATRIX_MULTIPLY + + Implements ``TOS = TOS1 @ TOS``. + + .. versionadded:: 3.5 + + .. opcode:: BINARY_FLOOR_DIVIDE Implements ``TOS = TOS1 // TOS``. @@ -433,6 +449,13 @@ the original TOS1. Implements in-place ``TOS = TOS1 * TOS``. +.. opcode:: INPLACE_MATRIX_MULTIPLY + + Implements in-place ``TOS = TOS1 @ TOS``. + + .. versionadded:: 3.5 + + .. opcode:: INPLACE_FLOOR_DIVIDE Implements in-place ``TOS = TOS1 // TOS``. @@ -493,6 +516,40 @@ the original TOS1. Implements ``del TOS1[TOS]``. +**Coroutine opcodes** + +.. opcode:: GET_AWAITABLE + + Implements ``TOS = get_awaitable(TOS)``, where ``get_awaitable(o)`` + returns ``o`` if ``o`` is a coroutine object or a generator object with + the CO_ITERABLE_COROUTINE flag, or resolves + ``o.__await__``. + + +.. opcode:: GET_AITER + + Implements ``TOS = get_awaitable(TOS.__aiter__())``. See ``GET_AWAITABLE`` + for details about ``get_awaitable`` + + +.. opcode:: GET_ANEXT + + Implements ``PUSH(get_awaitable(TOS.__anext__()))``. See ``GET_AWAITABLE`` + for details about ``get_awaitable`` + + +.. opcode:: BEFORE_ASYNC_WITH + + Resolves ``__aenter__`` and ``__aexit__`` from the object on top of the + stack. Pushes ``__aexit__`` and result of ``__aenter__()`` to the stack. + + +.. opcode:: SETUP_ASYNC_WITH + + Creates a new frame object. + + + **Miscellaneous opcodes** .. opcode:: PRINT_EXPR @@ -597,7 +654,7 @@ iterations of the loop. :opcode:`UNPACK_SEQUENCE`). -.. opcode:: WITH_CLEANUP +.. opcode:: WITH_CLEANUP_START Cleans up the stack when a :keyword:`with` statement block exits. TOS is the context manager's :meth:`__exit__` bound method. Below TOS are 1--3 values @@ -609,7 +666,13 @@ iterations of the loop. * (SECOND, THIRD, FOURTH) = exc_info() In the last case, ``TOS(SECOND, THIRD, FOURTH)`` is called, otherwise - ``TOS(None, None, None)``. In addition, TOS is removed from the stack. + ``TOS(None, None, None)``. Pushes SECOND and result of the call + to the stack. + + +.. opcode:: WITH_CLEANUP_FINISH + + Pops exception type and result of 'exit' function call from the stack. If the stack represents an exception, *and* the function call returns a 'true' value, this information is "zapped" and replaced with a single @@ -795,10 +858,6 @@ the more significant byte last. Pushes a try block from a try-except clause onto the block stack. *delta* points to the finally block. -.. opcode:: STORE_MAP - - Store a key and value pair in a dictionary. Pops the key and value while - leaving the dictionary on the stack. .. opcode:: LOAD_FAST (var_num) diff --git a/Doc/library/distribution.rst b/Doc/library/distribution.rst index c4954d1..3e6e84b 100644 --- a/Doc/library/distribution.rst +++ b/Doc/library/distribution.rst @@ -12,3 +12,4 @@ with a local index server, or without any index server at all. distutils.rst ensurepip.rst venv.rst + zipapp.rst diff --git a/Doc/library/doctest.rst b/Doc/library/doctest.rst index 2e372de..9aa9ea6 100644 --- a/Doc/library/doctest.rst +++ b/Doc/library/doctest.rst @@ -1058,15 +1058,9 @@ from text files and modules with doctests: This function uses the same search technique as :func:`testmod`. - .. note:: - Unlike :func:`testmod` and :class:`DocTestFinder`, this function raises - a :exc:`ValueError` if *module* contains no docstrings. You can prevent - this error by passing a :class:`DocTestFinder` instance as the - *test_finder* argument with its *exclude_empty* keyword argument set - to ``False``:: - - >>> finder = doctest.DocTestFinder(exclude_empty=False) - >>> suite = doctest.DocTestSuite(test_finder=finder) + .. versionchanged:: 3.5 + :func:`DocTestSuite` returns an empty :class:`unittest.TestSuite` if *module* + contains no docstrings instead of raising :exc:`ValueError`. Under the covers, :func:`DocTestSuite` creates a :class:`unittest.TestSuite` out diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst index aeea942..b91f26d 100644 --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -578,6 +578,15 @@ Here are the methods of the :class:`Message` class: will be *failobj*. + .. method:: get_content_disposition() + + Return the lowercased value (without parameters) of the message's + :mailheader:`Content-Disposition` header if it has one, or ``None``. The + possible values for this method are *inline*, *attachment* or ``None`` + if the message follows :rfc:`2183`. + + .. versionadded:: 3.5 + .. method:: walk() The :meth:`walk` method is an all-purpose generator which can be used to diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst index 1d70225..67d0a67 100644 --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -195,7 +195,8 @@ Here are the classes: set of the text and is passed as an argument to the :class:`~email.mime.nonmultipart.MIMENonMultipart` constructor; it defaults to ``us-ascii`` if the string contains only ``ascii`` code points, and - ``utf-8`` otherwise. + ``utf-8`` otherwise. The *_charset* parameter accepts either a string or a + :class:`~email.charset.Charset` instance. Unless the *_charset* argument is explicitly set to ``None``, the MIMEText object created will have both a :mailheader:`Content-Type` header @@ -206,3 +207,6 @@ Here are the classes: ``Content-Transfer-Encoding`` header, after which a ``set_payload`` call will automatically encode the new payload (and add a new :mailheader:`Content-Transfer-Encoding` header). + + .. versionchanged:: 3.5 + *_charset* also accepts :class:`~email.charset.Charset` instances. diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst index d4e3fc1..045b119 100644 --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -187,6 +187,18 @@ added matters. To illustrate:: :const:`False` (the default), defects will be passed to the :meth:`register_defect` method. + + + .. attribute:: mangle_from\_ + + If :const:`True`, lines starting with *"From "* in the body are + escaped by putting a ``>`` in front of them. This parameter is used when + the message is being serialized by a generator. + Default: :const:`False`. + + .. versionadded:: 3.5 + The *mangle_from_* parameter. + The following :class:`Policy` method is intended to be called by code using the email library to create policy instances with custom settings: @@ -319,6 +331,13 @@ added matters. To illustrate:: :const:`compat32`, that is used as the default policy. Thus the default behavior of the email package is to maintain compatibility with Python 3.2. + The following attributes have values that are different from the + :class:`Policy` default: + + .. attribute:: mangle_from_ + + The default is ``True``. + The class provides the following concrete implementations of the abstract methods of :class:`Policy`: @@ -356,6 +375,14 @@ added matters. To illustrate:: line breaks and any (RFC invalid) binary data it may contain. +An instance of :class:`Compat32` is provided as a module constant: + +.. data:: compat32 + + An instance of :class:`Compat32`, providing backward compatibility with the + behavior of the email package in Python 3.2. + + .. note:: The documentation below describes new policies that are included in the @@ -378,6 +405,14 @@ added matters. To illustrate:: In addition to the settable attributes listed above that apply to all policies, this policy adds the following additional attributes: + .. attribute:: utf8 + + If ``False``, follow :rfc:`5322`, supporting non-ASCII characters in + headers by encoding them as "encoded words". If ``True``, follow + :rfc:`6532` and use ``utf-8`` encoding for headers. Messages + formatted in this way may be passed to SMTP servers that support + the ``SMTPUTF8`` extension (:rfc:`6531`). + .. attribute:: refold_source If the value for a header in the ``Message`` object originated from a @@ -499,6 +534,14 @@ more closely to the RFCs relevant to their domains. Like ``default``, but with ``linesep`` set to ``\r\n``, which is RFC compliant. +.. data:: SMTPUTF8 + + The same as ``SMTP`` except that :attr:`~EmailPolicy.utf8` is ``True``. + Useful for serializing messages to a message store without using encoded + words in the headers. Should only be used for SMTP trasmission if the + sender or recipient addresses have non-ASCII characters (the + :meth:`smtplib.SMTP.send_message` method handles this automatically). + .. data:: HTTP Suitable for serializing headers with for use in HTTP traffic. Like diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index cf09559..d3b838c 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -314,11 +314,11 @@ Then:: >>> str(Mood.funky) 'my custom str! 1' -The rules for what is allowed are as follows: _sunder_ names (starting and -ending with a single underscore) are reserved by enum and cannot be used; -all other attributes defined within an enumeration will become members of this -enumeration, with the exception of *__dunder__* names and descriptors (methods -are also descriptors). +The rules for what is allowed are as follows: names that start and end with a +with a single underscore are reserved by enum and cannot be used; all other +attributes defined within an enumeration will become members of this +enumeration, with the exception of special methods (:meth:`__str__`, +:meth:`__add__`, etc.) and descriptors (methods are also descriptors). Note: if your enumeration defines :meth:`__new__` and/or :meth:`__init__` then whatever value(s) were given to the enum member will be passed into those @@ -400,7 +400,8 @@ The second argument is the *source* of enumeration member names. It can be a whitespace-separated string of names, a sequence of names, a sequence of 2-tuples with key/value pairs, or a mapping (e.g. dictionary) of names to values. The last two options enable assigning arbitrary values to -enumerations; the others auto-assign increasing integers starting with 1. A +enumerations; the others auto-assign increasing integers starting with 1 (use +the ``start`` parameter to specify a different starting value). A new class derived from :class:`Enum` is returned. In other words, the above assignment to :class:`Animal` is equivalent to:: @@ -438,12 +439,12 @@ SomeData in the global scope:: The complete signature is:: - Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=<mixed-in class>) + Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=<mixed-in class>, start=1) :value: What the new Enum class will record as its name. :names: The Enum members. This can be a whitespace or comma separated string - (values will start at 1):: + (values will start at 1 unless otherwise specified):: 'red green blue' | 'red,green,blue' | 'red, green, blue' @@ -465,6 +466,11 @@ The complete signature is:: :type: type to mix in to new Enum class. +:start: number to start counting at if only names are passed in + +.. versionchanged:: 3.5 + The *start* parameter was added. + Derived Enumerations -------------------- diff --git a/Doc/library/errno.rst b/Doc/library/errno.rst index d2163b6..22a5cbc 100644 --- a/Doc/library/errno.rst +++ b/Doc/library/errno.rst @@ -41,7 +41,10 @@ defined by the module. The specific list of defined symbols is available as .. data:: EINTR - Interrupted system call + Interrupted system call. + + .. seealso:: + This error is mapped to the exception :exc:`InterruptedError`. .. data:: EIO diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 271a5c8..1a9d029 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -162,7 +162,8 @@ The following exceptions are the exceptions that are usually raised. .. exception:: GeneratorExit - Raised when a :term:`generator`\'s :meth:`close` method is called. It + Raised when a :term:`generator` or :term:`coroutine` is closed; + see :meth:`generator.close` and :meth:`coroutine.close`. It directly inherits from :exc:`BaseException` instead of :exc:`Exception` since it is technically not an error. @@ -306,14 +307,30 @@ The following exceptions are the exceptions that are usually raised. given as an argument when constructing the exception, and defaults to :const:`None`. - When a generator function returns, a new :exc:`StopIteration` instance is + When a :term:`generator` or :term:`coroutine` function + returns, a new :exc:`StopIteration` instance is raised, and the value returned by the function is used as the :attr:`value` parameter to the constructor of the exception. + If a generator function defined in the presence of a ``from __future__ + import generator_stop`` directive raises :exc:`StopIteration`, it will be + converted into a :exc:`RuntimeError` (retaining the :exc:`StopIteration` + as the new exception's cause). + .. versionchanged:: 3.3 Added ``value`` attribute and the ability for generator functions to use it to return a value. + .. versionchanged:: 3.5 + Introduced the RuntimeError transformation. + +.. exception:: StopAsyncIteration + + Must be raised by :meth:`__anext__` method of an + :term:`asynchronous iterator` object to stop the iteration. + + .. versionadded:: 3.5 + .. exception:: SyntaxError Raised when the parser encounters a syntax error. This may occur in an @@ -536,7 +553,12 @@ depending on the system error code. .. exception:: InterruptedError Raised when a system call is interrupted by an incoming signal. - Corresponds to :c:data:`errno` ``EINTR``. + Corresponds to :c:data:`errno` :py:data:`~errno.EINTR`. + + .. versionchanged:: 3.5 + Python now retries system calls when a syscall is interrupted by a + signal, except if the signal handler raises an exception (see :pep:`475` + for the rationale), instead of raising :exc:`InterruptedError`. .. exception:: IsADirectoryError diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst index eb2016a..3a5badd 100644 --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -47,6 +47,9 @@ Dumping the traceback Dump the tracebacks of all threads into *file*. If *all_threads* is ``False``, dump only the current thread. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + Fault handler state ------------------- @@ -59,6 +62,12 @@ Fault handler state produce tracebacks for every running thread. Otherwise, dump only the current thread. + The *file* must be kept open until the fault handler is disabled: see + :ref:`issue with file descriptors <faulthandler-fd>`. + + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: disable() Disable the fault handler: uninstall the signal handlers installed by @@ -82,9 +91,16 @@ Dumping the tracebacks after a timeout call replaces previous parameters and resets the timeout. The timer has a sub-second resolution. + The *file* must be kept open until the traceback is dumped or + :func:`cancel_dump_traceback_later` is called: see :ref:`issue with file + descriptors <faulthandler-fd>`. + This function is implemented using a watchdog thread and therefore is not available if Python is compiled with threads disabled. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: cancel_dump_traceback_later() Cancel the last call to :func:`dump_traceback_later`. @@ -99,8 +115,14 @@ Dumping the traceback on a user signal the traceback of all threads, or of the current thread if *all_threads* is ``False``, into *file*. Call the previous handler if chain is ``True``. + The *file* must be kept open until the signal is unregistered by + :func:`unregister`: see :ref:`issue with file descriptors <faulthandler-fd>`. + Not available on Windows. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: unregister(signum) Unregister a user signal: uninstall the handler of the *signum* signal @@ -110,6 +132,8 @@ Dumping the traceback on a user signal Not available on Windows. +.. _faulthandler-fd: + Issue with file descriptors --------------------------- diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index 8e932fb..432140f 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -28,41 +28,41 @@ descriptor. The module defines the following functions: -.. function:: fcntl(fd, op[, arg]) +.. function:: fcntl(fd, cmd, arg=0) - Perform the operation *op* on file descriptor *fd* (file objects providing + Perform the operation *cmd* on file descriptor *fd* (file objects providing a :meth:`~io.IOBase.fileno` method are accepted as well). The values used - for *op* are operating system dependent, and are available as constants + for *cmd* are operating system dependent, and are available as constants in the :mod:`fcntl` module, using the same names as used in the relevant C - header files. The argument *arg* is optional, and defaults to the integer - value ``0``. When present, it can either be an integer value, or a string. - With the argument missing or an integer value, the return value of this function - is the integer return value of the C :c:func:`fcntl` call. When the argument is - a string it represents a binary structure, e.g. created by :func:`struct.pack`. - The binary data is copied to a buffer whose address is passed to the C - :c:func:`fcntl` call. The return value after a successful call is the contents - of the buffer, converted to a string object. The length of the returned string - will be the same as the length of the *arg* argument. This is limited to 1024 - bytes. If the information returned in the buffer by the operating system is - larger than 1024 bytes, this is most likely to result in a segmentation - violation or a more subtle data corruption. + header files. The argument *arg* can either be an integer value, or a + :class:`bytes` object. With an integer value, the return value of this + function is the integer return value of the C :c:func:`fcntl` call. When + the argument is bytes it represents a binary structure, e.g. created by + :func:`struct.pack`. The binary data is copied to a buffer whose address is + passed to the C :c:func:`fcntl` call. The return value after a successful + call is the contents of the buffer, converted to a :class:`bytes` object. + The length of the returned object will be the same as the length of the + *arg* argument. This is limited to 1024 bytes. If the information returned + in the buffer by the operating system is larger than 1024 bytes, this is + most likely to result in a segmentation violation or a more subtle data + corruption. If the :c:func:`fcntl` fails, an :exc:`OSError` is raised. -.. function:: ioctl(fd, op[, arg[, mutate_flag]]) +.. function:: ioctl(fd, request, arg=0, mutate_flag=True) This function is identical to the :func:`~fcntl.fcntl` function, except that the argument handling is even more complicated. - The op parameter is limited to values that can fit in 32-bits. - Additional constants of interest for use as the *op* argument can be + The *request* parameter is limited to values that can fit in 32-bits. + Additional constants of interest for use as the *request* argument can be found in the :mod:`termios` module, under the same names as used in the relevant C header files. - The parameter *arg* can be one of an integer, absent (treated identically to the - integer ``0``), an object supporting the read-only buffer interface (most likely - a plain Python string) or an object supporting the read-write buffer interface. + The parameter *arg* can be one of an integer, an object supporting the + read-only buffer interface (like :class:`bytes`) or an object supporting + the read-write buffer interface (like :class:`bytearray`). In all but the last case, behaviour is as for the :func:`~fcntl.fcntl` function. @@ -72,7 +72,7 @@ The module defines the following functions: If it is false, the buffer's mutability is ignored and behaviour is as for a read-only buffer, except that the 1024 byte limit mentioned above is avoided -- - so long as the buffer you pass is as least as long as what the operating system + so long as the buffer you pass is at least as long as what the operating system wants to put there, things should work. If *mutate_flag* is true (the default), then the buffer is (in effect) passed @@ -97,25 +97,25 @@ The module defines the following functions: array('h', [13341]) -.. function:: flock(fd, op) +.. function:: flock(fd, operation) - Perform the lock operation *op* on file descriptor *fd* (file objects providing + Perform the lock operation *operation* on file descriptor *fd* (file objects providing a :meth:`~io.IOBase.fileno` method are accepted as well). See the Unix manual :manpage:`flock(2)` for details. (On some systems, this function is emulated using :c:func:`fcntl`.) -.. function:: lockf(fd, operation, [length, [start, [whence]]]) +.. function:: lockf(fd, cmd, len=0, start=0, whence=0) This is essentially a wrapper around the :func:`~fcntl.fcntl` locking calls. - *fd* is the file descriptor of the file to lock or unlock, and *operation* + *fd* is the file descriptor of the file to lock or unlock, and *cmd* is one of the following values: * :const:`LOCK_UN` -- unlock * :const:`LOCK_SH` -- acquire a shared lock * :const:`LOCK_EX` -- acquire an exclusive lock - When *operation* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be + When *cmd* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be bitwise ORed with :const:`LOCK_NB` to avoid blocking on lock acquisition. If :const:`LOCK_NB` is used and the lock cannot be acquired, an :exc:`OSError` will be raised and the exception will have an *errno* @@ -124,7 +124,7 @@ The module defines the following functions: systems, :const:`LOCK_EX` can only be used if the file descriptor refers to a file opened for writing. - *length* is the number of bytes to lock, *start* is the byte offset at + *len* is the number of bytes to lock, *start* is the byte offset at which the lock starts, relative to *whence*, and *whence* is as with :func:`io.IOBase.seek`, specifically: @@ -133,7 +133,7 @@ The module defines the following functions: * :const:`2` -- relative to the end of the file (:data:`os.SEEK_END`) The default for *start* is 0, which means to start at the beginning of the file. - The default for *length* is 0 which means to lock to the end of the file. The + The default for *len* is 0 which means to lock to the end of the file. The default for *whence* is also 0. Examples (all on a SVR4 compliant system):: @@ -147,9 +147,9 @@ Examples (all on a SVR4 compliant system):: rv = fcntl.fcntl(f, fcntl.F_SETLKW, lockdata) Note that in the first example the return value variable *rv* will hold an -integer value; in the second example it will hold a string value. The structure -lay-out for the *lockdata* variable is system dependent --- therefore using the -:func:`flock` call may be better. +integer value; in the second example it will hold a :class:`bytes` object. The +structure lay-out for the *lockdata* variable is system dependent --- therefore +using the :func:`flock` call may be better. .. seealso:: diff --git a/Doc/library/formatter.rst b/Doc/library/formatter.rst index 1847a80..a515f74 100644 --- a/Doc/library/formatter.rst +++ b/Doc/library/formatter.rst @@ -5,7 +5,7 @@ :synopsis: Generic output formatter and device interface. :deprecated: -.. deprecated:: 3.4 +.. deprecated-removed:: 3.4 3.6 Due to lack of usage, the formatter module has been deprecated and is slated for removal in Python 3.6. diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst index c2c7401..e0f0682 100644 --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -172,6 +172,9 @@ another rational number, or from a string. sign as *b* if *b* is nonzero; otherwise it takes the sign of *a*. ``gcd(0, 0)`` returns ``0``. + .. deprecated:: 3.5 + Use :func:`math.gcd` instead. + .. seealso:: diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 5377335..409f6c4 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -156,11 +156,12 @@ are always available. They are listed here in alphabetical order. .. function:: chr(i) - Return the string representing a character whose Unicode code point is the integer - *i*. For example, ``chr(97)`` returns the string ``'a'``. This is the - inverse of :func:`ord`. The valid range for the argument is from 0 through - 1,114,111 (0x10FFFF in base 16). :exc:`ValueError` will be raised if *i* is - outside that range. + Return the string representing a character whose Unicode code point is the + integer *i*. For example, ``chr(97)`` returns the string ``'a'``, while + ``chr(957)`` returns the string ``'ν'``. This is the inverse of :func:`ord`. + + The valid range for the argument is from 0 through 1,114,111 (0x10FFFF in + base 16). :exc:`ValueError` will be raised if *i* is outside that range. .. function:: classmethod(function) @@ -972,9 +973,11 @@ are always available. They are listed here in alphabetical order. Characters not supported by the encoding are replaced with the appropriate XML character reference ``&#nnn;``. - * ``'backslashreplace'`` (also only supported when writing) - replaces unsupported characters with Python's backslashed escape - sequences. + * ``'backslashreplace'`` replaces malformed data by Python's backslashed + escape sequences. + + * ``'namereplace'`` (also only supported when writing) + replaces unsupported characters with ``\N{...}`` escape sequences. .. index:: single: universal newlines; open() built-in function @@ -999,8 +1002,8 @@ are always available. They are listed here in alphabetical order. If *closefd* is ``False`` and a file descriptor rather than a filename was given, the underlying file descriptor will be kept open when the file is - closed. If a filename is given *closefd* has no effect and must be ``True`` - (the default). + closed. If a filename is given *closefd* must be ``True`` (the default) + otherwise an error will be raised. A custom opener can be used by passing a callable as *opener*. The underlying file descriptor for the file object is then obtained by calling *opener* with @@ -1062,14 +1065,18 @@ are always available. They are listed here in alphabetical order. The ``'U'`` mode. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + -.. XXX works for bytes too, but should it? .. function:: ord(c) Given a string representing one Unicode character, return an integer - representing the Unicode code - point of that character. For example, ``ord('a')`` returns the integer ``97`` - and ``ord('\u2020')`` returns ``8224``. This is the inverse of :func:`chr`. + representing the Unicode code point of that character. For example, + ``ord('a')`` returns the integer ``97`` and ``ord('ν')`` returns ``957``. + This is the inverse of :func:`chr`. .. function:: pow(x, y[, z]) @@ -1186,6 +1193,9 @@ are always available. They are listed here in alphabetical order. The returned property object also has the attributes ``fget``, ``fset``, and ``fdel`` corresponding to the constructor arguments. + .. versionchanged:: 3.5 + The docstrings of property objects are now writeable. + .. _func-range: .. function:: range(stop) @@ -1218,8 +1228,8 @@ are always available. They are listed here in alphabetical order. .. function:: round(number[, ndigits]) Return the floating point value *number* rounded to *ndigits* digits after - the decimal point. If *ndigits* is omitted, it defaults to zero. Delegates - to ``number.__round__(ndigits)``. + the decimal point. If *ndigits* is omitted, it returns the nearest integer + to its input. Delegates to ``number.__round__(ndigits)``. For the built-in types supporting :func:`round`, values are rounded to the closest multiple of 10 to the power minus *ndigits*; if two multiples are @@ -1485,7 +1495,9 @@ are always available. They are listed here in alphabetical order. The left-to-right evaluation order of the iterables is guaranteed. This makes possible an idiom for clustering a data series into n-length groups - using ``zip(*[iter(s)]*n)``. + using ``zip(*[iter(s)]*n)``. This repeats the *same* iterator ``n`` times + so that each output tuple has the result of ``n`` calls to the iterator. + This has the effect of dividing the input into n-length chunks. :func:`zip` should only be used with unequal length inputs when you don't care about trailing, unmatched values from the longer iterables. If those diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 8135542..d11c2e1 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -186,7 +186,7 @@ values but should not rebind them): added to this list rather than freed. .. versionchanged:: 3.2 - If this list is non-empty at interpreter shutdown, a + If this list is non-empty at :term:`interpreter shutdown`, a :exc:`ResourceWarning` is emitted, which is silent by default. If :const:`DEBUG_UNCOLLECTABLE` is set, in addition all uncollectable objects are printed. @@ -252,8 +252,8 @@ The following constants are provided for use with :func:`set_debug`: to the ``garbage`` list. .. versionchanged:: 3.2 - Also print the contents of the :data:`garbage` list at interpreter - shutdown, if it isn't empty. + Also print the contents of the :data:`garbage` list at + :term:`interpreter shutdown`, if it isn't empty. .. data:: DEBUG_SAVEALL diff --git a/Doc/library/gettext.rst b/Doc/library/gettext.rst index ff23b59..514cc5a 100644 --- a/Doc/library/gettext.rst +++ b/Doc/library/gettext.rst @@ -344,9 +344,9 @@ will assume message ids as Unicode strings, not byte strings. The entire set of key/value pairs are placed into a dictionary and set as the "protected" :attr:`_info` instance variable. -If the :file:`.mo` file's magic number is invalid, or if other problems occur -while reading the file, instantiating a :class:`GNUTranslations` class can raise -:exc:`OSError`. +If the :file:`.mo` file's magic number is invalid, the major version number is +unexpected, or if other problems occur while reading the file, instantiating a +:class:`GNUTranslations` class can raise :exc:`OSError`. The following methods are overridden from the base class implementation: diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst index abcbf38..50f38a4 100644 --- a/Doc/library/glob.rst +++ b/Doc/library/glob.rst @@ -29,7 +29,7 @@ For example, ``'[?]'`` matches the character ``'?'``. The :mod:`pathlib` module offers high-level path objects. -.. function:: glob(pathname) +.. function:: glob(pathname, *, recursive=False) Return a possibly-empty list of path names that match *pathname*, which must be a string containing a path specification. *pathname* can be either absolute @@ -37,8 +37,19 @@ For example, ``'[?]'`` matches the character ``'?'``. :file:`../../Tools/\*/\*.gif`), and can contain shell-style wildcards. Broken symlinks are included in the results (as in the shell). + If *recursive* is true, the pattern "``**``" will match any files and zero or + more directories and subdirectories. If the pattern is followed by a + ``os.sep``, only directories and subdirectories match. -.. function:: iglob(pathname) + .. note:: + Using the "``**``" pattern in large directory trees may consume + an inordinate amount of time. + + .. versionchanged:: 3.5 + Support for recursive globs using "``**``". + + +.. function:: iglob(pathname, recursive=False) Return an :term:`iterator` which yields the same values as :func:`glob` without actually storing them all simultaneously. @@ -55,8 +66,9 @@ For example, ``'[?]'`` matches the character ``'?'``. .. versionadded:: 3.4 -For example, consider a directory containing only the following files: -:file:`1.gif`, :file:`2.txt`, and :file:`card.gif`. :func:`glob` will produce +For example, consider a directory containing the following files: +:file:`1.gif`, :file:`2.txt`, :file:`card.gif` and a subdirectory :file:`sub` +which contains only the file :file:`3.txt`. :func:`glob` will produce the following results. Notice how any leading components of the path are preserved. :: @@ -67,6 +79,10 @@ preserved. :: ['1.gif', 'card.gif'] >>> glob.glob('?.gif') ['1.gif'] + >>> glob.glob('**/*.txt', recursive=True) + ['2.txt', 'sub/3.txt'] + >>> glob.glob('./**/', recursive=True) + ['./', './sub/'] If the directory contains files starting with ``.`` they won't be matched by default. For example, consider a directory containing :file:`card.gif` and diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst index 78536fa..04c41d5 100644 --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -90,13 +90,9 @@ The module defines the following items: is no compression. The default is ``9``. The *mtime* argument is an optional numeric timestamp to be written to - the stream when compressing. All :program:`gzip` compressed streams are - required to contain a timestamp. If omitted or ``None``, the current - time is used. This module ignores the timestamp when decompressing; - however, some programs, such as :program:`gunzip`\ , make use of it. - The format of the timestamp is the same as that of the return value of - ``time.time()`` and of the ``st_mtime`` attribute of the object returned - by ``os.stat()``. + the last modification time field in the stream when compressing. It + should only be provided in compression mode. If omitted or ``None``, the + current time is used. See the :attr:`mtime` attribute for more details. Calling a :class:`GzipFile` object's :meth:`close` method does not close *fileobj*, since you might wish to append more material after the compressed @@ -108,9 +104,9 @@ The module defines the following items: including iteration and the :keyword:`with` statement. Only the :meth:`truncate` method isn't implemented. - :class:`GzipFile` also provides the following method: + :class:`GzipFile` also provides the following method and attribute: - .. method:: peek([n]) + .. method:: peek(n) Read *n* uncompressed bytes without advancing the file position. At most one single read on the compressed stream is done to satisfy @@ -124,9 +120,21 @@ The module defines the following items: .. versionadded:: 3.2 + .. attribute:: mtime + + When decompressing, the value of the last modification time field in + the most recently read header may be read from this attribute, as an + integer. The initial value before reading any headers is ``None``. + + All :program:`gzip` compressed streams are required to contain this + timestamp field. Some programs, such as :program:`gunzip`\ , make use + of the timestamp. The format is the same as the return value of + :func:`time.time` and the :attr:`~os.stat_result.st_mtime` attribute of + the object returned by :func:`os.stat`. + .. versionchanged:: 3.1 Support for the :keyword:`with` statement was added, along with the - *mtime* argument. + *mtime* constructor argument and :attr:`mtime` attribute. .. versionchanged:: 3.2 Support for zero-padded and unseekable files was added. @@ -137,6 +145,12 @@ The module defines the following items: .. versionchanged:: 3.4 Added support for the ``'x'`` and ``'xb'`` modes. + .. versionchanged:: 3.5 + Added support for writing arbitrary + :term:`bytes-like objects <bytes-like object>`. + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + .. function:: compress(data, compresslevel=9) @@ -175,9 +189,10 @@ Example of how to create a compressed GZIP file:: Example of how to GZIP compress an existing file:: import gzip + import shutil with open('/home/joe/file.txt', 'rb') as f_in: with gzip.open('/home/joe/file.txt.gz', 'wb') as f_out: - f_out.writelines(f_in) + shutil.copyfileobj(f_in, f_out) Example of how to GZIP compress a binary string:: diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst index f8970be..9fbbcc6 100644 --- a/Doc/library/heapq.rst +++ b/Doc/library/heapq.rst @@ -82,7 +82,7 @@ The following functions are provided: The module also offers three general purpose functions based on heaps. -.. function:: merge(*iterables) +.. function:: merge(*iterables, key=None, reverse=False) Merge multiple sorted inputs into a single sorted output (for example, merge timestamped entries from multiple log files). Returns an :term:`iterator` @@ -92,6 +92,18 @@ The module also offers three general purpose functions based on heaps. not pull the data into memory all at once, and assumes that each of the input streams is already sorted (smallest to largest). + Has two optional arguments which must be specified as keyword arguments. + + *key* specifies a :term:`key function` of one argument that is used to + extract a comparison key from each input element. The default value is + ``None`` (compare the elements directly). + + *reverse* is a boolean value. If set to ``True``, then the input elements + are merged as if each comparison were reversed. + + .. versionchanged:: 3.5 + Added the optional *key* and *reverse* parameters. + .. function:: nlargest(n, iterable, key=None) diff --git a/Doc/library/html.parser.rst b/Doc/library/html.parser.rst index 44b7d6e..b84c60b 100644 --- a/Doc/library/html.parser.rst +++ b/Doc/library/html.parser.rst @@ -16,21 +16,13 @@ This module defines a class :class:`HTMLParser` which serves as the basis for parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. -.. class:: HTMLParser(strict=False, *, convert_charrefs=False) +.. class:: HTMLParser(*, convert_charrefs=True) - Create a parser instance. + Create a parser instance able to parse invalid markup. - If *convert_charrefs* is ``True`` (default: ``False``), all character + If *convert_charrefs* is ``True`` (the default), all character references (except the ones in ``script``/``style`` elements) are automatically converted to the corresponding Unicode characters. - The use of ``convert_charrefs=True`` is encouraged and will become - the default in Python 3.5. - - If *strict* is ``False`` (the default), the parser will accept and parse - invalid markup. If *strict* is ``True`` the parser will raise an - :exc:`~html.parser.HTMLParseError` exception instead [#]_ when it's not - able to parse the markup. The use of ``strict=True`` is discouraged and - the *strict* argument is deprecated. An :class:`.HTMLParser` instance is fed HTML data and calls handler methods when start tags, end tags, text, comments, and other markup elements are @@ -40,31 +32,11 @@ parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. This parser does not check that end tags match start tags or call the end-tag handler for elements which are closed implicitly by closing an outer element. - .. versionchanged:: 3.2 - *strict* argument added. - - .. deprecated-removed:: 3.3 3.5 - The *strict* argument and the strict mode have been deprecated. - The parser is now able to accept and parse invalid markup too. - .. versionchanged:: 3.4 *convert_charrefs* keyword argument added. -An exception is defined as well: - - -.. exception:: HTMLParseError - - Exception raised by the :class:`HTMLParser` class when it encounters an error - while parsing and *strict* is ``True``. This exception provides three - attributes: :attr:`msg` is a brief message explaining the error, - :attr:`lineno` is the number of the line on which the broken construct was - detected, and :attr:`offset` is the number of characters into the line at - which the construct starts. - - .. deprecated-removed:: 3.3 3.5 - This exception has been deprecated because it's never raised by the parser - (when the default non-strict mode is used). + .. versionchanged:: 3.5 + The default value for argument *convert_charrefs* is now ``True``. Example HTML Parser Application @@ -246,8 +218,7 @@ implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`): The *data* parameter will be the entire contents of the declaration inside the ``<![...]>`` markup. It is sometimes useful to be overridden by a - derived class. The base class implementation raises an :exc:`HTMLParseError` - when *strict* is ``True``. + derived class. The base class implementation does nothing. .. _htmlparser-examples: @@ -358,9 +329,3 @@ Parsing invalid HTML (e.g. unquoted attributes) also works:: Data : tag soup End tag : p End tag : a - -.. rubric:: Footnotes - -.. [#] For backward compatibility reasons *strict* mode does not raise - exceptions for all non-compliant HTML. That is, some invalid HTML - is tolerated even in *strict* mode. diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index 807f685..d57649c 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -180,6 +180,17 @@ The following exceptions are raised as appropriate: is received in the HTTP protocol from the server. +.. exception:: RemoteDisconnected + + A subclass of :exc:`ConnectionResetError` and :exc:`BadStatusLine`. Raised + by :meth:`HTTPConnection.getresponse` when the attempt to read the response + results in no data read from the connection, indicating that the remote end + has closed the connection. + + .. versionadded:: 3.5 + Previously, :exc:`BadStatusLine`\ ``('')`` was raised. + + The constants defined in this module are: .. data:: HTTP_PORT @@ -191,221 +202,15 @@ The constants defined in this module are: The default port for the HTTPS protocol (always ``443``). -and also the following constants for integer status codes: - -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| Constant | Value | Definition | -+==========================================+=========+=======================================================================+ -| :const:`CONTINUE` | ``100`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SWITCHING_PROTOCOLS` | ``101`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROCESSING` | ``102`` | WEBDAV, `RFC 2518, Section 10.1 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_102>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`OK` | ``200`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CREATED` | ``201`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`ACCEPTED` | ``202`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NON_AUTHORITATIVE_INFORMATION` | ``203`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NO_CONTENT` | ``204`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`RESET_CONTENT` | ``205`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PARTIAL_CONTENT` | ``206`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.7 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.7>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTI_STATUS` | ``207`` | WEBDAV `RFC 2518, Section 10.2 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_207>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`IM_USED` | ``226`` | Delta encoding in HTTP, | -| | | :rfc:`3229`, Section 10.4.1 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTIPLE_CHOICES` | ``300`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MOVED_PERMANENTLY` | ``301`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FOUND` | ``302`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SEE_OTHER` | ``303`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_MODIFIED` | ``304`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`USE_PROXY` | ``305`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TEMPORARY_REDIRECT` | ``307`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.8 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.8>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_REQUEST` | ``400`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNAUTHORIZED` | ``401`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PAYMENT_REQUIRED` | ``402`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FORBIDDEN` | ``403`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_FOUND` | ``404`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`METHOD_NOT_ALLOWED` | ``405`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_ACCEPTABLE` | ``406`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.7 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.7>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROXY_AUTHENTICATION_REQUIRED` | ``407`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.8 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.8>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_TIMEOUT` | ``408`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.9 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.9>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CONFLICT` | ``409`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.10 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.10>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GONE` | ``410`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.11 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.11>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LENGTH_REQUIRED` | ``411`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.12 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.12>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_FAILED` | ``412`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.13 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.13>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_ENTITY_TOO_LARGE` | ``413`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.14 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.14>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_URI_TOO_LONG` | ``414`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.15 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.15>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNSUPPORTED_MEDIA_TYPE` | ``415`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.16 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.16>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUESTED_RANGE_NOT_SATISFIABLE` | ``416`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.17 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.17>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`EXPECTATION_FAILED` | ``417`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.18 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.18>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNPROCESSABLE_ENTITY` | ``422`` | WEBDAV, `RFC 2518, Section 10.3 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_422>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LOCKED` | ``423`` | WEBDAV `RFC 2518, Section 10.4 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_423>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FAILED_DEPENDENCY` | ``424`` | WEBDAV, `RFC 2518, Section 10.5 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_424>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UPGRADE_REQUIRED` | ``426`` | HTTP Upgrade to TLS, | -| | | :rfc:`2817`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_REQUIRED` | ``428`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 3 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TOO_MANY_REQUESTS` | ``429`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 4 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_HEADER_FIELDS_TOO_LARGE` | ``431`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 5 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INTERNAL_SERVER_ERROR` | ``500`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_IMPLEMENTED` | ``501`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_GATEWAY` | ``502`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SERVICE_UNAVAILABLE` | ``503`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GATEWAY_TIMEOUT` | ``504`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`HTTP_VERSION_NOT_SUPPORTED` | ``505`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INSUFFICIENT_STORAGE` | ``507`` | WEBDAV, `RFC 2518, Section 10.6 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_507>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_EXTENDED` | ``510`` | An HTTP Extension Framework, | -| | | :rfc:`2774`, Section 7 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NETWORK_AUTHENTICATION_REQUIRED` | ``511`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ - -.. versionchanged:: 3.3 - Added codes ``428``, ``429``, ``431`` and ``511`` from :rfc:`6585`. - - .. data:: responses This dictionary maps the HTTP 1.1 status codes to the W3C names. Example: ``http.client.responses[http.client.NOT_FOUND]`` is ``'Not Found'``. +See :ref:`http-status-codes` for a list of HTTP status codes that are +available in this module as constants. + .. _httpconnection-objects: @@ -458,6 +263,11 @@ HTTPConnection Objects Note that you must have read the whole response before you can send a new request to the server. + .. versionchanged:: 3.5 + If a :exc:`ConnectionError` or subclass is raised, the + :class:`HTTPConnection` object will be ready to reconnect when + a new request is sent. + .. method:: HTTPConnection.set_debuglevel(level) @@ -496,7 +306,9 @@ HTTPConnection Objects .. method:: HTTPConnection.connect() - Connect to the server specified when the object was created. + Connect to the server specified when the object was created. By default, + this is called automatically when making a request if the client does not + already have a connection. .. method:: HTTPConnection.close() diff --git a/Doc/library/http.cookies.rst b/Doc/library/http.cookies.rst index 646f2e8..7c85d09 100644 --- a/Doc/library/http.cookies.rst +++ b/Doc/library/http.cookies.rst @@ -143,26 +143,43 @@ Morsel Objects The keys are case-insensitive. + .. versionchanged:: 3.5 + :meth:`~Morsel.__eq__` now takes :attr:`~Morsel.key` and :attr:`~Morsel.value` + into account. + .. attribute:: Morsel.value The value of the cookie. + .. deprecated:: 3.5 + assigning to ``value``; use :meth:`~Morsel.set` instead. + .. attribute:: Morsel.coded_value The encoded value of the cookie --- this is what should be sent. + .. deprecated:: 3.5 + assigning to ``coded_value``; use :meth:`~Morsel.set` instead. + .. attribute:: Morsel.key The name of the cookie. + .. deprecated:: 3.5 + assigning to ``key``; use :meth:`~Morsel.set` instead. + .. method:: Morsel.set(key, value, coded_value) Set the *key*, *value* and *coded_value* attributes. + .. deprecated:: 3.5 + The undocumented *LegalChars* parameter is ignored and will be removed in + a future version. + .. method:: Morsel.isReservedKey(K) @@ -193,6 +210,30 @@ Morsel Objects The meaning for *attrs* is the same as in :meth:`output`. +.. method:: Morsel.update(values) + + Update the values in the Morsel dictionary with the values in the dictionary + *values*. Raise an error if any of the keys in the *values* dict is not a + valid :rfc:`2109` attribute. + + .. versionchanged:: 3.5 + an error is raised for invalid keys. + + +.. method:: Morsel.copy(value) + + Return a shallow copy of the Morsel object. + + .. versionchanged:: 3.5 + return a Morsel object instead of a dict. + + +.. method:: Morsel.setdefault(key, value=None) + + Raise an error if key is not a valid :rfc:`2109` attribute, otherwise + behave the same as :meth:`dict.setdefault`. + + .. _cookie-example: Example diff --git a/Doc/library/http.rst b/Doc/library/http.rst index a387a37..b6f2c58 100644 --- a/Doc/library/http.rst +++ b/Doc/library/http.rst @@ -1,7 +1,16 @@ :mod:`http` --- HTTP modules ============================ -``http`` is a package that collects several modules for working with the +.. module:: http + :synopsis: HTTP status codes and messages + +.. index:: + pair: HTTP; protocol + single: HTTP; http (standard module) + +**Source code:** :source:`Lib/http/__init__.py` + +:mod:`http` is a package that collects several modules for working with the HyperText Transfer Protocol: * :mod:`http.client` is a low-level HTTP protocol client; for high-level URL @@ -9,3 +18,105 @@ HyperText Transfer Protocol: * :mod:`http.server` contains basic HTTP server classes based on :mod:`socketserver` * :mod:`http.cookies` has utilities for implementing state management with cookies * :mod:`http.cookiejar` provides persistence of cookies + +:mod:`http` is also a module that defines a number of HTTP status codes and +associated messages through the :class:`http.HTTPStatus` enum: + +.. class:: HTTPStatus + + .. versionadded:: 3.5 + + A subclass of :class:`enum.IntEnum` that defines a set of HTTP status codes, + reason phrases and long descriptions written in English. + + Usage:: + + >>> from http import HTTPStatus + >>> HTTPStatus.OK + <HTTPStatus.OK: 200> + >>> HTTPStatus.OK == 200 + True + >>> http.HTTPStatus.OK.value + 200 + >>> HTTPStatus.OK.phrase + 'OK' + >>> HTTPStatus.OK.description + 'Request fulfilled, document follows' + >>> list(HTTPStatus) + [<HTTPStatus.CONTINUE: 100>, <HTTPStatus.SWITCHING_PROTOCOLS: 101>, ...] + +.. _http-status-codes: + +HTTP status codes +----------------- + +Supported, +`IANA-registered <http://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml>`_ +status codes available in :class:`http.HTTPStatus` are: + +======= =================================== ================================================================== +Code Enum Name Details +======= =================================== ================================================================== +``100`` ``CONTINUE`` HTTP/1.1 :rfc:`7231`, Section 6.2.1 +``101`` ``SWITCHING_PROTOCOLS`` HTTP/1.1 :rfc:`7231`, Section 6.2.2 +``102`` ``PROCESSING`` WebDAV :rfc:`2518`, Section 10.1 +``200`` ``OK`` HTTP/1.1 :rfc:`7231`, Section 6.3.1 +``201`` ``CREATED`` HTTP/1.1 :rfc:`7231`, Section 6.3.2 +``202`` ``ACCEPTED`` HTTP/1.1 :rfc:`7231`, Section 6.3.3 +``203`` ``NON_AUTHORITATIVE_INFORMATION`` HTTP/1.1 :rfc:`7231`, Section 6.3.4 +``204`` ``NO_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.5 +``205`` ``RESET_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.6 +``206`` ``PARTIAL_CONTENT`` HTTP/1.1 :rfc:`7233`, Section 4.1 +``207`` ``MULTI_STATUS`` WebDAV :rfc:`4918`, Section 11.1 +``208`` ``ALREADY_REPORTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.1 (Experimental) +``226`` ``IM_USED`` Delta Encoding in HTTP :rfc:`3229`, Section 10.4.1 +``300`` ``MULTIPLE_CHOICES`` HTTP/1.1 :rfc:`7231`, Section 6.4.1 +``301`` ``MOVED_PERMANENTLY`` HTTP/1.1 :rfc:`7231`, Section 6.4.2 +``302`` ``FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.4.3 +``303`` ``SEE_OTHER`` HTTP/1.1 :rfc:`7231`, Section 6.4.4 +``304`` ``NOT_MODIFIED`` HTTP/1.1 :rfc:`7232`, Section 4.1 +``305`` ``USE_PROXY`` HTTP/1.1 :rfc:`7231`, Section 6.4.5 +``307`` ``TEMPORARY_REDIRECT`` HTTP/1.1 :rfc:`7231`, Section 6.4.7 +``308`` ``PERMANENT_REDIRECT`` Permanent Redirect :rfc:`7238`, Section 3 (Experimental) +``400`` ``BAD_REQUEST`` HTTP/1.1 :rfc:`7231`, Section 6.5.1 +``401`` ``UNAUTHORIZED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.1 +``402`` ``PAYMENT_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.2 +``403`` ``FORBIDDEN`` HTTP/1.1 :rfc:`7231`, Section 6.5.3 +``404`` ``NOT_FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.5.4 +``405`` ``METHOD_NOT_ALLOWED`` HTTP/1.1 :rfc:`7231`, Section 6.5.5 +``406`` ``NOT_ACCEPTABLE`` HTTP/1.1 :rfc:`7231`, Section 6.5.6 +``407`` ``PROXY_AUTHENTICATION_REQUIRED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.2 +``408`` ``REQUEST_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.5.7 +``409`` ``CONFLICT`` HTTP/1.1 :rfc:`7231`, Section 6.5.8 +``410`` ``GONE`` HTTP/1.1 :rfc:`7231`, Section 6.5.9 +``411`` ``LENGTH_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.10 +``412`` ``PRECONDITION_FAILED`` HTTP/1.1 :rfc:`7232`, Section 4.2 +``413`` ``REQUEST_ENTITY_TOO_LARGE`` HTTP/1.1 :rfc:`7231`, Section 6.5.11 +``414`` ``REQUEST_URI_TOO_LONG`` HTTP/1.1 :rfc:`7231`, Section 6.5.12 +``415`` ``UNSUPPORTED_MEDIA_TYPE`` HTTP/1.1 :rfc:`7231`, Section 6.5.13 +``416`` ``REQUEST_RANGE_NOT_SATISFIABLE`` HTTP/1.1 Range Requests :rfc:`7233`, Section 4.4 +``417`` ``EXPECTATION_FAILED`` HTTP/1.1 :rfc:`7231`, Section 6.5.14 +``422`` ``UNPROCESSABLE_ENTITY`` WebDAV :rfc:`4918`, Section 11.2 +``423`` ``LOCKED`` WebDAV :rfc:`4918`, Section 11.3 +``424`` ``FAILED_DEPENDENCY`` WebDAV :rfc:`4918`, Section 11.4 +``426`` ``UPGRADE_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.15 +``428`` ``PRECONDITION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585` +``429`` ``TOO_MANY_REQUESTS`` Additional HTTP Status Codes :rfc:`6585` +``431`` ``REQUEST_HEADER_FIELDS_TOO_LARGE`` Additional HTTP Status Codes :rfc:`6585` +``500`` ``INTERNAL_SERVER_ERROR`` HTTP/1.1 :rfc:`7231`, Section 6.6.1 +``501`` ``NOT_IMPLEMENTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.2 +``502`` ``BAD_GATEWAY`` HTTP/1.1 :rfc:`7231`, Section 6.6.3 +``503`` ``SERVICE_UNAVAILABLE`` HTTP/1.1 :rfc:`7231`, Section 6.6.4 +``504`` ``GATEWAY_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.6.5 +``505`` ``HTTP_VERSION_NOT_SUPPORTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.6 +``506`` ``VARIANT_ALSO_NEGOTIATES`` Transparent Content Negotiation in HTTP :rfc:`2295`, Section 8.1 (Experimental) +``507`` ``INSUFFICIENT_STORAGE`` WebDAV :rfc:`4918`, Section 11.5 +``508`` ``LOOP_DETECTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.2 (Experimental) +``510`` ``NOT_EXTENDED`` An HTTP Extension Framework :rfc:`2774`, Section 7 (Experimental) +``511`` ``NETWORK_AUTHENTICATION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585`, Section 6 +======= =================================== ================================================================== + +In order to preserve backwards compatibility, enum values are also present +in the :mod:`http.client` module in the form of constants. The enum name is +equal to the constant name (i.e. ``http.HTTPStatus.OK`` is also available as +``http.client.OK``). diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index fa736fe..15b0932 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -37,6 +37,19 @@ base class: initialized. If *host* is not specified, ``''`` (the local host) is used. If *port* is omitted, the standard IMAP4 port (143) is used. + The :class:`IMAP4` class supports the :keyword:`with` statement. When used + like this, the IMAP4 ``LOGOUT`` command is issued automatically when the + :keyword:`with` statement exits. E.g.:: + + >>> from imaplib import IMAP4 + >>> with IMAP4("domain.org") as M: + ... M.noop() + ... + ('OK', [b'Nothing Accomplished. d25if65hy903weo.87']) + + .. versionchanged:: 3.5 + Support for the :keyword:`with` statement was added. + Three exceptions are defined as attributes of the :class:`IMAP4` class: @@ -64,7 +77,8 @@ Three exceptions are defined as attributes of the :class:`IMAP4` class: There's also a subclass for secure connections: -.. class:: IMAP4_SSL(host='', port=IMAP4_SSL_PORT, keyfile=None, certfile=None, ssl_context=None) +.. class:: IMAP4_SSL(host='', port=IMAP4_SSL_PORT, keyfile=None, \ + certfile=None, ssl_context=None) This is a subclass derived from :class:`IMAP4` that connects over an SSL encrypted socket (to use this class you need a socket module that was compiled @@ -198,6 +212,10 @@ An :class:`IMAP4` instance has the following methods: that will be base64 encoded and sent to the server. It should return ``None`` if the client abort response ``*`` should be sent instead. + .. versionchanged:: 3.5 + string usernames and passwords are now encoded to ``utf-8`` instead of + being limited to ASCII. + .. method:: IMAP4.check() @@ -230,6 +248,16 @@ An :class:`IMAP4` instance has the following methods: Delete the ACLs (remove any rights) set for who on mailbox. +.. method:: IMAP4.enable(capability) + + Enable *capability* (see :rfc:`5161`). Most capabilities do not need to be + enabled. Currently only the ``UTF8=ACCEPT`` capability is supported + (see :RFC:`6855`). + + .. versionadded:: 3.5 + The :meth:`enable` method itself, and :RFC:`6855` support. + + .. method:: IMAP4.expunge() Permanently remove deleted items from selected mailbox. Generates an ``EXPUNGE`` @@ -367,7 +395,9 @@ An :class:`IMAP4` instance has the following methods: Search mailbox for matching messages. *charset* may be ``None``, in which case no ``CHARSET`` will be specified in the request to the server. The IMAP protocol requires that at least one criterion be specified; an exception will be - raised when the server returns an error. + raised when the server returns an error. *charset* must be ``None`` if + the ``UTF8=ACCEPT`` capability was enabled using the :meth:`enable` + command. Example:: @@ -529,6 +559,15 @@ The following attributes are defined on instances of :class:`IMAP4`: the module variable ``Debug``. Values greater than three trace each command. +.. attribute:: IMAP4.utf8_enabled + + Boolean value that is normally ``False``, but is set to ``True`` if an + :meth:`enable` command is successfully issued for the ``UTF8=ACCEPT`` + capability. + + .. versionadded:: 3.5 + + .. _imap4-example: IMAP4 Example diff --git a/Doc/library/imghdr.rst b/Doc/library/imghdr.rst index 9e89523..c60df24 100644 --- a/Doc/library/imghdr.rst +++ b/Doc/library/imghdr.rst @@ -48,6 +48,16 @@ from :func:`what`: +------------+-----------------------------------+ | ``'png'`` | Portable Network Graphics | +------------+-----------------------------------+ +| ``'webp'`` | WebP files | ++------------+-----------------------------------+ +| ``'exr'`` | OpenEXR Files | ++------------+-----------------------------------+ + +.. versionadded:: 3.5 + The *exr* format was added. + +.. versionchanged:: 3.5 + The *webp* type was added. You can extend the list of file types :mod:`imghdr` can recognize by appending to this variable: diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst index 83a52e4..68a6b68 100644 --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -197,11 +197,9 @@ file paths. value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. The ``cpython-32`` string comes from the current magic tag (see :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then - :exc:`NotImplementedError` will be raised). The returned path will end in - ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python - (i.e. ``__debug__`` is ``False``). By passing in ``True`` or ``False`` for - *debug_override* you can override the system's value for ``__debug__`` for - extension selection. + :exc:`NotImplementedError` will be raised). By passing in ``True`` or + ``False`` for *debug_override* you can override the system's value for + ``__debug__``, leading to optimized bytecode. *path* need not exist. @@ -212,6 +210,9 @@ file paths. .. deprecated:: 3.4 Use :func:`importlib.util.cache_from_source` instead. + .. versionchanged:: 3.5 + The *debug_override* parameter no longer creates a ``.pyo`` file. + .. function:: source_from_cache(path) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index c947335..771c4c5 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -55,6 +55,12 @@ generically as an :term:`importer`) to participate in the import process. :pep:`451` A ModuleSpec Type for the Import System + :pep:`488` + Elimination of PYO files + + :pep:`489` + Multi-phase extension module initialization + :pep:`3120` Using UTF-8 as the Default Source Encoding @@ -69,6 +75,10 @@ Functions An implementation of the built-in :func:`__import__` function. + .. note:: + Programmatic importing of modules should use :func:`import_module` + instead of this function. + .. function:: import_module(name, package=None) Import a module. The *name* argument specifies what module to @@ -81,12 +91,15 @@ Functions The :func:`import_module` function acts as a simplifying wrapper around :func:`importlib.__import__`. This means all semantics of the function are - derived from :func:`importlib.__import__`, including requiring the package - from which an import is occurring to have been previously imported - (i.e., *package* must already be imported). The most important difference - is that :func:`import_module` returns the specified package or module - (e.g. ``pkg.mod``), while :func:`__import__` returns the - top-level package or module (e.g. ``pkg``). + derived from :func:`importlib.__import__`. The most important difference + between these two functions is that :func:`import_module` returns the + specified package or module (e.g. ``pkg.mod``), while :func:`__import__` + returns the top-level package or module (e.g. ``pkg``). + + If you are dynamically importing a module that was created since the + interpreter began execution (e.g., created a Python source file), you may + need to call :func:`invalidate_caches` in order for the new module to be + noticed by the import system. .. versionchanged:: 3.3 Parent packages are automatically imported. @@ -341,13 +354,16 @@ ABC hierarchy:: .. method:: create_module(spec) - An optional method that returns the module object to use when - importing a module. create_module() may also return ``None``, - indicating that the default module creation should take place - instead. + A method that returns the module object to use when + importing a module. This method may return ``None``, + indicating that default module creation semantics should take place. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + Starting in Python 3.6, this method will not be optional when + :meth:`exec_module` is defined. + .. method:: exec_module(module) An abstract method that executes the module in its own namespace @@ -411,7 +427,7 @@ ABC hierarchy:: .. deprecated:: 3.4 The recommended API for loading a module is :meth:`exec_module` - (and optionally :meth:`create_module`). Loaders should implement + (and :meth:`create_module`). Loaders should implement it instead of load_module(). The import machinery takes care of all the other responsibilities of load_module() when exec_module() is implemented. @@ -493,7 +509,7 @@ ABC hierarchy:: .. versionchanged:: 3.4 Raises :exc:`ImportError` instead of :exc:`NotImplementedError`. - .. method:: source_to_code(data, path='<string>') + .. staticmethod:: source_to_code(data, path='<string>') Create a code object from Python source. @@ -502,8 +518,14 @@ ABC hierarchy:: the "path" to where the source code originated from, which can be an abstract concept (e.g. location in a zip file). + With the subsequent code object one can execute it in a module by + running ``exec(code, module.__dict__)``. + .. versionadded:: 3.4 + .. versionchanged:: 3.5 + Made the method static. + .. method:: exec_module(module) Implementation of :meth:`Loader.exec_module`. @@ -689,6 +711,9 @@ find and load modules. .. versionadded:: 3.3 + .. deprecated:: 3.5 + Use :attr:`BYTECODE_SUFFIXES` instead. + .. attribute:: OPTIMIZED_BYTECODE_SUFFIXES A list of strings representing the file suffixes for optimized bytecode @@ -696,14 +721,19 @@ find and load modules. .. versionadded:: 3.3 + .. deprecated:: 3.5 + Use :attr:`BYTECODE_SUFFIXES` instead. + .. attribute:: BYTECODE_SUFFIXES A list of strings representing the recognized file suffixes for bytecode - modules. Set to either :attr:`DEBUG_BYTECODE_SUFFIXES` or - :attr:`OPTIMIZED_BYTECODE_SUFFIXES` based on whether ``__debug__`` is true. + modules (including the leading dot). .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The value is no longer dependent on ``__debug__``. + .. attribute:: EXTENSION_SUFFIXES A list of strings representing the recognized file suffixes for @@ -732,9 +762,9 @@ find and load modules. Only class methods are defined by this class to alleviate the need for instantiation. - .. note:: - Due to limitations in the extension module C-API, for now - BuiltinImporter does not implement :meth:`Loader.exec_module`. + .. versionchanged:: 3.5 + As part of :pep:`489`, the builtin importer now implements + :meth:`Loader.create_module` and :meth:`Loader.exec_module` .. class:: FrozenImporter @@ -782,6 +812,11 @@ find and load modules. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + If the current working directory -- represented by an empty string -- + is no longer valid then ``None`` is returned but no value is cached + in :data:`sys.path_importer_cache`. + .. classmethod:: find_module(fullname, path=None) A legacy wrapper around :meth:`find_spec`. @@ -944,14 +979,18 @@ find and load modules. Path to the extension module. - .. method:: load_module(name=None) + .. method:: create_module(spec) + + Creates the module object from the given specification in accordance + with :pep:`489`. + + .. versionadded:: 3.5 - Loads the extension module if and only if *fullname* is the same as - :attr:`name` or is ``None``. + .. method:: exec_module(module) - .. note:: - Due to limitations in the extension module C-API, for now - ExtensionFileLoader does not implement :meth:`Loader.exec_module`. + Initializes the given module object in accordance with :pep:`489`. + + .. versionadded:: 3.5 .. method:: is_package(fullname) @@ -1047,23 +1086,37 @@ an :term:`importer`. .. versionadded:: 3.4 -.. function:: cache_from_source(path, debug_override=None) +.. function:: cache_from_source(path, debug_override=None, *, optimization=None) - Return the :pep:`3147` path to the byte-compiled file associated with the - source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return + Return the :pep:`3147`/:pep:`488` path to the byte-compiled file associated + with the source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. The ``cpython-32`` string comes from the current magic tag (see :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then - :exc:`NotImplementedError` will be raised). The returned path will end in - ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python - (i.e. ``__debug__`` is ``False``). By passing in ``True`` or ``False`` for - *debug_override* you can override the system's value for ``__debug__`` for - extension selection. - - *path* need not exist. + :exc:`NotImplementedError` will be raised). + + The *optimization* parameter is used to specify the optimization level of the + bytecode file. An empty string represents no optimization, so + ``/foo/bar/baz.py`` with an *optimization* of ``''`` will result in a + bytecode path of ``/foo/bar/__pycache__/baz.cpython-32.pyc``. ``None`` causes + the interpter's optimization level to be used. Any other value's string + representation being used, so ``/foo/bar/baz.py`` with an *optimization* of + ``2`` will lead to the bytecode path of + ``/foo/bar/__pycache__/baz.cpython-32.opt-2.pyc``. The string representation + of *optimization* can only be alphanumeric, else :exc:`ValueError` is raised. + + The *debug_override* parameter is deprecated and can be used to override + the system's value for ``__debug__``. A ``True`` value is the equivalent of + setting *optimization* to the empty string. A ``False`` value is the same as + setting *optimization* to ``1``. If both *debug_override* an *optimization* + are not ``None`` then :exc:`TypeError` is raised. .. versionadded:: 3.4 + .. versionchanged ::3.5 + The *optimization* parameter was added and the *debug_override* parameter + was deprecated. + .. function:: source_from_cache(path) @@ -1071,7 +1124,7 @@ an :term:`importer`. file path. For example, if *path* is ``/foo/bar/__pycache__/baz.cpython-32.pyc`` the returned path would be ``/foo/bar/baz.py``. *path* need not exist, however if it does not conform - to :pep:`3147` format, a ``ValueError`` is raised. If + to :pep:`3147` or :pep:`488` format, a ``ValueError`` is raised. If :attr:`sys.implementation.cache_tag` is not defined, :exc:`NotImplementedError` is raised. @@ -1117,6 +1170,21 @@ an :term:`importer`. .. versionadded:: 3.4 +.. function:: module_from_spec(spec) + + Create a new module based on **spec** and ``spec.loader.create_module()``. + + If ``spec.loader.create_module()`` does not return ``None``, then any + pre-existing attributes will not be reset. Also, no :exc:`AttributeError` + will be raised if triggered while accessing **spec** or setting an attribute + on the module. + + This function is preferred over using :class:`types.ModuleType` to create a + new module as **spec** is used to set as many import-controlled attributes on + the module as possible. + + .. versionadded:: 3.5 + .. decorator:: module_for_loader A :term:`decorator` for :meth:`importlib.abc.Loader.load_module` @@ -1195,3 +1263,39 @@ an :term:`importer`. module will be file-based. .. versionadded:: 3.4 + +.. class:: LazyLoader(loader) + + A class which postpones the execution of the loader of a module until the + module has an attribute accessed. + + This class **only** works with loaders that define + :meth:`~importlib.abc.Loader.exec_module` as control over what module type + is used for the module is required. For those same reasons, the loader's + :meth:`~importlib.abc.Loader.create_module` method will be ignored (i.e., the + loader's method should only return ``None``). Finally, + modules which substitute the object placed into :attr:`sys.modules` will + not work as there is no way to properly replace the module references + throughout the interpreter safely; :exc:`ValueError` is raised if such a + substitution is detected. + + .. note:: + For projects where startup time is critical, this class allows for + potentially minimizing the cost of loading a module if it is never used. + For projects where startup time is not essential then use of this class is + **heavily** discouraged due to error messages created during loading being + postponed and thus occurring out of context. + + .. versionadded:: 3.5 + + .. classmethod:: factory(loader) + + A static method which returns a callable that creates a lazy loader. This + is meant to be used in situations where the loader is passed by class + instead of by instance. + :: + + suffixes = importlib.machinery.SOURCE_SUFFIXES + loader = importlib.machinery.SourceFileLoader + lazy_loader = importlib.util.LazyLoader.factory(loader) + finder = importlib.machinery.FileFinder(path, [(lazy_loader, suffixes)]) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 57eb4ff..d21672f 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -28,7 +28,7 @@ Types and members ----------------- The :func:`getmembers` function retrieves the members of an object such as a -class or module. The sixteen functions whose names begin with "is" are mainly +class or module. The functions whose names begin with "is" are mainly provided as convenient choices for the second argument to :func:`getmembers`. They also help you determine when you can expect to find the following special attributes: @@ -168,6 +168,33 @@ attributes: | | | arguments and local | | | | variables | +-----------+-----------------+---------------------------+ +| generator | __name__ | name | ++-----------+-----------------+---------------------------+ +| | __qualname__ | qualified name | ++-----------+-----------------+---------------------------+ +| | gi_frame | frame | ++-----------+-----------------+---------------------------+ +| | gi_running | is the generator running? | ++-----------+-----------------+---------------------------+ +| | gi_code | code | ++-----------+-----------------+---------------------------+ +| coroutine | __name__ | name | ++-----------+-----------------+---------------------------+ +| | __qualname__ | qualified name | ++-----------+-----------------+---------------------------+ +| | cr_await | object being awaited on, | +| | | or ``None`` | ++-----------+-----------------+---------------------------+ +| | cr_frame | frame | ++-----------+-----------------+---------------------------+ +| | cr_running | is the coroutine running? | ++-----------+-----------------+---------------------------+ +| | cr_code | code | ++-----------+-----------------+---------------------------+ +| | gi_yieldfrom | object being iterated by | +| | | ``yield from``, or | +| | | ``None`` | ++-----------+-----------------+---------------------------+ | builtin | __doc__ | documentation string | +-----------+-----------------+---------------------------+ | | __name__ | original name of this | @@ -180,6 +207,12 @@ attributes: | | | ``None`` | +-----------+-----------------+---------------------------+ +.. versionchanged:: 3.5 + + Add ``__qualname__`` attribute to generators. The ``__name__`` attribute of + generators is now set from the function name, instead of the code name, and + it can now be modified. + .. function:: getmembers(object[, predicate]) @@ -261,6 +294,22 @@ attributes: Return true if the object is a generator. +.. function:: iscoroutinefunction(object) + + Return true if the object is a :term:`coroutine function` + (a function defined with an :keyword:`async def` syntax). + + .. versionadded:: 3.5 + + +.. function:: iscoroutine(object) + + Return true if the object is a :term:`coroutine` created by an + :keyword:`async def` function. + + .. versionadded:: 3.5 + + .. function:: istraceback(object) Return true if the object is a traceback. @@ -351,6 +400,9 @@ Retrieving source code .. function:: getdoc(object) Get the documentation string for an object, cleaned up with :func:`cleandoc`. + If the documentation string for an object is not provided and the object is + a class, a method, a property or a descriptor, retrieve the documentation + string from the inheritance hierarchy. .. function:: getcomments(object) @@ -423,7 +475,7 @@ The Signature object represents the call signature of a callable object and its return annotation. To retrieve a Signature object, use the :func:`signature` function. -.. function:: signature(callable) +.. function:: signature(callable, \*, follow_wrapped=True) Return a :class:`Signature` object for the given ``callable``:: @@ -448,6 +500,11 @@ function. Raises :exc:`ValueError` if no signature can be provided, and :exc:`TypeError` if that type of object is not supported. + .. versionadded:: 3.5 + ``follow_wrapped`` parameter. Pass ``False`` to get a signature of + ``callable`` specifically (``callable.__wrapped__`` will not be used to + unwrap decorated callables.) + .. note:: Some callables may not be introspectable in certain implementations of @@ -473,6 +530,9 @@ function. Signature objects are *immutable*. Use :meth:`Signature.replace` to make a modified copy. + .. versionchanged:: 3.5 + Signature objects are picklable and hashable. + .. attribute:: Signature.empty A special class-level marker to specify absence of a return annotation. @@ -517,12 +577,30 @@ function. >>> str(new_sig) "(a, b) -> 'new return anno'" + .. classmethod:: Signature.from_callable(obj, \*, follow_wrapped=True) + + Return a :class:`Signature` (or its subclass) object for a given callable + ``obj``. Pass ``follow_wrapped=False`` to get a signature of ``obj`` + without unwrapping its ``__wrapped__`` chain. + + This method simplifies subclassing of :class:`Signature`:: + + class MySignature(Signature): + pass + sig = MySignature.from_callable(min) + assert isinstance(sig, MySignature) + + .. versionadded:: 3.5 + .. class:: Parameter(name, kind, \*, default=Parameter.empty, annotation=Parameter.empty) Parameter objects are *immutable*. Instead of modifying a Parameter object, you can use :meth:`Parameter.replace` to create a modified copy. + .. versionchanged:: 3.5 + Parameter objects are picklable and hashable. + .. attribute:: Parameter.empty A special class-level marker to specify absence of default values and @@ -639,27 +717,8 @@ function. Arguments for which :meth:`Signature.bind` or :meth:`Signature.bind_partial` relied on a default value are skipped. - However, if needed, it is easy to include them. - - :: - - >>> def foo(a, b=10): - ... pass - - >>> sig = signature(foo) - >>> ba = sig.bind(5) - - >>> ba.args, ba.kwargs - ((5,), {}) - - >>> for param in sig.parameters.values(): - ... if (param.name not in ba.arguments - ... and param.default is not param.empty): - ... ba.arguments[param.name] = param.default - - >>> ba.args, ba.kwargs - ((5, 10), {}) - + However, if needed, use :meth:`BoundArguments.apply_defaults` to add + them. .. attribute:: BoundArguments.args @@ -675,6 +734,26 @@ function. A reference to the parent :class:`Signature` object. + .. method:: BoundArguments.apply_defaults() + + Set default values for missing arguments. + + For variable-positional arguments (``*args``) the default is an + empty tuple. + + For variable-keyword arguments (``**kwargs``) the default is an + empty dict. + + :: + + >>> def foo(a, b='ham', *args): pass + >>> ba = inspect.signature(foo).bind('spam') + >>> ba.apply_defaults() + >>> ba.arguments + OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())]) + + .. versionadded:: 3.5 + The :attr:`args` and :attr:`kwargs` properties can be used to invoke functions:: @@ -719,8 +798,10 @@ Classes and functions *n* elements listed in *args*. .. deprecated:: 3.0 - Use :func:`getfullargspec` instead, which provides information about - keyword-only arguments and annotations. + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. This function will be removed + in Python 3.6. .. function:: getfullargspec(func) @@ -741,15 +822,16 @@ Classes and functions The first four items in the tuple correspond to :func:`getargspec`. - .. note:: - Consider using the new :ref:`Signature Object <inspect-signature-object>` - interface, which provides a better way of introspecting functions. - .. versionchanged:: 3.4 This function is now based on :func:`signature`, but still ignores ``__wrapped__`` attributes and includes the already bound first parameter in the signature output for bound methods. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: getargvalues(frame) @@ -759,6 +841,11 @@ Classes and functions are the names of the ``*`` and ``**`` arguments or ``None``. *locals* is the locals dictionary of the given frame. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: formatargspec(args[, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations[, formatarg, formatvarargs, formatvarkw, formatvalue, formatreturns, formatannotations]]) @@ -781,6 +868,11 @@ Classes and functions >>> formatargspec(*getfullargspec(f)) '(a: int, b: float)' + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: formatargvalues(args[, varargs, varkw, locals, formatarg, formatvarargs, formatvarkw, formatvalue]) @@ -788,6 +880,11 @@ Classes and functions :func:`getargvalues`. The format\* arguments are the corresponding optional formatting functions that are called to turn names and values into strings. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: getmro(cls) @@ -822,8 +919,8 @@ Classes and functions .. versionadded:: 3.2 - .. note:: - Consider using the new :meth:`Signature.bind` instead. + .. deprecated:: 3.5 + Use :meth:`Signature.bind` and :meth:`Signature.bind_partial` instead. .. function:: getclosurevars(func) @@ -864,11 +961,17 @@ Classes and functions The interpreter stack --------------------- -When the following functions return "frame records," each record is a tuple of -six items: the frame object, the filename, the line number of the current line, +When the following functions return "frame records," each record is a +:term:`named tuple` +``FrameInfo(frame, filename, lineno, function, code_context, index)``. +The tuple contains the frame object, the filename, the line number of the +current line, the function name, a list of lines of context from the source code, and the index of the current line within that list. +.. versionchanged:: 3.5 + Return a named tuple instead of a tuple. + .. note:: Keeping references to frame objects, as found in the first element of the frame @@ -1007,8 +1110,8 @@ code execution:: pass -Current State of a Generator ----------------------------- +Current State of Generators and Coroutines +------------------------------------------ When implementing coroutine schedulers and for other advanced uses of generators, it is useful to determine whether a generator is currently @@ -1028,6 +1131,21 @@ generator to be determined easily. .. versionadded:: 3.2 +.. function:: getcoroutinestate(coroutine) + + Get current state of a coroutine object. The function is intended to be + used with coroutine objects created by :keyword:`async def` functions, but + will accept any coroutine-like object that has ``cr_running`` and + ``cr_frame`` attributes. + + Possible states are: + * CORO_CREATED: Waiting to start execution. + * CORO_RUNNING: Currently being executed by the interpreter. + * CORO_SUSPENDED: Currently suspended at an await expression. + * CORO_CLOSED: Execution has completed. + + .. versionadded:: 3.5 + The current internal state of the generator can also be queried. This is mostly useful for testing purposes, to ensure that internal state is being updated as expected: @@ -1052,6 +1170,13 @@ updated as expected: .. versionadded:: 3.3 +.. function:: getcoroutinelocals(coroutine) + + This function is analogous to :func:`~inspect.getgeneratorlocals`, but + works for coroutine objects created by :keyword:`async def` functions. + + .. versionadded:: 3.5 + .. _inspect-module-cli: diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 3adf6e9..f009c65 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -339,8 +339,11 @@ I/O Base Classes if *size* is not specified). The current stream position isn't changed. This resizing can extend or reduce the current file size. In case of extension, the contents of the new file area depend on the platform - (on most systems, additional bytes are zero-filled, on Windows they're - undetermined). The new file size is returned. + (on most systems, additional bytes are zero-filled). The new file size + is returned. + + .. versionchanged:: 3.5 + Windows will now zero-fill files when extending. .. method:: writable() @@ -391,8 +394,8 @@ I/O Base Classes .. method:: readinto(b) Read up to ``len(b)`` bytes into :class:`bytearray` *b* and return the - number of bytes read. If the object is in non-blocking mode and no - bytes are available, ``None`` is returned. + number of bytes read. If the object is in non-blocking mode and no bytes + are available, ``None`` is returned. .. method:: write(b) @@ -465,9 +468,10 @@ I/O Base Classes .. method:: read1(size=-1) - Read and return up to *size* bytes, with at most one call to the underlying - raw stream's :meth:`~RawIOBase.read` method. This can be useful if you - are implementing your own buffering on top of a :class:`BufferedIOBase` + Read and return up to *size* bytes, with at most one call to the + underlying raw stream's :meth:`~RawIOBase.read` (or + :meth:`~RawIOBase.readinto`) method. This can be useful if you are + implementing your own buffering on top of a :class:`BufferedIOBase` object. .. method:: readinto(b) @@ -478,8 +482,19 @@ I/O Base Classes Like :meth:`read`, multiple reads may be issued to the underlying raw stream, unless the latter is interactive. - A :exc:`BlockingIOError` is raised if the underlying raw stream is in - non blocking-mode, and has no data available at the moment. + A :exc:`BlockingIOError` is raised if the underlying raw stream is in non + blocking-mode, and has no data available at the moment. + + .. method:: readinto1(b) + + Read up to ``len(b)`` bytes into bytearray *b*, ,using at most one call to + the underlying raw stream's :meth:`~RawIOBase.read` (or + :meth:`~RawIOBase.readinto`) method. Return the number of bytes read. + + A :exc:`BlockingIOError` is raised if the underlying raw stream is in non + blocking-mode, and has no data available at the moment. + + .. versionadded:: 3.5 .. method:: write(b) @@ -507,9 +522,12 @@ Raw File I/O The *name* can be one of two things: * a character string or :class:`bytes` object representing the path to the - file which will be opened; + file which will be opened. In this case closefd must be True (the default) + otherwise an error will be raised. * an integer representing the number of an existing OS-level file descriptor - to which the resulting :class:`FileIO` object will give access. + to which the resulting :class:`FileIO` object will give access. When the + FileIO object is closed this fd will be closed as well, unless *closefd* + is set to ``False``. The *mode* can be ``'r'``, ``'w'``, ``'x'`` or ``'a'`` for reading (default), writing, exclusive creation or appending. The file will be @@ -598,6 +616,11 @@ than raw I/O does. In :class:`BytesIO`, this is the same as :meth:`read`. + .. method:: readinto1() + + In :class:`BytesIO`, this is the same as :meth:`readinto`. + + .. versionadded:: 3.5 .. class:: BufferedReader(raw, buffer_size=DEFAULT_BUFFER_SIZE) @@ -807,11 +830,13 @@ Text I/O exception if there is an encoding error (the default of ``None`` has the same effect), or pass ``'ignore'`` to ignore errors. (Note that ignoring encoding errors can lead to data loss.) ``'replace'`` causes a replacement marker - (such as ``'?'``) to be inserted where there is malformed data. When - writing, ``'xmlcharrefreplace'`` (replace with the appropriate XML character - reference) or ``'backslashreplace'`` (replace with backslashed escape - sequences) can be used. Any other error handling name that has been - registered with :func:`codecs.register_error` is also valid. + (such as ``'?'``) to be inserted where there is malformed data. + ``'backslashreplace'`` causes malformed data to be replaced by a + backslashed escape sequence. When writing, ``'xmlcharrefreplace'`` + (replace with the appropriate XML character reference) or ``'namereplace'`` + (replace with ``\N{...}`` escape sequences) can be used. Any other error + handling name that has been registered with + :func:`codecs.register_error` is also valid. .. index:: single: universal newlines; io.TextIOWrapper class diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst index 99b5bea..7b59440 100644 --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -146,6 +146,20 @@ write code that handles both IP versions correctly. the appropriate length (most significant octet first). This is 4 bytes for IPv4 and 16 bytes for IPv6. + .. attribute:: reverse_pointer + + The name of the reverse DNS PTR record for the IP address, e.g.:: + + >>> ipaddress.ip_address("127.0.0.1").reverse_pointer + '1.0.0.127.in-addr.arpa' + >>> ipaddress.ip_address("2001:db8::1").reverse_pointer + '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa' + + This is the name that could be used for performing a PTR lookup, not the + resolved hostname itself. + + .. versionadded:: 3.5 + .. attribute:: is_multicast ``True`` if the address is reserved for multicast use. See @@ -226,6 +240,7 @@ write code that handles both IP versions correctly. :class:`IPv4Address` class: .. attribute:: packed + .. attribute:: reverse_pointer .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast @@ -377,6 +392,12 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. 3. An integer packed into a :class:`bytes` object of length 4, big-endian. The interpretation is similar to an integer *address*. + 4. A two-tuple of an address description and a netmask, where the address + description is either a string, a 32-bits integer, a 4-bytes packed + integer, or an existing IPv4Address object; and the netmask is either + an integer representing the prefix length (e.g. ``24``) or a string + representing the prefix mask (e.g. ``255.255.255.0``). + An :exc:`AddressValueError` is raised if *address* is not a valid IPv4 address. A :exc:`NetmaskValueError` is raised if the mask is not valid for an IPv4 address. @@ -389,6 +410,10 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. objects will raise :exc:`TypeError` if the argument's IP version is incompatible to ``self`` + .. versionchanged:: 3.5 + + Added the two-tuple form for the *address* constructor parameter. + .. attribute:: version .. attribute:: max_prefixlen @@ -553,6 +578,11 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. 3. An integer packed into a :class:`bytes` object of length 16, bit-endian. The interpretation is similar to an integer *address*. + 4. A two-tuple of an address description and a netmask, where the address + description is either a string, a 128-bits integer, a 16-bytes packed + integer, or an existing IPv4Address object; and the netmask is an + integer representing the prefix length. + An :exc:`AddressValueError` is raised if *address* is not a valid IPv6 address. A :exc:`NetmaskValueError` is raised if the mask is not valid for an IPv6 address. @@ -561,6 +591,10 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. then :exc:`ValueError` is raised. Otherwise, the host bits are masked out to determine the appropriate network address. + .. versionchanged:: 3.5 + + Added the two-tuple form for the *address* constructor parameter. + .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index f489535..8c7592d 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -87,10 +87,15 @@ loops that truncate the stream. .. function:: accumulate(iterable[, func]) - Make an iterator that returns accumulated sums. Elements may be any addable - type including :class:`~decimal.Decimal` or :class:`~fractions.Fraction`. - If the optional *func* argument is supplied, it should be a function of two - arguments and it will be used instead of addition. + Make an iterator that returns accumulated sums, or accumulated + results of other binary functions (specified via the optional + *func* argument). If *func* is supplied, it should be a function + of two arguments. Elements of the input *iterable* may be any type + that can be accepted as arguments to *func*. (For example, with + the default operation of addition, elements may be any addable + type including :class:`~decimal.Decimal` or + :class:`~fractions.Fraction`.) If the input iterable is empty, the + output iterable will also be empty. Equivalent to:: @@ -99,7 +104,10 @@ loops that truncate the stream. # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 it = iter(iterable) - total = next(it) + try: + total = next(it) + except StopIteration: + return yield total for element in it: total = func(total, element) @@ -400,7 +408,10 @@ loops that truncate the stream. def _grouper(self, tgtkey): while self.currkey == tgtkey: yield self.currvalue - self.currvalue = next(self.it) # Exit on StopIteration + try: + self.currvalue = next(self.it) + except StopIteration: + return self.currkey = self.keyfunc(self.currvalue) @@ -424,7 +435,10 @@ loops that truncate the stream. # islice('ABCDEFG', 0, None, 2) --> A C E G s = slice(*args) it = iter(range(s.start or 0, s.stop or sys.maxsize, s.step or 1)) - nexti = next(it) + try: + nexti = next(it) + except StopIteration: + return for i, element in enumerate(iterable): if i == nexti: yield element @@ -582,7 +596,10 @@ loops that truncate the stream. def gen(mydeque): while True: if not mydeque: # when the local deque is empty - newval = next(it) # fetch a new value and + try: + newval = next(it) # fetch a new value and + except StopIteration: + return for d in deques: # load it to all the deques d.append(newval) yield mydeque.popleft() @@ -657,6 +674,11 @@ which incur interpreter overhead. "Return function(0), function(1), ..." return map(function, count(start)) + def tail(n, iterable): + "Return an iterator over the last n items" + # tail(3, 'ABCDEFG') --> E F G + return iter(collections.deque(iterable, maxlen=n)) + def consume(iterator, n): "Advance the iterator n-steps ahead. If n is none, consume entirely." # Use functions that consume iterators at C speed. diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 6f5f8b1..49bb090 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -106,6 +106,8 @@ Using json.tool from the shell to validate and pretty-print:: $ echo '{1.2:3.4}' | python -m json.tool Expecting property name enclosed in double quotes: line 1 column 2 (char 1) +See :ref:`json-commandline` for detailed documentation. + .. highlight:: python3 .. note:: @@ -248,7 +250,7 @@ Basic Usage will be passed to the constructor of the class. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. function:: loads(s, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) @@ -259,7 +261,7 @@ Basic Usage *encoding* which is ignored and deprecated. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. Encoders and Decoders --------------------- @@ -332,13 +334,16 @@ Encoders and Decoders ``'\n'``, ``'\r'`` and ``'\0'``. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. method:: decode(s) Return the Python representation of *s* (a :class:`str` instance containing a JSON document) + :exc:`JSONDecodeError` will be raised if the given JSON document is not + valid. + .. method:: raw_decode(s) Decode a JSON document from *s* (a :class:`str` beginning with a @@ -467,6 +472,36 @@ Encoders and Decoders mysocket.write(chunk) +Exceptions +---------- + +.. exception:: JSONDecodeError(msg, doc, pos, end=None) + + Subclass of :exc:`ValueError` with the following additional attributes: + + .. attribute:: msg + + The unformatted error message. + + .. attribute:: doc + + The JSON document being parsed. + + .. attribute:: pos + + The start index of *doc* where parsing failed. + + .. attribute:: lineno + + The line corresponding to *pos*. + + .. attribute:: colno + + The column corresponding to *pos*. + + .. versionadded:: 3.5 + + Standard Compliance and Interoperability ---------------------------------------- @@ -588,6 +623,68 @@ when serializing Python :class:`int` values of extremely large magnitude, or when serializing instances of "exotic" numerical types such as :class:`decimal.Decimal`. +.. highlight:: bash +.. module:: json.tool + +.. _json-commandline: + +Command Line Interface +---------------------- + +The :mod:`json.tool` module provides a simple command line interface to validate +and pretty-print JSON objects. + +If the optional ``infile`` and ``outfile`` arguments are not +specified, :attr:`sys.stdin` and :attr:`sys.stdout` will be used respectively:: + + $ echo '{"json": "obj"}' | python -m json.tool + { + "json": "obj" + } + $ echo '{1.2:3.4}' | python -m json.tool + Expecting property name enclosed in double quotes: line 1 column 2 (char 1) + +.. versionchanged:: 3.5 + The output is now in the same order as the input. Use the + :option:`--sort-keys` option to sort the output of dictionaries + alphabetically by key. + +Command line options +^^^^^^^^^^^^^^^^^^^^ + +.. cmdoption:: infile + + The JSON file to be validated or pretty-printed:: + + $ python -m json.tool mp_films.json + [ + { + "title": "And Now for Something Completely Different", + "year": 1971 + }, + { + "title": "Monty Python and the Holy Grail", + "year": 1975 + } + ] + + If *infile* is not specified, read from :attr:`sys.stdin`. + +.. cmdoption:: outfile + + Write the output of the *infile* to the given *outfile*. Otherwise, write it + to :attr:`sys.stdout`. + +.. cmdoption:: --sort-keys + + Sort the output of dictionaries alphabetically by key. + + .. versionadded:: 3.5 + +.. cmdoption:: -h, --help + + Show the help message. + .. rubric:: Footnotes diff --git a/Doc/library/linecache.rst b/Doc/library/linecache.rst index f18b1cd..6c92cc5 100644 --- a/Doc/library/linecache.rst +++ b/Doc/library/linecache.rst @@ -47,6 +47,14 @@ The :mod:`linecache` module defines the following functions: changed on disk, and you require the updated version. If *filename* is omitted, it will check all the entries in the cache. +.. function:: lazycache(filename, module_globals) + + Capture enough detail about a non-file based module to permit getting its + lines later via :func:`getline` even if *module_globals* is None in the later + call. This avoids doing I/O until a line is actually needed, without having + to carry the module globals around indefinitely. + + .. versionadded:: 3.5 Example:: diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index 9600193..bc7f5f9 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -387,6 +387,14 @@ The :mod:`locale` module defines the following exception and functions: ``str(float)``, but takes the decimal point into account. +.. function:: delocalize(string) + + Converts a string into a normalized number string, following the + :const:`LC_NUMERIC` settings. + + .. versionadded:: 3.5 + + .. function:: atof(string) Converts a string to a floating point number, following the :const:`LC_NUMERIC` diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst index d946892..67403a9 100644 --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -853,7 +853,7 @@ supports sending logging messages to a Web server, using either ``GET`` or credentials, you should also specify secure=True so that your userid and password are not passed in cleartext across the wire. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 The *context* parameter was added. .. method:: mapLogRecord(record) @@ -953,13 +953,20 @@ applications where threads servicing clients need to respond as quickly as possible, while any potentially slow operations (such as sending an email via :class:`SMTPHandler`) are done on a separate thread. -.. class:: QueueListener(queue, *handlers) +.. class:: QueueListener(queue, *handlers, respect_handler_level=False) Returns a new instance of the :class:`QueueListener` class. The instance is initialized with the queue to send messages to and a list of handlers which will handle entries placed on the queue. The queue can be any queue- like object; it's passed as-is to the :meth:`dequeue` method, which needs - to know how to get messages from it. + to know how to get messages from it. If ``respect_handler_level`` is ``True``, + a handler's level is respected (compared with the level for the message) when + deciding whether to pass messages to that handler; otherwise, the behaviour + is as in previous Python versions - to always pass each message to each + handler. + + .. versionchanged:: 3.5 + The ``respect_handler_levels`` argument was added. .. method:: dequeue(block) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 90fb9b0..bf821ab 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -159,11 +159,13 @@ is the module's name in the Python package namespace. *msg* using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.) - There are three keyword arguments in *kwargs* which are inspected: *exc_info* - which, if it does not evaluate as false, causes exception information to be + There are three keyword arguments in *kwargs* which are inspected: + *exc_info*, *stack_info*, and *extra*. + + If *exc_info* does not evaluate as false, it causes exception information to be added to the logging message. If an exception tuple (in the format returned by - :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` - is called to get the exception information. + :func:`sys.exc_info`) or an exception instance is provided, it is used; + otherwise, :func:`sys.exc_info` is called to get the exception information. The second optional keyword argument is *stack_info*, which defaults to ``False``. If true, stack information is added to the logging @@ -220,6 +222,9 @@ is the module's name in the Python package namespace. .. versionadded:: 3.2 The *stack_info* parameter was added. + .. versionchanged:: 3.5 + The *exc_info* parameter can now accept exception instances. + .. method:: Logger.info(msg, *args, **kwargs) diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst index b71051d..0546005 100644 --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -110,6 +110,10 @@ Reading and writing compressed files .. versionchanged:: 3.4 Added support for the ``"x"`` and ``"xb"`` modes. + .. versionchanged:: 3.5 + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + Compressing and decompressing data in memory -------------------------------------------- @@ -221,13 +225,32 @@ Compressing and decompressing data in memory decompress a multi-stream input with :class:`LZMADecompressor`, you must create a new decompressor for each stream. - .. method:: decompress(data) + .. method:: decompress(data, max_length=-1) + + Decompress *data* (a :term:`bytes-like object`), returning + uncompressed data as bytes. Some of *data* may be buffered + internally, for use in later calls to :meth:`decompress`. The + returned data should be concatenated with the output of any + previous calls to :meth:`decompress`. + + If *max_length* is nonnegative, returns at most *max_length* + bytes of decompressed data. If this limit is reached and further + output can be produced, the :attr:`~.needs_input` attribute will + be set to ``False``. In this case, the next call to + :meth:`~.decompress` may provide *data* as ``b''`` to obtain + more of the output. + + If all of the input data was decompressed and returned (either + because this was less than *max_length* bytes, or because + *max_length* was negative), the :attr:`~.needs_input` attribute + will be set to ``True``. - Decompress *data* (a :class:`bytes` object), returning a :class:`bytes` - object containing the decompressed data for at least part of the input. - Some of *data* may be buffered internally, for use in later calls to - :meth:`decompress`. The returned data should be concatenated with the - output of any previous calls to :meth:`decompress`. + Attempting to decompress data after the end of stream is reached + raises an `EOFError`. Any data found after the end of the + stream is ignored and saved in the :attr:`~.unused_data` attribute. + + .. versionchanged:: 3.5 + Added the *max_length* parameter. .. attribute:: check @@ -245,6 +268,12 @@ Compressing and decompressing data in memory Before the end of the stream is reached, this will be ``b""``. + .. attribute:: needs_input + + ``False`` if the :meth:`.decompress` method can provide more + decompressed data before requiring new uncompressed input. + + .. versionadded:: 3.5 .. function:: compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None) diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 0fc7c7c..244663e 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -100,6 +100,48 @@ Number-theoretic and representation functions <http://code.activestate.com/recipes/393090/>`_\. +.. function:: gcd(a, b) + + Return the greatest common divisor of the integers *a* and *b*. If either + *a* or *b* is nonzero, then the value of ``gcd(a, b)`` is the largest + positive integer that divides both *a* and *b*. ``gcd(0, 0)`` returns + ``0``. + + .. versionadded:: 3.5 + + +.. function:: isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0) + + Return ``True`` if the values *a* and *b* are close to each other and + ``False`` otherwise. + + Whether or not two values are considered close is determined according to + given absolute and relative tolerances. + + *rel_tol* is the relative tolerance -- it is the maximum allowed difference + between *a* and *b*, relative to the larger absolute value of *a* or *b*. + For example, to set a tolerance of 5%, pass ``rel_tol=0.05``. The default + tolerance is ``1e-09``, which assures that the two values are the same + within about 9 decimal digits. *rel_tol* must be greater than zero. + + *abs_tol* is the minimum absolute tolerance -- useful for comparisons near + zero. *abs_tol* must be at least zero. + + If no errors occur, the result will be: + ``abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)``. + + The IEEE 754 special values of ``NaN``, ``inf``, and ``-inf`` will be + handled according to IEEE rules. Specifically, ``NaN`` is not considered + close to any other value, including ``NaN``. ``inf`` and ``-inf`` are only + considered close to themselves. + + .. versionadded:: 3.5 + + .. seealso:: + + :pep:`485` -- A function for testing approximate equality + + .. function:: isfinite(x) Return ``True`` if *x* is neither an infinity nor a NaN, and @@ -383,6 +425,22 @@ Constants The mathematical constant e = 2.718281..., to available precision. +.. data:: inf + + A floating-point positive infinity. (For negative infinity, use + ``-math.inf``.) Equivalent to the output of ``float('inf')``. + + .. versionadded:: 3.5 + + +.. data:: nan + + A floating-point "not a number" (NaN) value. Equivalent to the output of + ``float('nan')``. + + .. versionadded:: 3.5 + + .. impl-detail:: The :mod:`math` module consists mostly of thin wrappers around the platform C diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index 18e05e3..b74a823 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -174,6 +174,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Optional arguments *start* and *end* are interpreted as in slice notation. Returns ``-1`` on failure. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: flush([offset[, size]]) @@ -234,6 +237,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Optional arguments *start* and *end* are interpreted as in slice notation. Returns ``-1`` on failure. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: seek(pos[, whence]) @@ -261,6 +267,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length were written. If the mmap was created with :const:`ACCESS_READ`, then writing to it will raise a :exc:`TypeError` exception. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: write_byte(byte) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 4d5f308..8703f8f 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -1352,6 +1352,9 @@ processes. Note that accessing the ctypes object through the wrapper can be a lot slower than accessing the raw ctypes object. + .. versionchanged:: 3.5 + Synchronized objects support the :term:`context manager` protocol. + The table below compares the syntax for creating shared ctypes objects from shared memory with the normal ctypes syntax. (In the table ``MyStruct`` is some @@ -1839,9 +1842,9 @@ itself. This means, for example, that one shared object can contain a second: >>> l = manager.list(range(10)) >>> l._callmethod('__len__') 10 - >>> l._callmethod('__getitem__', (slice(2, 7),)) # equiv to `l[2:7]` + >>> l._callmethod('__getitem__', (slice(2, 7),)) # equivalent to l[2:7] [2, 3, 4, 5, 6] - >>> l._callmethod('__getitem__', (20,)) # equiv to `l[20]` + >>> l._callmethod('__getitem__', (20,)) # equivalent to l[20] Traceback (most recent call last): ... IndexError: list index out of range diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst index f9e2a3d..c01e63b 100644 --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -138,6 +138,14 @@ The mathematical and bitwise operations are the most numerous: Return ``a * b``, for *a* and *b* numbers. +.. function:: matmul(a, b) + __matmul__(a, b) + + Return ``a @ b``. + + .. versionadded:: 3.5 + + .. function:: neg(obj) __neg__(obj) @@ -391,6 +399,8 @@ Python syntax and the functions in the :mod:`operator` module. +-----------------------+-------------------------+---------------------------------------+ | Multiplication | ``a * b`` | ``mul(a, b)`` | +-----------------------+-------------------------+---------------------------------------+ +| Matrix Multiplication | ``a @ b`` | ``matmul(a, b)`` | ++-----------------------+-------------------------+---------------------------------------+ | Negation (Arithmetic) | ``- a`` | ``neg(a)`` | +-----------------------+-------------------------+---------------------------------------+ | Negation (Logical) | ``not a`` | ``not_(a)`` | @@ -499,6 +509,14 @@ will perform the update, so no subsequent assignment is necessary: ``a = imul(a, b)`` is equivalent to ``a *= b``. +.. function:: imatmul(a, b) + __imatmul__(a, b) + + ``a = imatmul(a, b)`` is equivalent to ``a @= b``. + + .. versionadded:: 3.5 + + .. function:: ior(a, b) __ior__(a, b) diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index 92631b2..e4fe44e 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -66,11 +66,24 @@ the :mod:`glob` module.) empty string (``''``). +.. function:: commonpath(paths) + + Return the longest common sub-path of each pathname in the sequence + *paths*. Raise ValueError if *paths* contains both absolute and relative + pathnames, or if *paths* is empty. Unlike :func:`commonprefix`, this + returns a valid path. + + Availability: Unix, Windows + + .. versionadded:: 3.5 + + .. function:: commonprefix(list) - Return the longest path prefix (taken character-by-character) that is a prefix - of all paths in *list*. If *list* is empty, return the empty string (``''``). - Note that this may return invalid paths because it works a character at a time. + Return the longest path prefix (taken character-by-character) that is a + prefix of all paths in *list*. If *list* is empty, return the empty string + (``''``). Note that this may return invalid paths because it works a + character at a time. To obtain a valid path, see :func:`commonpath`. .. function:: dirname(path) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index d677b39..16e5019 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -78,9 +78,10 @@ uses the file system encoding to perform this conversion (see .. versionchanged:: 3.1 On some systems, conversion using the file system encoding may fail. In this - case, Python uses the ``surrogateescape`` encoding error handler, which means - that undecodable bytes are replaced by a Unicode character U+DCxx on - decoding, and these are again translated to the original byte on encoding. + case, Python uses the :ref:`surrogateescape encoding error handler + <surrogateescape>`, which means that undecodable bytes are replaced by a + Unicode character U+DCxx on decoding, and these are again translated to the + original byte on encoding. The file system encoding must guarantee to successfully decode all bytes @@ -311,8 +312,6 @@ process and user. Return the current process id. - Availability: Unix, Windows. - .. function:: getppid() @@ -549,8 +548,6 @@ process and user. On platforms where :c:func:`strerror` returns ``NULL`` when given an unknown error number, :exc:`ValueError` is raised. - Availability: Unix, Windows. - .. data:: supports_bytes_environ @@ -564,8 +561,6 @@ process and user. Set the current numeric umask and return the previous umask. - Availability: Unix, Windows. - .. function:: uname() @@ -656,8 +651,6 @@ as internal buffering of data. Close file descriptor *fd*. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -677,8 +670,6 @@ as internal buffering of data. except OSError: pass - Availability: Unix, Windows. - .. function:: device_encoding(fd) @@ -695,8 +686,6 @@ as internal buffering of data. 2: stderr), the new file descriptor is :ref:`inheritable <fd_inheritance>`. - Availability: Unix, Windows. - .. versionchanged:: 3.4 The new file descriptor is now non-inheritable. @@ -707,8 +696,6 @@ as internal buffering of data. The file descriptor *fd2* is :ref:`inheritable <fd_inheritance>` by default, or non-inheritable if *inheritable* is ``False``. - Availability: Unix, Windows. - .. versionchanged:: 3.4 Add the optional *inheritable* parameter. @@ -774,8 +761,6 @@ as internal buffering of data. The :func:`.stat` function. - Availability: Unix, Windows. - .. function:: fstatvfs(fd) @@ -804,8 +789,21 @@ as internal buffering of data. most *length* bytes in size. As of Python 3.3, this is equivalent to ``os.truncate(fd, length)``. + Availability: Unix, Windows. + + .. versionchanged:: 3.5 + Added support for Windows + +.. function:: get_blocking(fd) + + Get the blocking mode of the file descriptor: ``False`` if the + :data:`O_NONBLOCK` flag is set, ``True`` if the flag is cleared. + + See also :func:`set_blocking` and :meth:`socket.socket.setblocking`. + Availability: Unix. + .. versionadded:: 3.5 .. function:: isatty(fd) @@ -846,8 +844,6 @@ as internal buffering of data. current position; :const:`SEEK_END` or ``2`` to set it relative to the end of the file. Return the new cursor position in bytes, starting from the beginning. - Availability: Unix, Windows. - .. data:: SEEK_SET SEEK_CUR @@ -856,8 +852,6 @@ as internal buffering of data. Parameters to the :func:`lseek` function. Their values are 0, 1, and 2, respectively. - Availability: Unix, Windows. - .. versionadded:: 3.3 Some operating systems could support additional values, like :data:`os.SEEK_HOLE` or :data:`os.SEEK_DATA`. @@ -878,8 +872,6 @@ as internal buffering of data. This function can support :ref:`paths relative to directory descriptors <dir_fd>` with the *dir_fd* parameter. - Availability: Unix, Windows. - .. versionchanged:: 3.4 The new file descriptor is now non-inheritable. @@ -893,6 +885,11 @@ as internal buffering of data. .. versionadded:: 3.3 The *dir_fd* argument. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + The following constants are options for the *flags* parameter to the :func:`~os.open` function. They can be combined using the bitwise OR operator ``|``. Some of them are not available on all platforms. For descriptions of @@ -1060,8 +1057,6 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window bytes read. If the end of the file referred to by *fd* has been reached, an empty bytes object is returned. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -1070,6 +1065,11 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :func:`popen` or :func:`fdopen`, or :data:`sys.stdin`, use its :meth:`~file.read` or :meth:`~file.readline` methods. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: sendfile(out, in, offset, nbytes) sendfile(out, in, offset, nbytes, headers=None, trailers=None, flags=0) @@ -1107,6 +1107,18 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window .. versionadded:: 3.3 +.. function:: set_blocking(fd, blocking) + + Set the blocking mode of the specified file descriptor. Set the + :data:`O_NONBLOCK` flag if blocking is ``False``, clear the flag otherwise. + + See also :func:`get_blocking` and :meth:`socket.socket.setblocking`. + + Availability: Unix. + + .. versionadded:: 3.5 + + .. data:: SF_NODISKIO SF_MNOWAIT SF_SYNC @@ -1163,8 +1175,6 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window Write the bytestring in *str* to file descriptor *fd*. Return the number of bytes actually written. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -1173,6 +1183,11 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :func:`fdopen`, or :data:`sys.stdout` or :data:`sys.stderr`, use its :meth:`~file.write` method. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: writev(fd, buffers) @@ -1335,8 +1350,6 @@ features: or not it is available using :data:`os.supports_effective_ids`. If it is unavailable, using it will raise a :exc:`NotImplementedError`. - Availability: Unix, Windows. - .. note:: Using :func:`access` to check if a user is authorized to e.g. open a file @@ -1389,8 +1402,6 @@ features: This function can support :ref:`specifying a file descriptor <path_fd>`. The descriptor must refer to an opened directory, not an open file. - Availability: Unix, Windows. - .. versionadded:: 3.3 Added support for specifying *path* as a file descriptor on some platforms. @@ -1452,8 +1463,6 @@ features: :ref:`paths relative to directory descriptors <dir_fd>` and :ref:`not following symlinks <follow_symlinks>`. - Availability: Unix, Windows. - .. note:: Although Windows supports :func:`chmod`, you can only set the file's @@ -1504,15 +1513,11 @@ features: Return a string representing the current working directory. - Availability: Unix, Windows. - .. function:: getcwdb() Return a bytestring representing the current working directory. - Availability: Unix, Windows. - .. function:: lchflags(path, flags) @@ -1575,7 +1580,11 @@ features: .. note:: To encode ``str`` filenames to ``bytes``, use :func:`~os.fsencode`. - Availability: Unix, Windows. + .. seealso:: + + The :func:`scandir` function returns directory entries along with + file attribute information, giving better performance for many + common use cases. .. versionchanged:: 3.2 The *path* parameter became optional. @@ -1624,8 +1633,6 @@ features: It is also possible to create temporary directories; see the :mod:`tempfile` module's :func:`tempfile.mkdtemp` function. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* argument. @@ -1784,8 +1791,6 @@ features: This function is identical to :func:`unlink`. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* argument. @@ -1819,8 +1824,6 @@ features: If you want cross-platform overwriting of the destination, use :func:`replace`. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *src_dir_fd* and *dst_dir_fd* arguments. @@ -1849,8 +1852,6 @@ features: This function can support specifying *src_dir_fd* and/or *dst_dir_fd* to supply :ref:`paths relative to directory descriptors <dir_fd>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 @@ -1863,12 +1864,180 @@ features: This function can support :ref:`paths relative to directory descriptors <dir_fd>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* parameter. +.. function:: scandir(path='.') + + Return an iterator of :class:`DirEntry` objects corresponding to the entries + in the directory given by *path*. The entries are yielded in arbitrary + order, and the special entries ``'.'`` and ``'..'`` are not included. + + Using :func:`scandir` instead of :func:`listdir` can significantly + increase the performance of code that also needs file type or file + attribute information, because :class:`DirEntry` objects expose this + information if the operating system provides it when scanning a directory. + All :class:`DirEntry` methods may perform a system call, but + :func:`~DirEntry.is_dir` and :func:`~DirEntry.is_file` usually only + require a system call for symbolic links; :func:`DirEntry.stat` + always requires a system call on Unix but only requires one for + symbolic links on Windows. + + On Unix, *path* can be of type :class:`str` or :class:`bytes` (use + :func:`~os.fsencode` and :func:`~os.fsdecode` to encode and decode + :class:`bytes` paths). On Windows, *path* must be of type :class:`str`. + On both sytems, the type of the :attr:`~DirEntry.name` and + :attr:`~DirEntry.path` attributes of each :class:`DirEntry` will be of + the same type as *path*. + + The following example shows a simple use of :func:`scandir` to display all + the files (excluding directories) in the given *path* that don't start with + ``'.'``. The ``entry.is_file()`` call will generally not make an additional + system call:: + + for entry in os.scandir(path): + if not entry.name.startswith('.') and entry.is_file(): + print(entry.name) + + .. note:: + + On Unix-based systems, :func:`scandir` uses the system's + `opendir() <http://pubs.opengroup.org/onlinepubs/009695399/functions/opendir.html>`_ + and + `readdir() <http://pubs.opengroup.org/onlinepubs/009695399/functions/readdir_r.html>`_ + functions. On Windows, it uses the Win32 + `FindFirstFileW <http://msdn.microsoft.com/en-us/library/windows/desktop/aa364418(v=vs.85).aspx>`_ + and + `FindNextFileW <http://msdn.microsoft.com/en-us/library/windows/desktop/aa364428(v=vs.85).aspx>`_ + functions. + + .. versionadded:: 3.5 + + +.. class:: DirEntry + + Object yielded by :func:`scandir` to expose the file path and other file + attributes of a directory entry. + + :func:`scandir` will provide as much of this information as possible without + making additional system calls. When a ``stat()`` or ``lstat()`` system call + is made, the ``DirEntry`` object will cache the result. + + ``DirEntry`` instances are not intended to be stored in long-lived data + structures; if you know the file metadata has changed or if a long time has + elapsed since calling :func:`scandir`, call ``os.stat(entry.path)`` to fetch + up-to-date information. + + Because the ``DirEntry`` methods can make operating system calls, they may + also raise :exc:`OSError`. If you need very fine-grained + control over errors, you can catch :exc:`OSError` when calling one of the + ``DirEntry`` methods and handle as appropriate. + + Attributes and methods on a ``DirEntry`` instance are as follows: + + .. attribute:: name + + The entry's base filename, relative to the :func:`scandir` *path* + argument. + + The :attr:`name` attribute will be of the same type (``str`` or + ``bytes``) as the :func:`scandir` *path* argument. Use + :func:`~os.fsdecode` to decode byte filenames. + + .. attribute:: path + + The entry's full path name: equivalent to ``os.path.join(scandir_path, + entry.name)`` where *scandir_path* is the :func:`scandir` *path* + argument. The path is only absolute if the :func:`scandir` *path* + argument was absolute. + + The :attr:`path` attribute will be of the same type (``str`` or + ``bytes``) as the :func:`scandir` *path* argument. Use + :func:`~os.fsdecode` to decode byte filenames. + + .. method:: inode() + + Return the inode number of the entry. + + The result is cached on the ``DirEntry`` object, use ``os.stat(entry.path, + follow_symlinks=False).st_ino`` to fetch up-to-date information. + + On Unix, no system call is required. + + .. method:: is_dir(\*, follow_symlinks=True) + + If *follow_symlinks* is ``True`` (the default), return ``True`` if the + entry is a directory or a symbolic link pointing to a directory; + return ``False`` if it is or points to any other kind of file, or if it + doesn't exist anymore. + + If *follow_symlinks* is ``False``, return ``True`` only if this entry + is a directory; return ``False`` if it is any other kind of file + or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object. Call :func:`os.stat` + along with :func:`stat.S_ISDIR` to fetch up-to-date information. + + This method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is caught and not raised. + + In most cases, no system call is required. + + .. method:: is_file(\*, follow_symlinks=True) + + If *follow_symlinks* is ``True`` (the default), return ``True`` if the + entry is a file or a symbolic link pointing to a file; return ``False`` + if it is or points to a directory or other non-file entry, or if it + doesn't exist anymore. + + If *follow_symlinks* is ``False``, return ``True`` only if this entry + is a file; return ``False`` if it is a directory or other non-file entry, + or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object. Call :func:`os.stat` + along with :func:`stat.S_ISREG` to fetch up-to-date information. + + This method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is caught and not raised. + + In most cases, no system call is required. + + .. method:: is_symlink() + + Return ``True`` if this entry is a symbolic link (even if broken); + return ``False`` if it points to a directory or any kind of file, + or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object. Call + :func:`os.path.islink` to fetch up-to-date information. + + The method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is caught and not raised. + + In most cases, no system call is required. + + .. method:: stat(\*, follow_symlinks=True) + + Return a :class:`stat_result` object for this entry. This method + follows symbolic links by default; to stat a symbolic link add the + ``follow_symlinks=False`` argument. + + On Unix, this method always requires a system call. On Windows, + ``DirEntry.stat()`` requires a system call only if the + entry is a symbolic link, and ``DirEntry.stat(follow_symlinks=False)`` + never requires a system call. + + On Windows, the ``st_ino``, ``st_dev`` and ``st_nlink`` attributes of the + :class:`stat_result` are always set to zero. Call :func:`os.stat` to + get these attributes. + + The result is cached on the ``DirEntry`` object. Call :func:`os.stat` + to fetch up-to-date information. + + .. versionadded:: 3.5 + + .. function:: stat(path, \*, dir_fd=None, follow_symlinks=True) Get the status of a file or a file descriptor. Perform the equivalent of a @@ -1895,8 +2064,6 @@ features: >>> statinfo.st_size 264 - Availability: Unix, Windows. - .. seealso:: :func:`fstat` and :func:`lstat` functions. @@ -2044,6 +2211,15 @@ features: File type. + On Windows systems, the following attribute is also available: + + .. attribute:: st_file_attributes + + Windows file attributes: ``dwFileAttributes`` member of the + ``BY_HANDLE_FILE_INFORMATION`` structure returned by + :c:func:`GetFileInformationByHandle`. See the ``FILE_ATTRIBUTE_*`` + constants in the :mod:`stat` module. + The standard module :mod:`stat` defines functions and constants that are useful for extracting information from a :c:type:`stat` structure. (On Windows, some items are filled with dummy values.) @@ -2061,6 +2237,9 @@ features: Added the :attr:`st_atime_ns`, :attr:`st_mtime_ns`, and :attr:`st_ctime_ns` members. + .. versionadded:: 3.5 + Added the :attr:`st_file_attributes` member on Windows. + .. function:: stat_float_times([newvalue]) @@ -2264,10 +2443,12 @@ features: This function can support :ref:`specifying a file descriptor <path_fd>`. - Availability: Unix. + Availability: Unix, Windows. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + Added support for Windows .. function:: unlink(path, *, dir_fd=None) @@ -2276,8 +2457,6 @@ features: name. Please see the documentation for :func:`remove` for further information. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* parameter. @@ -2314,8 +2493,6 @@ features: :ref:`paths relative to directory descriptors <dir_fd>` and :ref:`not following symlinks <follow_symlinks>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 Added support for specifying an open file descriptor for *path*, and the *dir_fd*, *follow_symlinks*, and *ns* parameters. @@ -2406,6 +2583,10 @@ features: for name in dirs: os.rmdir(os.path.join(root, name)) + .. versionchanged:: 3.5 + This function now calls :func:`os.scandir` instead of :func:`os.listdir`, + making it faster by reducing the number of calls to :func:`os.stat`. + .. function:: fwalk(top='.', topdown=True, onerror=None, *, follow_symlinks=False, dir_fd=None) @@ -2562,8 +2743,6 @@ to be ignored. Python signal handler registered for :const:`SIGABRT` with :func:`signal.signal`. - Availability: Unix, Windows. - .. function:: execl(path, arg0, arg1, ...) execle(path, arg0, arg1, ..., env) @@ -2627,8 +2806,6 @@ to be ignored. Exit the process with status *n*, without calling cleanup handlers, flushing stdio buffers, etc. - Availability: Unix, Windows. - .. note:: The standard way to exit is ``sys.exit(n)``. :func:`_exit` should @@ -2990,6 +3167,10 @@ written in Python, such as a mail server's external command delivery program. doesn't work if it is. Use the :func:`os.path.normpath` function to ensure that the path is properly encoded for Win32. + To reduce interpreter startup overhead, the Win32 :c:func:`ShellExecute` + function is not resolved until this function is first called. If the function + cannot be resolved, :exc:`NotImplementedError` will be raised. + Availability: Windows. @@ -3137,6 +3318,11 @@ written in Python, such as a mail server's external command delivery program. id is known, not necessarily a child process. The :func:`spawn\* <spawnl>` functions called with :const:`P_NOWAIT` return suitable process handles. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: wait3(options) diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst index bb5081a..c60d596 100644 --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -148,21 +148,30 @@ and (read-only) attributes: .. method:: oss_audio_device.write(data) - Write the Python string *data* to the audio device and return the number of - bytes written. If the audio device is in blocking mode (the default), the - entire string is always written (again, this is different from usual Unix device - semantics). If the device is in non-blocking mode, some data may not be written + Write a :term:`bytes-like object` *data* to the audio device and return the + number of bytes written. If the audio device is in blocking mode (the + default), the entire data is always written (again, this is different from + usual Unix device semantics). If the device is in non-blocking mode, some + data may not be written ---see :meth:`writeall`. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: oss_audio_device.writeall(data) - Write the entire Python string *data* to the audio device: waits until the audio - device is able to accept data, writes as much data as it will accept, and - repeats until *data* has been completely written. If the device is in blocking - mode (the default), this has the same effect as :meth:`write`; :meth:`writeall` - is only useful in non-blocking mode. Has no return value, since the amount of - data written is always equal to the amount of data supplied. + Write a :term:`bytes-like object` *data* to the audio device: waits until + the audio device is able to accept data, writes as much data as it will + accept, and repeats until *data* has been completely written. If the device + is in blocking mode (the default), this has the same effect as + :meth:`write`; :meth:`writeall` is only useful in non-blocking mode. Has + no return value, since the amount of data written is always equal to the + amount of data supplied. + + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. versionchanged:: 3.2 Audio device objects also support the context management protocol, i.e. they can diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 24e2a30..2f06544 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -628,6 +628,17 @@ call fails (for example because the path doesn't exist): PosixPath('/home/antoine/pathlib') +.. classmethod:: Path.home() + + Return a new path object representing the user's home directory (as + returned by :func:`os.path.expanduser` with ``~`` construct):: + + >>> Path.home() + PosixPath('/home/antoine') + + .. versionadded:: 3.5 + + .. method:: Path.stat() Return information about this path (similarly to :func:`os.stat`). @@ -670,6 +681,18 @@ call fails (for example because the path doesn't exist): symlink *points to* an existing file or directory. +.. method:: Path.expanduser() + + Return a new path with expanded ``~`` and ``~user`` constructs, + as returned by :meth:`os.path.expanduser`:: + + >>> p = PosixPath('~/films/Monty Python') + >>> p.expanduser() + PosixPath('/home/eric/films/Monty Python') + + .. versionadded:: 3.5 + + .. method:: Path.glob(pattern) Glob the given *pattern* in the directory represented by this path, @@ -791,7 +814,7 @@ call fails (for example because the path doesn't exist): the symbolic link's information rather than its target's. -.. method:: Path.mkdir(mode=0o777, parents=False) +.. method:: Path.mkdir(mode=0o777, parents=False, exist_ok=False) Create a new directory at this given path. If *mode* is given, it is combined with the process' ``umask`` value to determine the file mode @@ -805,6 +828,16 @@ call fails (for example because the path doesn't exist): If *parents* is false (the default), a missing parent raises :exc:`FileNotFoundError`. + If *exist_ok* is false (the default), an :exc:`FileExistsError` is + raised if the target directory already exists. + + If *exist_ok* is true, :exc:`FileExistsError` exceptions will be + ignored (same behavior as the POSIX ``mkdir -p`` command), but only if the + last path component is not an existing non-directory file. + + .. versionchanged:: 3.5 + The *exist_ok* parameter was added. + .. method:: Path.open(mode='r', buffering=-1, encoding=None, errors=None, newline=None) @@ -824,6 +857,34 @@ call fails (for example because the path doesn't exist): if the file's uid isn't found in the system database. +.. method:: Path.read_bytes() + + Return the binary contents of the pointed-to file as a bytes object:: + + >>> p = Path('my_binary_file') + >>> p.write_bytes(b'Binary file contents') + 20 + >>> p.read_bytes() + b'Binary file contents' + + .. versionadded:: 3.5 + + +.. method:: Path.read_text(encoding=None, errors=None) + + Return the decoded contents of the pointed-to file as a string:: + + >>> p = Path('my_text_file') + >>> p.write_text('Text file contents') + 18 + >>> p.read_text() + 'Text file contents' + + The optional parameters have the same meaning as in :func:`open`. + + .. versionadded:: 3.5 + + .. method:: Path.rename(target) Rename this file or directory to the given *target*. *target* can be @@ -884,6 +945,25 @@ call fails (for example because the path doesn't exist): Remove this directory. The directory must be empty. +.. method:: Path.samefile(other_path) + + Return whether this path points to the same file as *other_path*, which + can be either a Path object, or a string. The semantics are similar + to :func:`os.path.samefile` and :func:`os.path.samestat`. + + An :exc:`OSError` can be raised if either file cannot be accessed for some + reason. + + >>> p = Path('spam') + >>> q = Path('eggs') + >>> p.samefile(q) + False + >>> p.samefile('spam') + True + + .. versionadded:: 3.5 + + .. method:: Path.symlink_to(target, target_is_directory=False) Make this path a symbolic link to *target*. Under Windows, @@ -917,3 +997,33 @@ call fails (for example because the path doesn't exist): Remove this file or symbolic link. If the path points to a directory, use :func:`Path.rmdir` instead. + + +.. method:: Path.write_bytes(data) + + Open the file pointed to in bytes mode, write *data* to it, and close the + file:: + + >>> p = Path('my_binary_file') + >>> p.write_bytes(b'Binary file contents') + 20 + >>> p.read_bytes() + b'Binary file contents' + + An existing file of the same name is overwritten. + + .. versionadded:: 3.5 + + +.. method:: Path.write_text(data, encoding=None, errors=None) + + Open the file pointed to in text mode, write *data* to it, and close the + file:: + + >>> p = Path('my_text_file') + >>> p.write_text('Text file contents') + 18 + >>> p.read_text() + 'Text file contents' + + .. versionadded:: 3.5 diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index 47356f9..4ce4d34 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -859,7 +859,7 @@ For the simplest code, use the :func:`dump` and :func:`load` functions. :: data = { 'a': [1, 2.0, 3, 4+6j], 'b': ("character string", b"byte string"), - 'c': set([None, True, False]) + 'c': {None, True, False} } with open('data.pickle', 'wb') as f: diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 13ea7b9..5d3295d 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -58,7 +58,7 @@ support. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. class:: ImpLoader(fullname, file, filename, etc) @@ -67,7 +67,7 @@ support. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. function:: find_loader(fullname) diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index e27f2ad..679cc6b 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -247,6 +247,8 @@ Unix Platforms This is another name for :func:`linux_distribution`. + .. deprecated-removed:: 3.5 3.7 + .. function:: linux_distribution(distname='', version='', id='', supported_dists=('SuSE','debian','redhat','mandrake',...), full_distribution_name=1) Tries to determine the name of the Linux OS distribution name. @@ -263,6 +265,8 @@ Unix Platforms parameters. ``id`` is the item in parentheses after the version number. It is usually the version codename. + .. deprecated-removed:: 3.5 3.7 + .. function:: libc_ver(executable=sys.executable, lib='', version='', chunksize=2048) Tries to determine the libc version against which the file executable (defaults diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst index 45baad9..8468f4c 100644 --- a/Doc/library/poplib.rst +++ b/Doc/library/poplib.rst @@ -194,6 +194,15 @@ An :class:`POP3` instance has the following methods: the unique id for that message in the form ``'response mesgnum uid``, otherwise result is list ``(response, ['mesgnum uid', ...], octets)``. + +.. method:: POP3.utf8() + + Try to switch to UTF-8 mode. Returns the server response if successful, + raises :class:`error_proto` if not. Specified in :RFC:`6856`. + + .. versionadded:: 3.5 + + .. method:: POP3.stls(context=None) Start a TLS session on the active connection as specified in :rfc:`2595`. diff --git a/Doc/library/pprint.rst b/Doc/library/pprint.rst index c0589a3..0b44dc8 100644 --- a/Doc/library/pprint.rst +++ b/Doc/library/pprint.rst @@ -235,10 +235,10 @@ In its basic form, :func:`pprint` shows the whole object:: 'classifiers': ['Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 2 :: Only'], - 'description': 'An extensible framework for Python programming, ' - 'with special focus\r\n' - 'on event-based network programming and ' - 'multiprotocol integration.', + 'description': 'An extensible framework for Python programming, with ' + 'special focus\r\n' + 'on event-based network programming and multiprotocol ' + 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', 'home_page': 'http://twistedmatrix.com/', @@ -288,10 +288,10 @@ contents):: 'cheesecake_documentation_id': None, 'cheesecake_installability_id': None, 'classifiers': [...], - 'description': 'An extensible framework for Python programming, ' - 'with special focus\r\n' - 'on event-based network programming and ' - 'multiprotocol integration.', + 'description': 'An extensible framework for Python programming, with ' + 'special focus\r\n' + 'on event-based network programming and multiprotocol ' + 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', 'home_page': 'http://twistedmatrix.com/', @@ -323,13 +323,12 @@ cannot be split, the specified width will be exceeded:: 'cheesecake_installability_id': None, 'classifiers': [...], 'description': 'An extensible ' - 'framework for ' - 'Python programming, ' - 'with special ' - 'focus\r\n' - 'on event-based ' - 'network programming ' - 'and multiprotocol ' + 'framework for Python ' + 'programming, with ' + 'special focus\r\n' + 'on event-based network ' + 'programming and ' + 'multiprotocol ' 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', @@ -344,8 +343,8 @@ cannot be split, the specified width will be exceeded:: 'release_url': 'http://pypi.python.org/pypi/Twisted/12.3.0', 'requires_python': None, 'stable_version': None, - 'summary': 'An asynchronous ' - 'networking framework ' - 'written in Python', + 'summary': 'An asynchronous networking ' + 'framework written in ' + 'Python', 'version': '12.3.0'}, 'urls': [{...}, {...}]} diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst index bae8450..97f2b20 100644 --- a/Doc/library/py_compile.rst +++ b/Doc/library/py_compile.rst @@ -29,9 +29,9 @@ byte-code cache files in the directory containing the source code. .. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1) Compile a source file to byte-code and write out the byte-code cache file. - The source code is loaded from the file name *file*. The byte-code is - written to *cfile*, which defaults to the :PEP:`3147` path, ending in - ``.pyc`` (``.pyo`` if optimization is enabled in the current interpreter). + The source code is loaded from the file name *file*. The byte-code is + written to *cfile*, which defaults to the :pep:`3147`/:pep:`488` path, ending + in ``.pyc``. For example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. If *dfile* is specified, it is used as the name of the source file in error messages when @@ -68,7 +68,7 @@ byte-code cache files in the directory containing the source code. .. function:: main(args=None) Compile several source files. The files named in *args* (or on the command - line, if *args* is ``None``) are compiled and the resulting bytecode is + line, if *args* is ``None``) are compiled and the resulting byte-code is cached in the normal manner. This function does not search a directory structure to locate source files; it only compiles files named explicitly. If ``'-'`` is the only parameter in args, the list of files is taken from @@ -86,4 +86,3 @@ could not be compiled. Module :mod:`compileall` Utilities to compile all Python source files in a directory tree. - diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 680d690..1cb0935 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -158,22 +158,32 @@ fully processed by daemon consumer threads. Example of how to wait for enqueued tasks to be completed:: - def worker(): - while True: - item = q.get() - do_work(item) - q.task_done() - - q = Queue() - for i in range(num_worker_threads): - t = Thread(target=worker) - t.daemon = True + def worker(): + while True: + item = q.get() + if item is None: + break + do_work(item) + q.task_done() + + q = queue.Queue() + threads = [] + for i in range(num_worker_threads): + t = threading.Thread(target=worker) t.start() + threads.append(t) - for item in source(): - q.put(item) + for item in source(): + q.put(item) - q.join() # block until all tasks are done + # block until all tasks are done + q.join() + + # stop workers + for i in range(num_worker_threads): + q.put(None) + for t in threads: + t.join() .. seealso:: diff --git a/Doc/library/random.rst b/Doc/library/random.rst index 11dd367..f8b7727 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -46,8 +46,7 @@ from sources provided by the operating system. .. warning:: The pseudo-random generators of this module should not be used for - security purposes. Use :func:`os.urandom` or :class:`SystemRandom` if - you require a cryptographically secure pseudo-random number generator. + security purposes. Bookkeeping functions: diff --git a/Doc/library/re.rst b/Doc/library/re.rst index c3c8b65..8884584 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -281,9 +281,7 @@ The special characters are: assertion`. ``(?<=abc)def`` will find a match in ``abcdef``, since the lookbehind will back up 3 characters and check if the contained pattern matches. The contained pattern must only match strings of some fixed length, meaning that - ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not. Group - references are not supported even if they match strings of some fixed length. - Note that + ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not. Note that patterns which start with positive lookbehind assertions will not match at the beginning of the string being searched; you will most likely want to use the :func:`search` function rather than the :func:`match` function: @@ -299,12 +297,14 @@ The special characters are: >>> m.group(0) 'egg' + .. versionchanged: 3.5 + Added support for group references of fixed length. + ``(?<!...)`` Matches if the current position in the string is not preceded by a match for ``...``. This is called a :dfn:`negative lookbehind assertion`. Similar to positive lookbehind assertions, the contained pattern must only match strings of - some fixed length and shouldn't contain group references. - Patterns which start with negative lookbehind assertions may + some fixed length. Patterns which start with negative lookbehind assertions may match at the beginning of the string being searched. ``(?(id/name)yes-pattern|no-pattern)`` @@ -438,6 +438,10 @@ three digits in length. .. versionchanged:: 3.3 The ``'\u'`` and ``'\U'`` escape sequences have been added. +.. deprecated-removed:: 3.5 3.6 + Unknown escapes consist of ``'\'`` and ASCII letter now raise a + deprecation warning and will be forbidden in Python 3.6. + .. seealso:: @@ -524,7 +528,11 @@ form. current locale. The use of this flag is discouraged as the locale mechanism is very unreliable, and it only handles one "culture" at a time anyway; you should use Unicode matching instead, which is the default in Python 3 - for Unicode (str) patterns. + for Unicode (str) patterns. This flag makes sense only with bytes patterns. + + .. deprecated-removed:: 3.5 3.6 + Deprecated the use of :const:`re.LOCALE` with string patterns or + :const:`re.ASCII`. .. data:: M @@ -625,17 +633,37 @@ form. That way, separator components are always found at the same relative indices within the result list. - Note that *split* will never split a string on an empty pattern match. - For example: + .. note:: + + :func:`split` doesn't currently split a string on an empty pattern match. + For example: - >>> re.split('x*', 'foo') - ['foo'] - >>> re.split("(?m)^$", "foo\n\nbar\n") - ['foo\n\nbar\n'] + >>> re.split('x*', 'axbc') + ['a', 'bc'] + + Even though ``'x*'`` also matches 0 'x' before 'a', between 'b' and 'c', + and after 'c', currently these matches are ignored. The correct behavior + (i.e. splitting on empty matches too and returning ``['', 'a', 'b', 'c', + '']``) will be implemented in future versions of Python, but since this + is a backward incompatible change, a :exc:`FutureWarning` will be raised + in the meanwhile. + + Patterns that can only match empty strings currently never split the + string. Since this doesn't match the expected behavior, a + :exc:`ValueError` will be raised starting from Python 3.5:: + + >>> re.split("^$", "foo\n\nbar\n", flags=re.M) + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ... + ValueError: split() requires a non-empty pattern match. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Splitting on a pattern that could match an empty string now raises + a warning. Patterns that can only match empty strings are now rejected. .. function:: findall(pattern, string, flags=0) @@ -663,7 +691,7 @@ form. *string* is returned unchanged. *repl* can be a string or a function; if it is a string, any backslash escapes in it are processed. That is, ``\n`` is converted to a single newline character, ``\r`` is converted to a carriage return, and - so forth. Unknown escapes such as ``\j`` are left alone. Backreferences, such + so forth. Unknown escapes such as ``\&`` are left alone. Backreferences, such as ``\6``, are replaced with the substring matched by group 6 in the pattern. For example: @@ -705,6 +733,13 @@ form. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. + + .. deprecated-removed:: 3.5 3.6 + Unknown escapes consist of ``'\'`` and ASCII letter now raise a + deprecation warning and will be forbidden in Python 3.6. + .. function:: subn(pattern, repl, string, count=0, flags=0) @@ -714,6 +749,9 @@ form. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. + .. function:: escape(string) @@ -730,13 +768,36 @@ form. Clear the regular expression cache. -.. exception:: error +.. exception:: error(msg, pattern=None, pos=None) Exception raised when a string passed to one of the functions here is not a valid regular expression (for example, it might contain unmatched parentheses) or when some other error occurs during compilation or matching. It is never an - error if a string contains no match for a pattern. + error if a string contains no match for a pattern. The error instance has + the following additional attributes: + + .. attribute:: msg + + The unformatted error message. + + .. attribute:: pattern + + The regular expression pattern. + + .. attribute:: pos + + The index of *pattern* where compilation failed. + + .. attribute:: lineno + + The line corresponding to *pos*. + + .. attribute:: colno + + The column corresponding to *pos*. + .. versionchanged:: 3.5 + Added additional attributes. .. _re-objects: @@ -889,6 +950,8 @@ Match objects support the following methods and attributes: (``\g<1>``, ``\g<name>``) are replaced by the contents of the corresponding group. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. .. method:: match.group([group1, ...]) diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index 692310b..3864f0d 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -59,6 +59,14 @@ The :mod:`readline` module defines the following functions: Save a readline history file. The default filename is :file:`~/.history`. +.. function:: append_history_file(nelements[, filename]) + + Append the last *nelements* of history to a file. The default filename is + :file:`~/.history`. The file must already exist. + + .. versionadded:: 3.5 + + .. function:: clear_history() Clear the current history. (Note: this function is not available if the @@ -209,6 +217,26 @@ from the user's :envvar:`PYTHONSTARTUP` file. :: This code is actually automatically run when Python is run in :ref:`interactive mode <tut-interactive>` (see :ref:`rlcompleter-config`). +The following example achieves the same goal but supports concurrent interactive +sessions, by only appending the new history. :: + + import atexit + import os + import realine + histfile = os.path.join(os.path.expanduser("~"), ".python_history") + + try: + readline.read_history_file(histfile) + h_len = readline.get_history_length() + except FileNotFoundError: + open(histfile, 'wb').close() + h_len = 0 + + def save(prev_h_len, histfile): + new_h_len = readline.get_history_length() + readline.append_history_file(new_h_len - prev_h_len, histfile) + atexit.register(save, h_len, histfile) + The following example extends the :class:`code.InteractiveConsole` class to support history save/restore. :: @@ -234,4 +262,3 @@ support history save/restore. :: def save_history(self, histfile): readline.write_history_file(histfile) - diff --git a/Doc/library/select.rst b/Doc/library/select.rst index 5334af8..a62dc84 100644 --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -145,6 +145,13 @@ The module defines the following: library, and does not handle file descriptors that don't originate from WinSock. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + + .. attribute:: PIPE_BUF The minimum number of bytes which can be written without blocking to a pipe @@ -242,6 +249,12 @@ object. returning. If *timeout* is omitted, -1, or :const:`None`, the call will block until there is an event for this poll object. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _epoll-objects: @@ -322,6 +335,12 @@ Edge and Level Trigger Polling (epoll) Objects Wait for events. timeout in seconds (float) + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _poll-objects: @@ -401,6 +420,12 @@ linearly scanned again. :c:func:`select` is O(highest file descriptor), while returning. If *timeout* is omitted, negative, or :const:`None`, the call will block until there is an event for this poll object. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _kqueue-objects: @@ -435,6 +460,12 @@ Kqueue Objects - max_events must be 0 or a positive integer - timeout in seconds (floats possible) + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _kevent-objects: diff --git a/Doc/library/selectors.rst b/Doc/library/selectors.rst index 98377c8..f6ef24b 100644 --- a/Doc/library/selectors.rst +++ b/Doc/library/selectors.rst @@ -45,6 +45,7 @@ Classes hierarchy:: +-- SelectSelector +-- PollSelector +-- EpollSelector + +-- DevpollSelector +-- KqueueSelector @@ -158,6 +159,12 @@ below: timeout has elapsed if the current process receives a signal: in this case, an empty list will be returned. + .. versionchanged:: 3.5 + The selector is now retried with a recomputed timeout when interrupted + by a signal if the signal handler did not raise an exception (see + :pep:`475` for the rationale), instead of returning an empty list + of events before the timeout. + .. method:: close() Close the selector. @@ -207,6 +214,16 @@ below: This returns the file descriptor used by the underlying :func:`select.epoll` object. +.. class:: DevpollSelector() + + :func:`select.devpoll`-based selector. + + .. method:: fileno() + + This returns the file descriptor used by the underlying + :func:`select.devpoll` object. + + .. versionadded:: 3.5 .. class:: KqueueSelector() diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 04afe92..3b467e0 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -191,7 +191,8 @@ Directory and files operations match one of the glob-style *patterns* provided. See the example below. -.. function:: copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, ignore_dangling_symlinks=False) +.. function:: copytree(src, dst, symlinks=False, ignore=None, \ + copy_function=copy2, ignore_dangling_symlinks=False) Recursively copy an entire directory tree rooted at *src*, returning the destination directory. The destination @@ -282,7 +283,7 @@ Directory and files operations .. versionadded:: 3.3 -.. function:: move(src, dst) +.. function:: move(src, dst, copy_function=copy2) Recursively move a file or directory (*src*) to another location (*dst*) and return the destination. @@ -292,15 +293,26 @@ Directory and files operations be overwritten depending on :func:`os.rename` semantics. If the destination is on the current filesystem, then :func:`os.rename` is - used. Otherwise, *src* is copied (using :func:`shutil.copy2`) to *dst* and - then removed. In case of symlinks, a new symlink pointing to the target of - *src* will be created in or as *dst* and *src* will be removed. + used. Otherwise, *src* is copied to *dst* using *copy_function* and then + removed. In case of symlinks, a new symlink pointing to the target of *src* + will be created in or as *dst* and *src* will be removed. + + If *copy_function* is given, it must be a callable that takes two arguments + *src* and *dst*, and will be used to copy *src* to *dest* if + :func:`os.rename` cannot be used. If the source is a directory, + :func:`copytree` is called, passing it the :func:`copy_function`. The + default *copy_function* is :func:`copy2`. Using :func:`copy` as the + *copy_function* allows the move to succeed when it is not possible to also + copy the metadata, at the expense of not copying any of the metadata. .. versionchanged:: 3.3 Added explicit symlink handling for foreign filesystems, thus adapting it to the behavior of GNU's :program:`mv`. Now returns *dst*. + .. versionchanged:: 3.5 + Added the *copy_function* keyword argument. + .. function:: disk_usage(path) Return disk usage statistics about the given path as a :term:`named tuple` @@ -418,6 +430,26 @@ Another example that uses the *ignore* argument to add a logging call:: copytree(source, destination, ignore=_logpath) +.. _shutil-rmtree-example: + +rmtree example +~~~~~~~~~~~~~~ + +This example shows how to remove a directory tree on Windows where some +of the files have their read-only bit set. It uses the onerror callback +to clear the readonly bit and reattempt the remove. Any subsequent failure +will propagate. :: + + import os, stat + import shutil + + def remove_readonly(func, path, _): + "Clear the readonly bit and reattempt the removal" + os.chmod(path, stat.S_IWRITE) + func(path) + + shutil.rmtree(directory, onerror=remove_readonly) + .. _archiving-operations: Archiving operations @@ -434,7 +466,8 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. *base_name* is the name of the file to create, including the path, minus any format-specific extension. *format* is the archive format: one of - "zip", "tar", "bztar" (if the :mod:`bz2` module is available) or "gztar". + "zip", "tar", "bztar" (if the :mod:`bz2` module is available), "xztar" + (if the :mod:`lzma` module is available) or "gztar". *root_dir* is a directory that will be the root directory of the archive; for example, we typically chdir into *root_dir* before creating the @@ -457,6 +490,9 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. The *verbose* argument is unused and deprecated. + .. versionchanged:: 3.5 + Added support for the *xztar* format. + .. function:: get_archive_formats() @@ -467,6 +503,7 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. - *gztar*: gzip'ed tar-file - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available.) - *tar*: uncompressed tar file - *zip*: ZIP file @@ -542,6 +579,7 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. - *gztar*: gzip'ed tar-file - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available.) - *tar*: uncompressed tar file - *zip*: ZIP file diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 84e2836..9549838 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -65,6 +65,16 @@ Besides, only the main thread is allowed to set a new signal handler. Module contents --------------- +.. versionchanged:: 3.5 + signal (SIG*), handler (:const:`SIG_DFL`, :const:`SIG_IGN`) and sigmask + (:const:`SIG_BLOCK`, :const:`SIG_UNBLOCK`, :const:`SIG_SETMASK`) + related constants listed below were turned into + :class:`enums <enum.IntEnum>`. + :func:`getsignal`, :func:`pthread_sigmask`, :func:`sigpending` and + :func:`sigwait` functions return human-readable + :class:`enums <enum.IntEnum>`. + + The variables defined in the :mod:`signal` module are: @@ -209,21 +219,21 @@ The :mod:`signal` module defines the following functions: :func:`sigpending`. -.. function:: pthread_kill(thread_id, signum) +.. function:: pthread_kill(thread_id, signalnum) - Send the signal *signum* to the thread *thread_id*, another thread in the + Send the signal *signalnum* to the thread *thread_id*, another thread in the same process as the caller. The target thread can be executing any code (Python or not). However, if the target thread is executing the Python interpreter, the Python signal handlers will be :ref:`executed by the main - thread <signals-and-threads>`. Therefore, the only point of sending a signal to a particular - Python thread would be to force a running system call to fail with - :exc:`InterruptedError`. + thread <signals-and-threads>`. Therefore, the only point of sending a + signal to a particular Python thread would be to force a running system call + to fail with :exc:`InterruptedError`. Use :func:`threading.get_ident()` or the :attr:`~threading.Thread.ident` attribute of :class:`threading.Thread` objects to get a suitable value for *thread_id*. - If *signum* is 0, then no signal is sent, but error checking is still + If *signalnum* is 0, then no signal is sent, but error checking is still performed; this can be used to check if the target thread is still running. Availability: Unix (see the man page :manpage:`pthread_kill(3)` for further @@ -308,6 +318,9 @@ The :mod:`signal` module defines the following functions: attempting to call it from other threads will cause a :exc:`ValueError` exception to be raised. + .. versionchanged:: 3.5 + On Windows, the function now also supports socket handles. + .. function:: siginterrupt(signalnum, flag) @@ -395,6 +408,11 @@ The :mod:`signal` module defines the following functions: .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now retried if interrupted by a signal not in *sigset* + and the signal handler does not raise an exception (see :pep:`475` for + the rationale). + .. function:: sigtimedwait(sigset, timeout) @@ -409,6 +427,11 @@ The :mod:`signal` module defines the following functions: .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now retried with the recomputed *timeout* if interrupted + by a signal not in *sigset* and the signal handler does not raise an + exception (see :pep:`475` for the rationale). + .. _signal-example: diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 51e5da8..43daf79 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -26,24 +26,23 @@ additions, call the :func:`site.main` function. :option:`-S`. .. index:: - pair: site-python; directory pair: site-packages; directory It starts by constructing up to four directories from a head and a tail part. For the head part, it uses ``sys.prefix`` and ``sys.exec_prefix``; empty heads are skipped. For the tail part, it uses the empty string and then :file:`lib/site-packages` (on Windows) or -:file:`lib/python{X.Y}/site-packages` and then :file:`lib/site-python` (on -Unix and Macintosh). For each of the distinct head-tail combinations, it sees -if it refers to an existing directory, and if so, adds it to ``sys.path`` and -also inspects the newly added path for configuration files. +:file:`lib/python{X.Y}/site-packages` (on Unix and Macintosh). For each +of the distinct head-tail combinations, it sees if it refers to an existing +directory, and if so, adds it to ``sys.path`` and also inspects the newly +added path for configuration files. -.. deprecated:: 3.4 - Support for the "site-python" directory will be removed in 3.5. +.. versionchanged:: 3.5 + Support for the "site-python" directory has been removed. If a file named "pyvenv.cfg" exists one directory above sys.executable, sys.prefix and sys.exec_prefix are set to that directory and -it is also checked for site-packages and site-python (sys.base_prefix and +it is also checked for site-packages (sys.base_prefix and sys.base_exec_prefix will always be the "real" prefixes of the Python installation). If "pyvenv.cfg" (a bootstrap configuration file) contains the key "include-system-site-packages" set to anything other than "false" @@ -195,8 +194,7 @@ Module contents .. function:: getsitepackages() - Return a list containing all global site-packages directories (and possibly - site-python). + Return a list containing all global site-packages directories. .. versionadded:: 3.2 diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst index 3ebed06..977f9a8 100644 --- a/Doc/library/smtpd.rst +++ b/Doc/library/smtpd.rst @@ -20,7 +20,8 @@ specific mail-sending strategies. Additionally the SMTPChannel may be extended to implement very specific interaction behaviour with SMTP clients. -The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE extension. +The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE and :rfc:`6531` +SMTPUTF8 extensions. SMTPServer Objects @@ -28,7 +29,7 @@ SMTPServer Objects .. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432,\ - map=None) + map=None, enable_SMTPUTF8=False, decode_data=True) Create a new :class:`SMTPServer` object, which binds to local address *localaddr*. It will treat *remoteaddr* as an upstream SMTP relayer. It @@ -39,25 +40,77 @@ SMTPServer Objects accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no limit. - A dictionary can be specified in *map* to avoid using a global socket map. - - .. method:: process_message(peer, mailfrom, rcpttos, data) - - Raise :exc:`NotImplementedError` exception. Override this in subclasses to + *map* is the socket map to use for connections (an initially empty + dictionary is a suitable value). If not specified the :mod:`asyncore` + global socket map is used. + + *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined + in :RFC:`6531`) should be enabled. The default is ``False``. If set to + ``True``, *decode_data* must be ``False`` (otherwise an error is raised). + When ``True``, ``SMTPUTF8`` is accepted as a parameter to the ``MAIL`` + command and when present is passed to :meth:`process_message` in the + ``kwargs['mail_options']`` list. + + *decode_data* specifies whether the data portion of the SMTP transaction + should be decoded using UTF-8. The default is ``True`` for backward + compatibility reasons, but will change to ``False`` in Python 3.6; specify + the keyword value explicitly to avoid the :exc:`DeprecationWarning`. When + *decode_data* is set to ``False`` the server advertises the ``8BITMIME`` + extension (:rfc:`6152`), accepts the ``BODY=8BITMIME`` parameter to + the ``MAIL`` command, and when present passes it to :meth:`process_message` + in the ``kwargs['mail_options']`` list. + + .. method:: process_message(peer, mailfrom, rcpttos, data, **kwargs) + + Raise a :exc:`NotImplementedError` exception. Override this in subclasses to do something useful with this message. Whatever was passed in the constructor as *remoteaddr* will be available as the :attr:`_remoteaddr` attribute. *peer* is the remote host's address, *mailfrom* is the envelope originator, *rcpttos* are the envelope recipients and *data* is a string - containing the contents of the e-mail (which should be in :rfc:`2822` + containing the contents of the e-mail (which should be in :rfc:`5321` format). + If the *decode_data* constructor keyword is set to ``True``, the *data* + argument will be a unicode string. If it is set to ``False``, it + will be a bytes object. + + *kwargs* is a dictionary containing additional information. It is empty + unless at least one of ``decode_data=False`` or ``enable_SMTPUTF8=True`` + was given as an init parameter, in which case it contains the following + keys: + + *mail_options*: + a list of all received parameters to the ``MAIL`` + command (the elements are uppercase strings; example: + ``['BODY=8BITMIME', 'SMTPUTF8']``). + + *rcpt_options*: + same as *mail_options* but for the ``RCPT`` command. + Currently no ``RCPT TO`` options are supported, so for now + this will always be an empty list. + + Implementations of ``process_message`` should use the ``**kwargs`` + signature to accept arbitrary keyword arguments, since future feature + enhancements may add keys to the kwargs dictionary. + + Return ``None`` to request a normal ``250 Ok`` response; otherwise + return the desired response string in :RFC:`5321` format. + .. attribute:: channel_class Override this in subclasses to use a custom :class:`SMTPChannel` for managing SMTP clients. - .. versionchanged:: 3.4 - The *map* argument was added. + .. versionadded:: 3.4 + The *map* constructor argument. + + .. versionchanged:: 3.5 + *localaddr* and *remoteaddr* may now contain IPv6 addresses. + + .. versionadded:: 3.5 + the *decode_data* and *enable_SMTPUTF8* constructor arguments, and the + *kwargs* argument to :meth:`process_message` when one or more of these is + specified. DebuggingServer Objects @@ -97,7 +150,7 @@ SMTPChannel Objects ------------------- .. class:: SMTPChannel(server, conn, addr, data_size_limit=33554432,\ - map=None)) + map=None, enable_SMTPUTF8=False, decode_data=True) Create a new :class:`SMTPChannel` object which manages the communication between the server and a single SMTP client. @@ -108,11 +161,24 @@ SMTPChannel Objects accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no limit. + *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined + in :RFC:`6531`) should be enabled. The default is ``False``. A + :exc:`ValueError` is raised if both *enable_SMTPUTF8* and *decode_data* are + set to ``True`` at the same time. + A dictionary can be specified in *map* to avoid using a global socket map. + *decode_data* specifies whether the data portion of the SMTP transaction + should be decoded using UTF-8. The default is ``True`` for backward + compatibility reasons, but will change to ``False`` in Python 3.6. Specify + the keyword value explicitly to avoid the :exc:`DeprecationWarning`. + To use a custom SMTPChannel implementation you need to override the :attr:`SMTPServer.channel_class` of your :class:`SMTPServer`. + .. versionchanged:: 3.5 + the *decode_data* and *enable_SMTPUTF8* arguments were added. + The :class:`SMTPChannel` has the following instance variables: .. attribute:: smtp_server diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index 8e1bfb5..25279f2 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -61,6 +61,10 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). .. versionchanged:: 3.3 source_address argument was added. + .. versionadded:: 3.5 + The SMTPUTF8 extension (:rfc:`6531`) is now supported. + + .. class:: SMTP_SSL(host='', port=0, local_hostname=None, keyfile=None, \ certfile=None [, timeout], context=None, \ source_address=None) @@ -161,6 +165,13 @@ A nice selection of exceptions is defined as well: The server refused our ``HELO`` message. +.. exception:: SMTPNotSupportedError + + The command or option attempted is not supported by the server. + + .. versionadded:: 3.5 + + .. exception:: SMTPAuthenticationError SMTP authentication went wrong. Most probably the server didn't accept the @@ -189,8 +200,12 @@ An :class:`SMTP` instance has the following methods: .. method:: SMTP.set_debuglevel(level) - Set the debug output level. A true value for *level* results in debug messages - for connection and for all messages sent to and received from the server. + Set the debug output level. A value of 1 or ``True`` for *level* results in + debug messages for connection and for all messages sent to and received from + the server. A value of 2 for *level* results in these messages being + timestamped. + + .. versionchanged:: 3.5 Added debuglevel 2. .. method:: SMTP.docmd(cmd, args='') @@ -240,8 +255,7 @@ An :class:`SMTP` instance has the following methods: the server is stored as the :attr:`ehlo_resp` attribute, :attr:`does_esmtp` is set to true or false depending on whether the server supports ESMTP, and :attr:`esmtp_features` will be a dictionary containing the names of the - SMTP service extensions this server supports, and their - parameters (if any). + SMTP service extensions this server supports, and their parameters (if any). Unless you wish to use :meth:`has_extn` before sending mail, it should not be necessary to call this method explicitly. It will be implicitly called by @@ -288,9 +302,51 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPAuthenticationError` The server didn't accept the username/password combination. + :exc:`SMTPNotSupportedError` + The ``AUTH`` command is not supported by the server. + :exc:`SMTPException` No suitable authentication method was found. + Each of the authentication methods supported by :mod:`smtplib` are tried in + turn if they are advertised as supported by the server (see :meth:`auth` + for a list of supported authentication methods). + + .. versionchanged:: 3.5 + :exc:`SMTPNotSupportedError` may be raised. + + +.. method:: SMTP.auth(mechanism, authobject) + + Issue an ``SMTP`` ``AUTH`` command for the specified authentication + *mechanism*, and handle the challenge response via *authobject*. + + *mechanism* specifies which authentication mechanism is to + be used as argument to the ``AUTH`` command; the valid values are + those listed in the ``auth`` element of :attr:`esmtp_features`. + + *authobject* must be a callable object taking a single argument: + + data = authobject(challenge) + + It will be called to process the server's challenge response; the + *challenge* argument it is passed will be a ``bytes``. It should return + ``bytes`` *data* that will be base64 encoded and sent to the server. + + The ``SMTP`` class provides ``authobjects`` for the ``CRAM-MD5``, ``PLAIN``, + and ``LOGIN`` mechanisms; they are named ``SMTP.auth_cram_md5``, + ``SMTP.auth_plain``, and ``SMTP.auth_login`` respectively. They all require + that the ``user`` and ``password`` properties of the ``SMTP`` instance are + set to appropriate values. + + User code does not normally need to call ``auth`` directly, but can instead + call the :meth:`login` method, which will try each of the above mechanisms in + turn, in the order listed. ``auth`` is exposed to facilitate the + implementation of authentication methods not (or not yet) supported directly + by :mod:`smtplib`. + + .. versionadded:: 3.5 + .. method:: SMTP.starttls(keyfile=None, certfile=None, context=None) @@ -310,7 +366,7 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPHeloError` The server didn't reply properly to the ``HELO`` greeting. - :exc:`SMTPException` + :exc:`SMTPNotSupportedError` The server does not support the STARTTLS extension. :exc:`RuntimeError` @@ -324,6 +380,11 @@ An :class:`SMTP` instance has the following methods: :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see :data:`~ssl.HAS_SNI`). + .. versionchanged:: 3.5 + The error raised for lack of STARTTLS support is now the + :exc:`SMTPNotSupportedError` subclass instead of the base + :exc:`SMTPException`. + .. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]) @@ -360,6 +421,9 @@ An :class:`SMTP` instance has the following methods: recipient that was refused. Each entry contains a tuple of the SMTP error code and the accompanying error message sent by the server. + If ``SMTPUTF8`` is included in *mail_options*, and the server supports it, + *from_addr* and *to_addr* may contain non-ASCII characters. + This method may raise the following exceptions: :exc:`SMTPRecipientsRefused` @@ -378,12 +442,20 @@ An :class:`SMTP` instance has the following methods: The server replied with an unexpected error code (other than a refusal of a recipient). + :exc:`SMTPNotSupportedError` + ``SMTPUTF8`` was given in the *mail_options* but is not supported by the + server. + Unless otherwise noted, the connection will be open even after an exception is raised. .. versionchanged:: 3.2 *msg* may be a byte string. + .. versionchanged:: 3.5 + ``SMTPUTF8`` support added, and :exc:`SMTPNotSupportedError` may be + raised if ``SMTPUTF8`` is specified but the server does not support it. + .. method:: SMTP.send_message(msg, from_addr=None, to_addrs=None, \ mail_options=[], rcpt_options=[]) @@ -395,7 +467,7 @@ An :class:`SMTP` instance has the following methods: If *from_addr* is ``None`` or *to_addrs* is ``None``, ``send_message`` fills those arguments with addresses extracted from the headers of *msg* as - specified in :rfc:`2822`\: *from_addr* is set to the :mailheader:`Sender` + specified in :rfc:`5322`\: *from_addr* is set to the :mailheader:`Sender` field if it is present, and otherwise to the :mailheader:`From` field. *to_adresses* combines the values (if any) of the :mailheader:`To`, :mailheader:`Cc`, and :mailheader:`Bcc` fields from *msg*. If exactly one @@ -410,10 +482,18 @@ An :class:`SMTP` instance has the following methods: calls :meth:`sendmail` to transmit the resulting message. Regardless of the values of *from_addr* and *to_addrs*, ``send_message`` does not transmit any :mailheader:`Bcc` or :mailheader:`Resent-Bcc` headers that may appear - in *msg*. + in *msg*. If any of the addresses in *from_addr* and *to_addrs* contain + non-ASCII characters and the server does not advertise ``SMTPUTF8`` support, + an :exc:`SMTPNotSupported` error is raised. Otherwise the ``Message`` is + serialized with a clone of its :mod:`~email.policy` with the + :attr:`~email.policy.EmailPolicy.utf8` attribute set to ``True``, and + ``SMTPUTF8`` and ``BODY=8BITMIME`` are added to *mail_options*. .. versionadded:: 3.2 + .. versionadded:: 3.5 + Support for internationalized addresses (``SMTPUTF8``). + .. method:: SMTP.quit() diff --git a/Doc/library/sndhdr.rst b/Doc/library/sndhdr.rst index f36df68..f8b5d8b 100644 --- a/Doc/library/sndhdr.rst +++ b/Doc/library/sndhdr.rst @@ -16,8 +16,9 @@ The :mod:`sndhdr` provides utility functions which attempt to determine the type of sound data which is in a file. When these functions are able to determine -what type of sound data is stored in a file, they return a tuple ``(type, -sampling_rate, channels, frames, bits_per_sample)``. The value for *type* +what type of sound data is stored in a file, they return a +:func:`~collections.namedtuple`, containing five attributes: (``filetype``, +``framerate``, ``nchannels``, ``nframes``, ``sampwidth``). The value for *type* indicates the data type and will be one of the strings ``'aifc'``, ``'aiff'``, ``'au'``, ``'hcom'``, ``'sndr'``, ``'sndt'``, ``'voc'``, ``'wav'``, ``'8svx'``, ``'sb'``, ``'ub'``, or ``'ul'``. The *sampling_rate* will be either the actual @@ -31,13 +32,19 @@ be the sample size in bits or ``'A'`` for A-LAW or ``'U'`` for u-LAW. .. function:: what(filename) Determines the type of sound data stored in the file *filename* using - :func:`whathdr`. If it succeeds, returns a tuple as described above, otherwise + :func:`whathdr`. If it succeeds, returns a namedtuple as described above, otherwise ``None`` is returned. + .. versionchanged:: 3.5 + Result changed from a tuple to a namedtuple. + .. function:: whathdr(filename) Determines the type of sound data stored in a file based on the file header. - The name of the file is given by *filename*. This function returns a tuple as + The name of the file is given by *filename*. This function returns a namedtuple as described above on success, or ``None``. + .. versionchanged:: 3.5 + Result changed from a tuple to a namedtuple. + diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 827191e..1dcdb2f 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -46,17 +46,20 @@ created. Socket addresses are represented as follows: - The address of an :const:`AF_UNIX` socket bound to a file system node is represented as a string, using the file system encoding and the ``'surrogateescape'`` error handler (see :pep:`383`). An address in - Linux's abstract namespace is returned as a :class:`bytes` object with + Linux's abstract namespace is returned as a :term:`bytes-like object` with an initial null byte; note that sockets in this namespace can communicate with normal file system sockets, so programs intended to run on Linux may need to deal with both types of address. A string or - :class:`bytes` object can be used for either type of address when + bytes-like object can be used for either type of address when passing it as an argument. .. versionchanged:: 3.3 Previously, :const:`AF_UNIX` socket paths were assumed to use UTF-8 encoding. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + - A pair ``(host, port)`` is used for the :const:`AF_INET` address family, where *host* is a string representing either a hostname in Internet domain notation like ``'daring.cwi.nl'`` or an IPv4 address like ``'100.50.200.5'``, @@ -276,6 +279,18 @@ Constants .. versionadded:: 3.4 +.. data:: CAN_RAW_FD_FRAMES + + Enables CAN FD support in a CAN_RAW socket. This is disabled by default. + This allows your application to send both CAN and CAN FD frames; however, + you one must accept both CAN and CAN FD frames when reading from the socket. + + This constant is documented in the Linux documentation. + + Availability: Linux >= 3.6. + + .. versionadded:: 3.5 + .. data:: AF_RDS PF_RDS SOL_RDS @@ -352,7 +367,6 @@ The following functions all create :ref:`socket objects <socket-objects>`. type, and protocol number. Address family, socket type, and protocol number are as for the :func:`.socket` function above. The default family is :const:`AF_UNIX` if defined on the platform; otherwise, the default is :const:`AF_INET`. - Availability: Unix. The newly created sockets are :ref:`non-inheritable <fd_inheritance>`. @@ -363,6 +377,9 @@ The following functions all create :ref:`socket objects <socket-objects>`. .. versionchanged:: 3.4 The returned sockets are now non-inheritable. + .. versionchanged:: 3.5 + Windows support added. + .. function:: create_connection(address[, timeout[, source_address]]) @@ -609,8 +626,8 @@ The :mod:`socket` module also offers various network-related services: .. function:: inet_ntoa(packed_ip) - Convert a 32-bit packed IPv4 address (a bytes object four characters in - length) to its standard dotted-quad string representation (for example, + Convert a 32-bit packed IPv4 address (a :term:`bytes-like object` four + bytes in length) to its standard dotted-quad string representation (for example, '123.45.67.89'). This is useful when conversing with a program that uses the standard C library and needs objects of type :c:type:`struct in_addr`, which is the C type for the 32-bit packed binary data this function takes as an @@ -621,6 +638,9 @@ The :mod:`socket` module also offers various network-related services: support IPv6, and :func:`inet_ntop` should be used instead for IPv4/v6 dual stack support. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. function:: inet_pton(address_family, ip_string) @@ -643,22 +663,26 @@ The :mod:`socket` module also offers various network-related services: .. function:: inet_ntop(address_family, packed_ip) - Convert a packed IP address (a bytes object of some number of characters) to its - standard, family-specific string representation (for example, ``'7.10.0.5'`` or - ``'5aef:2b::8'``). :func:`inet_ntop` is useful when a library or network protocol - returns an object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) - or :c:type:`struct in6_addr`. + Convert a packed IP address (a :term:`bytes-like object` of some number of + bytes) to its standard, family-specific string representation (for + example, ``'7.10.0.5'`` or ``'5aef:2b::8'``). + :func:`inet_ntop` is useful when a library or network protocol returns an + object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) or + :c:type:`struct in6_addr`. Supported values for *address_family* are currently :const:`AF_INET` and - :const:`AF_INET6`. If the string *packed_ip* is not the correct length for the - specified address family, :exc:`ValueError` will be raised. A - :exc:`OSError` is raised for errors from the call to :func:`inet_ntop`. + :const:`AF_INET6`. If the bytes object *packed_ip* is not the correct + length for the specified address family, :exc:`ValueError` will be raised. + A :exc:`OSError` is raised for errors from the call to :func:`inet_ntop`. Availability: Unix (maybe not all platforms), Windows. .. versionchanged:: 3.4 Windows support added + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. XXX: Are sendmsg(), recvmsg() and CMSG_*() available on any @@ -783,6 +807,11 @@ to sockets. .. versionchanged:: 3.4 The socket is now non-inheritable. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.bind(address) @@ -815,6 +844,19 @@ to sockets. Connect to a remote socket at *address*. (The format of *address* depends on the address family --- see above.) + If the connection is interrupted by a signal, the method waits until the + connection completes, or raise a :exc:`socket.timeout` on timeout, if the + signal handler doesn't raise an exception and the socket is blocking or has + a timeout. For non-blocking sockets, the method raises an + :exc:`InterruptedError` exception if the connection is interrupted by a + signal (or the exception raised by the signal handler). + + .. versionchanged:: 3.5 + The method now waits until the connection completes instead of raising an + :exc:`InterruptedError` exception if the connection is interrupted by a + signal, the signal handler doesn't raise an exception and the socket is + blocking or has a timeout (see the :pep:`475` for the rationale). + .. method:: socket.connect_ex(address) @@ -910,12 +952,15 @@ to sockets. On other platforms, the generic :func:`fcntl.fcntl` and :func:`fcntl.ioctl` functions may be used; they accept a socket object as their first argument. -.. method:: socket.listen(backlog) +.. method:: socket.listen([backlog]) - Listen for connections made to the socket. The *backlog* argument specifies the - maximum number of queued connections and should be at least 0; the maximum value - is system-dependent (usually 5), the minimum value is forced to 0. + Enable a server to accept connections. If *backlog* is specified, it must + be at least 0 (if it is lower, it is set to 0); it specifies the number of + unaccepted connections that the system will allow before refusing new + connections. If not specified, a default reasonable value is chosen. + .. versionchanged:: 3.5 + The *backlog* parameter is now optional. .. method:: socket.makefile(mode='r', buffering=None, *, encoding=None, \ errors=None, newline=None) @@ -953,6 +998,11 @@ to sockets. For best match with hardware and network realities, the value of *bufsize* should be a relatively small power of 2, for example, 4096. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvfrom(bufsize[, flags]) @@ -962,6 +1012,11 @@ to sockets. :manpage:`recv(2)` for the meaning of the optional argument *flags*; it defaults to zero. (The format of *address* depends on the address family --- see above.) + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvmsg(bufsize[, ancbufsize[, flags]]) @@ -1028,6 +1083,11 @@ to sockets. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvmsg_into(buffers[, ancbufsize[, flags]]) @@ -1094,6 +1154,11 @@ to sockets. application needs to attempt delivery of the remaining data. For further information on this topic, consult the :ref:`socket-howto`. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendall(bytes[, flags]) @@ -1104,6 +1169,15 @@ to sockets. success. On error, an exception is raised, and there is no way to determine how much data, if any, was successfully sent. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time data is sent successfuly. + The socket timeout is now the maximum total duration to send all data. + + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendto(bytes, address) socket.sendto(bytes, flags, address) @@ -1114,6 +1188,11 @@ to sockets. bytes sent. (The format of *address* depends on the address family --- see above.) + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendmsg(buffers[, ancdata[, flags[, address]]]) @@ -1150,6 +1229,26 @@ to sockets. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + +.. method:: socket.sendfile(file, offset=0, count=None) + + Send a file until EOF is reached by using high-performance + :mod:`os.sendfile` and return the total number of bytes which were sent. + *file* must be a regular file object opened in binary mode. If + :mod:`os.sendfile` is not available (e.g. Windows) or *file* is not a + regular file :meth:`send` will be used instead. *offset* tells from where to + start reading the file. If specified, *count* is the total number of bytes + to transmit as opposed to sending the file until EOF is reached. File + position is updated on return or also in case of error in which case + :meth:`file.tell() <io.IOBase.tell>` can be used to figure out the number of + bytes which were sent. The socket must be of :const:`SOCK_STREAM` type. Non- + blocking sockets are not supported. + + .. versionadded:: 3.5 .. method:: socket.set_inheritable(inheritable) @@ -1189,11 +1288,15 @@ to sockets. Set the value of the given socket option (see the Unix manual page :manpage:`setsockopt(2)`). The needed symbolic constants are defined in the - :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer or a - bytes object representing a buffer. In the latter case it is up to the caller to + :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer or + a :term:`bytes-like object` representing a buffer. In the latter case it is + up to the caller to ensure that the bytestring contains the proper bits (see the optional built-in module :mod:`struct` for a way to encode C structures as bytestrings). + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: socket.shutdown(how) diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index 1ec4438..9db36d5 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -113,7 +113,7 @@ the request handler class :meth:`handle` method. Another approach to handling multiple simultaneous requests in an environment that supports neither threads nor :func:`~os.fork` (or where these are too expensive or inappropriate for the service) is to maintain an explicit table of -partially finished requests and to use :func:`~select.select` to decide which +partially finished requests and to use :mod:`selectors` to decide which request to work on next (or whether to handle a new incoming request). This is particularly important for stream services where each client can potentially be connected for a long time (if threads or subprocesses cannot be used). See @@ -136,7 +136,7 @@ Server Objects .. method:: BaseServer.fileno() Return an integer file descriptor for the socket on which the server is - listening. This function is most commonly passed to :func:`select.select`, to + listening. This function is most commonly passed to :mod:`selectors`, to allow monitoring multiple servers in the same process. diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 6097e7a..fc69a80 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -649,6 +649,9 @@ Row Objects This method returns a list of column names. Immediately after a query, it is the first member of each tuple in :attr:`Cursor.description`. + .. versionchanged:: 3.5 + Added support of slicing. + Let's assume we initialize a table as in the example given above:: conn = sqlite3.connect(":memory:") diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 233de8d..4075134 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -315,6 +315,8 @@ Random generation For almost all applications :func:`os.urandom` is preferable. + For almost all applications :func:`os.urandom` is preferable. + .. versionadded:: 3.3 .. function:: RAND_status() @@ -335,6 +337,8 @@ Random generation See http://egd.sourceforge.net/ or http://prngd.sourceforge.net/ for sources of entropy-gathering daemons. + Availability: not available with LibreSSL. + .. function:: RAND_add(bytes, entropy) Mix the given *bytes* into the SSL pseudo-random number generator. The @@ -342,6 +346,9 @@ Random generation string (so you can always use :const:`0.0`). See :rfc:`1750` for more information on sources of entropy. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + Certificate handling ^^^^^^^^^^^^^^^^^^^^ @@ -350,10 +357,9 @@ Certificate handling Verify that *cert* (in decoded format as returned by :meth:`SSLSocket.getpeercert`) matches the given *hostname*. The rules applied are those for checking the identity of HTTPS servers as outlined - in :rfc:`2818` and :rfc:`6125`, except that IP addresses are not currently - supported. In addition to HTTPS, this function should be suitable for - checking the identity of servers in various SSL-based protocols such as - FTPS, IMAPS, POPS and others. + in :rfc:`2818` and :rfc:`6125`. In addition to HTTPS, this function + should be suitable for checking the identity of servers in various + SSL-based protocols such as FTPS, IMAPS, POPS and others. :exc:`CertificateError` is raised on failure. On success, the function returns nothing:: @@ -375,22 +381,38 @@ Certificate handling IDN A-labels such as ``www*.xn--pthon-kva.org`` are still supported, but ``x*.python.org`` no longer matches ``xn--tda.python.org``. -.. function:: cert_time_to_seconds(timestring) + .. versionchanged:: 3.5 + Matching of IP addresses, when present in the subjectAltName field + of the certificate, is now supported. + +.. function:: cert_time_to_seconds(cert_time) - Returns a floating-point value containing a normal seconds-after-the-epoch - time value, given the time-string representing the "notBefore" or "notAfter" - date from a certificate. + Return the time in seconds since the Epoch, given the ``cert_time`` + string representing the "notBefore" or "notAfter" date from a + certificate in ``"%b %d %H:%M:%S %Y %Z"`` strptime format (C + locale). - Here's an example:: + Here's an example: - >>> import ssl - >>> ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT") - 1178694000.0 - >>> import time - >>> time.ctime(ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT")) - 'Wed May 9 00:00:00 2007' + .. doctest:: newcontext -.. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv3, ca_certs=None) + >>> import ssl + >>> timestamp = ssl.cert_time_to_seconds("Jan 5 09:34:43 2018 GMT") + >>> timestamp + 1515144883 + >>> from datetime import datetime + >>> print(datetime.utcfromtimestamp(timestamp)) + 2018-01-05 09:34:43 + + "notBefore" or "notAfter" dates must use GMT (:rfc:`5280`). + + .. versionchanged:: 3.5 + Interpret the input time as a time in UTC as specified by 'GMT' + timezone in the input string. Local timezone was used + previously. Return an integer (no fractions of a second in the + input format) + +.. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv23, ca_certs=None) Given the address ``addr`` of an SSL-protected server, as a (*hostname*, *port-number*) pair, fetches the server's certificate, and returns it as a @@ -404,6 +426,10 @@ Certificate handling .. versionchanged:: 3.3 This function is now IPv6-compatible. + .. versionchanged:: 3.5 + The default *ssl_version* is changed from :data:`PROTOCOL_SSLv3` to + :data:`PROTOCOL_SSLv23` for maximum compatibility with modern servers. + .. function:: DER_cert_to_PEM_cert(DER_cert_bytes) Given a certificate as a DER-encoded blob of bytes, returns a PEM-encoded @@ -671,6 +697,13 @@ Constants .. versionadded:: 3.3 +.. data:: HAS_ALPN + + Whether the OpenSSL library has built-in support for the *Application-Layer + Protocol Negotiation* TLS extension as described in :rfc:`7301`. + + .. versionadded:: 3.5 + .. data:: HAS_ECDH Whether the OpenSSL library has built-in support for Elliptic Curve-based @@ -788,6 +821,8 @@ SSL Sockets (but passing a non-zero ``flags`` argument is not allowed) - :meth:`~socket.socket.send()`, :meth:`~socket.socket.sendall()` (with the same limitation) + - :meth:`~socket.socket.sendfile()` (but :mod:`os.sendfile` will be used + for plain-text sockets only, else :meth:`~socket.socket.send()` will be used) - :meth:`~socket.socket.shutdown()` However, since the SSL (and TLS) protocol has its own framing atop @@ -798,6 +833,15 @@ SSL Sockets Usually, :class:`SSLSocket` are not created directly, but using the :func:`wrap_socket` function or the :meth:`SSLContext.wrap_socket` method. + .. versionchanged:: 3.5 + The :meth:`sendfile` method was added. + + .. versionchanged:: 3.5 + The :meth:`shutdown` does not reset the socket timeout each time bytes + are received or sent. The socket timeout is now to maximum total duration + of the shutdown. + + SSL sockets also have the following additional methods and attributes: .. method:: SSLSocket.read(len=0, buffer=None) @@ -812,6 +856,11 @@ SSL sockets also have the following additional methods and attributes: As at any time a re-negotiation is possible, a call to :meth:`read` can also cause write operations. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration to read up to *len* + bytes. + .. method:: SSLSocket.write(buf) Write *buf* to the SSL socket and return the number of bytes written. The @@ -823,6 +872,10 @@ SSL sockets also have the following additional methods and attributes: As at any time a re-negotiation is possible, a call to :meth:`write` can also cause read operations. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration to write *buf*. + .. note:: The :meth:`~SSLSocket.read` and :meth:`~SSLSocket.write` methods are the @@ -844,6 +897,10 @@ SSL sockets also have the following additional methods and attributes: :attr:`~SSLContext.check_hostname` attribute of the socket's :attr:`~SSLSocket.context` is true. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration of the handshake. + .. method:: SSLSocket.getpeercert(binary_form=False) If there is no certificate for the peer on the other end of the connection, @@ -917,6 +974,17 @@ SSL sockets also have the following additional methods and attributes: version of the SSL protocol that defines its use, and the number of secret bits being used. If no connection has been established, returns ``None``. +.. method:: SSLSocket.shared_ciphers() + + Return the list of ciphers shared by the client during the handshake. Each + entry of the returned list is a three-value tuple containing the name of the + cipher, the version of the SSL protocol that defines its use, and the number + of secret bits the cipher uses. :meth:`~SSLSocket.shared_ciphers` returns + ``None`` if no connection has been established or the socket is a client + socket. + + .. versionadded:: 3.5 + .. method:: SSLSocket.compression() Return the compression algorithm being used as a string, or ``None`` @@ -940,12 +1008,22 @@ SSL sockets also have the following additional methods and attributes: .. versionadded:: 3.3 +.. method:: SSLSocket.selected_alpn_protocol() + + Return the protocol that was selected during the TLS handshake. If + :meth:`SSLContext.set_alpn_protocols` was not called, if the other party does + not support ALPN, if this socket does not support any of the client's + proposed protocols, or if the handshake has not happened yet, ``None`` is + returned. + + .. versionadded:: 3.5 + .. method:: SSLSocket.selected_npn_protocol() - Returns the protocol that was selected during the TLS/SSL handshake. If - :meth:`SSLContext.set_npn_protocols` was not called, or if the other party - does not support NPN, or if the handshake has not yet happened, this will - return ``None``. + Return the higher-level protocol that was selected during the TLS/SSL + handshake. If :meth:`SSLContext.set_npn_protocols` was not called, or + if the other party does not support NPN, or if the handshake has not yet + happened, this will return ``None``. .. versionadded:: 3.3 @@ -957,6 +1035,16 @@ SSL sockets also have the following additional methods and attributes: returned socket should always be used for further communication with the other side of the connection, rather than the original socket. +.. method:: SSLSocket.version() + + Return the actual SSL protocol version negotiated by the connection + as a string, or ``None`` is no secure connection is established. + As of this writing, possible return values include ``"SSLv2"``, + ``"SSLv3"``, ``"TLSv1"``, ``"TLSv1.1"`` and ``"TLSv1.2"``. + Recent OpenSSL versions may define more return values. + + .. versionadded:: 3.5 + .. method:: SSLSocket.pending() Returns the number of already decrypted bytes available for read, pending on @@ -1135,6 +1223,20 @@ to speed up repeated connections from the same clients. when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will give the currently selected cipher. +.. method:: SSLContext.set_alpn_protocols(protocols) + + Specify which protocols the socket should advertise during the SSL/TLS + handshake. It should be a list of ASCII strings, like ``['http/1.1', + 'spdy/2']``, ordered by preference. The selection of a protocol will happen + during the handshake, and will play out according to :rfc:`7301`. After a + successful handshake, the :meth:`SSLSocket.selected_alpn_protocol` method will + return the agreed-upon protocol. + + This method will raise :exc:`NotImplementedError` if :data:`HAS_ALPN` is + False. + + .. versionadded:: 3.5 + .. method:: SSLContext.set_npn_protocols(protocols) Specify which protocols the socket should advertise during the SSL/TLS @@ -1175,7 +1277,7 @@ to speed up repeated connections from the same clients. Due to the early negotiation phase of the TLS connection, only limited methods and attributes are usable like - :meth:`SSLSocket.selected_npn_protocol` and :attr:`SSLSocket.context`. + :meth:`SSLSocket.selected_alpn_protocol` and :attr:`SSLSocket.context`. :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.cipher` and :meth:`SSLSocket.compress` methods require that the TLS connection has progressed beyond the TLS Client Hello and therefore @@ -1251,10 +1353,20 @@ to speed up repeated connections from the same clients. quite similarly to HTTP virtual hosts. Specifying *server_hostname* will raise a :exc:`ValueError` if *server_side* is true. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 Always allow a server_hostname to be passed, even if OpenSSL does not have SNI. +.. method:: SSLContext.wrap_bio(incoming, outgoing, server_side=False, \ + server_hostname=None) + + Create a new :class:`SSLObject` instance by wrapping the BIO objects + *incoming* and *outgoing*. The SSL routines will read input data from the + incoming BIO and write data to the outgoing BIO. + + The *server_side* and *server_hostname* parameters have the same meaning as + in :meth:`SSLContext.wrap_socket`. + .. method:: SSLContext.session_stats() Get statistics about the SSL sessions created or managed by this context. @@ -1627,7 +1739,7 @@ are finished with the client (or the client is finished with you):: And go back to listening for new client connections (of course, a real server would probably handle each client connection in a separate thread, or put -the sockets in non-blocking mode and use an event loop). +the sockets in :ref:`non-blocking mode <ssl-nonblocking>` and use an event loop). .. _ssl-nonblocking: @@ -1649,6 +1761,12 @@ thus several things you need to be aware of: socket first, and attempts to *read* from the SSL socket may require a prior *write* to the underlying socket. + .. versionchanged:: 3.5 + + In earlier Python versions, the :meth:`!SSLSocket.send` method + returned zero instead of raising :exc:`SSLWantWriteError` or + :exc:`SSLWantReadError`. + - Calling :func:`~select.select` tells you that the OS-level socket can be read from (or written to), but it does not imply that there is sufficient data at the upper SSL layer. For example, only part of an SSL frame might @@ -1681,13 +1799,143 @@ thus several things you need to be aware of: .. seealso:: - The :mod:`asyncio` module supports non-blocking SSL sockets and provides a + The :mod:`asyncio` module supports :ref:`non-blocking SSL sockets + <ssl-nonblocking>` and provides a higher level API. It polls for events using the :mod:`selectors` module and handles :exc:`SSLWantWriteError`, :exc:`SSLWantReadError` and :exc:`BlockingIOError` exceptions. It runs the SSL handshake asynchronously as well. +Memory BIO Support +------------------ + +.. versionadded:: 3.5 + +Ever since the SSL module was introduced in Python 2.6, the :class:`SSLSocket` +class has provided two related but distinct areas of functionality: + +- SSL protocol handling +- Network IO + +The network IO API is identical to that provided by :class:`socket.socket`, +from which :class:`SSLSocket` also inherits. This allows an SSL socket to be +used as a drop-in replacement for a regular socket, making it very easy to add +SSL support to an existing application. + +Combining SSL protocol handling and network IO usually works well, but there +are some cases where it doesn't. An example is async IO frameworks that want to +use a different IO multiplexing model than the "select/poll on a file +descriptor" (readiness based) model that is assumed by :class:`socket.socket` +and by the internal OpenSSL socket IO routines. This is mostly relevant for +platforms like Windows where this model is not efficient. For this purpose, a +reduced scope variant of :class:`SSLSocket` called :class:`SSLObject` is +provided. + +.. class:: SSLObject + + A reduced-scope variant of :class:`SSLSocket` representing an SSL protocol + instance that does not contain any network IO methods. This class is + typically used by framework authors that want to implement asynchronous IO + for SSL through memory buffers. + + This class implements an interface on top of a low-level SSL object as + implemented by OpenSSL. This object captures the state of an SSL connection + but does not provide any network IO itself. IO needs to be performed through + separate "BIO" objects which are OpenSSL's IO abstraction layer. + + An :class:`SSLObject` instance can be created using the + :meth:`~SSLContext.wrap_bio` method. This method will create the + :class:`SSLObject` instance and bind it to a pair of BIOs. The *incoming* + BIO is used to pass data from Python to the SSL protocol instance, while the + *outgoing* BIO is used to pass data the other way around. + + The following methods are available: + + - :attr:`~SSLSocket.context` + - :attr:`~SSLSocket.server_side` + - :attr:`~SSLSocket.server_hostname` + - :meth:`~SSLSocket.read` + - :meth:`~SSLSocket.write` + - :meth:`~SSLSocket.getpeercert` + - :meth:`~SSLSocket.selected_npn_protocol` + - :meth:`~SSLSocket.cipher` + - :meth:`~SSLSocket.shared_ciphers` + - :meth:`~SSLSocket.compression` + - :meth:`~SSLSocket.pending` + - :meth:`~SSLSocket.do_handshake` + - :meth:`~SSLSocket.unwrap` + - :meth:`~SSLSocket.get_channel_binding` + + When compared to :class:`SSLSocket`, this object lacks the following + features: + + - Any form of network IO incluging methods such as ``recv()`` and + ``send()``. + + - There is no *do_handshake_on_connect* machinery. You must always manually + call :meth:`~SSLSocket.do_handshake` to start the handshake. + + - There is no handling of *suppress_ragged_eofs*. All end-of-file conditions + that are in violation of the protocol are reported via the + :exc:`SSLEOFError` exception. + + - The method :meth:`~SSLSocket.unwrap` call does not return anything, + unlike for an SSL socket where it returns the underlying socket. + + - The *server_name_callback* callback passed to + :meth:`SSLContext.set_servername_callback` will get an :class:`SSLObject` + instance instead of a :class:`SSLSocket` instance as its first parameter. + + Some notes related to the use of :class:`SSLObject`: + + - All IO on an :class:`SSLObject` is :ref:`non-blocking <ssl-nonblocking>`. + This means that for example :meth:`~SSLSocket.read` will raise an + :exc:`SSLWantReadError` if it needs more data than the incoming BIO has + available. + + - There is no module-level ``wrap_bio()`` call like there is for + :meth:`~SSLContext.wrap_socket`. An :class:`SSLObject` is always created + via an :class:`SSLContext`. + +An SSLObject communicates with the outside world using memory buffers. The +class :class:`MemoryBIO` provides a memory buffer that can be used for this +purpose. It wraps an OpenSSL memory BIO (Basic IO) object: + +.. class:: MemoryBIO + + A memory buffer that can be used to pass data between Python and an SSL + protocol instance. + + .. attribute:: MemoryBIO.pending + + Return the number of bytes currently in the memory buffer. + + .. attribute:: MemoryBIO.eof + + A boolean indicating whether the memory BIO is current at the end-of-file + position. + + .. method:: MemoryBIO.read(n=-1) + + Read up to *n* bytes from the memory buffer. If *n* is not specified or + negative, all bytes are returned. + + .. method:: MemoryBIO.write(buf) + + Write the bytes from *buf* to the memory BIO. The *buf* argument must be an + object supporting the buffer protocol. + + The return value is the number of bytes written, which is always equal to + the length of *buf*. + + .. method:: MemoryBIO.write_eof() + + Write an EOF marker to the memory BIO. After this method has been called, it + is illegal to call :meth:`~MemoryBIO.write`. The attribute :attr:`eof` will + become true after all data currently in the buffer has been read. + + .. _ssl-security: Security considerations diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index 24769f6..845b2ef 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -126,7 +126,7 @@ Example:: if __name__ == '__main__': walktree(sys.argv[1], visitfile) -An additional utility function is provided to covert a file's mode in a human +An additional utility function is provided to convert a file's mode in a human readable string: .. function:: filemode(mode) @@ -399,3 +399,29 @@ The following flags can be used in the *flags* argument of :func:`os.chflags`: The file is a snapshot file. See the \*BSD or Mac OS systems man page :manpage:`chflags(2)` for more information. + +On Windows, the following file attribute constants are available for use when +testing bits in the ``st_file_attributes`` member returned by :func:`os.stat`. +See the `Windows API documentation +<http://msdn.microsoft.com/en-us/library/windows/desktop/gg258117.aspx>`_ +for more detail on the meaning of these constants. + +.. data:: FILE_ATTRIBUTE_ARCHIVE + FILE_ATTRIBUTE_COMPRESSED + FILE_ATTRIBUTE_DEVICE + FILE_ATTRIBUTE_DIRECTORY + FILE_ATTRIBUTE_ENCRYPTED + FILE_ATTRIBUTE_HIDDEN + FILE_ATTRIBUTE_INTEGRITY_STREAM + FILE_ATTRIBUTE_NORMAL + FILE_ATTRIBUTE_NOT_CONTENT_INDEXED + FILE_ATTRIBUTE_NO_SCRUB_DATA + FILE_ATTRIBUTE_OFFLINE + FILE_ATTRIBUTE_READONLY + FILE_ATTRIBUTE_REPARSE_POINT + FILE_ATTRIBUTE_SPARSE_FILE + FILE_ATTRIBUTE_SYSTEM + FILE_ATTRIBUTE_TEMPORARY + FILE_ATTRIBUTE_VIRTUAL + + .. versionadded:: 3.5 diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 057e3af..2c6e7c6 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -354,7 +354,7 @@ Notes: The numeric literals accepted include the digits ``0`` to ``9`` or any Unicode equivalent (code points with the ``Nd`` property). - See http://www.unicode.org/Public/6.3.0/ucd/extracted/DerivedNumericType.txt + See http://www.unicode.org/Public/8.0.0/ucd/extracted/DerivedNumericType.txt for a complete list of code points with the ``Nd`` property. @@ -1937,6 +1937,16 @@ expression support in the :mod:`re` module). >>> 'www.example.com'.strip('cmowz.') 'example' + The outermost leading and trailing *chars* argument values are stripped + from the string. Characters are removed from the leading end until + reaching a string character that is not contained in the set of + characters in *chars*. A similar action takes place on the trailing end. + For example:: + + >>> comment_string = '#....... Section 3.2.1 Issue #32 .......' + >>> comment_string.strip('.#! ') + 'Section 3.2.1 Issue #32' + .. method:: str.swapcase() @@ -2289,6 +2299,19 @@ the bytes type has an additional class method to read data in that format: >>> bytes.fromhex('2Ef0 F1f2 ') b'.\xf0\xf1\xf2' +A reverse conversion function exists to transform a bytes object into its +hexadecimal representation. + +.. method:: bytes.hex() + + Return a string object containing two hexadecimal digits for each + byte in the instance. + + >>> b'\xf0\xf1\xf2'.hex() + 'f0f1f2' + + .. versionadded:: 3.5 + Since bytes objects are sequences of integers (akin to a tuple), for a bytes object *b*, ``b[0]`` will be an integer, while ``b[0:1]`` will be a bytes object of length 1. (This contrasts with text strings, where both indexing @@ -2344,6 +2367,19 @@ the bytearray type has an additional class method to read data in that format: >>> bytearray.fromhex('2Ef0 F1f2 ') bytearray(b'.\xf0\xf1\xf2') +A reverse conversion function exists to transform a bytearray object into its +hexadecimal representation. + +.. method:: bytearray.hex() + + Return a string object containing two hexadecimal digits for each + byte in the instance. + + >>> bytearray(b'\xf0\xf1\xf2').hex() + 'f0f1f2' + + .. versionadded:: 3.5 + Since bytearray objects are sequences of integers (akin to a list), for a bytearray object *b*, ``b[0]`` will be an integer, while ``b[0:1]`` will be a bytearray object of length 1. (This contrasts with text strings, where @@ -3089,6 +3125,203 @@ place, and instead produce new objects. always produces a new object, even if no changes were made. +.. _bytes-formatting: + +``printf``-style Bytes Formatting +---------------------------------- + +.. index:: + single: formatting, bytes (%) + single: formatting, bytearray (%) + single: interpolation, bytes (%) + single: interpolation, bytearray (%) + single: bytes; formatting + single: bytearray; formatting + single: bytes; interpolation + single: bytearray; interpolation + single: printf-style formatting + single: sprintf-style formatting + single: % formatting + single: % interpolation + +.. note:: + + The formatting operations described here exhibit a variety of quirks that + lead to a number of common errors (such as failing to display tuples and + dictionaries correctly). If the value being printed may be a tuple or + dictionary, wrap it in a tuple. + +Bytes objects (``bytes``/``bytearray``) have one unique built-in operation: +the ``%`` operator (modulo). +This is also known as the bytes *formatting* or *interpolation* operator. +Given ``format % values`` (where *format* is a bytes object), ``%`` conversion +specifications in *format* are replaced with zero or more elements of *values*. +The effect is similar to using the :c:func:`sprintf` in the C language. + +If *format* requires a single argument, *values* may be a single non-tuple +object. [5]_ Otherwise, *values* must be a tuple with exactly the number of +items specified by the format bytes object, or a single mapping object (for +example, a dictionary). + +A conversion specifier contains two or more characters and has the following +components, which must occur in this order: + +#. The ``'%'`` character, which marks the start of the specifier. + +#. Mapping key (optional), consisting of a parenthesised sequence of characters + (for example, ``(somename)``). + +#. Conversion flags (optional), which affect the result of some conversion + types. + +#. Minimum field width (optional). If specified as an ``'*'`` (asterisk), the + actual width is read from the next element of the tuple in *values*, and the + object to convert comes after the minimum field width and optional precision. + +#. Precision (optional), given as a ``'.'`` (dot) followed by the precision. If + specified as ``'*'`` (an asterisk), the actual precision is read from the next + element of the tuple in *values*, and the value to convert comes after the + precision. + +#. Length modifier (optional). + +#. Conversion type. + +When the right argument is a dictionary (or other mapping type), then the +formats in the bytes object *must* include a parenthesised mapping key into that +dictionary inserted immediately after the ``'%'`` character. The mapping key +selects the value to be formatted from the mapping. For example: + + >>> print(b'%(language)s has %(number)03d quote types.' % + ... {b'language': b"Python", b"number": 2}) + b'Python has 002 quote types.' + +In this case no ``*`` specifiers may occur in a format (since they require a +sequential parameter list). + +The conversion flag characters are: + ++---------+---------------------------------------------------------------------+ +| Flag | Meaning | ++=========+=====================================================================+ +| ``'#'`` | The value conversion will use the "alternate form" (where defined | +| | below). | ++---------+---------------------------------------------------------------------+ +| ``'0'`` | The conversion will be zero padded for numeric values. | ++---------+---------------------------------------------------------------------+ +| ``'-'`` | The converted value is left adjusted (overrides the ``'0'`` | +| | conversion if both are given). | ++---------+---------------------------------------------------------------------+ +| ``' '`` | (a space) A blank should be left before a positive number (or empty | +| | string) produced by a signed conversion. | ++---------+---------------------------------------------------------------------+ +| ``'+'`` | A sign character (``'+'`` or ``'-'``) will precede the conversion | +| | (overrides a "space" flag). | ++---------+---------------------------------------------------------------------+ + +A length modifier (``h``, ``l``, or ``L``) may be present, but is ignored as it +is not necessary for Python -- so e.g. ``%ld`` is identical to ``%d``. + +The conversion types are: + ++------------+-----------------------------------------------------+-------+ +| Conversion | Meaning | Notes | ++============+=====================================================+=======+ +| ``'d'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'i'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'o'`` | Signed octal value. | \(1) | ++------------+-----------------------------------------------------+-------+ +| ``'u'`` | Obsolete type -- it is identical to ``'d'``. | \(8) | ++------------+-----------------------------------------------------+-------+ +| ``'x'`` | Signed hexadecimal (lowercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'X'`` | Signed hexadecimal (uppercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'e'`` | Floating point exponential format (lowercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'E'`` | Floating point exponential format (uppercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'f'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'F'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'g'`` | Floating point format. Uses lowercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'G'`` | Floating point format. Uses uppercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'c'`` | Single byte (accepts integer or single | | +| | byte objects). | | ++------------+-----------------------------------------------------+-------+ +| ``'b'`` | Bytes (any object that follows the | \(5) | +| | :ref:`buffer protocol <bufferobjects>` or has | | +| | :meth:`__bytes__`). | | ++------------+-----------------------------------------------------+-------+ +| ``'s'`` | ``'s'`` is an alias for ``'b'`` and should only | \(6) | +| | be used for Python2/3 code bases. | | ++------------+-----------------------------------------------------+-------+ +| ``'a'`` | Bytes (converts any Python object using | \(5) | +| | ``repr(obj).encode('ascii','backslashreplace)``). | | ++------------+-----------------------------------------------------+-------+ +| ``'r'`` | ``'r'`` is an alias for ``'a'`` and should only | \(7) | +| | be used for Python2/3 code bases. | | ++------------+-----------------------------------------------------+-------+ +| ``'%'`` | No argument is converted, results in a ``'%'`` | | +| | character in the result. | | ++------------+-----------------------------------------------------+-------+ + +Notes: + +(1) + The alternate form causes a leading zero (``'0'``) to be inserted between + left-hand padding and the formatting of the number if the leading character + of the result is not already a zero. + +(2) + The alternate form causes a leading ``'0x'`` or ``'0X'`` (depending on whether + the ``'x'`` or ``'X'`` format was used) to be inserted between left-hand padding + and the formatting of the number if the leading character of the result is not + already a zero. + +(3) + The alternate form causes the result to always contain a decimal point, even if + no digits follow it. + + The precision determines the number of digits after the decimal point and + defaults to 6. + +(4) + The alternate form causes the result to always contain a decimal point, and + trailing zeroes are not removed as they would otherwise be. + + The precision determines the number of significant digits before and after the + decimal point and defaults to 6. + +(5) + If precision is ``N``, the output is truncated to ``N`` characters. + +(6) + ``b'%s'`` is deprecated, but will not be removed during the 3.x series. + +(7) + ``b'%r'`` is deprecated, but will not be removed during the 3.x series. + +(8) + See :pep:`237`. + +.. note:: + + The bytearray version of this method does *not* operate in place - it + always produces a new object, even if no changes were made. + +.. seealso:: :pep:`461`. +.. versionadded:: 3.5 + .. _typememoryview: Memory Views @@ -3117,10 +3350,8 @@ copying. the view. The :class:`~memoryview.itemsize` attribute will give you the number of bytes in a single element. - A :class:`memoryview` supports slicing to expose its data. If - :class:`~memoryview.format` is one of the native format specifiers - from the :mod:`struct` module, indexing will return a single element - with the correct type. Full slicing will result in a subview:: + A :class:`memoryview` supports slicing and indexing to expose its data. + One-dimensional slicing will result in a subview:: >>> v = memoryview(b'abcefg') >>> v[1] @@ -3132,25 +3363,29 @@ copying. >>> bytes(v[1:4]) b'bce' - Other native formats:: + If :class:`~memoryview.format` is one of the native format specifiers + from the :mod:`struct` module, indexing with an integer or a tuple of + integers is also supported and returns a single *element* with + the correct type. One-dimensional memoryviews can be indexed + with an integer or a one-integer tuple. Multi-dimensional memoryviews + can be indexed with tuples of exactly *ndim* integers where *ndim* is + the number of dimensions. Zero-dimensional memoryviews can be indexed + with the empty tuple. + + Here is an example with a non-byte format:: >>> import array >>> a = array.array('l', [-11111111, 22222222, -33333333, 44444444]) - >>> a[0] + >>> m = memoryview(a) + >>> m[0] -11111111 - >>> a[-1] + >>> m[-1] 44444444 - >>> a[2:3].tolist() - [-33333333] - >>> a[::2].tolist() + >>> m[::2].tolist() [-11111111, -33333333] - >>> a[::-1].tolist() - [44444444, -33333333, 22222222, -11111111] - .. versionadded:: 3.3 - - If the underlying object is writable, the memoryview supports slice - assignment. Resizing is not allowed:: + If the underlying object is writable, the memoryview supports + one-dimensional slice assignment. Resizing is not allowed:: >>> data = bytearray(b'abcefg') >>> v = memoryview(data) @@ -3183,12 +3418,16 @@ copying. True .. versionchanged:: 3.3 + One-dimensional memoryviews can now be sliced. One-dimensional memoryviews with formats 'B', 'b' or 'c' are now hashable. .. versionchanged:: 3.4 memoryview is now registered automatically with :class:`collections.abc.Sequence` + .. versionchanged:: 3.5 + memoryviews can now be indexed with tuple of integers. + :class:`memoryview` has several methods: .. method:: __eq__(exporter) @@ -3255,6 +3494,17 @@ copying. supports all format strings, including those that are not in :mod:`struct` module syntax. + .. method:: hex() + + Return a string object containing two hexadecimal digits for each + byte in the buffer. :: + + >>> m = memoryview(b"abc") + >>> m.hex() + '616263' + + .. versionadded:: 3.5 + .. method:: tolist() Return the data in the buffer as a list of elements. :: diff --git a/Doc/library/string.rst b/Doc/library/string.rst index 19bdb21..2bd8dfd 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -95,6 +95,10 @@ implementation as the built-in :meth:`format` method. an arbitrary set of positional and keyword arguments. :meth:`format` is just a wrapper that calls :meth:`vformat`. + .. deprecated:: 3.5 + Passing a format string as keyword argument *format_string* has been + deprecated. + .. method:: vformat(format_string, args, kwargs) This function does the actual work of formatting. It is exposed as a diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 36cbf3c..9a2f4de 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -25,160 +25,99 @@ modules and functions can be found in the following sections. Using the :mod:`subprocess` Module ---------------------------------- -The recommended approach to invoking subprocesses is to use the following -convenience functions for all use cases they can handle. For more advanced -use cases, the underlying :class:`Popen` interface can be used directly. +The recommended approach to invoking subprocesses is to use the :func:`run` +function for all use cases it can handle. For more advanced use cases, the +underlying :class:`Popen` interface can be used directly. +The :func:`run` function was added in Python 3.5; if you need to retain +compatibility with older versions, see the :ref:`call-function-trio` section. -.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + +.. function:: run(args, *, stdin=None, input=None, stdout=None, stderr=None,\ + shell=False, timeout=None, check=False) Run the command described by *args*. Wait for command to complete, then - return the :attr:`returncode` attribute. + return a :class:`CompletedProcess` instance. The arguments shown above are merely the most common ones, described below in :ref:`frequently-used-arguments` (hence the use of keyword-only notation in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this function passes all - supplied arguments other than *timeout* directly through to that interface. + same as that of the :class:`Popen` constructor - apart from *timeout*, + *input* and *check*, all the arguments to this function are passed through to + that interface. - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The + This does not capture stdout or stderr by default. To do so, pass + :data:`PIPE` for the *stdout* and/or *stderr* arguments. + + The *timeout* argument is passed to :meth:`Popen.communicate`. If the timeout + expires, the child process will be killed and waited for. The :exc:`TimeoutExpired` exception will be re-raised after the child process has terminated. - Examples:: - - >>> subprocess.call(["ls", "-l"]) - 0 - - >>> subprocess.call("exit 1", shell=True) - 1 - - .. note:: - - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. - - .. versionchanged:: 3.3 - *timeout* was added. - - -.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) - - Run command with arguments. Wait for command to complete. If the return - code was zero then return, otherwise raise :exc:`CalledProcessError`. The - :exc:`CalledProcessError` object will have the return code in the - :attr:`~CalledProcessError.returncode` attribute. - - The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the use of keyword-only notation - in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this function passes all - supplied arguments other than *timeout* directly through to that interface. + The *input* argument is passed to :meth:`Popen.communicate` and thus to the + subprocess's stdin. If used it must be a byte sequence, or a string if + ``universal_newlines=True``. When used, the internal :class:`Popen` object + is automatically created with ``stdin=PIPE``, and the *stdin* argument may + not be used as well. - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The - :exc:`TimeoutExpired` exception will be re-raised after the child process - has terminated. + If *check* is True, and the process exits with a non-zero exit code, a + :exc:`CalledProcessError` exception will be raised. Attributes of that + exception hold the arguments, the exit code, and stdout and stderr if they + were captured. Examples:: - >>> subprocess.check_call(["ls", "-l"]) - 0 + >>> subprocess.run(["ls", "-l"]) # doesn't capture output + CompletedProcess(args=['ls', '-l'], returncode=0) - >>> subprocess.check_call("exit 1", shell=True) + >>> subprocess.run("exit 1", shell=True, check=True) Traceback (most recent call last): - ... + ... subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 - .. note:: + >>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE) + CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0, + stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n') - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. + .. versionadded:: 3.5 - .. versionchanged:: 3.3 - *timeout* was added. +.. class:: CompletedProcess + The return value from :func:`run`, representing a process that has finished. -.. function:: check_output(args, *, input=None, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) + .. attribute:: args - Run command with arguments and return its output. + The arguments used to launch the process. This may be a list or a string. - If the return code was non-zero it raises a :exc:`CalledProcessError`. The - :exc:`CalledProcessError` object will have the return code in the - :attr:`~CalledProcessError.returncode` attribute and any output in the - :attr:`~CalledProcessError.output` attribute. + .. attribute:: returncode - The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the use of keyword-only notation - in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this functions passes all - supplied arguments other than *input* and *timeout* directly through to - that interface. In addition, *stdout* is not permitted as an argument, as - it is used internally to collect the output from the subprocess. + Exit status of the child process. Typically, an exit status of 0 indicates + that it ran successfully. - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The - :exc:`TimeoutExpired` exception will be re-raised after the child process - has terminated. + A negative value ``-N`` indicates that the child was terminated by signal + ``N`` (POSIX only). - The *input* argument is passed to :meth:`Popen.communicate` and thus to the - subprocess's stdin. If used it must be a byte sequence, or a string if - ``universal_newlines=True``. When used, the internal :class:`Popen` object - is automatically created with ``stdin=PIPE``, and the *stdin* argument may - not be used as well. + .. attribute:: stdout - Examples:: + Captured stdout from the child process. A bytes sequence, or a string if + :func:`run` was called with ``universal_newlines=True``. None if stdout + was not captured. - >>> subprocess.check_output(["echo", "Hello World!"]) - b'Hello World!\n' + If you ran the process with ``stderr=subprocess.STDOUT``, stdout and + stderr will be combined in this attribute, and :attr:`stderr` will be + None. - >>> subprocess.check_output(["echo", "Hello World!"], universal_newlines=True) - 'Hello World!\n' + .. attribute:: stderr - >>> subprocess.check_output(["sed", "-e", "s/foo/bar/"], - ... input=b"when in the course of fooman events\n") - b'when in the course of barman events\n' + Captured stderr from the child process. A bytes sequence, or a string if + :func:`run` was called with ``universal_newlines=True``. None if stderr + was not captured. - >>> subprocess.check_output("exit 1", shell=True) - Traceback (most recent call last): - ... - subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 + .. method:: check_returncode() - By default, this function will return the data as encoded bytes. The actual - encoding of the output data may depend on the command being invoked, so the - decoding to text will often need to be handled at the application level. + If :attr:`returncode` is non-zero, raise a :exc:`CalledProcessError`. - This behaviour may be overridden by setting *universal_newlines* to - ``True`` as described below in :ref:`frequently-used-arguments`. - - To also capture standard error in the result, use - ``stderr=subprocess.STDOUT``:: - - >>> subprocess.check_output( - ... "ls non_existent_file; exit 0", - ... stderr=subprocess.STDOUT, - ... shell=True) - 'ls: non_existent_file: No such file or directory\n' - - .. note:: - - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. - - .. versionadded:: 3.1 - - .. versionchanged:: 3.3 - *timeout* was added. - - .. versionchanged:: 3.4 - *input* was added. + .. versionadded:: 3.5 .. data:: DEVNULL @@ -225,11 +164,22 @@ use cases, the underlying :class:`Popen` interface can be used directly. .. attribute:: output - Output of the child process if this exception is raised by + Output of the child process if it was captured by :func:`run` or :func:`check_output`. Otherwise, ``None``. + .. attribute:: stdout + + Alias for output, for symmetry with :attr:`stderr`. + + .. attribute:: stderr + + Stderr output of the child process if it was captured by :func:`run`. + Otherwise, ``None``. + .. versionadded:: 3.3 + .. versionchanged:: 3.5 + *stdout* and *stderr* attributes added .. exception:: CalledProcessError @@ -246,9 +196,20 @@ use cases, the underlying :class:`Popen` interface can be used directly. .. attribute:: output - Output of the child process if this exception is raised by + Output of the child process if it was captured by :func:`run` or :func:`check_output`. Otherwise, ``None``. + .. attribute:: stdout + + Alias for output, for symmetry with :attr:`stderr`. + + .. attribute:: stderr + + Stderr output of the child process if it was captured by :func:`run`. + Otherwise, ``None``. + + .. versionchanged:: 3.5 + *stdout* and *stderr* attributes added .. _frequently-used-arguments: @@ -635,6 +596,7 @@ Instances of the :class:`Popen` class have the following methods: must be bytes or, if *universal_newlines* was ``True``, a string. :meth:`communicate` returns a tuple ``(stdout_data, stderr_data)``. + The data will be bytes or, if *universal_newlines* was ``True``, strings. Note that if you want to send data to the process's stdin, you need to create the Popen object with ``stdin=PIPE``. Similarly, to get anything other than @@ -851,6 +813,96 @@ The :mod:`subprocess` module exposes the following constants. This flag is ignored if :data:`CREATE_NEW_CONSOLE` is specified. +.. _call-function-trio: + +Older high-level API +-------------------- + +Prior to Python 3.5, these three functions comprised the high level API to +subprocess. You can now use :func:`run` in many cases, but lots of existing code +calls these functions. + +.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + + Run the command described by *args*. Wait for command to complete, then + return the :attr:`returncode` attribute. + + This is equivalent to:: + + run(...).returncode + + (except that the *input* and *check* parameters are not supported) + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this + function. The child process will block if it generates enough + output to a pipe to fill up the OS pipe buffer as the pipes are + not being read from. + + .. versionchanged:: 3.3 + *timeout* was added. + +.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + + Run command with arguments. Wait for command to complete. If the return + code was zero then return, otherwise raise :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`~CalledProcessError.returncode` attribute. + + This is equivalent to:: + + run(..., check=True) + + (except that the *input* parameter is not supported) + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this + function. The child process will block if it generates enough + output to a pipe to fill up the OS pipe buffer as the pipes are + not being read from. + + .. versionchanged:: 3.3 + *timeout* was added. + + +.. function:: check_output(args, *, input=None, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) + + Run command with arguments and return its output. + + If the return code was non-zero it raises a :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`~CalledProcessError.returncode` attribute and any output in the + :attr:`~CalledProcessError.output` attribute. + + This is equivalent to:: + + run(..., check=True, stdout=PIPE).stdout + + By default, this function will return the data as encoded bytes. The actual + encoding of the output data may depend on the command being invoked, so the + decoding to text will often need to be handled at the application level. + + This behaviour may be overridden by setting *universal_newlines* to + ``True`` as described above in :ref:`frequently-used-arguments`. + + To also capture standard error in the result, use + ``stderr=subprocess.STDOUT``:: + + >>> subprocess.check_output( + ... "ls non_existent_file; exit 0", + ... stderr=subprocess.STDOUT, + ... shell=True) + 'ls: non_existent_file: No such file or directory\n' + + .. versionadded:: 3.1 + + .. versionchanged:: 3.3 + *timeout* was added. + + .. versionchanged:: 3.4 + *input* was added. .. _subprocess-replacements: diff --git a/Doc/library/symtable.rst b/Doc/library/symtable.rst index 2503d33..ba2caff 100644 --- a/Doc/library/symtable.rst +++ b/Doc/library/symtable.rst @@ -71,10 +71,6 @@ Examining Symbol Tables Return ``True`` if the block uses ``exec``. - .. method:: has_import_star() - - Return ``True`` if the block uses a starred from-import. - .. method:: get_identifiers() Return a list of names of symbols in this table. diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 3024086..545e674 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -167,7 +167,7 @@ always available. .. data:: dont_write_bytecode - If this is true, Python won't try to write ``.pyc`` or ``.pyo`` files on the + If this is true, Python won't try to write ``.pyc`` files on the import of source modules. This value is initially set to ``True`` or ``False`` depending on the :option:`-B` command line option and the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable, but you can set it @@ -576,6 +576,18 @@ always available. *service_pack_major*, *suite_mask*, and *product_type*. +.. function:: get_coroutine_wrapper() + + Returns ``None``, or a wrapper set by :func:`set_coroutine_wrapper`. + + .. versionadded:: 3.5 + See :pep:`492` for more details. + + .. note:: + This function has been added on a provisional basis (see :pep:`411` + for details.) Use it only for debugging purposes. + + .. data:: hash_info A :term:`struct sequence` giving parameters of the numeric hash @@ -718,6 +730,14 @@ always available. value of :func:`intern` around to benefit from it. +.. function:: is_finalizing() + + Return :const:`True` if the Python interpreter is + :term:`shutting down <interpreter shutdown>`, :const:`False` otherwise. + + .. versionadded:: 3.5 + + .. data:: last_type last_value last_traceback @@ -1053,6 +1073,46 @@ always available. thus not likely to be implemented elsewhere. +.. function:: set_coroutine_wrapper(wrapper) + + Allows intercepting creation of :term:`coroutine` objects (only ones that + are created by an :keyword:`async def` function; generators decorated with + :func:`types.coroutine` or :func:`asyncio.coroutine` will not be + intercepted). + + The *wrapper* argument must be either: + + * a callable that accepts one argument (a coroutine object); + * ``None``, to reset the wrapper. + + If called twice, the new wrapper replaces the previous one. The function + is thread-specific. + + The *wrapper* callable cannot define new coroutines directly or indirectly:: + + def wrapper(coro): + async def wrap(coro): + return await coro + return wrap(coro) + sys.set_coroutine_wrapper(wrapper) + + async def foo(): + pass + + # The following line will fail with a RuntimeError, because + # `wrapper` creates a `wrap(coro)` coroutine: + foo() + + See also :func:`get_coroutine_wrapper`. + + .. versionadded:: 3.5 + See :pep:`492` for more details. + + .. note:: + This function has been added on a provisional basis (see :pep:`411` + for details.) Use it only for debugging purposes. + + .. data:: stdin stdout stderr @@ -1223,4 +1283,3 @@ always available. .. rubric:: Citations .. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf\ . - diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index 05f29ad..adacb0a 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -62,6 +62,23 @@ Some facts and figures: +------------------+---------------------------------------------+ | ``'r:xz'`` | Open for reading with lzma compression. | +------------------+---------------------------------------------+ + | ``'x'`` or | Create a tarfile exclusively without | + | ``'x:'`` | compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:gz'`` | Create a tarfile with gzip compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:bz2'`` | Create a tarfile with bzip2 compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:xz'`` | Create a tarfile with lzma compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ | ``'a' or 'a:'`` | Open for appending with no compression. The | | | file is created if it does not exist. | +------------------+---------------------------------------------+ @@ -82,9 +99,9 @@ Some facts and figures: If *fileobj* is specified, it is used as an alternative to a :term:`file object` opened in binary mode for *name*. It is supposed to be at position 0. - For modes ``'w:gz'``, ``'r:gz'``, ``'w:bz2'``, ``'r:bz2'``, :func:`tarfile.open` - accepts the keyword argument *compresslevel* to specify the compression level of - the file. + For modes ``'w:gz'``, ``'r:gz'``, ``'w:bz2'``, ``'r:bz2'``, ``'x:gz'``, + ``'x:bz2'``, :func:`tarfile.open` accepts the keyword argument + *compresslevel* to specify the compression level of the file. For special purposes, there is a second format for *mode*: ``'filemode|[compression]'``. :func:`tarfile.open` will return a :class:`TarFile` @@ -127,6 +144,8 @@ Some facts and figures: | | writing. | +-------------+--------------------------------------------+ + .. versionchanged:: 3.5 + The ``'x'`` (exclusive creation) mode was added. .. class:: TarFile @@ -252,8 +271,8 @@ be finalized; only the internally used file object will be closed. See the In this case, the file object's :attr:`name` attribute is used if it exists. *mode* is either ``'r'`` to read from an existing archive, ``'a'`` to append - data to an existing file or ``'w'`` to create a new file overwriting an existing - one. + data to an existing file, ``'w'`` to create a new file overwriting an existing + one or ``'x'`` to create a new file only if it's not exists. If *fileobj* is given, it is used for reading or writing data. If it can be determined, *mode* is overridden by *fileobj*'s mode. *fileobj* will be used @@ -292,12 +311,14 @@ be finalized; only the internally used file object will be closed. See the to be handled. The default settings will work for most users. See section :ref:`tar-unicode` for in-depth information. - .. versionchanged:: 3.2 - Use ``'surrogateescape'`` as the default for the *errors* argument. - The *pax_headers* argument is an optional dictionary of strings which will be added as a pax global header if *format* is :const:`PAX_FORMAT`. + .. versionchanged:: 3.2 + Use ``'surrogateescape'`` as the default for the *errors* argument. + + .. versionchanged:: 3.5 + The ``'x'`` (exclusive creation) mode was added. .. classmethod:: TarFile.open(...) @@ -328,11 +349,15 @@ be finalized; only the internally used file object will be closed. See the returned by :meth:`getmembers`. -.. method:: TarFile.list(verbose=True) +.. method:: TarFile.list(verbose=True, *, members=None) Print a table of contents to ``sys.stdout``. If *verbose* is :const:`False`, only the names of the members are printed. If it is :const:`True`, output - similar to that of :program:`ls -l` is produced. + similar to that of :program:`ls -l` is produced. If optional *members* is + given, it must be a subset of the list returned by :meth:`getmembers`. + + .. versionchanged:: 3.5 + Added the *members* parameter. .. method:: TarFile.next() @@ -342,7 +367,7 @@ be finalized; only the internally used file object will be closed. See the available. -.. method:: TarFile.extractall(path=".", members=None) +.. method:: TarFile.extractall(path=".", members=None, *, numeric_owner=False) Extract all members from the archive to the current working directory or directory *path*. If optional *members* is given, it must be a subset of the @@ -352,6 +377,10 @@ be finalized; only the internally used file object will be closed. See the reset each time a file is created in it. And, if a directory's permissions do not allow writing, extracting files to it will fail. + If *numeric_owner* is :const:`True`, the uid and gid numbers from the tarfile + are used to set the owner/group for the extracted files. Otherwise, the named + values from the tarfile are used. + .. warning:: Never extract archives from untrusted sources without prior inspection. @@ -359,8 +388,11 @@ be finalized; only the internally used file object will be closed. See the that have absolute filenames starting with ``"/"`` or filenames with two dots ``".."``. + .. versionchanged:: 3.5 + Added the *numeric_only* parameter. -.. method:: TarFile.extract(member, path="", set_attrs=True) + +.. method:: TarFile.extract(member, path="", set_attrs=True, *, numeric_owner=False) Extract a member from the archive to the current working directory, using its full name. Its file information is extracted as accurately as possible. *member* @@ -368,6 +400,10 @@ be finalized; only the internally used file object will be closed. See the directory using *path*. File attributes (owner, mtime, mode) are set unless *set_attrs* is false. + If *numeric_owner* is :const:`True`, the uid and gid numbers from the tarfile + are used to set the owner/group for the extracted files. Otherwise, the named + values from the tarfile are used. + .. note:: The :meth:`extract` method does not take care of several extraction issues. @@ -380,6 +416,9 @@ be finalized; only the internally used file object will be closed. See the .. versionchanged:: 3.2 Added the *set_attrs* parameter. + .. versionchanged:: 3.5 + Added the *numeric_only* parameter. + .. method:: TarFile.extractfile(member) Extract a member from the archive as a file object. *member* may be a filename @@ -802,4 +841,3 @@ In case of :const:`PAX_FORMAT` archives, *encoding* is generally not needed because all the metadata is stored using *UTF-8*. *encoding* is only used in the rare cases when binary pax headers are decoded or when strings with surrogate characters are stored. - diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 44d025d..0387fb1 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -54,6 +54,13 @@ The module defines the following user-callable items: underlying true file object. This file-like object can be used in a :keyword:`with` statement, just like a normal file. + The :py:data:`os.O_TMPFILE` flag is used if it is available and works + (Linux-specific, require Linux kernel 3.11 or later). + + .. versionchanged:: 3.5 + + The :py:data:`os.O_TMPFILE` flag is now used if available. + .. function:: NamedTemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None, delete=True) @@ -112,7 +119,7 @@ The module defines the following user-callable items: .. versionadded:: 3.2 -.. function:: mkstemp(suffix='', prefix='tmp', dir=None, text=False) +.. function:: mkstemp(suffix=None, prefix=None, dir=None, text=False) Creates a temporary file in the most secure manner possible. There are no race conditions in the file's creation, assuming that the platform @@ -141,6 +148,16 @@ The module defines the following user-callable items: filename will have any nice properties, such as not requiring quoting when passed to external commands via ``os.popen()``. + *suffix*, *prefix*, and *dir* must all contain the same type, if specified. + If they are bytes, the returned name will be bytes instead of str. + If you want to force a bytes return value with otherwise default behavior, + pass ``suffix=b''``. + + A *prefix* value of ``None`` means use the return value of + :func:`gettempprefix` or :func:`gettempprefixb` as appropriate. + + A *suffix* value of ``None`` means use an appropriate empty value. + If *text* is specified, it indicates whether to open the file in binary mode (the default) or text mode. On some platforms, this makes no difference. @@ -149,8 +166,14 @@ The module defines the following user-callable items: file (as would be returned by :func:`os.open`) and the absolute pathname of that file, in that order. + .. versionchanged:: 3.5 + *suffix*, *prefix*, and *dir* may now be supplied in bytes in order to + obtain a bytes return value. Prior to this, only str was allowed. + *suffix* and *prefix* now accept and default to ``None`` to cause + an appropriate default value to be used. + -.. function:: mkdtemp(suffix='', prefix='tmp', dir=None) +.. function:: mkdtemp(suffix=None, prefix=None, dir=None) Creates a temporary directory in the most secure manner possible. There are no race conditions in the directory's creation. The directory is @@ -164,6 +187,12 @@ The module defines the following user-callable items: :func:`mkdtemp` returns the absolute pathname of the new directory. + .. versionchanged:: 3.5 + *suffix*, *prefix*, and *dir* may now be supplied in bytes in order to + obtain a bytes return value. Prior to this, only str was allowed. + *suffix* and *prefix* now accept and default to ``None`` to cause + an appropriate default value to be used. + .. function:: mktemp(suffix='', prefix='tmp', dir=None) @@ -232,12 +261,23 @@ the appropriate function arguments, instead. :data:`tempdir` is not ``None``, this simply returns its contents; otherwise, the search described above is performed, and the result returned. +.. function:: gettempdirb() + + Same as :func:`gettempdir` but the return value is in bytes. + + .. versionadded:: 3.5 .. function:: gettempprefix() Return the filename prefix used to create temporary files. This does not contain the directory component. +.. function:: gettempprefixb() + + Same as :func:`gettempprefixb` but the return value is in bytes. + + .. versionadded:: 3.5 + Examples -------- diff --git a/Doc/library/test.rst b/Doc/library/test.rst index 974909e..8f2df7c 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -568,6 +568,17 @@ The :mod:`test.support` module defines the following functions: def load_tests(*args): return load_package_tests(os.path.dirname(__file__), *args) +.. function:: detect_api_mismatch(ref_api, other_api, *, ignore=()): + + Returns the set of attributes, functions or methods of `ref_api` not + found on `other_api`, except for a defined list of items to be + ignored in this check specified in `ignore`. + + By default this skips private attributes beginning with '_' but + includes all magic methods, i.e. those starting and ending in '__'. + + .. versionadded:: 3.5 + The :mod:`test.support` module defines the following classes: diff --git a/Doc/library/time.rst b/Doc/library/time.rst index 8d8b7d4..3d335c8 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -314,9 +314,9 @@ The module defines the following functions and data items: processes running for more than 49 days. On more recent versions of Windows and on other operating systems, :func:`monotonic` is system-wide. - Availability: Windows, Mac OS X, Linux, FreeBSD, OpenBSD, Solaris. - .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now always available. .. function:: perf_counter() @@ -350,6 +350,11 @@ The module defines the following functions and data items: requested by an arbitrary amount because of the scheduling of other activity in the system. + .. versionchanged:: 3.5 + The function now sleeps at least *secs* even if the sleep is interrupted + by a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale). + .. function:: strftime(format[, t]) diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst index 70df409..d1051f6 100644 --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -59,18 +59,26 @@ Python Interface The module defines three convenience functions and a public class: -.. function:: timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000) +.. function:: timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000, globals=None) Create a :class:`Timer` instance with the given statement, *setup* code and *timer* function and run its :meth:`.timeit` method with *number* executions. + The optional *globals* argument specifies a namespace in which to execute the + code. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. -.. function:: repeat(stmt='pass', setup='pass', timer=<default timer>, repeat=3, number=1000000) + +.. function:: repeat(stmt='pass', setup='pass', timer=<default timer>, repeat=3, number=1000000, globals=None) Create a :class:`Timer` instance with the given statement, *setup* code and *timer* function and run its :meth:`.repeat` method with the given *repeat* - count and *number* executions. + count and *number* executions. The optional *globals* argument specifies a + namespace in which to execute the code. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. .. function:: default_timer() @@ -80,7 +88,7 @@ The module defines three convenience functions and a public class: :func:`time.perf_counter` is now the default timer. -.. class:: Timer(stmt='pass', setup='pass', timer=<timer function>) +.. class:: Timer(stmt='pass', setup='pass', timer=<timer function>, globals=None) Class for timing execution speed of small code snippets. @@ -88,7 +96,9 @@ The module defines three convenience functions and a public class: for setup, and a timer function. Both statements default to ``'pass'``; the timer function is platform-dependent (see the module doc string). *stmt* and *setup* may also contain multiple statements separated by ``;`` - or newlines, as long as they don't contain multi-line string literals. + or newlines, as long as they don't contain multi-line string literals. The + statement will by default be executed within timeit's namespace; this behavior + can be controlled by passing a namespace to *globals*. To measure the execution time of the first statement, use the :meth:`.timeit` method. The :meth:`.repeat` method is a convenience to call :meth:`.timeit` @@ -101,6 +111,8 @@ The module defines three convenience functions and a public class: will then be executed by :meth:`.timeit`. Note that the timing overhead is a little larger in this case because of the extra function calls. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. .. method:: Timer.timeit(number=1000000) @@ -169,7 +181,7 @@ Command-Line Interface When called as a program from the command line, the following form is used:: - python -m timeit [-n N] [-r N] [-s S] [-t] [-c] [-h] [statement ...] + python -m timeit [-n N] [-r N] [-u U] [-s S] [-t] [-c] [-h] [statement ...] Where the following options are understood: @@ -198,6 +210,12 @@ Where the following options are understood: use :func:`time.time` (deprecated) +.. cmdoption:: -u, --unit=U + + specify a time unit for timer output; can select usec, msec, or sec + + .. versionadded:: 3.5 + .. cmdoption:: -c, --clock use :func:`time.clock` (deprecated) @@ -320,3 +338,17 @@ To give the :mod:`timeit` module access to functions you define, you can pass a if __name__ == '__main__': import timeit print(timeit.timeit("test()", setup="from __main__ import test")) + +Another option is to pass :func:`globals` to the *globals* parameter, which will cause the code +to be executed within your current global namespace. This can be more convenient +than individually specifying imports:: + + def f(x): + return x**2 + def g(x): + return x**4 + def h(x): + return x**8 + + import timeit + print(timeit.timeit('[func(42) for func in (f,g,h)]', globals=globals())) diff --git a/Doc/library/token.rst b/Doc/library/token.rst index 4cd7098..88fb38b 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -93,6 +93,7 @@ The token constants are: DOUBLESLASH DOUBLESLASHEQUAL AT + ATEQUAL RARROW ELLIPSIS OP diff --git a/Doc/library/traceback.rst b/Doc/library/traceback.rst index 15fbedc..8d216d0 100644 --- a/Doc/library/traceback.rst +++ b/Doc/library/traceback.rst @@ -22,15 +22,20 @@ The module defines the following functions: .. function:: print_tb(traceback, limit=None, file=None) - Print up to *limit* stack trace entries from *traceback*. If *limit* is omitted - or ``None``, all entries are printed. If *file* is omitted or ``None``, the - output goes to ``sys.stderr``; otherwise it should be an open file or file-like - object to receive the output. + Print up to *limit* stack trace entries from *traceback* (starting from + the caller's frame) if *limit* is positive. Otherwise, print the last + ``abs(limit)`` entries. If *limit* is omitted or ``None``, all entries + are printed. If *file* is omitted or ``None``, the output goes to + ``sys.stderr``; otherwise it should be an open file or file-like object + to receive the output. + + .. versionchanged:: 3.5 + Added negative *limit* support. .. function:: print_exception(type, value, traceback, limit=None, file=None, chain=True) - Print exception information and up to *limit* stack trace entries from + Print exception information and stack trace entries from *traceback* to *file*. This differs from :func:`print_tb` in the following ways: @@ -41,6 +46,7 @@ The module defines the following functions: prints the line where the syntax error occurred with a caret indicating the approximate position of the error. + The optional *limit* argument has the same meaning as for :func:`print_tb`. If *chain* is true (the default), then chained exceptions (the :attr:`__cause__` or :attr:`__context__` attributes of the exception) will be printed as well, like the interpreter itself does when printing an unhandled @@ -49,33 +55,41 @@ The module defines the following functions: .. function:: print_exc(limit=None, file=None, chain=True) - This is a shorthand for ``print_exception(*sys.exc_info())``. + This is a shorthand for ``print_exception(*sys.exc_info(), limit, file, + chain)``. .. function:: print_last(limit=None, file=None, chain=True) This is a shorthand for ``print_exception(sys.last_type, sys.last_value, - sys.last_traceback, limit, file)``. In general it will work only after - an exception has reached an interactive prompt (see :data:`sys.last_type`). + sys.last_traceback, limit, file, chain)``. In general it will work only + after an exception has reached an interactive prompt (see + :data:`sys.last_type`). .. function:: print_stack(f=None, limit=None, file=None) - This function prints a stack trace from its invocation point. The optional *f* - argument can be used to specify an alternate stack frame to start. The optional - *limit* and *file* arguments have the same meaning as for - :func:`print_exception`. + Print up to *limit* stack trace entries (starting from the invocation + point) if *limit* is positive. Otherwise, print the last ``abs(limit)`` + entries. If *limit* is omitted or ``None``, all entries are printed. + The optional *f* argument can be used to specify an alternate stack frame + to start. The optional *file* argument has the same meaning as for + :func:`print_tb`. + + .. versionchanged:: 3.5 + Added negative *limit* support. .. function:: extract_tb(traceback, limit=None) - Return a list of up to *limit* "pre-processed" stack trace entries extracted - from the traceback object *traceback*. It is useful for alternate formatting of - stack traces. If *limit* is omitted or ``None``, all entries are extracted. A - "pre-processed" stack trace entry is a 4-tuple (*filename*, *line number*, - *function name*, *text*) representing the information that is usually printed - for a stack trace. The *text* is a string with leading and trailing whitespace - stripped; if the source is not available it is ``None``. + Return a list of "pre-processed" stack trace entries extracted from the + traceback object *traceback*. It is useful for alternate formatting of + stack traces. The optional *limit* argument has the same meaning as for + :func:`print_tb`. A "pre-processed" stack trace entry is a 4-tuple + (*filename*, *line number*, *function name*, *text*) representing the + information that is usually printed for a stack trace. The *text* is a + string with leading and trailing whitespace stripped; if the source is + not available it is ``None``. .. function:: extract_stack(f=None, limit=None) @@ -136,6 +150,162 @@ The module defines the following functions: .. versionadded:: 3.4 +.. function:: walk_stack(f) + + Walk a stack following ``f.f_back`` from the given frame, yielding the frame + and line number for each frame. If *f* is ``None``, the current stack is + used. This helper is used with :meth:`StackSummary.extract`. + + .. versionadded:: 3.5 + +.. function:: walk_tb(tb) + + Walk a traceback following ``tb_next`` yielding the frame and line number + for each frame. This helper is used with :meth:`StackSummary.extract`. + + .. versionadded:: 3.5 + +The module also defines the following classes: + +:class:`TracebackException` Objects +----------------------------------- + +.. versionadded:: 3.5 + +:class:`TracebackException` objects are created from actual exceptions to +capture data for later printing in a lightweight fashion. + +.. class:: TracebackException(exc_type, exc_value, exc_traceback, *, limit=None, lookup_lines=True, capture_locals=False) + + Capture an exception for later rendering. *limit*, *lookup_lines* and + *capture_locals* are as for the :class:`StackSummary` class. + + Note that when locals are captured, they are also shown in the traceback. + + .. attribute:: __cause__ + + A :class:`TracebackException` of the original ``__cause__``. + + .. attribute:: __context__ + + A :class:`TracebackException` of the original ``__context__``. + + .. attribute:: __suppress_context__ + + The ``__suppress_context__`` value from the original exception. + + .. attribute:: stack + + A :class:`StackSummary` representing the traceback. + + .. attribute:: exc_type + + The class of the original traceback. + + .. attribute:: filename + + For syntax errors - the file name where the error occurred. + + .. attribute:: lineno + + For syntax errors - the line number where the error occurred. + + .. attribute:: text + + For syntax errors - the text where the error occurred. + + .. attribute:: offset + + For syntax errors - the offset into the text where the error occurred. + + .. attribute:: msg + + For syntax errors - the compiler error message. + + .. classmethod:: from_exception(exc, *, limit=None, lookup_lines=True, capture_locals=False) + + Capture an exception for later rendering. *limit*, *lookup_lines* and + *capture_locals* are as for the :class:`StackSummary` class. + + Note that when locals are captured, they are also shown in the traceback. + + .. method:: format(*, chain=True) + + Format the exception. + + If *chain* is not ``True``, ``__cause__`` and ``__context__`` will not + be formatted. + + The return value is a generator of strings, each ending in a newline and + some containing internal newlines. :func:`~traceback.print_exception` + is a wrapper around this method which just prints the lines to a file. + + The message indicating which exception occurred is always the last + string in the output. + + .. method:: format_exception_only() + + Format the exception part of the traceback. + + The return value is a generator of strings, each ending in a newline. + + Normally, the generator emits a single string; however, for + :exc:`SyntaxError` exceptions, it emits several lines that (when + printed) display detailed information about where the syntax + error occurred. + + The message indicating which exception occurred is always the last + string in the output. + + +:class:`StackSummary` Objects +----------------------------- + +.. versionadded:: 3.5 + +:class:`StackSummary` objects represent a call stack ready for formatting. + +.. class:: StackSummary + + .. classmethod:: extract(frame_gen, *, limit=None, lookup_lines=True, capture_locals=False) + + Construct a :class:`StackSummary` object from a frame generator (such as + is returned by :func:`~traceback.walk_stack` or + :func:`~traceback.walk_tb`). + + If *limit* is supplied, only this many frames are taken from *frame_gen*. + If *lookup_lines* is ``False``, the returned :class:`FrameSummary` + objects will not have read their lines in yet, making the cost of + creating the :class:`StackSummary` cheaper (which may be valuable if it + may not actually get formatted). If *capture_locals* is ``True`` the + local variables in each :class:`FrameSummary` are captured as object + representations. + + .. classmethod:: from_list(a_list) + + Construct a :class:`StackSummary` object from a supplied old-style list + of tuples. Each tuple should be a 4-tuple with filename, lineno, name, + line as the elements. + + +:class:`FrameSummary` Objects +----------------------------- + +.. versionadded:: 3.5 + +:class:`FrameSummary` objects represent a single frame in a traceback. + +.. class:: FrameSummary(filename, lineno, name, lookup_line=True, locals=None, line=None) + + Represent a single frame in the traceback or stack that is being formatted + or printed. It may optionally have a stringified version of the frames + locals included in it. If *lookup_line* is ``False``, the source code is not + looked up until the :class:`FrameSummary` has the :attr:`~FrameSummary.line` + attribute accessed (which also happens when casting it to a tuple). + :attr:`~FrameSummary.line` may be directly provided, and will prevent line + lookups happening at all. *locals* is an optional local variable + dictionary, and if supplied the variable representations are stored in the + summary for later display. .. _traceback-example: diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst index a04a432..13c81a7 100644 --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -363,7 +363,7 @@ Filter Filter on traces of memory blocks. See the :func:`fnmatch.fnmatch` function for the syntax of - *filename_pattern*. The ``'.pyc'`` and ``'.pyo'`` file extensions are + *filename_pattern*. The ``'.pyc'`` file extension is replaced with ``'.py'``. Examples: @@ -374,6 +374,10 @@ Filter :mod:`tracemalloc` module * ``Filter(False, "<unknown>")`` excludes empty tracebacks + + .. versionchanged:: 3.5 + The ``'.pyo'`` file extension is no longer replaced with ``'.py'``. + .. attribute:: inclusive If *inclusive* is ``True`` (include), only trace memory blocks allocated @@ -631,4 +635,3 @@ Traceback obj = Object() File "test.py", line 12 tb = tracemalloc.get_object_traceback(f()) - diff --git a/Doc/library/tulip_coro.dia b/Doc/library/tulip_coro.dia Binary files differindex eccf4fa..70a33e3 100644 --- a/Doc/library/tulip_coro.dia +++ b/Doc/library/tulip_coro.dia diff --git a/Doc/library/tulip_coro.png b/Doc/library/tulip_coro.png Binary files differindex 65b6951..36ced8d 100644 --- a/Doc/library/tulip_coro.png +++ b/Doc/library/tulip_coro.png diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index efe5c54..30dd6ef 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -1879,7 +1879,7 @@ Settings and special methods >>> cv = screen.getcanvas() >>> cv - <turtle.ScrolledCanvas object at ...> + <turtle.ScrolledCanvas object ...> .. function:: getshapes() @@ -2351,6 +2351,9 @@ The demo scripts are: | | pairwise in opposite | shapesize, tilt, | | | direction | get_shapepoly, update | +----------------+------------------------------+-----------------------+ +| sorting_animate| visual demonstration of | simple alignment, | +| | different sorting methods | randomization | ++----------------+------------------------------+-----------------------+ | tree | a (graphical) breadth | :func:`clone` | | | first tree (using generators)| | +----------------+------------------------------+-----------------------+ diff --git a/Doc/library/types.rst b/Doc/library/types.rst index abdb939..ff75de1 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -90,6 +90,14 @@ Standard names are defined for the following types: generator function. +.. data:: CoroutineType + + The type of :term:`coroutine` objects, produced by calling a + function defined with an :keyword:`async def` statement. + + .. versionadded:: 3.5 + + .. data:: CodeType .. index:: builtin: compile @@ -115,6 +123,10 @@ Standard names are defined for the following types: The type of :term:`modules <module>`. Constructor takes the name of the module to be created and optionally its :term:`docstring`. + .. note:: + Use :func:`importlib.util.module_from_spec` to create a new module if you + wish to set the various import-controlled attributes. + .. attribute:: __doc__ The :term:`docstring` of the module. Defaults to ``None``. @@ -267,3 +279,25 @@ Additional Utility Classes and Functions attributes on the class with the same name (see Enum for an example). .. versionadded:: 3.4 + + +Coroutine Utility Functions +--------------------------- + +.. function:: coroutine(gen_func) + + This function transforms a :term:`generator` function into a + :term:`coroutine function` which returns a generator-based coroutine. + The generator-based coroutine is still a :term:`generator iterator`, + but is also considered to be a :term:`coroutine` object and is + :term:`awaitable`. However, it may not necessarily implement + the :meth:`__await__` method. + + If *gen_func* is a generator function, it will be modified in-place. + + If *gen_func* is not a generator function, it will be wrapped. If it + returns an instance of :class:`collections.abc.Generator`, the instance + will be wrapped in an *awaitable* proxy object. All other types + of objects will be returned as is. + + .. versionadded:: 3.5 diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst new file mode 100644 index 0000000..c94bea1 --- /dev/null +++ b/Doc/library/typing.rst @@ -0,0 +1,15 @@ +:mod:`typing` --- Support for type hints +======================================== + +.. module:: typing + :synopsis: Support for type hints (see PEP 484). + +**Source code:** :source:`Lib/typing.py` + +-------------- + +This module supports type hints as specified by :pep:`484`. The most +fundamental support consists of the type :class:`Any`, :class:`Union`, +:class:`Tuple`, :class:`Callable`, :class:`TypeVar`, and +:class:`Generic`. For full specification please see :pep:`484`. For +a simplified introduction to type hints see :pep:`483`. diff --git a/Doc/library/unicodedata.rst b/Doc/library/unicodedata.rst index 3b3d3a0..1430d9b 100644 --- a/Doc/library/unicodedata.rst +++ b/Doc/library/unicodedata.rst @@ -15,8 +15,8 @@ This module provides access to the Unicode Character Database (UCD) which defines character properties for all Unicode characters. The data contained in -this database is compiled from the `UCD version 6.3.0 -<http://www.unicode.org/Public/6.3.0/ucd>`_. +this database is compiled from the `UCD version 8.0.0 +<http://www.unicode.org/Public/8.0.0/ucd>`_. The module uses the same names and symbols as defined by Unicode Standard Annex #44, `"Unicode Character Database" @@ -166,6 +166,6 @@ Examples: .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NameAliases.txt +.. [#] http://www.unicode.org/Public/8.0.0/ucd/NameAliases.txt -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NamedSequences.txt +.. [#] http://www.unicode.org/Public/8.0.0/ucd/NamedSequences.txt diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index 91d5a27..f76c301 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -181,71 +181,77 @@ The Mock Class -------------- -`Mock` is a flexible mock object intended to replace the use of stubs and +:class:`Mock` is a flexible mock object intended to replace the use of stubs and test doubles throughout your code. Mocks are callable and create attributes as new mocks when you access them [#]_. Accessing the same attribute will always return the same mock. Mocks record how you use them, allowing you to make assertions about what your code has done to them. -:class:`MagicMock` is a subclass of `Mock` with all the magic methods +:class:`MagicMock` is a subclass of :class:`Mock` with all the magic methods pre-created and ready to use. There are also non-callable variants, useful when you are mocking out objects that aren't callable: :class:`NonCallableMock` and :class:`NonCallableMagicMock` The :func:`patch` decorators makes it easy to temporarily replace classes -in a particular module with a `Mock` object. By default `patch` will create -a `MagicMock` for you. You can specify an alternative class of `Mock` using -the `new_callable` argument to `patch`. +in a particular module with a :class:`Mock` object. By default :func:`patch` will create +a :class:`MagicMock` for you. You can specify an alternative class of :class:`Mock` using +the *new_callable* argument to :func:`patch`. -.. class:: Mock(spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, **kwargs) +.. class:: Mock(spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, unsafe=False, **kwargs) - Create a new `Mock` object. `Mock` takes several optional arguments + Create a new :class:`Mock` object. :class:`Mock` takes several optional arguments that specify the behaviour of the Mock object: - * `spec`: This can be either a list of strings or an existing object (a + * *spec*: This can be either a list of strings or an existing object (a class or instance) that acts as the specification for the mock object. If you pass in an object then a list of strings is formed by calling dir on the object (excluding unsupported magic attributes and methods). - Accessing any attribute not in this list will raise an `AttributeError`. + Accessing any attribute not in this list will raise an :exc:`AttributeError`. - If `spec` is an object (rather than a list of strings) then + If *spec* is an object (rather than a list of strings) then :attr:`~instance.__class__` returns the class of the spec object. This - allows mocks to pass `isinstance` tests. + allows mocks to pass :func:`isinstance` tests. - * `spec_set`: A stricter variant of `spec`. If used, attempting to *set* + * *spec_set*: A stricter variant of *spec*. If used, attempting to *set* or get an attribute on the mock that isn't on the object passed as - `spec_set` will raise an `AttributeError`. + *spec_set* will raise an :exc:`AttributeError`. - * `side_effect`: A function to be called whenever the Mock is called. See + * *side_effect*: A function to be called whenever the Mock is called. See the :attr:`~Mock.side_effect` attribute. Useful for raising exceptions or dynamically changing return values. The function is called with the same arguments as the mock, and unless it returns :data:`DEFAULT`, the return value of this function is used as the return value. - Alternatively `side_effect` can be an exception class or instance. In + Alternatively *side_effect* can be an exception class or instance. In this case the exception will be raised when the mock is called. - If `side_effect` is an iterable then each call to the mock will return + If *side_effect* is an iterable then each call to the mock will return the next value from the iterable. - A `side_effect` can be cleared by setting it to `None`. + A *side_effect* can be cleared by setting it to ``None``. - * `return_value`: The value returned when the mock is called. By default + * *return_value*: The value returned when the mock is called. By default this is a new Mock (created on first access). See the :attr:`return_value` attribute. - * `wraps`: Item for the mock object to wrap. If `wraps` is not None then + * *unsafe*: By default if any attribute starts with *assert* or + *assret* will raise an :exc:`AttributeError`. Passing ``unsafe=True`` + will allow access to these attributes. + + .. versionadded:: 3.5 + + * *wraps*: Item for the mock object to wrap. If *wraps* is not None then calling the Mock will pass the call through to the wrapped object (returning the real result). Attribute access on the mock will return a Mock object that wraps the corresponding attribute of the wrapped object (so attempting to access an attribute that doesn't exist will - raise an `AttributeError`). + raise an :exc:`AttributeError`). - If the mock has an explicit `return_value` set then calls are not passed - to the wrapped object and the `return_value` is returned instead. + If the mock has an explicit *return_value* set then calls are not passed + to the wrapped object and the *return_value* is returned instead. - * `name`: If the mock has a name then it will be used in the repr of the + * *name*: If the mock has a name then it will be used in the repr of the mock. This can be useful for debugging. The name is propagated to child mocks. @@ -315,6 +321,20 @@ the `new_callable` argument to `patch`. >>> calls = [call(4), call(2), call(3)] >>> mock.assert_has_calls(calls, any_order=True) + .. method:: assert_not_called(*args, **kwargs) + + Assert the mock was never called. + + >>> m = Mock() + >>> m.hello.assert_not_called() + >>> obj = m.hello() + >>> m.hello.assert_not_called() + Traceback (most recent call last): + ... + AssertionError: Expected 'hello' to not have been called. Called 1 times. + + .. versionadded:: 3.5 + .. method:: reset_mock() @@ -1032,25 +1052,31 @@ patch default because it can be dangerous. With it switched on you can write passing tests against APIs that don't actually exist! - Patch can be used as a `TestCase` class decorator. It works by + .. note:: + + .. versionchanged:: 3.5 + If you are patching builtins in a module then you don't + need to pass ``create=True``, it will be added by default. + + Patch can be used as a :class:`TestCase` class decorator. It works by decorating each test method in the class. This reduces the boilerplate - code when your test methods share a common patchings set. `patch` finds - tests by looking for method names that start with `patch.TEST_PREFIX`. - By default this is `test`, which matches the way `unittest` finds tests. - You can specify an alternative prefix by setting `patch.TEST_PREFIX`. + code when your test methods share a common patchings set. :func:`patch` finds + tests by looking for method names that start with ``patch.TEST_PREFIX``. + By default this is ``'test'``, which matches the way :mod:`unittest` finds tests. + You can specify an alternative prefix by setting ``patch.TEST_PREFIX``. Patch can be used as a context manager, with the with statement. Here the patching applies to the indented block after the with statement. If you use "as" then the patched object will be bound to the name after the - "as"; very useful if `patch` is creating a mock object for you. + "as"; very useful if :func:`patch` is creating a mock object for you. - `patch` takes arbitrary keyword arguments. These will be passed to - the `Mock` (or `new_callable`) on construction. + :func:`patch` takes arbitrary keyword arguments. These will be passed to + the :class:`Mock` (or *new_callable*) on construction. - `patch.dict(...)`, `patch.multiple(...)` and `patch.object(...)` are + ``patch.dict(...)``, ``patch.multiple(...)`` and ``patch.object(...)`` are available for alternate use-cases. -`patch` as function decorator, creating the mock for you and passing it into +:func:`patch` as function decorator, creating the mock for you and passing it into the decorated function: >>> @patch('__main__.SomeClass') @@ -1392,15 +1418,31 @@ method of a :class:`TestCase`: ... assert package.module.Class is self.MockClass ... - As an added bonus you no longer need to keep a reference to the `patcher` + As an added bonus you no longer need to keep a reference to the ``patcher`` object. It is also possible to stop all patches which have been started by using -`patch.stopall`. +:func:`patch.stopall`. .. function:: patch.stopall - Stop all active patches. Only stops patches started with `start`. + Stop all active patches. Only stops patches started with ``start``. + + +.. _patch-builtins: + +patch builtins +~~~~~~~~~~~~~~ +You can patch any builtins within a module. The following example patches +builtin :func:`ord`: + + >>> @patch('__main__.ord') + ... def test(mock_ord): + ... mock_ord.return_value = 101 + ... print(ord('c')) + ... + >>> test() + 101 TEST_PREFIX @@ -1587,7 +1629,7 @@ The full list of supported magic methods is: * Context manager: ``__enter__`` and ``__exit__`` * Unary numeric methods: ``__neg__``, ``__pos__`` and ``__invert__`` * The numeric methods (including right hand and in-place variants): - ``__add__``, ``__sub__``, ``__mul__``, ``__div__``,``__truediv__``, + ``__add__``, ``__sub__``, ``__mul__``, ``__matmul__``, ``__div__``, ``__truediv__``, ``__floordiv__``, ``__mod__``, ``__divmod__``, ``__lshift__``, ``__rshift__``, ``__and__``, ``__xor__``, ``__or__``, and ``__pow__`` * Numeric conversion methods: ``__complex__``, ``__int__``, ``__float__`` @@ -2013,7 +2055,7 @@ Mocking context managers with a :class:`MagicMock` is common enough and fiddly enough that a helper function is useful. >>> m = mock_open() - >>> with patch('__main__.open', m, create=True): + >>> with patch('__main__.open', m): ... with open('foo', 'w') as h: ... h.write('some stuff') ... @@ -2028,7 +2070,7 @@ enough that a helper function is useful. And for reading files: - >>> with patch('__main__.open', mock_open(read_data='bibble'), create=True) as m: + >>> with patch('__main__.open', mock_open(read_data='bibble')) as m: ... with open('foo') as h: ... result = h.read() ... diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index bdb18bc..70f99a0 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -214,9 +214,16 @@ Command-line options Stop the test run on the first error or failure. +.. cmdoption:: --locals + + Show local variables in tracebacks. + .. versionadded:: 3.2 The command-line options ``-b``, ``-c`` and ``-f`` were added. +.. versionadded:: 3.5 + The command-line option ``--locals``. + The command line can also be used for test discovery, for running all of the tests in a project or just a subset. @@ -1543,6 +1550,20 @@ Loading and running tests :data:`unittest.defaultTestLoader`. Using a subclass or instance, however, allows customization of some configurable properties. + :class:`TestLoader` objects have the following attributes: + + + .. attribute:: errors + + A list of the non-fatal errors encountered while loading tests. Not reset + by the loader at any point. Fatal errors are signalled by the relevant + a method raising an exception to the caller. Non-fatal errors are also + indicated by a synthetic test that will raise the original error when + run. + + .. versionadded:: 3.5 + + :class:`TestLoader` objects have the following methods: @@ -1552,7 +1573,7 @@ Loading and running tests :class:`testCaseClass`. - .. method:: loadTestsFromModule(module) + .. method:: loadTestsFromModule(module, pattern=None) Return a suite of all tests cases contained in the given module. This method searches *module* for classes derived from :class:`TestCase` and @@ -1569,11 +1590,18 @@ Loading and running tests If a module provides a ``load_tests`` function it will be called to load the tests. This allows modules to customize test loading. - This is the `load_tests protocol`_. + This is the `load_tests protocol`_. The *pattern* argument is passed as + the third argument to ``load_tests``. .. versionchanged:: 3.2 Support for ``load_tests`` added. + .. versionchanged:: 3.5 + The undocumented and unofficial *use_load_tests* default argument is + deprecated and ignored, although it is still accepted for backward + compatibility. The method also now accepts a keyword-only argument + *pattern* which is passed to ``load_tests`` as the third argument. + .. method:: loadTestsFromName(name, module=None) @@ -1599,6 +1627,12 @@ Loading and running tests The method optionally resolves *name* relative to the given *module*. + .. versionchanged:: 3.5 + If an :exc:`ImportError` or :exc:`AttributeError` occurs while traversing + *name* then a synthetic test that raises that error when run will be + returned. These errors are included in the errors accumulated by + self.errors. + .. method:: loadTestsFromNames(names, module=None) @@ -1625,18 +1659,22 @@ Loading and running tests the start directory is not the top level directory then the top level directory must be specified separately. - If importing a module fails, for example due to a syntax error, then this - will be recorded as a single error and discovery will continue. If the - import failure is due to :exc:`SkipTest` being raised, it will be recorded - as a skip instead of an error. + If importing a module fails, for example due to a syntax error, then + this will be recorded as a single error and discovery will continue. If + the import failure is due to :exc:`SkipTest` being raised, it will be + recorded as a skip instead of an error. - If a test package name (directory with :file:`__init__.py`) matches the - pattern then the package will be checked for a ``load_tests`` - function. If this exists then it will be called with *loader*, *tests*, - *pattern*. + If a package (a directory containing a file named :file:`__init__.py`) is + found, the package will be checked for a ``load_tests`` function. If this + exists then it will be called + ``package.load_tests(loader, tests, pattern)``. Test discovery takes care + to ensure that a package is only checked for tests once during an + invocation, even if the load_tests function itself calls + ``loader.discover``. - If ``load_tests`` exists then discovery does *not* recurse into the package, - ``load_tests`` is responsible for loading all tests in the package. + If ``load_tests`` exists then discovery does *not* recurse into the + package, ``load_tests`` is responsible for loading all tests in the + package. The pattern is deliberately not stored as a loader attribute so that packages can continue discovery themselves. *top_level_dir* is stored so @@ -1655,6 +1693,11 @@ Loading and running tests the same even if the underlying file system's ordering is not dependent on file name. + .. versionchanged:: 3.5 + Found packages are now checked for ``load_tests`` regardless of + whether their path matches *pattern*, because it is impossible for + a package name to match the default pattern. + The following attributes of a :class:`TestLoader` can be configured either by subclassing or assignment on an instance: @@ -1737,12 +1780,10 @@ Loading and running tests Set to ``True`` when the execution of tests should stop by :meth:`stop`. - .. attribute:: testsRun The total number of tests run so far. - .. attribute:: buffer If set to true, ``sys.stdout`` and ``sys.stderr`` will be buffered in between @@ -1752,7 +1793,6 @@ Loading and running tests .. versionadded:: 3.2 - .. attribute:: failfast If set to true :meth:`stop` will be called on the first failure or error, @@ -1760,6 +1800,11 @@ Loading and running tests .. versionadded:: 3.2 + .. attribute:: tb_locals + + If set to true then local variables will be shown in tracebacks. + + .. versionadded:: 3.5 .. method:: wasSuccessful() @@ -1770,7 +1815,6 @@ Loading and running tests Returns ``False`` if there were any :attr:`unexpectedSuccesses` from tests marked with the :func:`expectedFailure` decorator. - .. method:: stop() This method can be called to signal that the set of tests being run should @@ -1902,12 +1946,14 @@ Loading and running tests .. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, failfast=False, \ - buffer=False, resultclass=None, warnings=None) + buffer=False, resultclass=None, warnings=None, *, tb_locals=False) A basic test runner implementation that outputs results to a stream. If *stream* is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class has a few configurable parameters, but is essentially very simple. Graphical - applications which run test suites should provide alternate implementations. + applications which run test suites should provide alternate implementations. Such + implementations should accept ``**kwargs`` as the interface to construct runners + changes when features are added to unittest. By default this runner shows :exc:`DeprecationWarning`, :exc:`PendingDeprecationWarning`, :exc:`ResourceWarning` and @@ -1926,6 +1972,9 @@ Loading and running tests The default stream is set to :data:`sys.stderr` at instantiation time rather than import time. + .. versionchanged:: 3.5 + Added the tb_locals parameter. + .. method:: _makeResult() This method returns the instance of ``TestResult`` used by :meth:`run`. @@ -2023,7 +2072,10 @@ test runs or test discovery by implementing a function called ``load_tests``. If a test module defines ``load_tests`` it will be called by :meth:`TestLoader.loadTestsFromModule` with the following arguments:: - load_tests(loader, standard_tests, None) + load_tests(loader, standard_tests, pattern) + +where *pattern* is passed straight through from ``loadTestsFromModule``. It +defaults to ``None``. It should return a :class:`TestSuite`. @@ -2045,21 +2097,12 @@ A typical ``load_tests`` function that loads tests from a specific set of suite.addTests(tests) return suite -If discovery is started, either from the command line or by calling -:meth:`TestLoader.discover`, with a pattern that matches a package -name then the package :file:`__init__.py` will be checked for ``load_tests``. - -.. note:: - - The default pattern is ``'test*.py'``. This matches all Python files - that start with ``'test'`` but *won't* match any test directories. - - A pattern like ``'test*'`` will match test packages as well as - modules. - -If the package :file:`__init__.py` defines ``load_tests`` then it will be -called and discovery not continued into the package. ``load_tests`` -is called with the following arguments:: +If discovery is started in a directory containing a package, either from the +command line or by calling :meth:`TestLoader.discover`, then the package +:file:`__init__.py` will be checked for ``load_tests``. If that function does +not exist, discovery will recurse into the package as though it were just +another directory. Otherwise, discovery of the package's tests will be left up +to ``load_tests`` which is called with the following arguments:: load_tests(loader, standard_tests, pattern) @@ -2078,6 +2121,11 @@ continue (and potentially modify) test discovery. A 'do nothing' standard_tests.addTests(package_tests) return standard_tests +.. versionchanged:: 3.5 + Discovery no longer checks package names for matching *pattern* due to the + impossibility of package names matching the default pattern. + + Class and Module Fixtures ------------------------- diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index fbbabca..40098d0 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -270,6 +270,11 @@ or on combining URL components into a URL string. :func:`urlunsplit`, removing possible *scheme* and *netloc* parts. + .. versionchanged:: 3.5 + + Behaviour updated to match the semantics defined in :rfc:`3986`. + + .. function:: urldefrag(url) If *url* contains a fragment identifier, return a modified version of *url* @@ -516,7 +521,8 @@ task isn't already covered by the URL parsing functions above. Example: ``unquote_to_bytes('a%26%EF')`` yields ``b'a&\xef'``. -.. function:: urlencode(query, doseq=False, safe='', encoding=None, errors=None) +.. function:: urlencode(query, doseq=False, safe='', encoding=None, \ + errors=None, quote_via=quote_plus) Convert a mapping object or a sequence of two-element tuples, which may contain :class:`str` or :class:`bytes` objects, to a "percent-encoded" @@ -525,8 +531,16 @@ task isn't already covered by the URL parsing functions above. properly encoded to bytes, otherwise it would result in a :exc:`TypeError`. The resulting string is a series of ``key=value`` pairs separated by ``'&'`` - characters, where both *key* and *value* are quoted using :func:`quote_plus` - above. When a sequence of two-element tuples is used as the *query* + characters, where both *key* and *value* are quoted using the *quote_via* + function. By default, :func:`quote_plus` is used to quote the values, which + means spaces are quoted as a ``'+'`` character and '/' characters are + encoded as ``%2F``, which follows the standard for GET requests + (``application/x-www-form-urlencoded``). An alternate function that can be + passed as *quote_via* is :func:`quote`, which will encode spaces as ``%20`` + and not encode '/' characters. For maximum control of what is quoted, use + ``quote`` and specify a value for *safe*. + + When a sequence of two-element tuples is used as the *query* argument, the first element of each tuple is a key and the second is a value. The value element in itself can be a sequence and in that case, if the optional parameter *doseq* is evaluates to *True*, individual @@ -535,7 +549,7 @@ task isn't already covered by the URL parsing functions above. string will match the order of parameter tuples in the sequence. The *safe*, *encoding*, and *errors* parameters are passed down to - :func:`quote_plus` (the *encoding* and *errors* parameters are only passed + *quote_via* (the *encoding* and *errors* parameters are only passed when a query element is a :class:`str`). To reverse this encoding process, :func:`parse_qs` and :func:`parse_qsl` are @@ -547,6 +561,9 @@ task isn't already covered by the URL parsing functions above. .. versionchanged:: 3.2 Query parameter supports bytes and string objects. + .. versionadded:: 3.5 + *quote_via* parameter. + .. seealso:: diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 75b95d9..349ba70 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -115,6 +115,7 @@ The :mod:`urllib.request` module defines the following functions: .. versionchanged:: 3.4.3 *context* was added. + .. function:: install_opener(opener) Install an :class:`OpenerDirector` instance as the default global opener. @@ -287,13 +288,37 @@ The following classes are provided: fits. +.. class:: HTTPPasswordMgrWithPriorAuth() + + A variant of :class:`HTTPPasswordMgrWithDefaultRealm` that also has a + database of ``uri -> is_authenticated`` mappings. Can be used by a + BasicAuth handler to determine when to send authentication credentials + immediately instead of waiting for a ``401`` response first. + + .. versionadded:: 3.5 + + .. class:: AbstractBasicAuthHandler(password_mgr=None) This is a mixin class that helps with HTTP authentication, both to the remote host and to a proxy. *password_mgr*, if given, should be something that is compatible with :class:`HTTPPasswordMgr`; refer to section :ref:`http-password-mgr` for information on the interface that must be - supported. + supported. If *passwd_mgr* also provides ``is_authenticated`` and + ``update_authenticated`` methods (see + :ref:`http-password-mgr-with-prior-auth`), then the handler will use the + ``is_authenticated`` result for a given URI to determine whether or not to + send authentication credentials with the request. If ``is_authenticated`` + returns ``True`` for the URI, credentials are sent. If ``is_authenticated`` + is ``False``, credentials are not sent, and then if a ``401`` response is + received the request is re-sent with the authentication credentials. If + authentication succeeds, ``update_authenticated`` is called to set + ``is_authenticated`` ``True`` for the URI, so that subsequent requests to + the URI or any of its super-URIs will automatically include the + authentication credentials. + + .. versionadded:: 3.5 + Added ``is_authenticated`` support. .. class:: HTTPBasicAuthHandler(password_mgr=None) @@ -845,6 +870,42 @@ These methods are available on :class:`HTTPPasswordMgr` and searched if the given *realm* has no matching user/password. +.. _http-password-mgr-with-prior-auth: + +HTTPPasswordMgrWithPriorAuth Objects +------------------------------------ + +This password manager extends :class:`HTTPPasswordMgrWithDefaultRealm` to support +tracking URIs for which authentication credentials should always be sent. + + +.. method:: HTTPPasswordMgrWithPriorAuth.add_password(realm, uri, user, \ + passwd, is_authenticated=False) + + *realm*, *uri*, *user*, *passwd* are as for + :meth:`HTTPPasswordMgr.add_password`. *is_authenticated* sets the initial + value of the ``is_authenticated`` flag for the given URI or list of URIs. + If *is_authenticated* is specified as ``True``, *realm* is ignored. + + +.. method:: HTTPPasswordMgr.find_user_password(realm, authuri) + + Same as for :class:`HTTPPasswordMgrWithDefaultRealm` objects + + +.. method:: HTTPPasswordMgrWithPriorAuth.update_authenticated(self, uri, \ + is_authenticated=False) + + Update the ``is_authenticated`` flag for the given *uri* or list + of URIs. + + +.. method:: HTTPPasswordMgrWithPriorAuth.is_authenticated(self, authuri) + + Returns the current state of the ``is_authenticated`` flag for + the given URI. + + .. _abstract-basic-auth-handler: AbstractBasicAuthHandler Objects diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index cc883b1..2e16077 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -258,7 +258,7 @@ These method have the same issues as the and :meth:`keyrefs` method of are called in reverse order of creation. A finalizer will never invoke its callback during the later part of - the interpreter shutdown when module globals are liable to have + the :term:`interpreter shutdown` when module globals are liable to have been replaced by :const:`None`. .. method:: __call__() @@ -527,8 +527,8 @@ follows:: Starting with Python 3.4, :meth:`__del__` methods no longer prevent reference cycles from being garbage collected, and module globals are -no longer forced to :const:`None` during interpreter shutdown. So this -code should work without any issues on CPython. +no longer forced to :const:`None` during :term:`interpreter shutdown`. +So this code should work without any issues on CPython. However, handling of :meth:`__del__` methods is notoriously implementation specific, since it depends on internal details of the interpreter's garbage diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index a9e19da..de74c30 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -184,10 +184,11 @@ This module provides a single class, :class:`Headers`, for convenient manipulation of WSGI response headers using a mapping-like interface. -.. class:: Headers(headers) +.. class:: Headers([headers]) Create a mapping-like object wrapping *headers*, which must be a list of header - name/value tuples as described in :pep:`3333`. + name/value tuples as described in :pep:`3333`. The default value of *headers* is + an empty list. :class:`Headers` objects support typical mapping operations including :meth:`__getitem__`, :meth:`get`, :meth:`__setitem__`, :meth:`setdefault`, @@ -251,6 +252,10 @@ manipulation of WSGI response headers using a mapping-like interface. Content-Disposition: attachment; filename="bud.gif" + .. versionchanged:: 3.5 + *headers* parameter is optional. + + :mod:`wsgiref.simple_server` -- a simple WSGI HTTP server --------------------------------------------------------- diff --git a/Doc/library/xml.sax.reader.rst b/Doc/library/xml.sax.reader.rst index 3ab6063..31ca260 100644 --- a/Doc/library/xml.sax.reader.rst +++ b/Doc/library/xml.sax.reader.rst @@ -100,8 +100,10 @@ The :class:`XMLReader` interface supports the following methods: system identifier (a string identifying the input source -- typically a file name or an URL), a file-like object, or an :class:`InputSource` object. When :meth:`parse` returns, the input is completely processed, and the parser object - can be discarded or reset. As a limitation, the current implementation only - accepts byte streams; processing of character streams is for further study. + can be discarded or reset. + + .. versionchanged:: 3.5 + Added support of character streams. .. method:: XMLReader.getContentHandler() @@ -288,8 +290,7 @@ InputSource Objects .. method:: InputSource.setByteStream(bytefile) - Set the byte stream (a Python file-like object which does not perform - byte-to-character conversion) for this input source. + Set the byte stream (a :term:`binary file`) for this input source. The SAX parser will ignore this if there is also a character stream specified, but it will use a byte stream in preference to opening a URI connection itself. @@ -308,8 +309,7 @@ InputSource Objects .. method:: InputSource.setCharacterStream(charfile) - Set the character stream for this input source. (The stream must be a Python 1.6 - Unicode-wrapped file-like that performs conversion to strings.) + Set the character stream (a :term:`text file`) for this input source. If there is a character stream specified, the SAX parser will ignore any byte stream and will not attempt to open a URI connection to the system identifier. diff --git a/Doc/library/xml.sax.rst b/Doc/library/xml.sax.rst index e95d6b0..55f9799 100644 --- a/Doc/library/xml.sax.rst +++ b/Doc/library/xml.sax.rst @@ -47,7 +47,11 @@ The convenience functions are: .. function:: parseString(string, handler, error_handler=handler.ErrorHandler()) Similar to :func:`parse`, but parses from a buffer *string* received as a - parameter. + parameter. *string* must be a :class:`str` instance or a + :term:`bytes-like object`. + + .. versionchanged:: 3.5 + Added support of :class:`str` instances. A typical SAX application uses three kinds of objects: readers, handlers and input sources. "Reader" in this context is another term for parser, i.e. some diff --git a/Doc/library/xmlrpc.client.rst b/Doc/library/xmlrpc.client.rst index cc5e83a..e199931 100644 --- a/Doc/library/xmlrpc.client.rst +++ b/Doc/library/xmlrpc.client.rst @@ -27,7 +27,7 @@ between conformable Python objects and XML on the wire. constructed data. If you need to parse untrusted or unauthenticated data see :ref:`xml-vulnerabilities`. -.. versionchanged:: 3.4.3 +.. versionchanged:: 3.5 For https URIs, :mod:`xmlrpc.client` now performs all the necessary certificate and hostname checks by default @@ -129,7 +129,7 @@ between conformable Python objects and XML on the wire. :class:`Server` is retained as an alias for :class:`ServerProxy` for backwards compatibility. New code should use :class:`ServerProxy`. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 Added the *context* argument. @@ -200,6 +200,11 @@ grouped under the reserved :attr:`system` attribute: no such string is available, an empty string is returned. The documentation string may contain HTML markup. +.. versionchanged:: 3.5 + + Instances of :class:`ServerProxy` support the :term:`context manager` protocol + for closing the underlying transport. + A working example follows. The server code:: @@ -217,9 +222,9 @@ The client code for the preceding server:: import xmlrpc.client - proxy = xmlrpc.client.ServerProxy("http://localhost:8000/") - print("3 is even: %s" % str(proxy.is_even(3))) - print("100 is even: %s" % str(proxy.is_even(100))) + with xmlrpc.client.ServerProxy("http://localhost:8000/") as proxy: + print("3 is even: %s" % str(proxy.is_even(3))) + print("100 is even: %s" % str(proxy.is_even(100))) .. _datetime-objects: @@ -527,14 +532,14 @@ Example of Client Usage from xmlrpc.client import ServerProxy, Error # server = ServerProxy("http://localhost:8000") # local server - server = ServerProxy("http://betty.userland.com") + with ServerProxy("http://betty.userland.com") as proxy: - print(server) + print(proxy) - try: - print(server.examples.getStateName(41)) - except Error as v: - print("ERROR", v) + try: + print(proxy.examples.getStateName(41)) + except Error as v: + print("ERROR", v) To access an XML-RPC server through a proxy, you need to define a custom transport. The following example shows how: diff --git a/Doc/library/zipapp.rst b/Doc/library/zipapp.rst new file mode 100644 index 0000000..b4484c8 --- /dev/null +++ b/Doc/library/zipapp.rst @@ -0,0 +1,259 @@ +:mod:`zipapp` --- Manage executable python zip archives +======================================================= + +.. module:: zipapp + :synopsis: Manage executable python zip archives + + +.. index:: + single: Executable Zip Files + +.. versionadded:: 3.5 + +**Source code:** :source:`Lib/zipapp.py` + +-------------- + +This module provides tools to manage the creation of zip files containing +Python code, which can be :ref:`executed directly by the Python interpreter +<using-on-interface-options>`. The module provides both a +:ref:`zipapp-command-line-interface` and a :ref:`zipapp-python-api`. + + +Basic Example +------------- + +The following example shows how the :ref:`command-line-interface` +can be used to create an executable archive from a directory containing +Python code. When run, the archive will execute the ``main`` function from +the module ``myapp`` in the archive. + +.. code-block:: sh + + $ python -m zipapp myapp -m "myapp:main" + $ python myapp.pyz + <output from myapp> + + +.. _zipapp-command-line-interface: + +Command-Line Interface +---------------------- + +When called as a program from the command line, the following form is used: + +.. code-block:: sh + + $ python -m zipapp source [options] + +If *source* is a directory, this will create an archive from the contents of +*source*. If *source* is a file, it should be an archive, and it will be +copied to the target archive (or the contents of its shebang line will be +displayed if the --info option is specified). + +The following options are understood: + +.. program:: zipapp + +.. cmdoption:: -o <output>, --output=<output> + + Write the output to a file named *output*. If this option is not specified, + the output filename will be the same as the input *source*, with the + extension ``.pyz`` added. If an explicit filename is given, it is used as + is (so a ``.pyz`` extension should be included if required). + + An output filename must be specified if the *source* is an archive (and in + that case, *output* must not be the same as *source*). + +.. cmdoption:: -p <interpreter>, --python=<interpreter> + + Add a ``#!`` line to the archive specifying *interpreter* as the command + to run. Also, on POSIX, make the archive executable. The default is to + write no ``#!`` line, and not make the file executable. + +.. cmdoption:: -m <mainfn>, --main=<mainfn> + + Write a ``__main__.py`` file to the archive that executes *mainfn*. The + *mainfn* argument should have the form "pkg.mod:fn", where "pkg.mod" is a + package/module in the archive, and "fn" is a callable in the given module. + The ``__main__.py`` file will execute that callable. + + :option:`--main` cannot be specified when copying an archive. + +.. cmdoption:: --info + + Display the interpreter embedded in the archive, for diagnostic purposes. In + this case, any other options are ignored and SOURCE must be an archive, not a + directory. + +.. cmdoption:: -h, --help + + Print a short usage message and exit. + + +.. _zipapp-python-api: + +Python API +---------- + +The module defines two convenience functions: + + +.. function:: create_archive(source, target=None, interpreter=None, main=None) + + Create an application archive from *source*. The source can be any + of the following: + + * The name of a directory, or a :class:`pathlib.Path` object referring + to a directory, in which case a new application archive will be + created from the content of that directory. + * The name of an existing application archive file, or a :class:`pathlib.Path` + object referring to such a file, in which case the file is copied to + the target (modifying it to reflect the value given for the *interpreter* + argument). The file name should include the ``.pyz`` extension, if required. + * A file object open for reading in bytes mode. The content of the + file should be an application archive, and the file object is + assumed to be positioned at the start of the archive. + + The *target* argument determines where the resulting archive will be + written: + + * If it is the name of a file, or a :class:`pathlb.Path` object, + the archive will be written to that file. + * If it is an open file object, the archive will be written to that + file object, which must be open for writing in bytes mode. + * If the target is omitted (or None), the source must be a directory + and the target will be a file with the same name as the source, with + a ``.pyz`` extension added. + + The *interpreter* argument specifies the name of the Python + interpreter with which the archive will be executed. It is written as + a "shebang" line at the start of the archive. On POSIX, this will be + interpreted by the OS, and on Windows it will be handled by the Python + launcher. Omitting the *interpreter* results in no shebang line being + written. If an interpreter is specified, and the target is a + filename, the executable bit of the target file will be set. + + The *main* argument specifies the name of a callable which will be + used as the main program for the archive. It can only be specified if + the source is a directory, and the source does not already contain a + ``__main__.py`` file. The *main* argument should take the form + "pkg.module:callable" and the archive will be run by importing + "pkg.module" and executing the given callable with no arguments. It + is an error to omit *main* if the source is a directory and does not + contain a ``__main__.py`` file, as otherwise the resulting archive + would not be executable. + + If a file object is specified for *source* or *target*, it is the + caller's responsibility to close it after calling create_archive. + + When copying an existing archive, file objects supplied only need + ``read`` and ``readline``, or ``write`` methods. When creating an + archive from a directory, if the target is a file object it will be + passed to the ``zipfile.ZipFile`` class, and must supply the methods + needed by that class. + +.. function:: get_interpreter(archive) + + Return the interpreter specified in the ``#!`` line at the start of the + archive. If there is no ``#!`` line, return :const:`None`. + The *archive* argument can be a filename or a file-like object open + for reading in bytes mode. It is assumed to be at the start of the archive. + + +.. _zipapp-examples: + +Examples +-------- + +Pack up a directory into an archive, and run it. + +.. code-block:: sh + + $ python -m zipapp myapp + $ python myapp.pyz + <output from myapp> + +The same can be done using the :func:`create_archive` functon:: + + >>> import zipapp + >>> zipapp.create_archive('myapp.pyz', 'myapp') + +To make the application directly executable on POSIX, specify an interpreter +to use. + +.. code-block:: sh + + $ python -m zipapp myapp -p "/usr/bin/env python" + $ ./myapp.pyz + <output from myapp> + +To replace the shebang line on an existing archive, create a modified archive +using the :func:`create_archive` function:: + + >>> import zipapp + >>> zipapp.create_archive('old_archive.pyz', 'new_archive.pyz', '/usr/bin/python3') + +To update the file in place, do the replacement in memory using a :class:`BytesIO` +object, and then overwrite the source afterwards. Note that there is a risk +when overwriting a file in place that an error will result in the loss of +the original file. This code does not protect against such errors, but +production code should do so. Also, this method will only work if the archive +fits in memory:: + + >>> import zipapp + >>> import io + >>> temp = io.BytesIO() + >>> zipapp.create_archive('myapp.pyz', temp, '/usr/bin/python2') + >>> with open('myapp.pyz', 'wb') as f: + >>> f.write(temp.getvalue()) + +Note that if you specify an interpreter and then distribute your application +archive, you need to ensure that the interpreter used is portable. The Python +launcher for Windows supports most common forms of POSIX ``#!`` line, but there +are other issues to consider: + +* If you use "/usr/bin/env python" (or other forms of the "python" command, + such as "/usr/bin/python"), you need to consider that your users may have + either Python 2 or Python 3 as their default, and write your code to work + under both versions. +* If you use an explicit version, for example "/usr/bin/env python3" your + application will not work for users who do not have that version. (This + may be what you want if you have not made your code Python 2 compatible). +* There is no way to say "python X.Y or later", so be careful of using an + exact version like "/usr/bin/env python3.4" as you will need to change your + shebang line for users of Python 3.5, for example. + +The Python Zip Application Archive Format +----------------------------------------- + +Python has been able to execute zip files which contain a ``__main__.py`` file +since version 2.6. In order to be executed by Python, an application archive +simply has to be a standard zip file containing a ``__main__.py`` file which +will be run as the entry point for the application. As usual for any Python +script, the parent of the script (in this case the zip file) will be placed on +:data:`sys.path` and thus further modules can be imported from the zip file. + +The zip file format allows arbitrary data to be prepended to a zip file. The +zip application format uses this ability to prepend a standard POSIX "shebang" +line to the file (``#!/path/to/interpreter``). + +Formally, the Python zip application format is therefore: + +1. An optional shebang line, containing the characters ``b'#!'`` followed by an + interpreter name, and then a newline (``b'\n'``) character. The interpreter + name can be anything acceptable to the OS "shebang" processing, or the Python + launcher on Windows. The interpreter should be encoded in UTF-8 on Windows, + and in :func:`sys.getfilesystemencoding()` on POSIX. +2. Standard zipfile data, as generated by the :mod:`zipfile` module. The + zipfile content *must* include a file called ``__main__.py`` (which must be + in the "root" of the zipfile - i.e., it cannot be in a subdirectory). The + zipfile data can be compressed or uncompressed. + +If an application archive has a shebang line, it may have the executable bit set +on POSIX systems, to allow it to be executed directly. + +There is no requirement that the tools in this module are used to create +application archives - the module is a convenience, but archives in the above +format created by any means are acceptable to Python. + diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index 10a094f..d40315e 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -134,12 +134,16 @@ ZipFile Objects Open a ZIP file, where *file* can be either a path to a file (a string) or a file-like object. The *mode* parameter should be ``'r'`` to read an existing - file, ``'w'`` to truncate and write a new file, or ``'a'`` to append to an - existing file. If *mode* is ``'a'`` and *file* refers to an existing ZIP + file, ``'w'`` to truncate and write a new file, ``'x'`` to exclusive create + and write a new file, or ``'a'`` to append to an existing file. + If *mode* is ``'x'`` and *file* refers to an existing file, + a :exc:`FileExistsError` will be raised. + If *mode* is ``'a'`` and *file* refers to an existing ZIP file, then additional files are added to it. If *file* does not refer to a ZIP file, then a new ZIP archive is appended to the file. This is meant for adding a ZIP archive to another file (such as :file:`python.exe`). If *mode* is ``a`` and the file does not exist at all, it is created. + If *mode* is ``r`` or ``a``, the file should be seekable. *compression* is the ZIP compression method to use when writing the archive, and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized @@ -151,7 +155,7 @@ ZipFile Objects extensions when the zipfile is larger than 2 GiB. If it is false :mod:`zipfile` will raise an exception when the ZIP file would require ZIP64 extensions. - If the file is created with mode ``'a'`` or ``'w'`` and then + If the file is created with mode ``'w'``, ``'x'`` or ``'a'`` and then :meth:`closed <close>` without adding any files to the archive, the appropriate ZIP structures for an empty archive will be written to the file. @@ -171,6 +175,10 @@ ZipFile Objects .. versionchanged:: 3.4 ZIP64 extensions are enabled by default. + .. versionchanged:: 3.5 + Added support for writing to unseekable streams. + Added support for the ``'x'`` mode. + .. method:: ZipFile.close() @@ -226,14 +234,8 @@ ZipFile Objects .. note:: - If the ZipFile was created by passing in a file-like object as the first - argument to the constructor, then the object returned by :meth:`.open` shares the - ZipFile's file pointer. Under these circumstances, the object returned by - :meth:`.open` should not be used after any additional operations are performed - on the ZipFile object. If the ZipFile was created by passing in a string (the - filename) as the first argument to the constructor, then :meth:`.open` will - create a new file object that will be held by the ZipExtFile, allowing it to - operate independently of the ZipFile. + Objects returned by :meth:`.open` can operate independently of the + ZipFile. .. note:: @@ -318,7 +320,8 @@ ZipFile Objects *arcname* (by default, this will be the same as *filename*, but without a drive letter and with leading path separators removed). If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for - the new entry. The archive must be open with mode ``'w'`` or ``'a'`` -- calling + the new entry. + The archive must be open with mode ``'w'``, ``'x'`` or ``'a'`` -- calling :meth:`write` on a ZipFile created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling :meth:`write` on a closed ZipFile will raise a :exc:`RuntimeError`. @@ -340,16 +343,16 @@ ZipFile Objects If ``arcname`` (or ``filename``, if ``arcname`` is not given) contains a null byte, the name of the file in the archive will be truncated at the null byte. - .. method:: ZipFile.writestr(zinfo_or_arcname, bytes[, compress_type]) Write the string *bytes* to the archive; *zinfo_or_arcname* is either the file name it will be given in the archive, or a :class:`ZipInfo` instance. If it's an instance, at least the filename, date, and time must be given. If it's a - name, the date and time is set to the current date and time. The archive must be - opened with mode ``'w'`` or ``'a'`` -- calling :meth:`writestr` on a ZipFile - created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling - :meth:`writestr` on a closed ZipFile will raise a :exc:`RuntimeError`. + name, the date and time is set to the current date and time. + The archive must be opened with mode ``'w'``, ``'x'`` or ``'a'`` -- calling + :meth:`writestr` on a ZipFile created with mode ``'r'`` will raise a + :exc:`RuntimeError`. Calling :meth:`writestr` on a closed ZipFile will + raise a :exc:`RuntimeError`. If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for the new entry, or in the *zinfo_or_arcname* @@ -377,7 +380,8 @@ The following data attributes are also available: .. attribute:: ZipFile.comment The comment text associated with the ZIP file. If assigning a comment to a - :class:`ZipFile` instance created with mode 'a' or 'w', this should be a + :class:`ZipFile` instance created with mode ``'w'``, ``'x'`` or ``'a'``, + this should be a string no longer than 65535 bytes. Comments longer than this will be truncated in the written archive when :meth:`close` is called. @@ -407,8 +411,7 @@ The :class:`PyZipFile` constructor takes the same parameters as the archive. If the *optimize* parameter to :class:`PyZipFile` was not given or ``-1``, - the corresponding file is a :file:`\*.pyo` file if available, else a - :file:`\*.pyc` file, compiling if necessary. + the corresponding file is a :file:`\*.pyc` file, compiling if necessary. If the *optimize* parameter to :class:`PyZipFile` was ``0``, ``1`` or ``2``, only files with that optimization level (see :func:`compile`) are @@ -571,4 +574,3 @@ Instances have the following attributes: .. attribute:: ZipInfo.file_size Size of the uncompressed file. - diff --git a/Doc/library/zipimport.rst b/Doc/library/zipimport.rst index 2cf508b..8a5d5d1 100644 --- a/Doc/library/zipimport.rst +++ b/Doc/library/zipimport.rst @@ -20,10 +20,10 @@ subdirectory. For example, the path :file:`example.zip/lib/` would only import from the :file:`lib/` subdirectory within the archive. Any files may be present in the ZIP archive, but only files :file:`.py` and -:file:`.py[co]` are available for import. ZIP import of dynamic modules +:file:`.pyc` are available for import. ZIP import of dynamic modules (:file:`.pyd`, :file:`.so`) is disallowed. Note that if an archive only contains :file:`.py` files, Python will not attempt to modify the archive by adding the -corresponding :file:`.pyc` or :file:`.pyo` file, meaning that if a ZIP archive +corresponding :file:`.pyc` file, meaning that if a ZIP archive doesn't contain :file:`.pyc` files, importing may be rather slow. ZIP archives with an archive comment are currently not supported. @@ -161,4 +161,3 @@ Here is an example that imports a module from a ZIP archive - note that the >>> import jwzthreading >>> jwzthreading.__file__ 'example.zip/jwzthreading.py' - diff --git a/Doc/make.bat b/Doc/make.bat index 251f822..3ff91f2 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -8,9 +8,17 @@ set this=%~n0 if "%SPHINXBUILD%" EQU "" set SPHINXBUILD=sphinx-build
if "%PYTHON%" EQU "" set PYTHON=py
-if DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles(x86)%
-if NOT DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles%
-if "%HTMLHELP%" EQU "" set HTMLHELP=%_PRGMFLS%\HTML Help Workshop\hhc.exe
+if "%1" NEQ "htmlhelp" goto :skiphhcsearch
+if exist "%HTMLHELP%" goto :skiphhcsearch
+
+rem Search for HHC in likely places
+set HTMLHELP=
+where hhc /q && set HTMLHELP=hhc && goto :skiphhcsearch
+where /R ..\externals hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" where /R "%ProgramFiles(x86)%" hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" where /R "%ProgramFiles%" hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" echo Cannot find HHC on PATH or in externals & exit /B 1
+:skiphhcsearch
if "%DISTVERSION%" EQU "" for /f "usebackq" %%v in (`%PYTHON% tools/extensions/patchlevel.py`) do set DISTVERSION=%%v
@@ -36,7 +44,8 @@ if errorlevel 9009 ( echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
- goto end
+ popd
+ exit /B 1
)
rem Targets that do require sphinx-build and have their own label
diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 5dd17eb..76b3850 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -51,6 +51,9 @@ Summarizing: : | `with_stmt` : | `funcdef` : | `classdef` + : | `async_with_stmt` + : | `async_for_stmt` + : | `async_funcdef` suite: `stmt_list` NEWLINE | NEWLINE INDENT `statement`+ DEDENT statement: `stmt_list` NEWLINE | `compound_stmt` stmt_list: `simple_stmt` (";" `simple_stmt`)* [";"] @@ -660,6 +663,132 @@ can be used to create instance variables with different implementation details. :pep:`3129` - Class Decorators +Coroutines +========== + +.. versionadded:: 3.5 + +.. index:: statement: async def +.. _`async def`: + +Coroutine function definition +----------------------------- + +.. productionlist:: + async_funcdef: "async" `funcdef` + +.. index:: + keyword: async + keyword: await + +Execution of Python coroutines can be suspended and resumed at many points +(see :term:`coroutine`). In the body of a coroutine, any ``await`` and +``async`` identifiers become reserved keywords; :keyword:`await` expressions, +:keyword:`async for` and :keyword:`async with` can only be used in +coroutine bodies. However, to simplify the parser, these keywords cannot +be used on the same line as a function or coroutine (:keyword:`def` +statement) header. + +Functions defined with ``async def`` syntax are always coroutine functions, +even if they do not contain ``await`` or ``async`` keywords. + +It is a :exc:`SyntaxError` to use :keyword:`yield` expressions in +``async def`` coroutines. + +An example of a coroutine function:: + + async def func(param1, param2): + do_stuff() + await some_coroutine() + + +.. index:: statement: async for +.. _`async for`: + +The :keyword:`async for` statement +---------------------------------- + +.. productionlist:: + async_for_stmt: "async" `for_stmt` + +An :term:`asynchronous iterable` is able to call asynchronous code in its +*iter* implementation, and :term:`asynchronous iterator` can call asynchronous +code in its *next* method. + +The ``async for`` statement allows convenient iteration over asynchronous +iterators. + +The following code:: + + async for TARGET in ITER: + BLOCK + else: + BLOCK2 + +Is semantically equivalent to:: + + iter = (ITER) + iter = await type(iter).__aiter__(iter) + running = True + while running: + try: + TARGET = await type(iter).__anext__(iter) + except StopAsyncIteration: + running = False + else: + BLOCK + else: + BLOCK2 + +See also :meth:`__aiter__` and :meth:`__anext__` for details. + +It is a :exc:`SyntaxError` to use ``async for`` statement outside of an +:keyword:`async def` function. + + +.. index:: statement: async with +.. _`async with`: + +The :keyword:`async with` statement +----------------------------------- + +.. productionlist:: + async_with_stmt: "async" `with_stmt` + +An :term:`asynchronous context manager` is a :term:`context manager` that is +able to suspend execution in its *enter* and *exit* methods. + +The following code:: + + async with EXPR as VAR: + BLOCK + +Is semantically equivalent to:: + + mgr = (EXPR) + aexit = type(mgr).__aexit__ + aenter = type(mgr).__aenter__(mgr) + exc = True + + VAR = await aenter + try: + BLOCK + except: + if not await aexit(mgr, *sys.exc_info()): + raise + else: + await aexit(mgr, None, None, None) + +See also :meth:`__aenter__` and :meth:`__aexit__` for details. + +It is a :exc:`SyntaxError` to use ``async with`` statement outside of an +:keyword:`async def` function. + +.. seealso:: + + :pep:`492` - Coroutines with async and await syntax + + .. rubric:: Footnotes .. [#] The exception is propagated to the invocation stack unless diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index dda18ba..2ca1194 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -616,6 +616,16 @@ Callable types exception is raised and the iterator will have reached the end of the set of values to be returned. + Coroutine functions + .. index:: + single: coroutine; function + + A function or method which is defined using :keyword:`async def` is called + a :dfn:`coroutine function`. Such a function, when called, returns a + :term:`coroutine` object. It may contain :keyword:`await` expressions, + as well as :keyword:`async with` and :keyword:`async for` statements. See + also the :ref:`coroutine-objects` section. + Built-in functions .. index:: object: built-in function @@ -1986,6 +1996,7 @@ left undefined. .. method:: object.__add__(self, other) object.__sub__(self, other) object.__mul__(self, other) + object.__matmul__(self, other) object.__truediv__(self, other) object.__floordiv__(self, other) object.__mod__(self, other) @@ -2002,15 +2013,16 @@ left undefined. builtin: pow builtin: pow - These methods are called to implement the binary arithmetic operations (``+``, - ``-``, ``*``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, ``<<``, - ``>>``, ``&``, ``^``, ``|``). For instance, to evaluate the expression - ``x + y``, where *x* is an instance of a class that has an :meth:`__add__` - method, ``x.__add__(y)`` is called. The :meth:`__divmod__` method should be the - equivalent to using :meth:`__floordiv__` and :meth:`__mod__`; it should not be - related to :meth:`__truediv__`. Note that :meth:`__pow__` should be defined - to accept an optional third argument if the ternary version of the built-in - :func:`pow` function is to be supported. + These methods are called to implement the binary arithmetic operations + (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, + :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``). For instance, to + evaluate the expression ``x + y``, where *x* is an instance of a class that + has an :meth:`__add__` method, ``x.__add__(y)`` is called. The + :meth:`__divmod__` method should be the equivalent to using + :meth:`__floordiv__` and :meth:`__mod__`; it should not be related to + :meth:`__truediv__`. Note that :meth:`__pow__` should be defined to accept + an optional third argument if the ternary version of the built-in :func:`pow` + function is to be supported. If one of those methods does not support the operation with the supplied arguments, it should return ``NotImplemented``. @@ -2019,6 +2031,7 @@ left undefined. .. method:: object.__radd__(self, other) object.__rsub__(self, other) object.__rmul__(self, other) + object.__rmatmul__(self, other) object.__rtruediv__(self, other) object.__rfloordiv__(self, other) object.__rmod__(self, other) @@ -2034,14 +2047,14 @@ left undefined. builtin: divmod builtin: pow - These methods are called to implement the binary arithmetic operations (``+``, - ``-``, ``*``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, - ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected (swapped) operands. - These functions are only called if the left operand does not support the - corresponding operation and the operands are of different types. [#]_ For - instance, to evaluate the expression ``x - y``, where *y* is an instance of - a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` is called if - ``x.__sub__(y)`` returns *NotImplemented*. + These methods are called to implement the binary arithmetic operations + (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, + :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected + (swapped) operands. These functions are only called if the left operand does + not support the corresponding operation and the operands are of different + types. [#]_ For instance, to evaluate the expression ``x - y``, where *y* is + an instance of a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` + is called if ``x.__sub__(y)`` returns *NotImplemented*. .. index:: builtin: pow @@ -2059,6 +2072,7 @@ left undefined. .. method:: object.__iadd__(self, other) object.__isub__(self, other) object.__imul__(self, other) + object.__imatmul__(self, other) object.__itruediv__(self, other) object.__ifloordiv__(self, other) object.__imod__(self, other) @@ -2070,17 +2084,17 @@ left undefined. object.__ior__(self, other) These methods are called to implement the augmented arithmetic assignments - (``+=``, ``-=``, ``*=``, ``/=``, ``//=``, ``%=``, ``**=``, ``<<=``, ``>>=``, - ``&=``, ``^=``, ``|=``). These methods should attempt to do the operation - in-place (modifying *self*) and return the result (which could be, but does - not have to be, *self*). If a specific method is not defined, the augmented - assignment falls back to the normal methods. For instance, if *x* is an - instance of a class with an :meth:`__iadd__` method, ``x += y`` is equivalent - to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and ``y.__radd__(x)`` - are considered, as with the evaluation of ``x + y``. In certain situations, - augmented assignment can result in unexpected errors (see - :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in - fact part of the data model. + (``+=``, ``-=``, ``*=``, ``@=``, ``/=``, ``//=``, ``%=``, ``**=``, ``<<=``, + ``>>=``, ``&=``, ``^=``, ``|=``). These methods should attempt to do the + operation in-place (modifying *self*) and return the result (which could be, + but does not have to be, *self*). If a specific method is not defined, the + augmented assignment falls back to the normal methods. For instance, if *x* + is an instance of a class with an :meth:`__iadd__` method, ``x += y`` is + equivalent to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and + ``y.__radd__(x)`` are considered, as with the evaluation of ``x + y``. In + certain situations, augmented assignment can result in unexpected errors (see + :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in fact + part of the data model. .. method:: object.__neg__(self) @@ -2250,6 +2264,155 @@ special methods (the special method *must* be set on the class object itself in order to be consistently invoked by the interpreter). +.. index:: + single: coroutine + +Coroutines +========== + + +Awaitable Objects +----------------- + +An :term:`awaitable` object generally implements an :meth:`__await__` method. +:term:`Coroutine` objects returned from :keyword:`async def` functions +are awaitable. + +.. note:: + + The :term:`generator iterator` objects returned from generators + decorated with :func:`types.coroutine` or :func:`asyncio.coroutine` + are also awaitable, but they do not implement :meth:`__await__`. + +.. method:: object.__await__(self) + + Must return an :term:`iterator`. Should be used to implement + :term:`awaitable` objects. For instance, :class:`asyncio.Future` implements + this method to be compatible with the :keyword:`await` expression. + +.. versionadded:: 3.5 + +.. seealso:: :pep:`492` for additional information about awaitable objects. + + +.. _coroutine-objects: + +Coroutine Objects +----------------- + +:term:`Coroutine` objects are :term:`awaitable` objects. +A coroutine's execution can be controlled by calling :meth:`__await__` and +iterating over the result. When the coroutine has finished executing and +returns, the iterator raises :exc:`StopIteration`, and the exception's +:attr:`~StopIteration.value` attribute holds the return value. If the +coroutine raises an exception, it is propagated by the iterator. Coroutines +should not directly raise unhandled :exc:`StopIteration` exceptions. + +Coroutines also have the methods listed below, which are analogous to +those of generators (see :ref:`generator-methods`). However, unlike +generators, coroutines do not directly support iteration. + +.. method:: coroutine.send(value) + + Starts or resumes execution of the coroutine. If *value* is ``None``, + this is equivalent to advancing the iterator returned by + :meth:`__await__`. If *value* is not ``None``, this method delegates + to the :meth:`~generator.send` method of the iterator that caused + the coroutine to suspend. The result (return value, + :exc:`StopIteration`, or other exception) is the same as when + iterating over the :meth:`__await__` return value, described above. + +.. method:: coroutine.throw(type[, value[, traceback]]) + + Raises the specified exception in the coroutine. This method delegates + to the :meth:`~generator.throw` method of the iterator that caused + the coroutine to suspend, if it has such a method. Otherwise, + the exception is raised at the suspension point. The result + (return value, :exc:`StopIteration`, or other exception) is the same as + when iterating over the :meth:`__await__` return value, described + above. If the exception is not caught in the coroutine, it propagates + back to the caller. + +.. method:: coroutine.close() + + Causes the coroutine to clean itself up and exit. If the coroutine + is suspended, this method first delegates to the :meth:`~generator.close` + method of the iterator that caused the coroutine to suspend, if it + has such a method. Then it raises :exc:`GeneratorExit` at the + suspension point, causing the coroutine to immediately clean itself up. + Finally, the coroutine is marked as having finished executing, even if + it was never started. + + Coroutine objects are automatically closed using the above process when + they are about to be destroyed. + + +Asynchronous Iterators +---------------------- + +An *asynchronous iterable* is able to call asynchronous code in its +``__aiter__`` implementation, and an *asynchronous iterator* can call +asynchronous code in its ``__anext__`` method. + +Asynchronous iterators can be used in a :keyword:`async for` statement. + +.. method:: object.__aiter__(self) + + Must return an *awaitable* resulting in an *asynchronous iterator* object. + +.. method:: object.__anext__(self) + + Must return an *awaitable* resulting in a next value of the iterator. Should + raise a :exc:`StopAsyncIteration` error when the iteration is over. + +An example of an asynchronous iterable object:: + + class Reader: + async def readline(self): + ... + + async def __aiter__(self): + return self + + async def __anext__(self): + val = await self.readline() + if val == b'': + raise StopAsyncIteration + return val + +.. versionadded:: 3.5 + + +Asynchronous Context Managers +----------------------------- + +An *asynchronous context manager* is a *context manager* that is able to +suspend execution in its ``__aenter__`` and ``__aexit__`` methods. + +Asynchronous context managers can be used in a :keyword:`async with` statement. + +.. method:: object.__aenter__(self) + + This method is semantically similar to the :meth:`__enter__`, with only + difference that it must return an *awaitable*. + +.. method:: object.__aexit__(self, exc_type, exc_value, traceback) + + This method is semantically similar to the :meth:`__exit__`, with only + difference that it must return an *awaitable*. + +An example of an asynchronous context manager class:: + + class AsyncContextManager: + async def __aenter__(self): + await log('entering context') + + async def __aexit__(self, exc_type, exc, tb): + await log('exiting context') + +.. versionadded:: 3.5 + + .. rubric:: Footnotes .. [#] It *is* possible in some cases to change an object's type, under certain diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index dd3cb9e..d3face7 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -390,6 +390,7 @@ on the right hand side of an assignment statement. to sub-generators easy. .. index:: object: generator +.. _generator-methods: Generator-iterator methods ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -443,12 +444,12 @@ is already executing raises a :exc:`ValueError` exception. .. method:: generator.close() Raises a :exc:`GeneratorExit` at the point where the generator function was - paused. If the generator function then raises :exc:`StopIteration` (by - exiting normally, or due to already being closed) or :exc:`GeneratorExit` (by - not catching the exception), close returns to its caller. If the generator - yields a value, a :exc:`RuntimeError` is raised. If the generator raises any - other exception, it is propagated to the caller. :meth:`close` does nothing - if the generator has already exited due to an exception or normal exit. + paused. If the generator function then exits gracefully, is already closed, + or raises :exc:`GeneratorExit` (by not catching the exception), close + returns to its caller. If the generator yields a value, a + :exc:`RuntimeError` is raised. If the generator raises any other exception, + it is propagated to the caller. :meth:`close` does nothing if the generator + has already exited due to an exception or normal exit. .. index:: single: yield; examples @@ -811,6 +812,20 @@ a class instance: if that method was called. +.. _await: + +Await expression +================ + +Suspend the execution of :term:`coroutine` on an :term:`awaitable` object. +Can only be used inside a :term:`coroutine function`. + +.. productionlist:: + await: ["await"] `primary` + +.. versionadded:: 3.5 + + .. _power: The power operator @@ -820,7 +835,7 @@ The power operator binds more tightly than unary operators on its left; it binds less tightly than unary operators on its right. The syntax is: .. productionlist:: - power: `primary` ["**" `u_expr`] + power: `await` ["**" `u_expr`] Thus, in an unparenthesized sequence of power and unary operators, the operators are evaluated from right to left (this does not constrain the evaluation order @@ -891,8 +906,9 @@ from the power operator, there are only two levels, one for multiplicative operators and one for additive operators: .. productionlist:: - m_expr: `u_expr` | `m_expr` "*" `u_expr` | `m_expr` "//" `u_expr` | `m_expr` "/" `u_expr` - : | `m_expr` "%" `u_expr` + m_expr: `u_expr` | `m_expr` "*" `u_expr` | `m_expr` "@" `m_expr` | + : `m_expr` "//" `u_expr`| `m_expr` "/" `u_expr` | + : `m_expr` "%" `u_expr` a_expr: `m_expr` | `a_expr` "+" `m_expr` | `a_expr` "-" `m_expr` .. index:: single: multiplication @@ -903,6 +919,13 @@ the other must be a sequence. In the former case, the numbers are converted to a common type and then multiplied together. In the latter case, sequence repetition is performed; a negative repetition factor yields an empty sequence. +.. index:: single: matrix multiplication + +The ``@`` (at) operator is intended to be used for matrix multiplication. No +builtin Python types implement this operator. + +.. versionadded:: 3.5 + .. index:: exception: ZeroDivisionError single: division @@ -1346,13 +1369,16 @@ precedence and have a left-to-right chaining feature as described in the +-----------------------------------------------+-------------------------------------+ | ``+``, ``-`` | Addition and subtraction | +-----------------------------------------------+-------------------------------------+ -| ``*``, ``/``, ``//``, ``%`` | Multiplication, division, remainder | -| | [#]_ | +| ``*``, ``@``, ``/``, ``//``, ``%`` | Multiplication, matrix | +| | multiplication division, | +| | remainder [#]_ | +-----------------------------------------------+-------------------------------------+ | ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT | +-----------------------------------------------+-------------------------------------+ | ``**`` | Exponentiation [#]_ | +-----------------------------------------------+-------------------------------------+ +| ``await`` ``x`` | Await expression | ++-----------------------------------------------+-------------------------------------+ | ``x[index]``, ``x[index:index]``, | Subscription, slicing, | | ``x(arguments...)``, ``x.attribute`` | call, attribute reference | +-----------------------------------------------+-------------------------------------+ diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 7966bc5..50e4688 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -339,6 +339,7 @@ of what happens during the loading portion of import:: module = None if spec.loader is not None and hasattr(spec.loader, 'create_module'): + # It is assumed 'exec_module' will also be defined on the loader. module = spec.loader.create_module(spec) if module is None: module = ModuleType(spec.name) @@ -427,7 +428,7 @@ Module loaders may opt in to creating the module object during loading by implementing a :meth:`~importlib.abc.Loader.create_module` method. It takes one argument, the module spec, and returns the new module object to use during loading. ``create_module()`` does not need to set any attributes -on the module object. If the loader does not define ``create_module()``, the +on the module object. If the method returns ``None``, the import machinery will create the new module itself. .. versionadded:: 3.4 @@ -459,7 +460,13 @@ import machinery will create the new module itself. * If loading fails, the loader must remove any modules it has inserted into :data:`sys.modules`, but it must remove **only** the failing - module, and only if the loader itself has loaded it explicitly. + module(s), and only if the loader itself has loaded the module(s) + explicitly. + +.. versionchanged:: 3.5 + A :exc:`DeprecationWarning` is raised when ``exec_module()`` is defined but + ``create_module()`` is not. Starting in Python 3.6 it will be an error to not + define ``create_module()`` on a loader attached to a ModuleSpec. Submodules ---------- @@ -674,7 +681,7 @@ path entry finder that knows how to handle that particular kind of path. The default set of path entry finders implement all the semantics for finding modules on the file system, handling special file types such as Python source -code (``.py`` files), Python byte code (``.pyc`` and ``.pyo`` files) and +code (``.py`` files), Python byte code (``.pyc`` files) and shared libraries (e.g. ``.so`` files). When supported by the :mod:`zipimport` module in the standard library, the default path entry finders also handle loading all of these file types (other than shared libraries) from zipfiles. @@ -788,6 +795,15 @@ hook` callables on :data:`sys.path_hooks`, then the following protocol is used to ask the finder for a module spec, which is then used when loading the module. +The current working directory -- denoted by an empty string -- is handled +slightly differently from other entries on :data:`sys.path`. First, if the +current working directory is found to not exist, no value is stored in +:data:`sys.path_importer_cache`. Second, the value for the current working +directory is looked up fresh for each module lookup. Third, the path used for +:data:`sys.path_importer_cache` and returned by +:meth:`importlib.machinery.PathFinder.find_spec` will be the actual current +working directory and not the empty string. + Path entry finder protocol -------------------------- diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index 373be16..27d6a8a 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -311,7 +311,7 @@ The Unicode category codes mentioned above stand for: * *Nd* - decimal numbers * *Pc* - connector punctuations * *Other_ID_Start* - explicit list of characters in `PropList.txt - <http://www.unicode.org/Public/6.3.0/ucd/PropList.txt>`_ to support backwards + <http://www.unicode.org/Public/8.0.0/ucd/PropList.txt>`_ to support backwards compatibility * *Other_ID_Continue* - likewise @@ -690,7 +690,7 @@ Operators The following tokens are operators:: - + - * ** / // % + + - * ** / // % @ << >> & | ^ ~ < > <= >= == != @@ -706,7 +706,7 @@ The following tokens serve as delimiters in the grammar:: ( ) [ ] { } , : . ; @ = -> - += -= *= /= //= %= + += -= *= /= //= %= @= &= |= ^= >>= <<= **= The period can also occur in floating-point and imaginary literals. A sequence @@ -727,4 +727,4 @@ occurrence outside string literals and comments is an unconditional error:: .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NameAliases.txt +.. [#] http://www.unicode.org/Public/8.0.0/ucd/NameAliases.txt diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 8946b4f..5f60540 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -281,7 +281,7 @@ operation and an assignment statement: .. productionlist:: augmented_assignment_stmt: `augtarget` `augop` (`expression_list` | `yield_expression`) augtarget: `identifier` | `attributeref` | `subscription` | `slicing` - augop: "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**=" + augop: "+=" | "-=" | "*=" | "@=" | "/=" | "//=" | "%=" | "**=" : | ">>=" | "<<=" | "&=" | "^=" | "|=" (See section :ref:`primaries` for the syntax definitions of the last three diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 64a5665..d44b052 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -34,7 +34,7 @@ import suspicious ISSUE_URI = 'https://bugs.python.org/issue%s' -SOURCE_URI = 'https://hg.python.org/cpython/file/3.4/%s' +SOURCE_URI = 'https://hg.python.org/cpython/file/3.5/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists from docutils.parsers.rst.states import Body diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index 48653af8..8432f78 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -122,6 +122,8 @@ library/ipaddress,,:db8,>>> ipaddress.IPv6Address('2001:db8::1000') library/ipaddress,,::,>>> ipaddress.IPv6Address('2001:db8::1000') library/ipaddress,,:db8,IPv6Address('2001:db8::1000') library/ipaddress,,::,IPv6Address('2001:db8::1000') +library/ipaddress,,:db8,">>> ipaddress.ip_address(""2001:db8::1"").reverse_pointer" +library/ipaddress,,::,">>> ipaddress.ip_address(""2001:db8::1"").reverse_pointer" library/ipaddress,,::,"""::abc:7:def""" library/ipaddress,,:def,"""::abc:7:def""" library/ipaddress,,::,::FFFF/96 @@ -136,12 +138,9 @@ library/ipaddress,,:db00,2001:db00::0/ffff:ff00:: library/ipaddress,,::,2001:db00::0/ffff:ff00:: library/itertools,,:step,elements from seq[start:stop:step] library/itertools,,:stop,elements from seq[start:stop:step] -library/linecache,,:sys,"sys:x:3:3:sys:/dev:/bin/sh" library/logging.handlers,,:port,host:port library/mmap,,:i2,obj[i1:i2] library/multiprocessing,,`,# Add more tasks using `put()` -library/multiprocessing,,`,">>> l._callmethod('__getitem__', (20,)) # equiv to `l[20]`" -library/multiprocessing,,`,">>> l._callmethod('__getslice__', (2, 7)) # equiv to `l[2:7]`" library/multiprocessing,,:queue,">>> QueueManager.register('get_queue', callable=lambda:queue)" library/multiprocessing,,`,# register the Foo class; make `f()` and `g()` accessible via proxy library/multiprocessing,,`,# register the Foo class; make `g()` and `_h()` accessible via proxy @@ -179,8 +178,6 @@ library/ssl,,:MyState,State or Province Name (full name) [Some-State]:MyState library/ssl,,:ops,Email Address []:ops@myserver.mygroup.myorganization.com library/ssl,,:Some,"Locality Name (eg, city) []:Some City" library/ssl,,:US,Country Name (2 letter code) [AU]:US -library/stdtypes,,::,>>> a[::-1].tolist() -library/stdtypes,,::,>>> a[::2].tolist() library/stdtypes,,:end,s[start:end] library/stdtypes,,::,>>> hash(v[::-2]) == hash(b'abcefg'[::-2]) library/stdtypes,,:len,s[len(s):len(s)] @@ -256,6 +253,11 @@ whatsnew/2.4,,:System, whatsnew/2.5,,:memory,:memory: whatsnew/2.5,,:step,[start:stop:step] whatsnew/2.5,,:stop,[start:stop:step] +whatsnew/2.7,,::,"ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]'," +whatsnew/2.7,,::,>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') +whatsnew/2.7,,:Sunday,'2009:4:Sunday' +whatsnew/2.7,,:Cookie,"export PYTHONWARNINGS=all,error:::Cookie:0" +whatsnew/2.7,,::,"export PYTHONWARNINGS=all,error:::Cookie:0" whatsnew/3.2,,:affe,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," whatsnew/3.2,,:affe,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') whatsnew/3.2,,:beef,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," @@ -271,14 +273,20 @@ whatsnew/3.2,,:feed,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe: whatsnew/3.2,,:gz,">>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:" whatsnew/3.2,,:location,zope9-location = ${zope9:location} whatsnew/3.2,,:prefix,zope-conf = ${custom:prefix}/etc/zope.conf -whatsnew/changelog,,:platform,:platform: whatsnew/changelog,,:gz,": TarFile opened with external fileobj and ""w:gz"" mode didn't" -whatsnew/changelog,,:PythonCmd,"With Tk < 8.5 _tkinter.c:PythonCmd() raised UnicodeDecodeError, caused" -whatsnew/changelog,,::,": Fix FTP tests for IPv6, bind to ""::1"" instead of ""localhost""." whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" -whatsnew/changelog,,:password,user:password -whatsnew/2.7,780,:Sunday,'2009:4:Sunday' -whatsnew/2.7,907,::,"export PYTHONWARNINGS=all,error:::Cookie:0" -whatsnew/2.7,907,:Cookie,"export PYTHONWARNINGS=all,error:::Cookie:0" -whatsnew/2.7,1657,::,>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') -whatsnew/2.7,1657,::,"ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]'," +library/tarfile,149,:xz,'x:xz' +library/xml.etree.elementtree,290,:sometag,prefix:sometag +library/xml.etree.elementtree,301,:fictional,"<actors xmlns:fictional=""http://characters.example.com""" +library/xml.etree.elementtree,301,:character,<fictional:character>Lancelot</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Archie Leach</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Sir Robin</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Gunther</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Commander Clement</fictional:character> +library/xml.etree.elementtree,332,:actor,"for actor in root.findall('real_person:actor', ns):" +library/xml.etree.elementtree,332,:name,"name = actor.find('real_person:name', ns)" +library/xml.etree.elementtree,332,:character,"for char in actor.findall('role:character', ns):" +library/zipapp,31,:main,"$ python -m zipapp myapp -m ""myapp:main""" +library/zipapp,82,:fn,"argument should have the form ""pkg.mod:fn"", where ""pkg.mod"" is a" +library/zipapp,155,:callable,"""pkg.module:callable"" and the archive will be run by importing" +library/stdtypes,3767,::,>>> m[::2].tolist() diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html index abdf070..78e9c4f 100644 --- a/Doc/tools/templates/indexsidebar.html +++ b/Doc/tools/templates/indexsidebar.html @@ -3,8 +3,7 @@ <h3>Docs for other versions</h3> <ul> <li><a href="https://docs.python.org/2.7/">Python 2.7 (stable)</a></li> - <li><a href="https://docs.python.org/3.3/">Python 3.3 (stable)</a></li> - <li><a href="https://docs.python.org/3.5/">Python 3.5 (in development)</a></li> + <li><a href="https://docs.python.org/3.4/">Python 3.4 (stable)</a></li> <li><a href="https://www.python.org/doc/versions/">Old versions</a></li> </ul> diff --git a/Doc/tutorial/appendix.rst b/Doc/tutorial/appendix.rst index 8670efc..67262a1 100644 --- a/Doc/tutorial/appendix.rst +++ b/Doc/tutorial/appendix.rst @@ -40,7 +40,7 @@ Executable Python Scripts On BSD'ish Unix systems, Python scripts can be made directly executable, like shell scripts, by putting the line :: - #!/usr/bin/env python3.4 + #!/usr/bin/env python3.5 (assuming that the interpreter is on the user's :envvar:`PATH`) at the beginning of the script and giving the file an executable mode. The ``#!`` must be the @@ -107,7 +107,7 @@ of your user site-packages directory. Start Python and run this code:: >>> import site >>> site.getusersitepackages() - '/home/user/.local/lib/python3.4/site-packages' + '/home/user/.local/lib/python3.5/site-packages' Now you can create a file named :file:`usercustomize.py` in that directory and put anything you want in it. It will affect every invocation of Python, unless diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst index 1ea299f..a2031ed 100644 --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -73,10 +73,11 @@ objects: Return the number of times *x* appears in the list. -.. method:: list.sort() +.. method:: list.sort(key=None, reverse=False) :noindex: - Sort the items of the list in place. + Sort the items of the list in place (the arguments can be used for sort + customization, see :func:`sorted` for their explanation). .. method:: list.reverse() diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst index 8051634..d5789a6 100644 --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -10,13 +10,13 @@ Using the Python Interpreter Invoking the Interpreter ======================== -The Python interpreter is usually installed as :file:`/usr/local/bin/python3.4` +The Python interpreter is usually installed as :file:`/usr/local/bin/python3.5` on those machines where it is available; putting :file:`/usr/local/bin` in your Unix shell's search path makes it possible to start it by typing the command: .. code-block:: text - python3.4 + python3.5 to the shell. [#]_ Since the choice of the directory where the interpreter lives is an installation option, other places are possible; check with your local @@ -24,11 +24,11 @@ Python guru or system administrator. (E.g., :file:`/usr/local/python` is a popular alternative location.) On Windows machines, the Python installation is usually placed in -:file:`C:\\Python34`, though you can change this when you're running the +:file:`C:\\Python35`, though you can change this when you're running the installer. To add this directory to your path, you can type the following command into the command prompt in a DOS box:: - set path=%path%;C:\python34 + set path=%path%;C:\python35 Typing an end-of-file character (:kbd:`Control-D` on Unix, :kbd:`Control-Z` on Windows) at the primary prompt causes the interpreter to exit with a zero exit @@ -96,8 +96,8 @@ with the *secondary prompt*, by default three dots (``...``). The interpreter prints a welcome message stating its version number and a copyright notice before printing the first prompt:: - $ python3.4 - Python 3.4 (default, Mar 16 2014, 09:25:04) + $ python3.5 + Python 3.5 (default, Sep 16 2015, 09:25:04) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index fd361ae..9ae64b0 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -216,15 +216,15 @@ Some tips for experts: statements, the ``-OO`` switch removes both assert statements and __doc__ strings. Since some programs may rely on having these available, you should only use this option if you know what you're doing. "Optimized" modules have - a .pyo rather than a .pyc suffix and are usually smaller. Future releases may + an ``opt-`` tag and are usually smaller. Future releases may change the effects of optimization. -* A program doesn't run any faster when it is read from a ``.pyc`` or ``.pyo`` +* A program doesn't run any faster when it is read from a ``.pyc`` file than when it is read from a ``.py`` file; the only thing that's faster - about ``.pyc`` or ``.pyo`` files is the speed with which they are loaded. + about ``.pyc`` files is the speed with which they are loaded. -* The module :mod:`compileall` can create .pyc files (or .pyo files when - :option:`-O` is used) for all modules in a directory. +* The module :mod:`compileall` can create .pyc files for all modules in a + directory. * There is more detail on this process, including a flow chart of the decisions, in PEP 3147. @@ -548,4 +548,3 @@ modules found in a package. .. [#] In fact function definitions are also 'statements' that are 'executed'; the execution of a module-level function definition enters the function name in the module's global symbol table. - diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index 72d51de..0954eba 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -15,7 +15,7 @@ operating system:: >>> import os >>> os.getcwd() # Return the current working directory - 'C:\\Python34' + 'C:\\Python35' >>> os.chdir('/server/accesslogs') # Change current working directory >>> os.system('mkdir today') # Run the command mkdir in the system shell 0 diff --git a/Doc/tutorial/stdlib2.rst b/Doc/tutorial/stdlib2.rst index c0197ea..f7d2a0a 100644 --- a/Doc/tutorial/stdlib2.rst +++ b/Doc/tutorial/stdlib2.rst @@ -18,7 +18,7 @@ abbreviated displays of large or deeply nested containers:: >>> import reprlib >>> reprlib.repr(set('supercalifragilisticexpialidocious')) - "set(['a', 'c', 'd', 'e', 'f', 'g', ...])" + "{'a', 'c', 'd', 'e', 'f', 'g', ...}" The :mod:`pprint` module offers more sophisticated control over printing both built-in and user defined objects in a way that is readable by the interpreter. @@ -277,7 +277,7 @@ applications include caching objects that are expensive to create:: Traceback (most recent call last): File "<stdin>", line 1, in <module> d['primary'] # entry was automatically removed - File "C:/python34/lib/weakref.py", line 46, in __getitem__ + File "C:/python35/lib/weakref.py", line 46, in __getitem__ o = self.data[key]() KeyError: 'primary' diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 4017ce8..701c624 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -190,13 +190,16 @@ Miscellaneous options .. cmdoption:: -b - Issue a warning when comparing str and bytes. Issue an error when the + Issue a warning when comparing :class:`bytes` or :class:`bytearray` with + :class:`str` or :class:`bytes` with :class:`int`. Issue an error when the option is given twice (:option:`-bb`). + .. versionchanged: 3.5 + Affects comparisons of :class:`bytes` with :class:`int`. .. cmdoption:: -B - If given, Python won't try to write ``.pyc`` or ``.pyo`` files on the + If given, Python won't try to write ``.pyc`` files on the import of source modules. See also :envvar:`PYTHONDONTWRITEBYTECODE`. diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc index 45bdd5a..02bcbee 100644 --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -21,11 +21,11 @@ subdirectory (on Windows, this is ``Lib\site-packages``). On Windows, you may have to invoke the ``pyvenv`` script as follows, if you don't have the relevant PATH and PATHEXT settings:: - c:\Temp>c:\Python34\python c:\Python34\Tools\Scripts\pyvenv.py myenv + c:\Temp>c:\Python35\python c:\Python35\Tools\Scripts\pyvenv.py myenv or equivalently:: - c:\Temp>c:\Python34\python -m venv myenv + c:\Temp>c:\Python35\python -m venv myenv The command, if run with ``-h``, will show the available options:: diff --git a/Doc/using/win_installer.png b/Doc/using/win_installer.png Binary files differnew file mode 100644 index 0000000..8e25892 --- /dev/null +++ b/Doc/using/win_installer.png diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index c05f72a..8e3c110 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -7,22 +7,186 @@ ************************* .. sectionauthor:: Robert Lehmann <lehmannro@gmail.com> +.. sectionauthor:: Steve Dower <steve.dower@microsoft.com> This document aims to give an overview of Windows-specific behaviour you should know about when using Python on Microsoft Windows. -.. XXX (ncoghlan) - - This looks rather stale to me... - - Installing Python ================= -Unlike most Unix systems and services, Windows does not require Python natively -and thus does not pre-install a version of Python. However, the CPython team +Unlike most Unix systems and services, Windows does not include a system +supported installation of Python. To make Python available, the CPython team has compiled Windows installers (MSI packages) with every `release -<https://www.python.org/download/releases/>`_ for many years. +<https://www.python.org/download/releases/>`_ for many years. These installers +are primarily intended to add a per-user installation of Python, with the +core interpreter and library being used by a single user. The installer is also +able to install for all users of a single machine, and a separate ZIP file is +available for application-local distributions. + +Installation Steps +------------------ + +Four Python 3.5 installers are available for download - two each for the 32-bit +and 64-bit versions of the interpreter. The *web installer* is a small initial +download, and it will automatically download the required components as +necessary. The *offline installer* includes the components necessary for a +default installation and only requires an internet connection for optional +features. See :ref:`install-layout-option` for other ways to avoid downloading +during installation. + +After starting the installer, one of two options may be selected: + +.. image:: win_installer.png + +If you select "Install Now": + +* You will *not* need to be an administrator (unless a system update for the + C Runtime Library is required) +* Python will be installed into your user directory +* The :ref:`launcher` will *also* be installed into your user directory +* The standard library, test suite, launcher and pip will be installed +* If selected, the install directory will be added to your :envvar:`PATH` +* Shortcuts will only be visible for the current user + +Selecting "Customize installation" will allow you to select the features to +install, the installation location and other options or post-install actions. +To install debugging symbols or binaries, you will need to use this option. + +To perform an all-users installation, you should select "Customize +installation". In this case: + +* You may be required to provide administrative credentials or approval +* Python will be installed into the Program Files directory +* The :ref:`launcher` will be installed into the Windows directory +* Optional features may be selected during installation +* The standard library will be pre-compiled to bytecode +* If selected, the install directory will be added to the system :envvar:`PATH` +* Shortcuts are available for all users + +.. _install-quiet-option: + +Installing Without UI +--------------------- + +All of the options available in the installer UI can also be specified from the +command line, allowing scripted installers to replicate an installation on many +machines without user interaction. These options may also be set without +suppressing the UI in order to change some of the defaults. + +To completely hide the installer UI and install Python silently, pass the +``/quiet`` (``/q``) option. To skip past the user interaction but still display +progress and errors, pass the ``/passive`` (``/p``) option. The ``/uninstall`` +option may be passed to immediately begin removing Python - no prompt will be +displayed. + +All other options are passed as ``name=value``, where the value is usually +``0`` to disable a feature, ``1`` to enable a feature, or a path. The full list +of available options is shown below. + ++---------------------------+--------------------------------------+--------------------------+ +| Name | Description | Default | ++===========================+======================================+==========================+ +| InstallAllUsers | Perform a system-wide installation. | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| TargetDir | The installation directory | Selected based on | +| | | InstallAllUsers | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultAllUsersTargetDir | The default installation directory | :file:`%ProgramFiles%\\\ | +| | for all-user installs | Python X.Y` or :file:`\ | +| | | %ProgramFiles(x86)%\\\ | +| | | Python X.Y` | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultJustForMeTargetDir | The default install directory for | :file:`%LocalAppData%\\\ | +| | just-for-me installs | Programs\\PythonXY` or | +| | | :file:`%LocalAppData%\\\ | +| | | Programs\\PythonXY-32` | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultCustomTargetDir | The default custom install directory | (empty) | +| | displayed in the UI | | ++---------------------------+--------------------------------------+--------------------------+ +| AssociateFiles | Create file associations if the | 1 | +| | launcher is also installed. | | ++---------------------------+--------------------------------------+--------------------------+ +| CompileAll | Compile all ``.py`` files to | 0 | +| | ``.pyc``. | | ++---------------------------+--------------------------------------+--------------------------+ +| PrependPath | Add install and Scripts directories | 0 | +| | tho :envvar:`PATH` and ``.PY`` to | | +| | :envvar:`PATHEXT` | | ++---------------------------+--------------------------------------+--------------------------+ +| Shortcuts | Create shortcuts for the interpreter,| 1 | +| | documentation and IDLE if installed. | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_doc | Install Python manual | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_debug | Install debug binaries | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_dev | Install developer headers and | 1 | +| | libraries | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_exe | Install :file:`python.exe` and | 1 | +| | related files | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_launcher | Install :ref:`launcher`. | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_lib | Install standard library and | 1 | +| | extension modules | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_pip | Install bundled pip and setuptools | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_symbols | Install debugging symbols (`*`.pdb) | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_tcltk | Install Tcl/Tk support and IDLE | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_test | Install standard library test suite | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_tools | Install utility scripts | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| SimpleInstall | Disable most install UI | 0 | ++---------------------------+--------------------------------------+--------------------------+ + +For example, to silently install a default, system-wide Python installation, +you could use the following command (from an elevated command prompt):: + + python-3.5.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0 + +To allow users to easily install a personal copy of Python without the test +suite, you could provide a shortcut with the following command:: + + python-3.5.0.exe /passive InstallAllUsers=0 Include_launcher=0 Include_test=0 SimpleInstall=1 + +(Note that omitting the launcher also omits file associations, and is only +recommended for per-user installs when there is also a system-wide installation +that included the launcher.) + +.. _install-layout-option: + +Installing Without Downloading +------------------------------ + +As some features of Python are not included in the initial installer download, +selecting those features may require an internet connection. To avoid this +need, all possible components may be downloaded on-demand to create a complete +*layout* that will no longer require an internet connection regardless of the +selected features. Note that this download may be bigger than required, but +where a large number of installations are going to be performed it is very +useful to have a locally cached copy. + +Execute the following command from Command Prompt to download all possible +required files. Remember to substitute ``python-3.5.0.exe`` for the actual +name of your installer, and to create layouts in their own directories to +avoid collisions between files with the same name. + +:: + + python-3.5.0.exe /layout [optional target directory] + +You may also specify the ``/quiet`` option to hide the progress display. + + +Other Platforms +--------------- With ongoing development of Python, some platforms that used to be supported earlier are no longer supported (due to the lack of users or developers). @@ -66,19 +230,31 @@ key features: `ActivePython <http://www.activestate.com/activepython/>`_ Installer with multi-platform compatibility, documentation, PyWin32 -`Enthought Python Distribution <https://www.enthought.com/products/epd/>`_ - Popular modules (such as PyWin32) with their respective documentation, tool - suite for building extensible Python applications +`Anaconda <http://www.continuum.io/downloads/>`_ + Popular scientific modules (such as numpy, scipy and pandas) and the + ``conda`` package manager. + +`Canopy <https://www.enthought.com/products/canopy/>`_ + A "comprehensive Python analysis environment" with editors and other + development tools. -Notice that these packages are likely to install *older* versions of Python. +`WinPython <https://winpython.github.io/>`_ + Windows-specific distribution with prebuilt scientific packages and + tools for building packages. + +Note that these packages may not include the latest versions of Python or +other libraries, and are not maintained or supported by the core Python team. Configuring Python ================== -In order to run Python flawlessly, you might have to change certain environment -settings in Windows. +To run Python conveniently from a command prompt, you might consider changing +some default environment variables in Windows. While the installer provides an +option to configure the PATH and PATHEXT variables for you, this is only +reliable for a single, system-wide installation. If you regularly use multiple +versions of Python, consider using the :ref:`launcher`. .. _setting-envvars: @@ -86,163 +262,86 @@ settings in Windows. Excursus: Setting environment variables --------------------------------------- -Windows has a built-in dialog for changing environment variables (following -guide applies to XP classical view): Right-click the icon for your machine -(usually located on your Desktop and called "My Computer") and choose -:menuselection:`Properties` there. Then, open the :guilabel:`Advanced` tab -and click the :guilabel:`Environment Variables` button. +Windows allows environment variables to be configured permanently at both the +User level and the System level, or temporarily in a command prompt. + +To temporarily set environment variables, open Command Prompt and use the +:command:`set` command:: + + C:\>set PATH=C:\Program Files\Python 3.5;%PATH% + C:\>set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib + C:\>python -In short, your path is: +These changes will apply to any further commands executed in that console, and +will be inherited by any applications started from the console. - :menuselection:`My Computer - --> Properties - --> Advanced - --> Environment Variables` +Including the variable name within percent signs will expand to the existing +value, allowing you to add your new value at either the start or the end. +Modifying :envvar:`PATH` by adding the directory containing +:program:`python.exe` to the start is a common way to ensure the correct version +of Python is launched. +To permanently modify the default environment variables, click Start and search +for 'edit environment variables', or open System properties, :guilabel:`Advanced +system settings` and click the :guilabel:`Environment Variables` button. In this dialog, you can add or modify User and System variables. To change System variables, you need non-restricted access to your machine (i.e. Administrator rights). -Another way of adding variables to your environment is using the :command:`set` -command:: - - set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib - -To make this setting permanent, you could add the corresponding command line to -your :file:`autoexec.bat`. :program:`msconfig` is a graphical interface to this -file. +.. note:: -Viewing environment variables can also be done more straight-forward: The -command prompt will expand strings wrapped into percent signs automatically:: + Windows will concatenate User variables *after* System variables, which may + cause unexpected results when modifying :envvar:`PATH`. - echo %PATH% - -Consult :command:`set /?` for details on this behaviour. + The :envvar:`PYTHONPATH` variable is used by all versions of Python 2 and + Python 3, so you should not permanently configure this variable unless it + only includes code that is compatible with all of your installed Python + versions. .. seealso:: - http://support.microsoft.com/kb/100843 + http://support.microsoft.com/kb/100843 Environment variables in Windows NT - http://support.microsoft.com/kb/310519 + http://technet.microsoft.com/en-us/library/cc754250.aspx + The SET command, for temporarily modifying environment variables + + http://technet.microsoft.com/en-us/library/cc755104.aspx + The SETX command, for permanently modifying environment variables + + http://support.microsoft.com/kb/310519 How To Manage Environment Variables in Windows XP - http://www.chem.gla.ac.uk/~louis/software/faq/q1.html + http://www.chem.gla.ac.uk/~louis/software/faq/q1.html Setting Environment variables, Louis J. Farrugia - .. _windows-path-mod: Finding the Python executable ----------------------------- -.. versionchanged:: 3.3 +.. versionchanged:: 3.5 Besides using the automatically created start menu entry for the Python -interpreter, you might want to start Python in the command prompt. As of -Python 3.3, the installer has an option to set that up for you. - -At the "Customize Python 3.3" screen, an option called -"Add python.exe to search path" can be enabled to have the installer place -your installation into the :envvar:`%PATH%`. This allows you to type -:command:`python` to run the interpreter. Thus, you can also execute your +interpreter, you might want to start Python in the command prompt. The +installer for Python 3.5 and later has an option to set that up for you. + +On the first page of the installer, an option labelled "Add Python 3.5 to +PATH" can be selected to have the installer add the install location into the +:envvar:`PATH`. The location of the :file:`Scripts\\` folder is also added. +This allows you to type :command:`python` to run the interpreter, and +:command:`pip` or . Thus, you can also execute your scripts with command line options, see :ref:`using-on-cmdline` documentation. If you don't enable this option at install time, you can always re-run the -installer to choose it. - -The alternative is manually modifying the :envvar:`%PATH%` using the -directions in :ref:`setting-envvars`. You need to set your :envvar:`%PATH%` -environment variable to include the directory of your Python distribution, -delimited by a semicolon from other entries. An example variable could look -like this (assuming the first two entries are Windows' default):: - - C:\WINDOWS\system32;C:\WINDOWS;C:\Python33 - - -Finding modules ---------------- - -Python usually stores its library (and thereby your site-packages folder) in the -installation directory. So, if you had installed Python to -:file:`C:\\Python\\`, the default library would reside in -:file:`C:\\Python\\Lib\\` and third-party modules should be stored in -:file:`C:\\Python\\Lib\\site-packages\\`. - -This is how :data:`sys.path` is populated on Windows: - -* An empty entry is added at the start, which corresponds to the current - directory. - -* If the environment variable :envvar:`PYTHONPATH` exists, as described in - :ref:`using-on-envvars`, its entries are added next. Note that on Windows, - paths in this variable must be separated by semicolons, to distinguish them - from the colon used in drive identifiers (``C:\`` etc.). - -* Additional "application paths" can be added in the registry as subkeys of - :samp:`\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath` under both the - ``HKEY_CURRENT_USER`` and ``HKEY_LOCAL_MACHINE`` hives. Subkeys which have - semicolon-delimited path strings as their default value will cause each path - to be added to :data:`sys.path`. (Note that all known installers only use - HKLM, so HKCU is typically empty.) - -* If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as - "Python Home". Otherwise, the path of the main Python executable is used to - locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home". If a - Python home is found, the relevant sub-directories added to :data:`sys.path` - (``Lib``, ``plat-win``, etc) are based on that folder. Otherwise, the core - Python path is constructed from the PythonPath stored in the registry. - -* If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in - the environment, and no registry entries can be found, a default path with - relative entries is used (e.g. ``.\Lib;.\plat-win``, etc). - -The end result of all this is: - -* When running :file:`python.exe`, or any other .exe in the main Python - directory (either an installed version, or directly from the PCbuild - directory), the core path is deduced, and the core paths in the registry are - ignored. Other "application paths" in the registry are always read. - -* When Python is hosted in another .exe (different directory, embedded via COM, - etc), the "Python Home" will not be deduced, so the core path from the - registry is used. Other "application paths" in the registry are always read. - -* If Python can't find its home and there is no registry (eg, frozen .exe, some - very strange installation setup) you get a path with some default, but - relative, paths. - - -Executing scripts ------------------ - -As of Python 3.3, Python includes a launcher which facilitates running Python -scripts. See :ref:`launcher` for more information. - -Executing scripts without the Python launcher ---------------------------------------------- - -Without the Python launcher installed, Python scripts (files with the extension -``.py``) will be executed by :program:`python.exe` by default. This executable -opens a terminal, which stays open even if the program uses a GUI. If you do -not want this to happen, use the extension ``.pyw`` which will cause the script -to be executed by :program:`pythonw.exe` by default (both executables are -located in the top-level of your Python installation directory). This -suppresses the terminal window on startup. - -You can also make all ``.py`` scripts execute with :program:`pythonw.exe`, -setting this through the usual facilities, for example (might require -administrative rights): - -#. Launch a command prompt. -#. Associate the correct file group with ``.py`` scripts:: - - assoc .py=Python.File - -#. Redirect all Python files to the new executable:: - - ftype Python.File=C:\Path\to\pythonw.exe "%1" %* +installer, select Modify, and enable it. Alternatively, you can manually +modify the :envvar:`PATH` using the directions in :ref:`setting-envvars`. You +need to set your :envvar:`PATH` environment variable to include the directory +of your Python installation, delimited by a semicolon from other entries. An +example variable could look like this (assuming the first two entries already +existed):: + C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.5 .. _launcher: @@ -251,21 +350,26 @@ Python Launcher for Windows .. versionadded:: 3.3 -The Python launcher for Windows is a utility which aids in the location and -execution of different Python versions. It allows scripts (or the +The Python launcher for Windows is a utility which aids in locating and +executing of different Python versions. It allows scripts (or the command-line) to indicate a preference for a specific Python version, and will locate and execute that version. +Unlike the :envvar:`PATH` variable, the launcher will correctly select the most +appropriate version of Python. It will prefer per-user installations over +system-wide ones, and orders by language version rather than using the most +recently installed version. + Getting started --------------- From the command-line ^^^^^^^^^^^^^^^^^^^^^ -You should ensure the launcher is on your PATH - depending on how it was -installed it may already be there, but check just in case it is not. - -From a command-prompt, execute the following command: +System-wide installations of Python 3.3 and later will put the launcher on your +:envvar:`PATH`. The launcher is compatible with all available versions of +Python, so it does not matter which version is installed. To check that the +launcher is available, execute the following command in Command Prompt: :: @@ -291,6 +395,28 @@ If you have a Python 3.x installed, try the command: You should find the latest version of Python 3.x starts. +If you see the following error, you do not have the launcher installed: + +:: + + 'py' is not recognized as an internal or external command, + operable program or batch file. + +Per-user installations of Python do not add the launcher to :envvar:`PATH` +unless the option was selected on installation. + +Virtual environments +^^^^^^^^^^^^^^^^^^^^ + +.. versionadded:: 3.5 + +If the launcher is run with no explicit Python version specification, and a +virtual environment (created with the standard library :mod:`venv` module or +the external ``virtualenv`` tool) active, the launcher will run the virtual +environment's interpreter rather than the global one. To run the global +interpreter, either deactivate the virtual environment, or explicitly specify +the global Python version. + From a script ^^^^^^^^^^^^^ @@ -326,7 +452,7 @@ From file associations ^^^^^^^^^^^^^^^^^^^^^^ The launcher should have been associated with Python files (i.e. ``.py``, -``.pyw``, ``.pyc``, ``.pyo`` files) when it was installed. This means that +``.pyw``, ``.pyc`` files) when it was installed. This means that when you double-click on one of these files from Windows explorer the launcher will be used, and therefore you can use the same facilities described above to have the script specify the version which should be used. @@ -365,6 +491,16 @@ be used by the launcher without modification. If you are writing a new script on Windows which you hope will be useful on Unix, you should use one of the shebang lines starting with ``/usr``. +Any of the above virtual commands can be suffixed with an explicit version +(either just the major version, or the major and minor version) - for example +``/usr/bin/python2.7`` - which will cause that specific version to be located +and used. + +The ``/usr/bin/env`` form of shebang line has one further special property. +Before looking for installed Python interpreters, this form will search the +executable :envvar:`PATH` for a Python executable. This corresponds to the +behaviour of the Unix ``env`` program, which performs a :envvar:`PATH` search. + Arguments in shebang lines -------------------------- @@ -383,17 +519,16 @@ Customization Customization via INI files ^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Two .ini files will be searched by the launcher - ``py.ini`` in the - current user's "application data" directory (i.e. the directory returned - by calling the Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA) - and ``py.ini`` in the same directory as the launcher. The same .ini - files are used for both the 'console' version of the launcher (i.e. - py.exe) and for the 'windows' version (i.e. pyw.exe) +Two .ini files will be searched by the launcher - ``py.ini`` in the current +user's "application data" directory (i.e. the directory returned by calling the +Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA) and ``py.ini`` in the +same directory as the launcher. The same .ini files are used for both the +'console' version of the launcher (i.e. py.exe) and for the 'windows' version +(i.e. pyw.exe) - Customization specified in the "application directory" will have - precedence over the one next to the executable, so a user, who may not - have write access to the .ini file next to the launcher, can override - commands in that global .ini file) +Customization specified in the "application directory" will have precedence over +the one next to the executable, so a user, who may not have write access to the +.ini file next to the launcher, can override commands in that global .ini file) Customizing default Python versions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -488,6 +623,93 @@ particular version was chosen and the exact command-line used to execute the target Python. + +Finding modules +=============== + +Python usually stores its library (and thereby your site-packages folder) in the +installation directory. So, if you had installed Python to +:file:`C:\\Python\\`, the default library would reside in +:file:`C:\\Python\\Lib\\` and third-party modules should be stored in +:file:`C:\\Python\\Lib\\site-packages\\`. + +This is how :data:`sys.path` is populated on Windows: + +* An empty entry is added at the start, which corresponds to the current + directory. + +* If the environment variable :envvar:`PYTHONPATH` exists, as described in + :ref:`using-on-envvars`, its entries are added next. Note that on Windows, + paths in this variable must be separated by semicolons, to distinguish them + from the colon used in drive identifiers (``C:\`` etc.). + +* Additional "application paths" can be added in the registry as subkeys of + :samp:`\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath` under both the + ``HKEY_CURRENT_USER`` and ``HKEY_LOCAL_MACHINE`` hives. Subkeys which have + semicolon-delimited path strings as their default value will cause each path + to be added to :data:`sys.path`. (Note that all known installers only use + HKLM, so HKCU is typically empty.) + +* If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as + "Python Home". Otherwise, the path of the main Python executable is used to + locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home". If a + Python home is found, the relevant sub-directories added to :data:`sys.path` + (``Lib``, ``plat-win``, etc) are based on that folder. Otherwise, the core + Python path is constructed from the PythonPath stored in the registry. + +* If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in + the environment, and no registry entries can be found, a default path with + relative entries is used (e.g. ``.\Lib;.\plat-win``, etc). + +If a ``pyvenv.cfg`` file is found alongside the main executable or in the +directory one level above the executable, the following variations apply: + +* If ``home`` is an absolute path and :envvar:`PYTHONHOME` is not set, this + path is used instead of the path to the main executable when deducing the + home location. + +* If ``applocal`` is set to true, the ``home`` property or the main executable + is always used as the home path, and all environment variables or registry + values affecting the path are ignored. The landmark file is not checked. + +The end result of all this is: + +* When running :file:`python.exe`, or any other .exe in the main Python + directory (either an installed version, or directly from the PCbuild + directory), the core path is deduced, and the core paths in the registry are + ignored. Other "application paths" in the registry are always read. + +* When Python is hosted in another .exe (different directory, embedded via COM, + etc), the "Python Home" will not be deduced, so the core path from the + registry is used. Other "application paths" in the registry are always read. + +* If Python can't find its home and there are no registry value (frozen .exe, + some very strange installation setup) you get a path with some default, but + relative, paths. + +For those who want to bundle Python into their application or distribution, the +following advice will prevent conflicts with other installations: + +* Include a ``pyvenv.cfg`` file alongside your executable containing + ``applocal = true``. This will ensure that your own directory will be used to + resolve paths even if you have included the standard library in a ZIP file. + +* If you are loading :file:`python3.dll` or :file:`python35.dll` in your own + executable, explicitly call :c:func:`Py_SetPath` or (at least) + :c:func:`Py_SetProgramName` before :c:func:`Py_Initialize`. + +* Clear and/or overwrite :envvar:`PYTHONPATH` and set :envvar:`PYTHONHOME` + before launching :file:`python.exe` from your application. + +* If you cannot use the previous suggestions (for example, you are a + distribution that allows people to run :file:`python.exe` directly), ensure + that the landmark file (:file:`Lib\\os.py`) exists in your install directory. + (Note that it will not be detected inside a ZIP file.) + +These will ensure that the files in a system-wide installation will not take +precedence over the copy of the standard library bundled with your application. +Otherwise, your users may experience problems using your application. + Additional modules ================== @@ -498,7 +720,6 @@ and external, and snippets exist to use these features. The Windows-specific standard modules are documented in :ref:`mswin-specific-services`. - PyWin32 ------- @@ -557,20 +778,8 @@ latest release's source or just grab a fresh `checkout <https://docs.python.org/devguide/setup.html#getting-the-source-code>`_. The source tree contains a build solution and project files for Microsoft -Visual C++, which is the compiler used to build the official Python releases. -View the :file:`readme.txt` in their respective directories: - -+--------------------+--------------+-----------------------+ -| Directory | MSVC version | Visual Studio version | -+====================+==============+=======================+ -| :file:`PC/VS9.0/` | 9.0 | 2008 | -+--------------------+--------------+-----------------------+ -| :file:`PCbuild/` | 10.0 | 2010 | -+--------------------+--------------+-----------------------+ - -Note that any build directories within the :file:`PC` directory are not -necessarily fully supported. The :file:`PCbuild` directory contains the files -for the compiler used to build the official release. +Visual Studio 2015, which is the compiler used to build the official Python +releases. These files are in the :file:`PCbuild` directory. Check :file:`PCbuild/readme.txt` for general information on the build process. @@ -603,5 +812,3 @@ Other resources :pep:`397` - Python launcher for Windows The proposal for the launcher to be included in the Python distribution. - - diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst new file mode 100644 index 0000000..bfefb2f --- /dev/null +++ b/Doc/whatsnew/3.5.rst @@ -0,0 +1,1122 @@ +**************************** + What's New In Python 3.5 +**************************** + +:Release: |release| +:Date: |today| + +.. Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten to some degree. + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. + + * It's helpful to add the bug/patch number as a comment: + + XXX Describe the transmogrify() function added to the socket + module. + (Contributed by P.Y. Developer in :issue:`12345`.) + + This saves the maintainer the effort of going through the Mercurial log + when researching a change. + +This article explains the new features in Python 3.5, compared to 3.4. + +For full details, see the :source:`Misc/NEWS` file. + +.. note:: + + Prerelease users should be aware that this document is currently in draft + form. It will be updated substantially as Python 3.5 moves towards release, + so it's worth checking back even after reading earlier versions. + + +.. seealso:: + + :pep:`478` - Python 3.5 Release Schedule + + +Summary -- Release highlights +============================= + +.. This section singles out the most important changes in Python 3.5. + Brevity is key. + +New syntax features: + +* :pep:`465`, a new matrix multiplication operator: ``a @ b``. +* :pep:`492`, coroutines with async and await syntax. + +New library modules: + +* :mod:`zipapp`: :ref:`Improving Python ZIP Application Support + <whatsnew-zipapp>` (:pep:`441`). + +New built-in features: + +* ``bytes % args``, ``bytearray % args``: :pep:`461` - Adding ``%`` formatting + to bytes and bytearray +* ``b'\xf0\x9f\x90\x8d'.hex()``, ``bytearray(b'\xf0\x9f\x90\x8d').hex()``, + ``memoryview(b'\xf0\x9f\x90\x8d').hex()``: :issue:`9951` - A ``hex`` method + has been added to bytes, bytearray, and memoryview. +* Generators have new ``gi_yieldfrom`` attribute, which returns the + object being iterated by ``yield from`` expressions. (Contributed + by Benno Leslie and Yury Selivanov in :issue:`24450`.) + +Implementation improvements: + +* When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), + :py:data:`sys.stdin` and :py:data:`sys.stdout` are now using the + ``surrogateescape`` error handler, instead of the ``strict`` error handler + (:issue:`19977`). + +* :pep:`488`, the elimination of ``.pyo`` files. +* :pep:`489`, multi-phase initialization of extension modules. + +Significantly Improved Library Modules: + +* :class:`collections.OrderedDict` is now implemented in C, which improves + its performance between 4x to 100x times. Contributed by Eric Snow in + :issue:`16991`. + +* You may now pass bytes to the :mod:`tempfile` module's APIs and it will + return the temporary pathname as bytes instead of str. It also accepts + a value of ``None`` on parameters where only str was accepted in the past to + do the right thing based on the types of the other inputs. Two functions, + :func:`gettempdirb` and :func:`gettempprefixb`, have been added to go along + with this. This behavior matches that of the :mod:`os` APIs. + +Security improvements: + +* None yet. + +Please read on for a comprehensive list of user-facing changes. + + +.. PEP-sized items next. + +.. _pep-4XX: + +.. PEP 4XX: Virtual Environments +.. ============================= + + +.. (Implemented by Foo Bar.) + +.. .. seealso:: + + :pep:`4XX` - Python Virtual Environments + PEP written by Carl Meyer + + +PEP 492 - Coroutines with async and await syntax +------------------------------------------------ + +The PEP added dedicated syntax for declaring :term:`coroutines <coroutine>`, +:keyword:`await` expressions, new asynchronous :keyword:`async for` +and :keyword:`async with` statements. + +Example:: + + async def read_data(db): + async with db.transaction(): + data = await db.fetch('SELECT ...') + +PEP written and implemented by Yury Selivanov. + +.. seealso:: + + :pep:`492` -- Coroutines with async and await syntax + + +PEP 461 - Formatting support for bytes and bytearray +---------------------------------------------------- + +This PEP proposes adding % formatting operations similar to Python 2's ``str`` +type to :class:`bytes` and :class:`bytearray`. + +Examples:: + + >>> b'Hello %s!' % b'World' + b'Hello World!' + >>> b'x=%i y=%f' % (1, 2.5) + b'x=1 y=2.500000' + +Unicode is not allowed for ``%s``, but it is accepted by ``%a`` (equivalent of +``repr(obj).encode('ascii', 'backslashreplace')``):: + + >>> b'Hello %s!' % 'World' + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: %b requires bytes, or an object that implements __bytes__, not 'str' + >>> b'price: %a' % '10€' + b"price: '10\\u20ac'" + +.. seealso:: + + :pep:`461` -- Adding % formatting to bytes and bytearray + + +PEP 465 - A dedicated infix operator for matrix multiplication +-------------------------------------------------------------- + +This PEP proposes a new binary operator to be used for matrix multiplication, +called ``@``. (Mnemonic: ``@`` is ``*`` for mATrices.) + +.. seealso:: + + :pep:`465` -- A dedicated infix operator for matrix multiplication + + +PEP 471 - os.scandir() function -- a better and faster directory iterator +------------------------------------------------------------------------- + +:pep:`471` adds a new directory iteration function, :func:`os.scandir`, +to the standard library. Additionally, :func:`os.walk` is now +implemented using :func:`os.scandir`, which speeds it up by 3-5 times +on POSIX systems and by 7-20 times on Windows systems. + +PEP and implementation written by Ben Hoyt with the help of Victor Stinner. + +.. seealso:: + + :pep:`471` -- os.scandir() function -- a better and faster directory + iterator + + +PEP 475: Retry system calls failing with EINTR +---------------------------------------------- + +:pep:`475` adds support for automatic retry of system calls failing with +:py:data:`~errno.EINTR`: this means that user code doesn't have to deal with +EINTR or :exc:`InterruptedError` manually, and should make it more robust +against asynchronous signal reception. + +.. seealso:: + + :pep:`475` -- Retry system calls failing with EINTR + + +PEP 479: Change StopIteration handling inside generators +-------------------------------------------------------- + +:pep:`479` changes the behavior of generators: when a :exc:`StopIteration` +exception is raised inside a generator, it is replaced with a +:exc:`RuntimeError`. To enable the feature a ``__future__`` import should +be used:: + + from __future__ import generator_stop + +Without a ``__future__`` import, a :exc:`PendingDeprecationWarning` will be +raised. + +PEP written by Chris Angelico and Guido van Rossum. Implemented by +Chris Angelico, Yury Selivanov and Nick Coghlan. + +.. seealso:: + + :pep:`479` -- Change StopIteration handling inside generators + + +PEP 486: Make the Python Launcher aware of virtual environments +--------------------------------------------------------------- + +:pep:`486` makes the Windows launcher (see :pep:`397`) aware of an active +virtual environment. When the default interpreter would be used and the +``VIRTUAL_ENV`` environment variable is set, the interpreter in the virtual +environment will be used. + +.. seealso:: + + :pep:`486` -- Make the Python Launcher aware of virtual environments + + +PEP 488: Elimination of PYO files +--------------------------------- + +:pep:`488` does away with the concept of ``.pyo`` files. This means that +``.pyc`` files represent both unoptimized and optimized bytecode. To prevent the +need to constantly regenerate bytecode files, ``.pyc`` files now have an +optional ``opt-`` tag in their name when the bytecode is optimized. This has the +side-effect of no more bytecode file name clashes when running under either +``-O`` or ``-OO``. Consequently, bytecode files generated from ``-O``, and +``-OO`` may now exist simultaneously. :func:`importlib.util.cache_from_source` +has an updated API to help with this change. + +.. seealso:: + + :pep:`488` -- Elimination of PYO files + + +PEP 489: Multi-phase extension module initialization +---------------------------------------------------- + +:pep:`489` updates extension module initialization to take advantage of the +two step module loading mechanism introduced by :pep:`451` in Python 3.4. + +This change brings the import semantics of extension modules that opt-in to +using the new mechanism much closer to those of Python source and bytecode +modules, including the ability to any valid identifier as a module name, +rather than being restricted to ASCII. + +.. seealso:: + + :pep:`488` -- Multi-phase extension module initialization + +PEP 485: A function for testing approximate equality +---------------------------------------------------- + +:pep:`485` adds the :func:`math.isclose` and :func:`cmath.isclose` +functions which tell whether two values are approximately equal or +"close" to each other. Whether or not two values are considered +close is determined according to given absolute and relative tolerances. + +.. seealso:: + + :pep:`485` -- A function for testing approximate equality + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* Added the ``'namereplace'`` error handlers. The ``'backslashreplace'`` + error handlers now works with decoding and translating. + (Contributed by Serhiy Storchaka in :issue:`19676` and :issue:`22286`.) + +* The :option:`-b` option now affects comparisons of :class:`bytes` with + :class:`int`. (Contributed by Serhiy Storchaka in :issue:`23681`) + +* New Kazakh :ref:`codec <standard-encodings>` ``kz1048``. (Contributed by + Serhiy Storchaka in :issue:`22682`.) + +* Property docstrings are now writable. This is especially useful for + :func:`collections.namedtuple` docstrings. + (Contributed by Berker Peksag in :issue:`24064`.) + +* New Tajik :ref:`codec <standard-encodings>` ``koi8_t``. (Contributed by + Serhiy Storchaka in :issue:`22681`.) + + +New Modules +=========== + +.. _whatsnew-zipapp: + +zipapp +------ + +The new :mod:`zipapp` module (specified in :pep:`441`) provides an API and +command line tool for creating executable Python Zip Applications, which +were introduced in Python 2.6 in :issue:`1739468` but which were not well +publicised, either at the time or since. + +With the new module, bundling your application is as simple as putting all +the files, including a ``__main__.py`` file, into a directory ``myapp`` +and running:: + + $ python -m zipapp myapp + $ python myapp.pyz + + +Improved Modules +================ + +argparse +-------- + +* :class:`~argparse.ArgumentParser` now allows to disable + :ref:`abbreviated usage <prefix-matching>` of long options by setting + :ref:`allow_abbrev` to ``False``. + (Contributed by Jonathan Paugh, Steven Bethard, paul j3 and Daniel Eriksson.) + +cgi +--- + +* :class:`~cgi.FieldStorage` now supports the context management protocol. + (Contributed by Berker Peksag in :issue:`20289`.) + +cmath +----- + +* :func:`cmath.isclose` function added. + (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) + + +code +---- + +* The :func:`code.InteractiveInterpreter.showtraceback` method now prints + the full chained traceback, just like the interactive interpreter. + (Contributed by Claudiu Popa in :issue:`17442`.) + +collections +----------- + +* You can now update docstrings produced by :func:`collections.namedtuple`:: + + Point = namedtuple('Point', ['x', 'y']) + Point.__doc__ = 'ordered pair' + Point.x.__doc__ = 'abscissa' + Point.y.__doc__ = 'ordinate' + + (Contributed by Berker Peksag in :issue:`24064`.) + +compileall +---------- + +* :func:`compileall.compile_dir` and :mod:`compileall`'s command-line interface + can now do parallel bytecode compilation. + (Contributed by Claudiu Popa in :issue:`16104`.) + +contextlib +---------- + +* The new :func:`contextlib.redirect_stderr` context manager(similar to + :func:`contextlib.redirect_stdout`) makes it easier for utility scripts to + handle inflexible APIs that write their output to :data:`sys.stderr` and + don't provide any options to redirect it. + (Contributed by Berker Peksag in :issue:`22389`.) + +curses +------ +* The new :func:`curses.update_lines_cols` function updates the variables + :envvar:`curses.LINES` and :envvar:`curses.COLS`. + +difflib +------- + +* The charset of the HTML document generated by :meth:`difflib.HtmlDiff.make_file` + can now be customized by using *charset* keyword-only parameter. The default + charset of HTML document changed from ``'ISO-8859-1'`` to ``'utf-8'``. + (Contributed by Berker Peksag in :issue:`2052`.) + +* It's now possible to compare lists of byte strings with + :func:`difflib.diff_bytes` (fixes a regression from Python 2). + +distutils +--------- + +* The ``build`` and ``build_ext`` commands now accept a ``-j`` + option to enable parallel building of extension modules. + (Contributed by Antoine Pitrou in :issue:`5309`.) + +* Added support for the LZMA compression. + (Contributed by Serhiy Storchaka in :issue:`16314`.) + +doctest +------- + +* :func:`doctest.DocTestSuite` returns an empty :class:`unittest.TestSuite` if + *module* contains no docstrings instead of raising :exc:`ValueError`. + (Contributed by Glenn Jones in :issue:`15916`.) + +email +----- + +* A new policy option :attr:`~email.policy.Policy.mangle_from_` controls + whether or not lines that start with "From " in email bodies are prefixed with + a '>' character by generators. The default is ``True`` for + :attr:`~email.policy.compat32` and ``False`` for all other policies. + (Contributed by Milan Oberkirch in :issue:`20098`.) + +* A new method :meth:`~email.message.Message.get_content_disposition` provides + easy access to a canonical value for the :mailheader:`Content-Disposition` + header (``None`` if there is no such header). (Contributed by Abhilash Raj + in :issue:`21083`.) + +* A new policy option :attr:`~email.policy.EmailPolicy.utf8` can be set + ``True`` to encode email headers using the utf8 charset instead of using + encoded words. This allows ``Messages`` to be formatted according to + :rfc:`6532` and used with an SMTP server that supports the :rfc:`6531` + ``SMTPUTF8`` extension. (Contributed by R. David Murray in :issue:`24211`.) + +glob +---- + +* :func:`~glob.iglob` and :func:`~glob.glob` now support recursive search in + subdirectories using the "``**``" pattern. + (Contributed by Serhiy Storchaka in :issue:`13968`.) + +idlelib and IDLE +---------------- + +Since idlelib implements the IDLE shell and editor and is not intended for +import by other programs, it gets improvements with every release. See +:file:`Lib/idlelib/NEWS.txt` for a cumulative list of changes since 3.4.0, +as well as changes made in future 3.5.x releases. This file is also available +from the IDLE Help -> About Idle dialog. + +imaplib +------- + +* :class:`IMAP4` now supports the context management protocol. When used in a + :keyword:`with` statement, the IMAP4 ``LOGOUT`` command will be called + automatically at the end of the block. (Contributed by Tarek Ziadé and + Serhiy Storchaka in :issue:`4972`.) + +* :mod:`imaplib` now supports :rfc:`5161`: the :meth:`~imaplib.IMAP4.enable` + extension), and :rfc:`6855`: utf-8 support (internationalized email, via the + ``UTF8=ACCEPT`` argument to :meth:`~imaplib.IMAP4.enable`). A new attribute, + :attr:`~imaplib.IMAP4.utf8_enabled`, tracks whether or not :rfc:`6855` + support is enabled. Milan Oberkirch, R. David Murray, and Maciej Szulik in + :issue:`21800`.) + +* :mod:`imaplib` now automatically encodes non-ASCII string usernames and + passwords using ``UTF8``, as recommended by the RFCs. (Contributed by Milan + Oberkirch in :issue:`21800`.) + +imghdr +------ + +* :func:`~imghdr.what` now recognizes the `OpenEXR <http://www.openexr.com>`_ + format. (Contributed by Martin Vignali and Claudiu Popa in :issue:`20295`.) + +importlib +--------- + +* :class:`importlib.util.LazyLoader` allows for the lazy loading of modules in + applications where startup time is paramount. + (Contributed by Brett Cannon in :issue:`17621`.) + +* :func:`importlib.abc.InspectLoader.source_to_code` is now a + static method to make it easier to work with source code in a string. + With a module object that you want to initialize you can then use + ``exec(code, module.__dict__)`` to execute the code in the module. + +* :func:`importlib.util.module_from_spec` is now the preferred way to create a + new module. Compared to :class:`types.ModuleType`, this new function will set + the various import-controlled attributes based on the passed-in spec object. + +inspect +------- + +* :class:`inspect.Signature` and :class:`inspect.Parameter` are now + picklable and hashable. (Contributed by Yury Selivanov in :issue:`20726` + and :issue:`20334`.) + +* New method :meth:`inspect.BoundArguments.apply_defaults`. (Contributed + by Yury Selivanov in :issue:`24190`.) + +* New class method :meth:`inspect.Signature.from_callable`, which makes + subclassing of :class:`~inspect.Signature` easier. (Contributed + by Yury Selivanov and Eric Snow in :issue:`17373`.) + +* New argument ``follow_wrapped`` for :func:`inspect.signature`. + (Contributed by Yury Selivanov in :issue:`20691`.) + +* New :func:`~inspect.iscoroutine` and :func:`~inspect.iscoroutinefunction` + functions. (Contributed by Yury Selivanov in :issue:`24017`.) + +* New :func:`~inspect.getcoroutinelocals` and :func:`~inspect.getcoroutinestate` + functions. (Contributed by Yury Selivanov in :issue:`24400`.) + +ipaddress +--------- + +* :class:`ipaddress.IPv4Network` and :class:`ipaddress.IPv6Network` now + accept an ``(address, netmask)`` tuple argument, so as to easily construct + network objects from existing addresses. (Contributed by Peter Moody + and Antoine Pitrou in :issue:`16531`.) + +json +---- + +* The output of :mod:`json.tool` command line interface is now in the same + order as the input. Use the :option:`--sort-keys` option to sort the output + of dictionaries alphabetically by key. (Contributed by Berker Peksag in + :issue:`21650`.) + +* JSON decoder now raises :exc:`json.JSONDecodeError` instead of + :exc:`ValueError`. (Contributed by Serhiy Storchaka in :issue:`19361`.) + +math +---- + +* :data:`math.inf` and :data:`math.nan` constants added. (Contributed by Mark + Dickinson in :issue:`23185`.) +* :func:`math.isclose` function added. + (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) + +os +-- + +* New :func:`os.scandir` function that exposes file information from + the operating system when listing a directory. :func:`os.scandir` + returns an iterator of :class:`os.DirEntry` objects corresponding to + the entries in the directory given by *path*. (Contributed by Ben + Hoyt with the help of Victor Stinner in :issue:`22524`.) + +* :class:`os.stat_result` now has a :attr:`~os.stat_result.st_file_attributes` + attribute on Windows. (Contributed by Ben Hoyt in :issue:`21719`.) + +os.path +------- + +* New :func:`~os.path.commonpath` function that extracts common path prefix. + Unlike the :func:`~os.path.commonprefix` function, it always returns a valid + patch. (Contributed by Rafik Draoui and Serhiy Storchaka in :issue:`10395`.) + +pickle +------ + +* Serializing more "lookupable" objects (such as unbound methods or nested + classes) now are supported with pickle protocols < 4. + (Contributed by Serhiy Storchaka in :issue:`23611`.) + +poplib +------ + +* A new command :meth:`~poplib.POP3.utf8` enables :rfc:`6856` + (internationalized email) support if the POP server supports it. (Contributed + by Milan OberKirch in :issue:`21804`.) + +re +-- + +* Number of capturing groups in regular expression is no longer limited by 100. + (Contributed by Serhiy Storchaka in :issue:`22437`.) + +* Now unmatched groups are replaced with empty strings in :func:`re.sub` + and :func:`re.subn`. (Contributed by Serhiy Storchaka in :issue:`1519638`.) + +shutil +------ + +* :func:`~shutil.move` now accepts a *copy_function* argument, allowing, + for example, :func:`~shutil.copy` to be used instead of the default + :func:`~shutil.copy2` if there is a need to ignore metadata. (Contributed by + Claudiu Popa in :issue:`19840`.) + +signal +------ + +* On Windows, :func:`signal.set_wakeup_fd` now also supports socket handles. + (Contributed by Victor Stinner in :issue:`22018`.) + +* Different constants of :mod:`signal` module are now enumeration values using + the :mod:`enum` module. This allows meaningful names to be printed during + debugging, instead of integer “magic numbers”. (Contributed by Giampaolo + Rodola' in :issue:`21076`.) + +smtpd +----- + +* Both :class:`~smtpd.SMTPServer` and :class:`smtpd.SMTPChannel` now accept a + *decode_data* keyword to determine if the DATA portion of the SMTP + transaction is decoded using the ``utf-8`` codec or is instead provided to + :meth:`~smtpd.SMTPServer.process_message` as a byte string. The default + is ``True`` for backward compatibility reasons, but will change to ``False`` + in Python 3.6. If *decode_data* is set to ``False``, the + :meth:`~smtpd.SMTPServer.process_message` method must be prepared to accept + keyword arguments. (Contributed by Maciej Szulik in :issue:`19662`.) + +* :class:`~smtpd.SMTPServer` now advertises the ``8BITMIME`` extension + (:rfc:`6152`) if if *decode_data* has been set ``True``. If the client + specifies ``BODY=8BITMIME`` on the ``MAIL`` command, it is passed to + :meth:`~smtpd.SMTPServer.process_message` via the ``mail_options`` keyword. + (Contributed by Milan Oberkirch and R. David Murray in :issue:`21795`.) + +* :class:`~smtpd.SMTPServer` now supports the ``SMTPUTF8`` extension + (:rfc:`6531`: Internationalized Email). If the client specified ``SMTPUTF8 + BODY=8BITMIME`` on the ``MAIL`` command, they are passed to + :meth:`~smtpd.SMTPServer.process_message` via the ``mail_options`` keyword. + It is the responsibility of the :meth:`~smtpd.SMTPServer.process_message` + method to correctly handle the ``SMTPUTF8`` data. (Contributed by Milan + Oberkirch in :issue:`21725`.) + +* It is now possible to provide, directly or via name resolution, IPv6 + addresses in the :class:`~smtpd.SMTPServer` constructor, and have it + successfully connect. (Contributed by Milan Oberkirch in :issue:`14758`.) + +smtplib +------- + +* A new :meth:`~smtplib.SMTP.auth` method provides a convenient way to + implement custom authentication mechanisms. + (Contributed by Milan Oberkirch in :issue:`15014`.) + +* Additional debuglevel (2) shows timestamps for debug messages in + :class:`smtplib.SMTP`. (Contributed by Gavin Chappell and Maciej Szulik in + :issue:`16914`.) + +* :mod:`smtplib` now supports :rfc:`6531` (SMTPUTF8) in both the + :meth:`~smtplib.SMTP.sendmail` and :meth:`~smtplib.SMTP.send_message` + commands. (Contributed by Milan Oberkirch and R. David Murray in + :issue:`22027`.) + +sndhdr +------ + +* :func:`~sndhdr.what` and :func:`~sndhdr.whathdr` now return + :func:`~collections.namedtuple`. + (Contributed by Claudiu Popa in :issue:`18615`.) + +socket +------ + +* New :meth:`socket.socket.sendfile` method allows to send a file over a socket + by using high-performance :func:`os.sendfile` function on UNIX resulting in + uploads being from 2x to 3x faster than when using plain + :meth:`socket.socket.send`. + (Contributed by Giampaolo Rodola' in :issue:`17552`.) + +subprocess +---------- + +* The new :func:`subprocess.run` function runs subprocesses and returns a + :class:`subprocess.CompletedProcess` object. It Provides a more consistent + API than :func:`~subprocess.call`, :func:`~subprocess.check_call` and + :func:`~subprocess.check_output`. + +sys +--- + +* New :func:`~sys.set_coroutine_wrapper` and :func:`~sys.get_coroutine_wrapper` + functions. (Contributed by Yury Selivanov in :issue:`24017`.) + +sysconfig +--------- + +* The user scripts directory on Windows is now versioned. + (Contributed by Paul Moore in :issue:`23437`.) + +tarfile +------- + +* The :func:`tarfile.open` function now supports ``'x'`` (exclusive creation) + mode. (Contributed by Berker Peksag in :issue:`21717`.) + +* The :meth:`~tarfile.TarFile.extractall` and :meth:`~tarfile.TarFile.extract` + methods now take a keyword parameter *numeric_only*. If set to ``True``, + the extracted files and directories will be owned by the numeric uid and gid + from the tarfile. If set to ``False`` (the default, and the behavior in + versions prior to 3.5), they will be owned bythe named user and group in the + tarfile. (Contributed by Michael Vogt and Eric Smith in :issue:`23193`.) + +time +---- + +* The :func:`time.monotonic` function is now always available. (Contributed by + Victor Stinner in :issue:`22043`.) + +tkinter +------- + +* The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment + on Windows has been replaced by a private function in the :mod:`_tkinter` + module which makes no permanent changes to environment variables. + (Contributed by Zachary Ware in :issue:`20035`.) + +types +----- + +* New :func:`~types.coroutine` function. (Contributed by Yury Selivanov + in :issue:`24017`.) + +* New :class:`~types.CoroutineType`. (Contributed by Yury Selivanov + in :issue:`24400`.) + +urllib +------ + +* A new :class:`~urllib.request.HTTPPasswordMgrWithPriorAuth` allows HTTP Basic + Authentication credentials to be managed so as to eliminate unnecessary + ``401`` response handling, or to unconditionally send credentials + on the first request in order to communicate with servers that return a + ``404`` response instead of a ``401`` if the ``Authorization`` header is not + sent. (Contributed by Matej Cepl in :issue:`19494` and Akshit Khurana in + :issue:`7159`.) + +* A new :func:`~urllib.parse.urlencode` parameter *quote_via* provides a way to + control the encoding of query parts if needed. (Contributed by Samwyse and + Arnon Yaari in :issue:`13866`.) + +unicodedata +----------- + +* The :mod:`unicodedata` module now uses data from `Unicode 8.0.0 +<http://unicode.org/versions/Unicode8.0.0/>`_. + + +wsgiref +------- + +* *headers* parameter of :class:`wsgiref.headers.Headers` is now optional. + (Contributed by Pablo Torres Navarrete and SilentGhost in :issue:`5800`.) + +xmlrpc +------ + +* :class:`xmlrpc.client.ServerProxy` is now a :term:`context manager`. + (Contributed by Claudiu Popa in :issue:`20627`.) + +xml.sax +------- + +* SAX parsers now support a character stream of + :class:`~xml.sax.xmlreader.InputSource` object. + (Contributed by Serhiy Storchaka in :issue:`2175`.) + +faulthandler +------------ + +* :func:`~faulthandler.enable`, :func:`~faulthandler.register`, + :func:`~faulthandler.dump_traceback` and + :func:`~faulthandler.dump_traceback_later` functions now accept file + descriptors. (Contributed by Wei Wu in :issue:`23566`.) + +zipfile +------- + +* Added support for writing ZIP files to unseekable streams. + (Contributed by Serhiy Storchaka in :issue:`23252`.) + +* The :func:`zipfile.ZipFile.open` function now supports ``'x'`` (exclusive + creation) mode. (Contributed by Serhiy Storchaka in :issue:`21717`.) + + +Optimizations +============= + +The following performance enhancements have been added: + +* :func:`os.walk` has been sped up by 3-5x on POSIX systems and 7-20x + on Windows. This was done using the new :func:`os.scandir` function, + which exposes file information from the underlying ``readdir`` and + ``FindFirstFile``/``FindNextFile`` system calls. (Contributed by + Ben Hoyt with help from Victor Stinner in :issue:`23605`.) + +* Construction of ``bytes(int)`` (filled by zero bytes) is faster and uses less + memory for large objects. ``calloc()`` is used instead of ``malloc()`` to + allocate memory for these objects. + +* Some operations on :class:`~ipaddress.IPv4Network` and + :class:`~ipaddress.IPv6Network` have been massively sped up, such as + :meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`, + :func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`. + The speed up can range from 3x to 15x. + (:issue:`21486`, :issue:`21487`, :issue:`20826`) + +* Many operations on :class:`io.BytesIO` are now 50% to 100% faster. + (Contributed by Serhiy Storchaka in :issue:`15381` and David Wilson in + :issue:`22003`.) + +* :func:`marshal.dumps` is now faster (65%-85% with versions 3--4, 20-25% with + versions 0--2 on typical data, and up to 5x in best cases). + (Contributed by Serhiy Storchaka in :issue:`20416` and :issue:`23344`.) + +* The UTF-32 encoder is now 3x to 7x faster. (Contributed by Serhiy Storchaka + in :issue:`15027`.) + + +Build and C API Changes +======================= + +Changes to Python's build process and to the C API include: + +* New ``calloc`` functions: + + * :c:func:`PyMem_RawCalloc` + * :c:func:`PyMem_Calloc` + * :c:func:`PyObject_Calloc` + * :c:func:`_PyObject_GC_Calloc` + + +Deprecated +========== + +New Keywords +------------ + +``async`` and ``await`` are not recommended to be used as variable, class or +function names. Introduced by :pep:`492` in Python 3.5, they will become +proper keywords in Python 3.7. + + +Unsupported Operating Systems +----------------------------- + +* Windows XP - Per :PEP:`11`, Microsoft support of Windows XP has ended. + + +Deprecated Python modules, functions and methods +------------------------------------------------ + +* The :mod:`formatter` module has now graduated to full deprecation and is still + slated for removal in Python 3.6. + +* :mod:`smtpd` has in the past always decoded the DATA portion of email + messages using the ``utf-8`` codec. This can now be controlled by the new + *decode_data* keyword to :class:`~smtpd.SMTPServer`. The default value is + ``True``, but this default is deprecated. Specify the *decode_data* keyword + with an appropriate value to avoid the deprecation warning. + +* Directly assigning values to the :attr:`~http.cookies.Morsel.key`, + :attr:`~http.cookies.Morsel.value` and + :attr:`~http.cookies.Morsel.coded_value` of :class:`~http.cookies.Morsel` + objects is deprecated. Use the :func:`~http.cookies.Morsel.set` method + instead. In addition, the undocumented *LegalChars* parameter of + :func:`~http.cookies.Morsel.set` is deprecated, and is now ignored. + +* Passing a format string as keyword argument *format_string* to the + :meth:`~string.Formatter.format` method of the :class:`string.Formatter` + class has been deprecated. + +* :func:`platform.dist` and :func:`platform.linux_distribution` functions are + now deprecated and will be removed in Python 3.7. Linux distributions use + too many different ways of describing themselves, so the functionality is + left to a package. + (Contributed by Vajrasky Kok and Berker Peksag in :issue:`1322`.) + +* The previously undocumented ``from_function`` and ``from_builtin`` methods of + :class:`inspect.Signature` are deprecated. Use new + :meth:`inspect.Signature.from_callable` instead. (Contributed by Yury + Selivanov in :issue:`24248`.) + +* :func:`inspect.getargspec` is deprecated and scheduled to be removed in + Python 3.6. (See :issue:`20438` for details.) + +* :func:`~inspect.getfullargspec`, :func:`~inspect.getargvalues`, + :func:`~inspect.getcallargs`, :func:`~inspect.getargvalues`, + :func:`~inspect.formatargspec`, and :func:`~inspect.formatargvalues` are + deprecated in favor of :func:`inspect.signature` API. (See :issue:`20438` + for details.) + + +Deprecated functions and types of the C API +------------------------------------------- + +* None yet. + + +Deprecated features +------------------- + +* None yet. + + +Removed +======= + +API and Feature Removals +------------------------ + +The following obsolete and previously deprecated APIs and features have been +removed: + +* The ``__version__`` attribute has been dropped from the email package. The + email code hasn't been shipped separately from the stdlib for a long time, + and the ``__version__`` string was not updated in the last few releases. + +* The internal ``Netrc`` class in the :mod:`ftplib` module was deprecated in + 3.4, and has now been removed. + (Contributed by Matt Chaput in :issue:`6623`.) + +* The concept of ``.pyo`` files has been removed. + +* The JoinableQueue class in the provisional asyncio module was deprecated + in 3.4.4 and is now removed (:issue:`23464`). + + +Porting to Python 3.5 +===================== + +This section lists previously described changes and other bugfixes +that may require changes to your code. + +Changes in the Python API +------------------------- + +* :pep:`475`: Examples of functions which are now retried when interrupted + instead of raising :exc:`InterruptedError` if the signal handler does not + raise an exception: + + - :func:`open`, :func:`os.open`, :func:`io.open` + - functions of the :mod:`faulthandler` module + - :mod:`os` functions: + + * :func:`os.fchdir` + * :func:`os.fchmod` + * :func:`os.fchown` + * :func:`os.fdatasync` + * :func:`os.fstat` + * :func:`os.fstatvfs` + * :func:`os.fsync` + * :func:`os.ftruncate` + * :func:`os.mkfifo` + * :func:`os.mknod` + * :func:`os.posix_fadvise` + * :func:`os.posix_fallocate` + * :func:`os.pread` + * :func:`os.pwrite` + * :func:`os.read` + * :func:`os.readv` + * :func:`os.sendfile` + * :func:`os.wait3` + * :func:`os.wait4` + * :func:`os.wait` + * :func:`os.waitid` + * :func:`os.waitpid` + * :func:`os.write` + * :func:`os.writev` + * special cases: :func:`os.close` and :func:`os.dup2` now ignore + :py:data:`~errno.EINTR` error, the syscall is not retried (see the PEP + for the rationale) + + - :func:`select.select`, :func:`select.poll.poll`, :func:`select.epoll.poll`, + :func:`select.kqueue.control`, :func:`select.devpoll.poll` + - :func:`socket.socket` methods: + + * :meth:`~socket.socket.accept` + * :meth:`~socket.socket.connect` (except for non-blocking sockets) + * :meth:`~socket.socket.recv` + * :meth:`~socket.socket.recvfrom` + * :meth:`~socket.socket.recvmsg` + * :meth:`~socket.socket.send` + * :meth:`~socket.socket.sendall` + * :meth:`~socket.socket.sendmsg` + * :meth:`~socket.socket.sendto` + + - :func:`signal.sigtimedwait`, :func:`signal.sigwaitinfo` + - :func:`time.sleep` + +* Before Python 3.5, a :class:`datetime.time` object was considered to be false + if it represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. + +* :meth:`ssl.SSLSocket.send()` now raises either :exc:`ssl.SSLWantReadError` + or :exc:`ssl.SSLWantWriteError` on a non-blocking socket if the operation + would block. Previously, it would return 0. See :issue:`20951`. + +* The ``__name__`` attribute of generator is now set from the function name, + instead of being set from the code name. Use ``gen.gi_code.co_name`` to + retrieve the code name. Generators also have a new ``__qualname__`` + attribute, the qualified name, which is now used for the representation + of a generator (``repr(gen)``). See :issue:`21205`. + +* The deprecated "strict" mode and argument of :class:`~html.parser.HTMLParser`, + :meth:`HTMLParser.error`, and the :exc:`HTMLParserError` exception have been + removed. (Contributed by Ezio Melotti in :issue:`15114`.) + The *convert_charrefs* argument of :class:`~html.parser.HTMLParser` is + now ``True`` by default. (Contributed by Berker Peksag in :issue:`21047`.) + +* Although it is not formally part of the API, it is worth noting for porting + purposes (ie: fixing tests) that error messages that were previously of the + form "'sometype' does not support the buffer protocol" are now of the form "a + bytes-like object is required, not 'sometype'". (Contributed by Ezio Melotti + in :issue:`16518`.) + +* If the current directory is set to a directory that no longer exists then + :exc:`FileNotFoundError` will no longer be raised and instead + :meth:`~importlib.machinery.FileFinder.find_spec` will return ``None`` + **without** caching ``None`` in :data:`sys.path_importer_cache` which is + different than the typical case (:issue:`22834`). + +* HTTP status code and messages from :mod:`http.client` and :mod:`http.server` + were refactored into a common :class:`~http.HTTPStatus` enum. The values in + :mod:`http.client` and :mod:`http.server` remain available for backwards + compatibility. (Contributed by Demian Brecht in :issue:`21793`.) + +* When an import loader defines :meth:`~importlib.machinery.Loader.exec_module` + it is now expected to also define + :meth:`~importlib.machinery.Loader.create_module` (raises a + :exc:`DeprecationWarning` now, will be an error in Python 3.6). If the loader + inherits from :class:`importlib.abc.Loader` then there is nothing to do, else + simply define :meth:`~importlib.machinery.Loader.create_module` to return + ``None`` (:issue:`23014`). + +* :func:`re.split` always ignored empty pattern matches, so the ``'x*'`` + pattern worked the same as ``'x+'``, and the ``'\b'`` pattern never worked. + Now :func:`re.split` raises a warning if the pattern could match + an empty string. For compatibility use patterns that never match an empty + string (e.g. ``'x+'`` instead of ``'x*'``). Patterns that could only match + an empty string (such as ``'\b'``) now raise an error. + +* The :class:`~http.cookies.Morsel` dict-like interface has been made self + consistent: morsel comparison now takes the :attr:`~http.cookies.Morsel.key` + and :attr:`~http.cookies.Morsel.value` into account, + :meth:`~http.cookies.Morsel.copy` now results in a + :class:`~http.cookies.Morsel` instance rather than a :class:`dict`, and + :meth:`~http.cookies.Morsel.update` will now raise an exception if any of the + keys in the update dictionary are invalid. In addition, the undocumented + *LegalChars* parameter of :func:`~http.cookies.Morsel.set` is deprecated and + is now ignored. (:issue:`2211`) + +* :pep:`488` has removed ``.pyo`` files from Python and introduced the optional + ``opt-`` tag in ``.pyc`` file names. The + :func:`importlib.util.cache_from_source` has gained an *optimization* + parameter to help control the ``opt-`` tag. Because of this, the + *debug_override* parameter of the function is now deprecated. `.pyo` files + are also no longer supported as a file argument to the Python interpreter and + thus serve no purpose when distributed on their own (i.e. sourcless code + distribution). Due to the fact that the magic number for bytecode has changed + in Python 3.5, all old `.pyo` files from previous versions of Python are + invalid regardless of this PEP. + +* The :mod:`socket` module now exports the CAN_RAW_FD_FRAMES constant on linux + 3.6 and greater. + +* The `pygettext.py` Tool now uses the standard +NNNN format for timezones in + the POT-Creation-Date header. + +* The :mod:`smtplib` module now uses :data:`sys.stderr` instead of previous + module level :data:`stderr` variable for debug output. If your (test) + program depends on patching the module level variable to capture the debug + output, you will need to update it to capture sys.stderr instead. + +* The :meth:`str.startswith` and :meth:`str.endswith` methods no longer return + ``True`` when finding the empty string and the indexes are completely out of + range. See :issue:`24284`. + +Changes in the C API +-------------------- + +* The undocumented :c:member:`~PyMemoryViewObject.format` member of the + (non-public) :c:type:`PyMemoryViewObject` structure has been removed. + + All extensions relying on the relevant parts in ``memoryobject.h`` + must be rebuilt. + +* The :c:type:`PyMemAllocator` structure was renamed to + :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. + +* Removed non-documented macro :c:macro:`PyObject_REPR` which leaked references. + Use format character ``%R`` in :c:func:`PyUnicode_FromFormat`-like functions + to format the :func:`repr` of the object. + +* Because the lack of the :attr:`__module__` attribute breaks pickling and + introspection, a deprecation warning now is raised for builtin type without + the :attr:`__module__` attribute. Would be an AttributeError in future. + (:issue:`20204`) + +* As part of :pep:`492` implementation, ``tp_reserved`` slot of + :c:type:`PyTypeObject` was replaced with a + :c:member:`tp_as_async` slot. Refer to :ref:`coro-objects` for + new types, structures and functions. diff --git a/Doc/whatsnew/index.rst b/Doc/whatsnew/index.rst index 29902e4..edb5502 100644 --- a/Doc/whatsnew/index.rst +++ b/Doc/whatsnew/index.rst @@ -11,6 +11,7 @@ anyone wishing to stay up-to-date after a new release. .. toctree:: :maxdepth: 2 + 3.5.rst 3.4.rst 3.3.rst 3.2.rst |
