diff options
Diffstat (limited to 'Doc')
121 files changed, 3503 insertions, 856 deletions
diff --git a/Doc/Makefile b/Doc/Makefile index 878685d..202e8e1 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -153,7 +153,7 @@ dist: cp -pPR build/epub/Python.epub dist/python-$(DISTVERSION)-docs.epub check: - $(PYTHON) tools/rstlint.py -i tools + $(PYTHON) tools/rstlint.py -i tools -i venv serve: ../Tools/scripts/serve.py build/html diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index d5e4703..f932997 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -223,8 +223,7 @@ which disallows mutable objects such as :class:`bytearray`. :c:func:`PyArg_ParseTuple` will use this location as the buffer and interpret the initial value of *\*buffer_length* as the buffer size. It will then copy the encoded data into the buffer and NUL-terminate it. If the buffer is not large - enough, a :exc:`TypeError` will be set. - Note: starting from Python 3.6 a :exc:`ValueError` will be set. + enough, a :exc:`ValueError` will be set. In both cases, *\*buffer_length* is set to the length of the encoded data without the trailing NUL byte. @@ -424,8 +423,15 @@ API Functions .. c:function:: int PyArg_ParseTupleAndKeywords(PyObject *args, PyObject *kw, const char *format, char *keywords[], ...) Parse the parameters of a function that takes both positional and keyword - parameters into local variables. Returns true on success; on failure, it - returns false and raises the appropriate exception. + parameters into local variables. The *keywords* argument is a + *NULL*-terminated array of keyword parameter names. Empty names denote + :ref:`positional-only parameters <positional-only_parameter>`. + Returns true on success; on failure, it returns false and raises the + appropriate exception. + + .. versionchanged:: 3.6 + Added support for :ref:`positional-only parameters + <positional-only_parameter>`. .. c:function:: int PyArg_VaParseTupleAndKeywords(PyObject *args, PyObject *kw, const char *format, char *keywords[], va_list vargs) diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index 19cbb3b..226b619 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -334,6 +334,14 @@ an error value). .. versionadded:: 3.2 +.. c:function:: int PyErr_ResourceWarning(PyObject *source, Py_ssize_t stack_level, const char *format, ...) + + Function similar to :c:func:`PyErr_WarnFormat`, but *category* is + :exc:`ResourceWarning` and pass *source* to :func:`warnings.WarningMessage`. + + .. versionadded:: 3.6 + + Querying the error indicator ============================ diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst index 2936f4f..5a083ce 100644 --- a/Doc/c-api/import.rst +++ b/Doc/c-api/import.rst @@ -207,13 +207,13 @@ Importing Modules .. c:function:: PyObject* PyImport_GetImporter(PyObject *path) - Return an importer object for a :data:`sys.path`/:attr:`pkg.__path__` item + Return a finder object for a :data:`sys.path`/:attr:`pkg.__path__` item *path*, possibly by fetching it from the :data:`sys.path_importer_cache` dict. If it wasn't yet cached, traverse :data:`sys.path_hooks` until a hook is found that can handle the path item. Return ``None`` if no hook could; - this tells our caller it should fall back to the built-in import mechanism. - Cache the result in :data:`sys.path_importer_cache`. Return a new reference - to the importer object. + this tells our caller that the :term:`path based finder` could not find a + finder for this path item. Cache the result in :data:`sys.path_importer_cache`. + Return a new reference to the finder object. .. c:function:: void _PyImport_Init() diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 639819c..d77836a 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -25,7 +25,7 @@ Initializing and finalizing the interpreter triple: module; search; path single: PySys_SetArgv() single: PySys_SetArgvEx() - single: Py_Finalize() + single: Py_FinalizeEx() Initialize the Python interpreter. In an application embedding Python, this should be called before using any other Python/C API functions; with the @@ -34,7 +34,7 @@ Initializing and finalizing the interpreter modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. It also initializes the module search path (``sys.path``). It does not set ``sys.argv``; use :c:func:`PySys_SetArgvEx` for that. This is a no-op when called for a second time - (without calling :c:func:`Py_Finalize` first). There is no return value; it is a + (without calling :c:func:`Py_FinalizeEx` first). There is no return value; it is a fatal error if the initialization fails. @@ -48,19 +48,20 @@ Initializing and finalizing the interpreter .. c:function:: int Py_IsInitialized() Return true (nonzero) when the Python interpreter has been initialized, false - (zero) if not. After :c:func:`Py_Finalize` is called, this returns false until + (zero) if not. After :c:func:`Py_FinalizeEx` is called, this returns false until :c:func:`Py_Initialize` is called again. -.. c:function:: void Py_Finalize() +.. c:function:: int Py_FinalizeEx() Undo all initializations made by :c:func:`Py_Initialize` and subsequent use of Python/C API functions, and destroy all sub-interpreters (see :c:func:`Py_NewInterpreter` below) that were created and not yet destroyed since the last call to :c:func:`Py_Initialize`. Ideally, this frees all memory allocated by the Python interpreter. This is a no-op when called for a second - time (without calling :c:func:`Py_Initialize` again first). There is no return - value; errors during finalization are ignored. + time (without calling :c:func:`Py_Initialize` again first). Normally the + return value is 0. If there were errors during finalization + (flushing buffered data), -1 is returned. This function is provided for a number of reasons. An embedding application might want to restart Python without having to restart the application itself. @@ -79,7 +80,15 @@ Initializing and finalizing the interpreter freed. Some memory allocated by extension modules may not be freed. Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls :c:func:`Py_Initialize` and - :c:func:`Py_Finalize` more than once. + :c:func:`Py_FinalizeEx` more than once. + + .. versionadded:: 3.6 + + +.. c:function:: void Py_Finalize() + + This is a backwards-compatible version of :c:func:`Py_FinalizeEx` that + disregards the return value. Process-wide parameters @@ -107,7 +116,7 @@ Process-wide parameters Note that :data:`sys.stderr` always uses the "backslashreplace" error handler, regardless of this (or any other) setting. - If :c:func:`Py_Finalize` is called, this function will need to be called + If :c:func:`Py_FinalizeEx` is called, this function will need to be called again in order to affect subsequent calls to :c:func:`Py_Initialize`. Returns 0 if successful, a nonzero value on error (e.g. calling after the @@ -918,7 +927,7 @@ using the following functions: entry.) .. index:: - single: Py_Finalize() + single: Py_FinalizeEx() single: Py_Initialize() Extension modules are shared between (sub-)interpreters as follows: the first @@ -928,7 +937,7 @@ using the following functions: and filled with the contents of this copy; the extension's ``init`` function is not called. Note that this is different from what happens when an extension is imported after the interpreter has been completely re-initialized by calling - :c:func:`Py_Finalize` and :c:func:`Py_Initialize`; in that case, the extension's + :c:func:`Py_FinalizeEx` and :c:func:`Py_Initialize`; in that case, the extension's ``initmodule`` function *is* called again. .. index:: single: close() (in module os) @@ -936,14 +945,14 @@ using the following functions: .. c:function:: void Py_EndInterpreter(PyThreadState *tstate) - .. index:: single: Py_Finalize() + .. index:: single: Py_FinalizeEx() Destroy the (sub-)interpreter represented by the given thread state. The given thread state must be the current thread state. See the discussion of thread states below. When the call returns, the current thread state is *NULL*. All thread states associated with this interpreter are destroyed. (The global interpreter lock must be held before calling this function and is still held - when it returns.) :c:func:`Py_Finalize` will destroy all sub-interpreters that + when it returns.) :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that haven't been explicitly destroyed at that point. diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst index bc3a752..74681d2 100644 --- a/Doc/c-api/intro.rst +++ b/Doc/c-api/intro.rst @@ -64,9 +64,10 @@ The header files are typically installed with Python. On Unix, these are located in the directories :file:`{prefix}/include/pythonversion/` and :file:`{exec_prefix}/include/pythonversion/`, where :envvar:`prefix` and :envvar:`exec_prefix` are defined by the corresponding parameters to Python's -:program:`configure` script and *version* is ``sys.version[:3]``. On Windows, -the headers are installed in :file:`{prefix}/include`, where :envvar:`prefix` is -the installation directory specified to the installer. +:program:`configure` script and *version* is +``'%d.%d' % sys.version_info[:2]``. On Windows, the headers are installed +in :file:`{prefix}/include`, where :envvar:`prefix` is the installation +directory specified to the installer. To include the headers, place both directories (if different) on your compiler's search path for includes. Do *not* place the parent directories on the search @@ -578,9 +579,9 @@ Sometimes, it is desirable to "uninitialize" Python. For instance, the application may want to start over (make another call to :c:func:`Py_Initialize`) or the application is simply done with its use of Python and wants to free memory allocated by Python. This can be accomplished -by calling :c:func:`Py_Finalize`. The function :c:func:`Py_IsInitialized` returns +by calling :c:func:`Py_FinalizeEx`. The function :c:func:`Py_IsInitialized` returns true if Python is currently in the initialized state. More information about -these functions is given in a later chapter. Notice that :c:func:`Py_Finalize` +these functions is given in a later chapter. Notice that :c:func:`Py_FinalizeEx` does *not* free all memory allocated by the Python interpreter, e.g. memory allocated by extension modules currently cannot be released. diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 290ef09..3ff5452 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -85,9 +85,12 @@ for the I/O buffer escapes completely the Python memory manager. .. seealso:: + The :envvar:`PYTHONMALLOC` environment variable can be used to configure + the memory allocators used by Python. + The :envvar:`PYTHONMALLOCSTATS` environment variable can be used to print - memory allocation statistics every time a new object arena is created, and - on shutdown. + statistics of the :ref:`pymalloc memory allocator <pymalloc>` every time a + new pymalloc object arena is created, and on shutdown. Raw Memory Interface @@ -162,15 +165,17 @@ The following function sets, modeled after the ANSI C standard, but specifying behavior when requesting zero bytes, are available for allocating and releasing memory from the Python heap. -The default memory block allocator uses the following functions: -:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call -``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes. +By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`. .. warning:: The :term:`GIL <global interpreter lock>` must be held when using these functions. +.. versionchanged:: 3.6 + + The default allocator is now pymalloc instead of system :c:func:`malloc`. + .. c:function:: void* PyMem_Malloc(size_t n) Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the @@ -292,15 +297,32 @@ Customize Memory Allocators Enum used to identify an allocator domain. Domains: - * :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`, - :c:func:`PyMem_RawRealloc`, :c:func:`PyMem_RawCalloc` and - :c:func:`PyMem_RawFree` - * :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`, - :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc` and :c:func:`PyMem_Free` - * :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`, - :c:func:`PyObject_Realloc`, :c:func:`PyObject_Calloc` and - :c:func:`PyObject_Free` + .. c:var:: PYMEM_DOMAIN_RAW + + Functions: + + * :c:func:`PyMem_RawMalloc` + * :c:func:`PyMem_RawRealloc` + * :c:func:`PyMem_RawCalloc` + * :c:func:`PyMem_RawFree` + + .. c:var:: PYMEM_DOMAIN_MEM + + Functions: + + * :c:func:`PyMem_Malloc`, + * :c:func:`PyMem_Realloc` + * :c:func:`PyMem_Calloc` + * :c:func:`PyMem_Free` + .. c:var:: PYMEM_DOMAIN_OBJ + + Functions: + + * :c:func:`PyObject_Malloc` + * :c:func:`PyObject_Realloc` + * :c:func:`PyObject_Calloc` + * :c:func:`PyObject_Free` .. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) @@ -325,43 +347,62 @@ Customize Memory Allocators .. c:function:: void PyMem_SetupDebugHooks(void) - Setup hooks to detect bugs in the following Python memory allocator - functions: - - - :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`, - :c:func:`PyMem_RawCalloc`, :c:func:`PyMem_RawFree` - - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc`, - :c:func:`PyMem_Free` - - :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`, - :c:func:`PyObject_Calloc`, :c:func:`PyObject_Free` + Setup hooks to detect bugs in the Python memory allocator functions. Newly allocated memory is filled with the byte ``0xCB``, freed memory is - filled with the byte ``0xDB``. Additional checks: + filled with the byte ``0xDB``. - - detect API violations, ex: :c:func:`PyObject_Free` called on a buffer + Runtime checks: + + - Detect API violations, ex: :c:func:`PyObject_Free` called on a buffer allocated by :c:func:`PyMem_Malloc` - - detect write before the start of the buffer (buffer underflow) - - detect write after the end of the buffer (buffer overflow) + - Detect write before the start of the buffer (buffer underflow) + - Detect write after the end of the buffer (buffer overflow) + - Check that the :term:`GIL <global interpreter lock>` is held when + allocator functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex: + :c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_MEM` (ex: + :c:func:`PyMem_Malloc`) domains are called + + On error, the debug hooks use the :mod:`tracemalloc` module to get the + traceback where a memory block was allocated. The traceback is only + displayed if :mod:`tracemalloc` is tracing Python memory allocations and the + memory block was traced. - The function does nothing if Python is not compiled is debug mode. + These hooks are installed by default if Python is compiled in debug + mode. The :envvar:`PYTHONMALLOC` environment variable can be used to install + debug hooks on a Python compiled in release mode. + .. versionchanged:: 3.6 + This function now also works on Python compiled in release mode. + On error, the debug hooks now use :mod:`tracemalloc` to get the traceback + where a memory block was allocated. The debug hooks now also check + if the GIL is held when functions of :c:data:`PYMEM_DOMAIN_OBJ` and + :c:data:`PYMEM_DOMAIN_MEM` domains are called. -Customize PyObject Arena Allocator -================================== -Python has a *pymalloc* allocator for allocations smaller than 512 bytes. This -allocator is optimized for small objects with a short lifetime. It uses memory -mappings called "arenas" with a fixed size of 256 KB. It falls back to -:c:func:`PyMem_RawMalloc` and :c:func:`PyMem_RawRealloc` for allocations larger -than 512 bytes. *pymalloc* is the default allocator used by -:c:func:`PyObject_Malloc`. +.. _pymalloc: -The default arena allocator uses the following functions: +The pymalloc allocator +====================== + +Python has a *pymalloc* allocator optimized for small objects (smaller or equal +to 512 bytes) with a short lifetime. It uses memory mappings called "arenas" +with a fixed size of 256 KB. It falls back to :c:func:`PyMem_RawMalloc` and +:c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes. + +*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_MEM` (ex: +:c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_OBJ` (ex: +:c:func:`PyObject_Malloc`) domains. + +The arena allocator uses the following functions: * :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows, * :c:func:`mmap` and :c:func:`munmap` if available, * :c:func:`malloc` and :c:func:`free` otherwise. +Customize pymalloc Arena Allocator +---------------------------------- + .. versionadded:: 3.4 .. c:type:: PyObjectArenaAllocator diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index cc84314..e9e8add 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -150,8 +150,9 @@ specific C type of the *self* object. The :attr:`ml_flags` field is a bitfield which can include the following flags. The individual flags indicate either a calling convention or a binding convention. Of the calling convention flags, only :const:`METH_VARARGS` and -:const:`METH_KEYWORDS` can be combined. Any of the calling convention flags -can be combined with a binding flag. +:const:`METH_KEYWORDS` can be combined (but note that :const:`METH_KEYWORDS` +alone is equivalent to ``METH_VARARGS | METH_KEYWORDS``). Any of the calling +convention flags can be combined with a binding flag. .. data:: METH_VARARGS diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index 3d83b27..035cdc1 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -5,6 +5,17 @@ Operating System Utilities ========================== +.. c:function:: PyObject* PyOS_FSPath(PyObject *path) + + Return the file system representation for *path*. If the object is a + :class:`str` or :class:`bytes` object, then its reference count is + incremented. If the object implements the :class:`os.PathLike` interface, + then :meth:`~os.PathLike.__fspath__` is returned as long as it is a + :class:`str` or :class:`bytes` object. Otherwise :exc:`TypeError` is raised + and ``NULL`` is returned. + + .. versionadded:: 3.6 + .. c:function:: int Py_FdIsInteractive(FILE *fp, const char *filename) @@ -212,20 +223,24 @@ Process Control .. c:function:: void Py_Exit(int status) .. index:: - single: Py_Finalize() + single: Py_FinalizeEx() single: exit() - Exit the current process. This calls :c:func:`Py_Finalize` and then calls the - standard C library function ``exit(status)``. + Exit the current process. This calls :c:func:`Py_FinalizeEx` and then calls the + standard C library function ``exit(status)``. If :c:func:`Py_FinalizeEx` + indicates an error, the exit status is set to 120. + + .. versionchanged:: 3.6 + Errors from finalization no longer ignored. .. c:function:: int Py_AtExit(void (*func) ()) .. index:: - single: Py_Finalize() + single: Py_FinalizeEx() single: cleanup functions - Register a cleanup function to be called by :c:func:`Py_Finalize`. The cleanup + Register a cleanup function to be called by :c:func:`Py_FinalizeEx`. The cleanup function will be called with no arguments and should return no value. At most 32 cleanup functions can be registered. When the registration is successful, :c:func:`Py_AtExit` returns ``0``; on failure, it returns ``-1``. The cleanup diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index e388195..8b469f4 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -921,6 +921,9 @@ PyNumber_Xor:PyObject*:o2:0: PyObject_AsFileDescriptor:int::: PyObject_AsFileDescriptor:PyObject*:o:0: +PyOS_FSPath:PyObject*::+1: +PyOS_FSPath:PyObject*:path:0: + PyObject_Call:PyObject*::+1: PyObject_Call:PyObject*:callable_object:0: PyObject_Call:PyObject*:args:0: diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst index 64033dc..ab2f616 100644 --- a/Doc/extending/embedding.rst +++ b/Doc/extending/embedding.rst @@ -67,7 +67,9 @@ perform some operation on a file. :: Py_Initialize(); PyRun_SimpleString("from time import time,ctime\n" "print('Today is', ctime(time()))\n"); - Py_Finalize(); + if (Py_FinalizeEx() < 0) { + exit(120); + } PyMem_RawFree(program); return 0; } @@ -76,7 +78,7 @@ The :c:func:`Py_SetProgramName` function should be called before :c:func:`Py_Initialize` to inform the interpreter about paths to Python run-time libraries. Next, the Python interpreter is initialized with :c:func:`Py_Initialize`, followed by the execution of a hard-coded Python script -that prints the date and time. Afterwards, the :c:func:`Py_Finalize` call shuts +that prints the date and time. Afterwards, the :c:func:`Py_FinalizeEx` call shuts the interpreter down, followed by the end of the program. In a real program, you may want to get the Python script from another source, perhaps a text-editor routine, a file, or a database. Getting the Python code from a file can better diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 694753e..9c5e20d 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -838,7 +838,8 @@ How do I convert a number to a string? To convert, e.g., the number 144 to the string '144', use the built-in type constructor :func:`str`. If you want a hexadecimal or octal representation, use the built-in functions :func:`hex` or :func:`oct`. For fancy formatting, see -the :ref:`formatstrings` section, e.g. ``"{:04d}".format(144)`` yields +the :ref:`f-strings` and :ref:`formatstrings` sections, +e.g. ``"{:04d}".format(144)`` yields ``'0144'`` and ``"{:.3f}".format(1.0/3.0)`` yields ``'0.333'``. diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 45b794f..3d05c14 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -718,6 +718,8 @@ Glossary def func(foo, bar=None): ... + .. _positional-only_parameter: + * :dfn:`positional-only`: specifies an argument that can be supplied only by position. Python has no syntax for defining positional-only parameters. However, some built-in functions have positional-only @@ -776,6 +778,16 @@ Glossary One of the default :term:`meta path finders <meta path finder>` which searches an :term:`import path` for modules. + path-like object + An object representing a file system path. A path-like object is either + a :class:`str` or :class:`bytes` object representing a path, or an object + implementing the :class:`os.PathLike` protocol. An object that supports + the :class:`os.PathLike` protocol can be converted to a :class:`str` or + :class:`bytes` file system path by calling the :func:`os.fspath` function; + :func:`os.fsdecode` and :func:`os.fsencode` can be used to guarantee a + :class:`str` or :class:`bytes` result instead, respectively. Introduced + by :pep:`519`. + portion A set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package, as defined in :pep:`420`. @@ -958,7 +970,7 @@ Glossary without interfering with the behaviour of other Python applications running on the same system. - See also :ref:`scripts-pyvenv`. + See also :mod:`venv`. virtual machine A computer defined entirely in software. Python's virtual machine diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst index cfe9868..7a60165 100644 --- a/Doc/howto/argparse.rst +++ b/Doc/howto/argparse.rst @@ -547,7 +547,8 @@ And this is what it gives: Traceback (most recent call last): File "prog.py", line 11, in <module> if args.verbosity >= 2: - TypeError: unorderable types: NoneType() >= int() + TypeError: '>=' not supported between instances of 'NoneType' and 'int' + * First output went well, and fixes the bug we had before. That is, we want any value >= 2 to be as verbose as possible. diff --git a/Doc/includes/run-func.c b/Doc/includes/run-func.c index 986d670..ead7bdd 100644 --- a/Doc/includes/run-func.c +++ b/Doc/includes/run-func.c @@ -63,6 +63,8 @@ main(int argc, char *argv[]) fprintf(stderr, "Failed to load \"%s\"\n", argv[1]); return 1; } - Py_Finalize(); + if (Py_FinalizeEx() < 0) { + return 120; + } return 0; } diff --git a/Doc/includes/test.py b/Doc/includes/test.py index 7ebf46a..9e9d4a6 100644 --- a/Doc/includes/test.py +++ b/Doc/includes/test.py @@ -204,7 +204,7 @@ Test cyclic gc(?) import os import sys from distutils.util import get_platform -PLAT_SPEC = "%s-%s" % (get_platform(), sys.version[0:3]) +PLAT_SPEC = "%s-%d.%d" % (get_platform(), *sys.version_info[:2]) src = os.path.join("build", "lib.%s" % PLAT_SPEC) sys.path.append(src) diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst index 1ef3149..b22465d 100644 --- a/Doc/installing/index.rst +++ b/Doc/installing/index.rst @@ -2,9 +2,9 @@ .. _installing-index: -***************************** - Installing Python Modules -***************************** +************************* +Installing Python Modules +************************* :Email: distutils-sig@python.org @@ -34,24 +34,24 @@ Key terms * ``pip`` is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers. -* a virtual environment is a semi-isolated Python environment that allows +* A *virtual environment* is a semi-isolated Python environment that allows packages to be installed for use by a particular application, rather than - being installed system wide -* ``pyvenv`` is the standard tool for creating virtual environments, and has + being installed system wide. +* ``venv`` is the standard tool for creating virtual environments, and has been part of Python since Python 3.3. Starting with Python 3.4, it - defaults to installing ``pip`` into all created virtual environments + defaults to installing ``pip`` into all created virtual environments. * ``virtualenv`` is a third party alternative (and predecessor) to - ``pyvenv``. It allows virtual environments to be used on versions of - Python prior to 3.4, which either don't provide ``pyvenv`` at all, or + ``venv``. It allows virtual environments to be used on versions of + Python prior to 3.4, which either don't provide ``venv`` at all, or aren't able to automatically install ``pip`` into created environments. -* the `Python Packaging Index <https://pypi.python.org/pypi>`__ is a public +* The `Python Packaging Index <https://pypi.python.org/pypi>`__ is a public repository of open source licensed packages made available for use by - other Python users + other Python users. * the `Python Packaging Authority <https://www.pypa.io/en/latest/>`__ are the group of developers and documentation authors responsible for the maintenance and evolution of the standard packaging tools and the associated metadata and - file format standards. They maintain a variety of tools, documentation + file format standards. They maintain a variety of tools, documentation, and issue trackers on both `GitHub <https://github.com/pypa>`__ and `BitBucket <https://bitbucket.org/pypa/>`__. * ``distutils`` is the original build and distribution system first added to @@ -62,6 +62,19 @@ Key terms of the mailing list used to coordinate Python packaging standards development). +.. deprecated:: 3.6 + ``pyvenv`` was the recommended tool for creating virtual environments for + Python 3.3 and 3.4, and is `deprecated in Python 3.6 + <https://docs.python.org/dev/whatsnew/3.6.html#deprecated-features>`_. + +.. versionchanged:: 3.5 + The use of ``venv`` is now recommended for creating virtual environments. + +.. seealso:: + + `Python Packaging User Guide: Creating and using virtual environments + <https://packaging.python.org/installing/#creating-virtual-environments>`__ + Basic usage =========== @@ -100,13 +113,14 @@ explicitly:: More information and resources regarding ``pip`` and its capabilities can be found in the `Python Packaging User Guide <https://packaging.python.org>`__. -``pyvenv`` has its own documentation at :ref:`scripts-pyvenv`. Installing -into an active virtual environment uses the commands shown above. +Creation of virtual environments is done through the :mod:`venv` module. +Installing packages into an active virtual environment uses the commands shown +above. .. seealso:: `Python Packaging User Guide: Installing Python Distribution Packages - <https://packaging.python.org/en/latest/installing/>`__ + <https://packaging.python.org/installing/>`__ How do I ...? @@ -124,7 +138,7 @@ User Guide. .. seealso:: `Python Packaging User Guide: Requirements for Installing Packages - <https://packaging.python.org/en/latest/installing/#requirements-for-installing-packages>`__ + <https://packaging.python.org/installing/#requirements-for-installing-packages>`__ .. installing-per-user-installation: @@ -142,20 +156,19 @@ package just for the current user, rather than for all users of the system. A number of scientific Python packages have complex binary dependencies, and aren't currently easy to install using ``pip`` directly. At this point in time, it will often be easier for users to install these packages by -`other means -<https://packaging.python.org/en/latest/science/>`__ +`other means <https://packaging.python.org/science/>`__ rather than attempting to install them with ``pip``. .. seealso:: `Python Packaging User Guide: Installing Scientific Packages - <https://packaging.python.org/en/latest/science/>`__ + <https://packaging.python.org/science/>`__ ... work with multiple versions of Python installed in parallel? ---------------------------------------------------------------- -On Linux, Mac OS X and other POSIX systems, use the versioned Python commands +On Linux, Mac OS X, and other POSIX systems, use the versioned Python commands in combination with the ``-m`` switch to run the appropriate copy of ``pip``:: @@ -164,7 +177,7 @@ in combination with the ``-m`` switch to run the appropriate copy of python3 -m pip install SomePackage # default Python 3 python3.4 -m pip install SomePackage # specifically Python 3.4 -(appropriately versioned ``pip`` commands may also be available) +Appropriately versioned ``pip`` commands may also be available. On Windows, use the ``py`` Python launcher in combination with the ``-m`` switch:: @@ -212,11 +225,11 @@ as users are more regularly able to install pre-built extensions rather than needing to build them themselves. Some of the solutions for installing `scientific software -<https://packaging.python.org/en/latest/science/>`__ -that is not yet available as pre-built ``wheel`` files may also help with +<https://packaging.python.org/science/>`__ +that are not yet available as pre-built ``wheel`` files may also help with obtaining other binary extensions without needing to build them locally. .. seealso:: `Python Packaging User Guide: Binary Extensions - <https://packaging.python.org/en/latest/extensions/>`__ + <https://packaging.python.org/extensions/>`__ diff --git a/Doc/library/asynchat.rst b/Doc/library/asynchat.rst index ae72d26..e02360c 100644 --- a/Doc/library/asynchat.rst +++ b/Doc/library/asynchat.rst @@ -58,13 +58,13 @@ connection requests. The asynchronous output buffer size (default ``4096``). Unlike :class:`asyncore.dispatcher`, :class:`async_chat` allows you to - define a first-in-first-out queue (fifo) of *producers*. A producer need + define a :abbr:`FIFO (first-in, first-out)` queue of *producers*. A producer need have only one method, :meth:`more`, which should return data to be transmitted on the channel. The producer indicates exhaustion (*i.e.* that it contains no more data) by having its :meth:`more` method return the empty bytes object. At this point - the :class:`async_chat` object removes the producer from the fifo and starts - using the next producer, if any. When the producer fifo is empty the + the :class:`async_chat` object removes the producer from the queue and starts + using the next producer, if any. When the producer queue is empty the :meth:`handle_write` method does nothing. You use the channel object's :meth:`set_terminator` method to describe how to recognize the end of, or an important breakpoint in, an incoming transmission from the remote @@ -78,8 +78,8 @@ connection requests. .. method:: async_chat.close_when_done() - Pushes a ``None`` on to the producer fifo. When this producer is popped off - the fifo it causes the channel to be closed. + Pushes a ``None`` on to the producer queue. When this producer is popped off + the queue it causes the channel to be closed. .. method:: async_chat.collect_incoming_data(data) @@ -92,7 +92,7 @@ connection requests. .. method:: async_chat.discard_buffers() In emergencies this method will discard any data held in the input and/or - output buffers and the producer fifo. + output buffers and the producer queue. .. method:: async_chat.found_terminator() @@ -110,7 +110,7 @@ connection requests. .. method:: async_chat.push(data) - Pushes data on to the channel's fifo to ensure its transmission. + Pushes data on to the channel's queue to ensure its transmission. This is all you need to do to have the channel write the data out to the network, although it is possible to use your own producers in more complex schemes to implement encryption and chunking, for example. @@ -118,7 +118,7 @@ connection requests. .. method:: async_chat.push_with_producer(producer) - Takes a producer object and adds it to the producer fifo associated with + Takes a producer object and adds it to the producer queue associated with the channel. When all currently-pushed producers have been exhausted the channel will consume this producer's data by calling its :meth:`more` method and send the data to the remote endpoint. diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index eed4f08..6706001 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -110,8 +110,9 @@ keywords to your callback, use :func:`functools.partial`. For example, called after :meth:`call_soon` returns, when control returns to the event loop. - This operates as a FIFO queue, callbacks are called in the order in - which they are registered. Each callback will be called exactly once. + This operates as a :abbr:`FIFO (first-in, first-out)` queue, callbacks + are called in the order in which they are registered. Each callback + will be called exactly once. Any positional arguments after the callback will be passed to the callback when it is called. diff --git a/Doc/library/binascii.rst b/Doc/library/binascii.rst index 878d8db..4f5f0f2 100644 --- a/Doc/library/binascii.rst +++ b/Doc/library/binascii.rst @@ -53,13 +53,14 @@ The :mod:`binascii` module defines the following functions: than one line may be passed at a time. -.. function:: b2a_base64(data) +.. function:: b2a_base64(data, \*, newline=True) Convert binary data to a line of ASCII characters in base64 coding. The return - value is the converted line, including a newline char. The newline is - added because the original use case for this function was to feed it a - series of 57 byte input lines to get output lines that conform to the - MIME-base64 standard. Otherwise the output conforms to :rfc:`3548`. + value is the converted line, including a newline char if *newline* is + true. The output of this function conforms to :rfc:`3548`. + + .. versionchanged:: 3.6 + Added the *newline* parameter. .. function:: a2b_qp(data, header=False) diff --git a/Doc/library/cmath.rst b/Doc/library/cmath.rst index 62ddb6b..85393dc 100644 --- a/Doc/library/cmath.rst +++ b/Doc/library/cmath.rst @@ -253,6 +253,12 @@ Constants The mathematical constant *e*, as a float. +.. data:: tau + + The mathematical constant *τ*, as a float. + + .. versionadded:: 3.6 + .. index:: module: math Note that the selection of functions is similar, but not identical, to that in @@ -276,5 +282,3 @@ cuts for numerical purposes, a good reference should be the following: Kahan, W: Branch cuts for complex elementary functions; or, Much ado about nothing's sign bit. In Iserles, A., and Powell, M. (eds.), The state of the art in numerical analysis. Clarendon Press (1987) pp165-211. - - diff --git a/Doc/library/code.rst b/Doc/library/code.rst index 443af69..4cce1fa 100644 --- a/Doc/library/code.rst +++ b/Doc/library/code.rst @@ -30,15 +30,19 @@ build applications which provide an interactive interpreter prompt. ``sys.ps1`` and ``sys.ps2``, and input buffering. -.. function:: interact(banner=None, readfunc=None, local=None) +.. function:: interact(banner=None, readfunc=None, local=None, exitmsg=None) Convenience function to run a read-eval-print loop. This creates a new instance of :class:`InteractiveConsole` and sets *readfunc* to be used as the :meth:`InteractiveConsole.raw_input` method, if provided. If *local* is provided, it is passed to the :class:`InteractiveConsole` constructor for use as the default namespace for the interpreter loop. The :meth:`interact` - method of the instance is then run with *banner* passed as the banner to - use, if provided. The console object is discarded after use. + method of the instance is then run with *banner* and *exitmsg* passed as the + banner and exit message to use, if provided. The console object is discarded + after use. + + .. versionchanged:: 3.6 + Added *exitmsg* parameter. .. function:: compile_command(source, filename="<input>", symbol="single") @@ -136,7 +140,7 @@ The :class:`InteractiveConsole` class is a subclass of interpreter objects as well as the following additions. -.. method:: InteractiveConsole.interact(banner=None) +.. method:: InteractiveConsole.interact(banner=None, exitmsg=None) Closely emulate the interactive Python console. The optional *banner* argument specify the banner to print before the first interaction; by default it prints a @@ -144,9 +148,16 @@ interpreter objects as well as the following additions. by the class name of the console object in parentheses (so as not to confuse this with the real interpreter -- since it's so close!). + The optional *exitmsg* argument specifies an exit message printed when exiting. + Pass the empty string to suppress the exit message. If *exitmsg* is not given or + None, a default message is printed. + .. versionchanged:: 3.4 To suppress printing any banner, pass an empty string. + .. versionchanged:: 3.6 + Print an exit message when exiting. + .. method:: InteractiveConsole.push(line) diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index aeb6a73..e4b75a0 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -41,13 +41,16 @@ ABC Inherits from Abstract Methods Mixin :class:`Hashable` ``__hash__`` :class:`Iterable` ``__iter__`` :class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__`` +:class:`Reversible` :class:`Iterable` ``__reversed__`` :class:`Generator` :class:`Iterator` ``send``, ``throw`` ``close``, ``__iter__``, ``__next__`` :class:`Sized` ``__len__`` :class:`Callable` ``__call__`` +:class:`Collection` :class:`Sized`, ``__contains__``, + :class:`Iterable`, ``__iter__``, + :class:`Container` ``__len__`` -:class:`Sequence` :class:`Sized`, ``__getitem__``, ``__contains__``, ``__iter__``, ``__reversed__``, - :class:`Iterable`, ``__len__`` ``index``, and ``count`` - :class:`Container` +:class:`Sequence` :class:`Reversible`, ``__getitem__``, ``__contains__``, ``__iter__``, ``__reversed__``, + :class:`Collection` ``__len__`` ``index``, and ``count`` :class:`MutableSequence` :class:`Sequence` ``__getitem__``, Inherited :class:`Sequence` methods and ``__setitem__``, ``append``, ``reverse``, ``extend``, ``pop``, @@ -58,9 +61,9 @@ ABC Inherits from Abstract Methods Mixin :class:`ByteString` :class:`Sequence` ``__getitem__``, Inherited :class:`Sequence` methods ``__len__`` -:class:`Set` :class:`Sized`, ``__contains__``, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``, - :class:`Iterable`, ``__iter__``, ``__gt__``, ``__ge__``, ``__and__``, ``__or__``, - :class:`Container` ``__len__`` ``__sub__``, ``__xor__``, and ``isdisjoint`` +:class:`Set` :class:`Collection` ``__contains__``, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``, + ``__iter__``, ``__gt__``, ``__ge__``, ``__and__``, ``__or__``, + ``__len__`` ``__sub__``, ``__xor__``, and ``isdisjoint`` :class:`MutableSet` :class:`Set` ``__contains__``, Inherited :class:`Set` methods and ``__iter__``, ``clear``, ``pop``, ``remove``, ``__ior__``, @@ -68,9 +71,9 @@ ABC Inherits from Abstract Methods Mixin ``add``, ``discard`` -:class:`Mapping` :class:`Sized`, ``__getitem__``, ``__contains__``, ``keys``, ``items``, ``values``, - :class:`Iterable`, ``__iter__``, ``get``, ``__eq__``, and ``__ne__`` - :class:`Container` ``__len__`` +:class:`Mapping` :class:`Collection` ``__getitem__``, ``__contains__``, ``keys``, ``items``, ``values``, + ``__iter__``, ``get``, ``__eq__``, and ``__ne__`` + ``__len__`` :class:`MutableMapping` :class:`Mapping` ``__getitem__``, Inherited :class:`Mapping` methods and ``__setitem__``, ``pop``, ``popitem``, ``clear``, ``update``, @@ -105,12 +108,25 @@ ABC Inherits from Abstract Methods Mixin ABC for classes that provide the :meth:`__iter__` method. See also the definition of :term:`iterable`. +.. class:: Collection + + ABC for sized iterable container classes. + + .. versionadded:: 3.6 + .. class:: Iterator ABC for classes that provide the :meth:`~iterator.__iter__` and :meth:`~iterator.__next__` methods. See also the definition of :term:`iterator`. +.. class:: Reversible + + ABC for iterable classes that also provide the :meth:`__reversed__` + method. + + .. versionadded:: 3.6 + .. class:: Generator ABC for generator classes that implement the protocol defined in diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index a147287..6daee6f 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -763,7 +763,7 @@ Named tuples assign meaning to each position in a tuple and allow for more reada self-documenting code. They can be used wherever regular tuples are used, and they add the ability to access fields by name instead of position index. -.. function:: namedtuple(typename, field_names, verbose=False, rename=False) +.. function:: namedtuple(typename, field_names, *, verbose=False, rename=False) Returns a new tuple subclass named *typename*. The new subclass is used to create tuple-like objects that have fields accessible by attribute lookup as @@ -794,7 +794,11 @@ they add the ability to access fields by name instead of position index. lightweight and require no more memory than regular tuples. .. versionchanged:: 3.1 - Added support for *rename*. + Added support for *rename*. + + .. versionchanged:: 3.6 + The *verbose* and *rename* parameters became + :ref:`keyword-only arguments <keyword-only_parameter>`. .. doctest:: @@ -846,7 +850,9 @@ field names, the method and attribute names start with an underscore. .. method:: somenamedtuple._asdict() Return a new :class:`OrderedDict` which maps field names to their corresponding - values:: + values: + + .. doctest:: >>> p = Point(x=11, y=22) >>> p._asdict() @@ -908,7 +914,9 @@ Since a named tuple is a regular Python class, it is easy to add or change functionality with a subclass. Here is how to add a calculated field and a fixed-width print format: - >>> class Point(namedtuple('Point', 'x y')): +.. doctest:: + + >>> class Point(namedtuple('Point', ['x', 'y'])): ... __slots__ = () ... @property ... def hypot(self): @@ -959,6 +967,11 @@ customize a prototype instance: constructor that is convenient for use cases where named tuples are being subclassed. + * See :meth:`types.SimpleNamespace` for a mutable namespace based on an + underlying dictionary instead of a tuple. + + * See :meth:`typing.NamedTuple` for a way to add type hints for named tuples. + :class:`OrderedDict` objects ---------------------------- @@ -980,8 +993,9 @@ the items are returned in the order their keys were first added. .. method:: popitem(last=True) The :meth:`popitem` method for ordered dictionaries returns and removes a - (key, value) pair. The pairs are returned in LIFO order if *last* is true - or FIFO order if false. + (key, value) pair. The pairs are returned in + :abbr:`LIFO (last-in, first-out)` order if *last* is true + or :abbr:`FIFO (first-in, first-out)` order if false. .. method:: move_to_end(key, last=True) diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 511c581..91bdd18 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -102,7 +102,8 @@ Public functions .. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1) Recursively descend the directory tree named by *dir*, compiling all :file:`.py` - files along the way. + files along the way. Return a true value if all the files compiled successfully, + and a false value otherwise. The *maxlevels* parameter is used to limit the depth of the recursion; it defaults to ``10``. @@ -154,7 +155,8 @@ Public functions .. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1) - Compile the file with path *fullname*. + Compile the file with path *fullname*. Return a true value if the file + compiled successfully, and a false value otherwise. If *ddir* is given, it is prepended to the path to the file being compiled for use in compilation time tracebacks, and is also compiled in to the @@ -190,8 +192,10 @@ Public functions .. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1) - Byte-compile all the :file:`.py` files found along ``sys.path``. If - *skip_curdir* is true (the default), the current directory is not included + Byte-compile all the :file:`.py` files found along ``sys.path``. Return a + true value if all the files compiled successfully, and a false value otherwise. + + If *skip_curdir* is true (the default), the current directory is not included in the search. All other parameters are passed to the :func:`compile_dir` function. Note that unlike the other compile functions, ``maxlevels`` defaults to ``0``. diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index ae03f4b..d85576b 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -124,7 +124,7 @@ And:: executor.submit(wait_on_future) -.. class:: ThreadPoolExecutor(max_workers=None) +.. class:: ThreadPoolExecutor(max_workers=None, thread_name_prefix='') An :class:`Executor` subclass that uses a pool of at most *max_workers* threads to execute calls asynchronously. @@ -137,6 +137,10 @@ And:: should be higher than the number of workers for :class:`ProcessPoolExecutor`. + .. versionadded:: 3.6 + The *thread_name_prefix* argument was added to allow users to + control the threading.Thread names for worker threads created by + the pool for easier debugging. .. _threadpoolexecutor-example: diff --git a/Doc/library/constants.rst b/Doc/library/constants.rst index d5a0f09..f0742ce 100644 --- a/Doc/library/constants.rst +++ b/Doc/library/constants.rst @@ -33,16 +33,22 @@ A small number of constants live in the built-in namespace. They are: (e.g. :meth:`__imul__`, :meth:`__iand__`, etc.) for the same purpose. Its truth value is true. -.. note:: + .. note:: + + When a binary (or in-place) method returns ``NotImplemented`` the + interpreter will try the reflected operation on the other type (or some + other fallback, depending on the operator). If all attempts return + ``NotImplemented``, the interpreter will raise an appropriate exception. + Incorrectly returning ``NotImplemented`` will result in a misleading + error message or the ``NotImplemented`` value being returned to Python code. + + See :ref:`implementing-the-arithmetic-operations` for examples. - When ``NotImplemented`` is returned, the interpreter will then try the - reflected operation on the other type, or some other fallback, depending - on the operator. If all attempted operations return ``NotImplemented``, the - interpreter will raise an appropriate exception. + .. note:: - See - :ref:`implementing-the-arithmetic-operations` - for more details. + ``NotImplentedError`` and ``NotImplemented`` are not interchangeable, + even though they have similar names and purposes. + See :exc:`NotImplementedError` for details on when to use it. .. data:: Ellipsis diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index cf85fcd..810cea8 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -18,6 +18,18 @@ Utilities Functions and classes provided: +.. class:: AbstractContextManager + + An :term:`abstract base class` for classes that implement + :meth:`object.__enter__` and :meth:`object.__exit__`. A default + implementation for :meth:`object.__enter__` is provided which returns + ``self`` while :meth:`object.__exit__` is an abstract method which by default + returns ``None``. See also the definition of :ref:`typecontextmanager`. + + .. versionadded:: 3.6 + + + .. decorator:: contextmanager This function is a :term:`decorator` that can be used to define a factory @@ -447,9 +459,9 @@ Here's an example of doing this for a context manager that accepts resource acquisition and release functions, along with an optional validation function, and maps them to the context management protocol:: - from contextlib import contextmanager, ExitStack + from contextlib import contextmanager, AbstractContextManager, ExitStack - class ResourceManager: + class ResourceManager(AbstractContextManager): def __init__(self, acquire_resource, release_resource, check_resource_ok=None): self.acquire_resource = acquire_resource diff --git a/Doc/library/crypt.rst b/Doc/library/crypt.rst index a21c1e7..dbd4274 100644 --- a/Doc/library/crypt.rst +++ b/Doc/library/crypt.rst @@ -68,7 +68,7 @@ Module Attributes A list of available password hashing algorithms, as ``crypt.METHOD_*`` objects. This list is sorted from strongest to - weakest, and is guaranteed to have at least ``crypt.METHOD_CRYPT``. + weakest. Module Functions diff --git a/Doc/library/crypto.rst b/Doc/library/crypto.rst index 1eddfdc..ae45549 100644 --- a/Doc/library/crypto.rst +++ b/Doc/library/crypto.rst @@ -16,3 +16,4 @@ Here's an overview: hashlib.rst hmac.rst + secrets.rst diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index a675790..2870940 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -1239,9 +1239,10 @@ When programming in a compiled language, shared libraries are accessed when compiling/linking a program, and when the program is run. The purpose of the :func:`find_library` function is to locate a library in a way -similar to what the compiler does (on platforms with several versions of a -shared library the most recent should be loaded), while the ctypes library -loaders act like when a program is run, and call the runtime loader directly. +similar to what the compiler or runtime loader does (on platforms with several +versions of a shared library the most recent should be loaded), while the ctypes +library loaders act like when a program is run, and call the runtime loader +directly. The :mod:`ctypes.util` module provides a function which can help to determine the library to load. @@ -1259,8 +1260,14 @@ the library to load. The exact functionality is system dependent. On Linux, :func:`find_library` tries to run external programs -(``/sbin/ldconfig``, ``gcc``, and ``objdump``) to find the library file. It -returns the filename of the library file. Here are some examples:: +(``/sbin/ldconfig``, ``gcc``, ``objdump`` and ``ld``) to find the library file. +It returns the filename of the library file. + +.. versionchanged:: 3.6 + On Linux, the value of the environment variable ``LD_LIBRARY_PATH`` is used + when searching for libraries, if a library cannot be found by any other means. + +Here are some examples:: >>> from ctypes.util import find_library >>> find_library("m") diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 9254ae8..a286fbe 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -610,7 +610,8 @@ Instance methods: .. method:: date.__format__(format) Same as :meth:`.date.strftime`. This makes it possible to specify a format - string for a :class:`.date` object when using :meth:`str.format`. For a + string for a :class:`.date` object in :ref:`formatted string + literals <f-strings>` and when using :meth:`str.format`. For a complete list of formatting directives, see :ref:`strftime-strptime-behavior`. @@ -793,16 +794,23 @@ Other constructors, all class methods: microsecond of the result are all 0, and :attr:`.tzinfo` is ``None``. -.. classmethod:: datetime.combine(date, time) +.. classmethod:: datetime.combine(date, time[, tzinfo]) Return a new :class:`.datetime` object whose date components are equal to the - given :class:`date` object's, and whose time components and :attr:`.tzinfo` - attributes are equal to the given :class:`.time` object's. For any - :class:`.datetime` object *d*, - ``d == datetime.combine(d.date(), d.timetz())``. If date is a + given :class:`date` object's, and whose time components + are equal to the given :class:`.time` object's. If the *tzinfo* + argument is provided, its value is used to set the :attr:`.tzinfo` attribute + of the result, otherwise the :attr:`~.time.tzinfo` attribute of the *time* argument + is used. + + For any :class:`.datetime` object *d*, + ``d == datetime.combine(d.date(), d.time(), d.tzinfo)``. If date is a :class:`.datetime` object, its time components and :attr:`.tzinfo` attributes are ignored. + .. versionchanged:: 3.6 + Added the *tzinfo* argument. + .. classmethod:: datetime.strptime(date_string, format) @@ -1138,7 +1146,7 @@ Instance methods: ``self.date().isocalendar()``. -.. method:: datetime.isoformat(sep='T') +.. method:: datetime.isoformat(sep='T', timespec='auto') Return a string representing the date and time in ISO 8601 format, YYYY-MM-DDTHH:MM:SS.mmmmmm or, if :attr:`microsecond` is 0, @@ -1159,6 +1167,37 @@ Instance methods: >>> datetime(2002, 12, 25, tzinfo=TZ()).isoformat(' ') '2002-12-25 00:00:00-06:39' + The optional argument *timespec* specifies the number of additional + components of the time to include (the default is ``'auto'``). + It can be one of the following: + + - ``'auto'``: Same as ``'seconds'`` if :attr:`microsecond` is 0, + same as ``'microseconds'`` otherwise. + - ``'hours'``: Include the :attr:`hour` in the two-digit HH format. + - ``'minutes'``: Include :attr:`hour` and :attr:`minute` in HH:MM format. + - ``'seconds'``: Include :attr:`hour`, :attr:`minute`, and :attr:`second` + in HH:MM:SS format. + - ``'milliseconds'``: Include full time, but truncate fractional second + part to milliseconds. HH:MM:SS.sss format. + - ``'microseconds'``: Include full time in HH:MM:SS.mmmmmm format. + + .. note:: + + Excluded time components are truncated, not rounded. + + :exc:`ValueError` will be raised on an invalid *timespec* argument. + + + >>> from datetime import datetime + >>> datetime.now().isoformat(timespec='minutes') + '2002-12-25T00:00' + >>> dt = datetime(2015, 1, 1, 12, 30, 59, 0) + >>> dt.isoformat(timespec='microseconds') + '2015-01-01T12:30:59.000000' + + .. versionadded:: 3.6 + Added the *timespec* argument. + .. method:: datetime.__str__() @@ -1185,7 +1224,8 @@ Instance methods: .. method:: datetime.__format__(format) Same as :meth:`.datetime.strftime`. This makes it possible to specify a format - string for a :class:`.datetime` object when using :meth:`str.format`. For a + string for a :class:`.datetime` object in :ref:`formatted string + literals <f-strings>` and when using :meth:`str.format`. For a complete list of formatting directives, see :ref:`strftime-strptime-behavior`. @@ -1407,13 +1447,46 @@ Instance methods: aware :class:`.time`, without conversion of the time data. -.. method:: time.isoformat() +.. method:: time.isoformat(timespec='auto') Return a string representing the time in ISO 8601 format, HH:MM:SS.mmmmmm or, if - self.microsecond is 0, HH:MM:SS If :meth:`utcoffset` does not return ``None``, a + :attr:`microsecond` is 0, HH:MM:SS If :meth:`utcoffset` does not return ``None``, a 6-character string is appended, giving the UTC offset in (signed) hours and minutes: HH:MM:SS.mmmmmm+HH:MM or, if self.microsecond is 0, HH:MM:SS+HH:MM + The optional argument *timespec* specifies the number of additional + components of the time to include (the default is ``'auto'``). + It can be one of the following: + + - ``'auto'``: Same as ``'seconds'`` if :attr:`microsecond` is 0, + same as ``'microseconds'`` otherwise. + - ``'hours'``: Include the :attr:`hour` in the two-digit HH format. + - ``'minutes'``: Include :attr:`hour` and :attr:`minute` in HH:MM format. + - ``'seconds'``: Include :attr:`hour`, :attr:`minute`, and :attr:`second` + in HH:MM:SS format. + - ``'milliseconds'``: Include full time, but truncate fractional second + part to milliseconds. HH:MM:SS.sss format. + - ``'microseconds'``: Include full time in HH:MM:SS.mmmmmm format. + + .. note:: + + Excluded time components are truncated, not rounded. + + :exc:`ValueError` will be raised on an invalid *timespec* argument. + + + >>> from datetime import time + >>> time(hour=12, minute=34, second=56, microsecond=123456).isoformat(timespec='minutes') + '12:34' + >>> dt = time(hour=12, minute=34, second=56, microsecond=0) + >>> dt.isoformat(timespec='microseconds') + '12:34:56.000000' + >>> dt.isoformat(timespec='auto') + '12:34:56' + + .. versionadded:: 3.6 + Added the *timespec* argument. + .. method:: time.__str__() @@ -1430,7 +1503,8 @@ Instance methods: .. method:: time.__format__(format) Same as :meth:`.time.strftime`. This makes it possible to specify a format string - for a :class:`.time` object when using :meth:`str.format`. For a + for a :class:`.time` object in :ref:`formatted string + literals <f-strings>` and when using :meth:`str.format`. For a complete list of formatting directives, see :ref:`strftime-strptime-behavior`. @@ -1741,10 +1815,7 @@ made to civil time. otherwise :exc:`ValueError` is raised. The *name* argument is optional. If specified it must be a string that - is used as the value returned by the ``tzname(dt)`` method. Otherwise, - ``tzname(dt)`` returns a string 'UTCsHH:MM', where s is the sign of - *offset*, HH and MM are two digits of ``offset.hours`` and - ``offset.minutes`` respectively. + will be used as the value returned by the :meth:`datetime.tzname` method. .. versionadded:: 3.2 @@ -1757,11 +1828,19 @@ made to civil time. .. method:: timezone.tzname(dt) - Return the fixed value specified when the :class:`timezone` instance is - constructed or a string 'UTCsHH:MM', where s is the sign of - *offset*, HH and MM are two digits of ``offset.hours`` and + Return the fixed value specified when the :class:`timezone` instance + is constructed. If *name* is not provided in the constructor, the + name returned by ``tzname(dt)`` is generated from the value of the + ``offset`` as follows. If *offset* is ``timedelta(0)``, the name + is "UTC", otherwise it is a string 'UTC±HH:MM', where ± is the sign + of ``offset``, HH and MM are two digits of ``offset.hours`` and ``offset.minutes`` respectively. + .. versionchanged:: 3.6 + Name generated from ``offset=timedelta(0)`` is now plain 'UTC', not + 'UTC+00:00'. + + .. method:: timezone.dst(dt) Always returns ``None``. @@ -1911,6 +1990,34 @@ format codes. | ``%%`` | A literal ``'%'`` character. | % | | +-----------+--------------------------------+------------------------+-------+ +Several additional directives not required by the C89 standard are included for +convenience. These parameters all correspond to ISO 8601 date values. These +may not be available on all platforms when used with the :meth:`strftime` +method. The ISO 8601 year and ISO 8601 week directives are not interchangeable +with the year and week number directives above. Calling :meth:`strptime` with +incomplete or ambiguous ISO 8601 directives will raise a :exc:`ValueError`. + ++-----------+--------------------------------+------------------------+-------+ +| Directive | Meaning | Example | Notes | ++===========+================================+========================+=======+ +| ``%G`` | ISO 8601 year with century | 0001, 0002, ..., 2013, | \(8) | +| | representing the year that | 2014, ..., 9998, 9999 | | +| | contains the greater part of | | | +| | the ISO week (``%V``). | | | ++-----------+--------------------------------+------------------------+-------+ +| ``%u`` | ISO 8601 weekday as a decimal | 1, 2, ..., 7 | | +| | number where 1 is Monday. | | | ++-----------+--------------------------------+------------------------+-------+ +| ``%V`` | ISO 8601 week as a decimal | 01, 02, ..., 53 | \(8) | +| | number with Monday as | | | +| | the first day of the week. | | | +| | Week 01 is the week containing | | | +| | Jan 4. | | | ++-----------+--------------------------------+------------------------+-------+ + +.. versionadded:: 3.6 + ``%G``, ``%u`` and ``%V`` were added. + Notes: (1) @@ -1975,7 +2082,14 @@ Notes: (7) When used with the :meth:`strptime` method, ``%U`` and ``%W`` are only used - in calculations when the day of the week and the year are specified. + in calculations when the day of the week and the calendar year (``%Y``) + are specified. + +(8) + Similar to ``%U`` and ``%W``, ``%V`` is only used in calculations when the + day of the week and the ISO year (``%G``) are specified in a + :meth:`strptime` format string. Also note that ``%G`` and ``%Y`` are not + interchangeable. .. rubric:: Footnotes diff --git a/Doc/library/dbm.rst b/Doc/library/dbm.rst index 2a1db91..32e80b2 100644 --- a/Doc/library/dbm.rst +++ b/Doc/library/dbm.rst @@ -351,6 +351,10 @@ The module defines the following: :func:`.open` always creates a new database when the flag has the value ``'n'``. + .. deprecated-removed:: 3.6 3.8 + Creating database in ``'r'`` and ``'w'`` modes. Modifying database in + ``'r'`` mode. + In addition to the methods provided by the :class:`collections.abc.MutableMapping` class, :class:`dumbdbm` objects provide the following methods: diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index b5ce0b1..ee746e9 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -447,6 +447,19 @@ Decimal objects ``Decimal('321e+5').adjusted()`` returns seven. Used for determining the position of the most significant digit with respect to the decimal point. + .. method:: as_integer_ratio() + + Return a pair ``(n, d)`` of integers that represent the given + :class:`Decimal` instance as a fraction, in lowest terms and + with a positive denominator:: + + >>> Decimal('-3.14').as_integer_ratio() + (-157, 50) + + The conversion is exact. Raise OverflowError on infinities and ValueError + on NaNs. + + .. versionadded:: 3.6 .. method:: as_tuple() diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index d2d8ac7..245b4d2 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -31,9 +31,9 @@ the following command can be used to display the disassembly of >>> dis.dis(myfunc) 2 0 LOAD_GLOBAL 0 (len) - 3 LOAD_FAST 0 (alist) - 6 CALL_FUNCTION 1 - 9 RETURN_VALUE + 2 LOAD_FAST 0 (alist) + 4 CALL_FUNCTION 1 + 6 RETURN_VALUE (The "2" is a line number). @@ -682,8 +682,7 @@ iterations of the loop. .. XXX explain the WHY stuff! -All of the following opcodes expect arguments. An argument is two bytes, with -the more significant byte last. +All of the following opcodes use their arguments. .. opcode:: STORE_NAME (namei) @@ -769,6 +768,15 @@ the more significant byte last. to hold *count* entries. +.. opcode:: BUILD_CONST_KEY_MAP (count) + + The version of :opcode:`BUILD_MAP` specialized for constant keys. *count* + values are consumed from the stack. The top element on the stack contains + a tuple of keys. + + .. versionadded:: 3.6 + + .. opcode:: LOAD_ATTR (namei) Replaces TOS with ``getattr(TOS, co_names[namei])``. @@ -929,27 +937,16 @@ the more significant byte last. .. opcode:: MAKE_FUNCTION (argc) Pushes a new function object on the stack. From bottom to top, the consumed - stack must consist of - - * ``argc & 0xFF`` default argument objects in positional order - * ``(argc >> 8) & 0xFF`` pairs of name and default argument, with the name - just below the object on the stack, for keyword-only parameters - * ``(argc >> 16) & 0x7FFF`` parameter annotation objects - * a tuple listing the parameter names for the annotations (only if there are - ony annotation objects) + stack must consist of values if the argument carries a specified flag value + + * ``0x01`` a tuple of default argument objects in positional order + * ``0x02`` a dictionary of keyword-only parameters' default values + * ``0x04`` an annotation dictionary + * ``0x08`` a tuple containing cells for free variables, making a closure * the code associated with the function (at TOS1) * the :term:`qualified name` of the function (at TOS) -.. opcode:: MAKE_CLOSURE (argc) - - Creates a new function object, sets its *__closure__* slot, and pushes it on - the stack. TOS is the :term:`qualified name` of the function, TOS1 is the - code associated with the function, and TOS2 is the tuple containing cells for - the closure's free variables. *argc* is interpreted as in ``MAKE_FUNCTION``; - the annotations and defaults are also in the same order below TOS2. - - .. opcode:: BUILD_SLICE (argc) .. index:: builtin: slice @@ -989,6 +986,28 @@ the more significant byte last. arguments. +.. opcode:: FORMAT_VALUE (flags) + + Used for implementing formatted literal strings (f-strings). Pops + an optional *fmt_spec* from the stack, then a required *value*. + *flags* is interpreted as follows: + + * ``(flags & 0x03) == 0x00``: *value* is formatted as-is. + * ``(flags & 0x03) == 0x01``: call :func:`str` on *value* before + formatting it. + * ``(flags & 0x03) == 0x02``: call :func:`repr` on *value* before + formatting it. + * ``(flags & 0x03) == 0x03``: call :func:`ascii` on *value* before + formatting it. + * ``(flags & 0x04) == 0x04``: pop *fmt_spec* from the stack and use + it, else use an empty *fmt_spec*. + + Formatting is performed using :c:func:`PyObject_Format`. The + result is pushed on the stack. + + .. versionadded:: 3.6 + + .. opcode:: HAVE_ARGUMENT This is not really an opcode. It identifies the dividing line between diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index a3d5afc..827bab0 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -257,7 +257,7 @@ members are not integers (but see `IntEnum`_ below):: >>> Color.red < Color.blue Traceback (most recent call last): File "<stdin>", line 1, in <module> - TypeError: unorderable types: Color() < Color() + TypeError: '<' not supported between instances of 'Color' and 'Color' Equality comparisons are defined though:: @@ -558,7 +558,8 @@ Some rules: 4. %-style formatting: `%s` and `%r` call the :class:`Enum` class's :meth:`__str__` and :meth:`__repr__` respectively; other codes (such as `%i` or `%h` for IntEnum) treat the enum member as its mixed-in type. -5. :meth:`str.format` (or :func:`format`) will use the mixed-in +5. :ref:`Formatted string literals <f-strings>`, :meth:`str.format`, + and :func:`format` will use the mixed-in type's :meth:`__format__`. If the :class:`Enum` class's :func:`str` or :func:`repr` is desired, use the `!s` or `!r` format codes. @@ -747,6 +748,15 @@ besides the :class:`Enum` member you looking for:: .. versionchanged:: 3.5 +Boolean evaluation: Enum classes that are mixed with non-Enum types (such as +:class:`int`, :class:`str`, etc.) are evaluated according to the mixed-in +type's rules; otherwise, all members evaluate as ``True``. To make your own +Enum's boolean evaluation depend on the member's value add the following to +your class:: + + def __bool__(self): + return bool(self.value) + The :attr:`__members__` attribute is only available on the class. If you give your :class:`Enum` subclass extra methods, like the `Planet`_ @@ -766,3 +776,22 @@ appropriately. If you wish to change how :class:`Enum` members are looked up you should either write a helper function or a :func:`classmethod` for the :class:`Enum` subclass. + +To help keep Python 2 / Python 3 code in sync a user-specified :attr:`_order_`, +if provided, will be checked to ensure the actual order of the enumeration +matches:: + + >>> class Color(Enum): + ... _order_ = 'red green blue' + ... red = 1 + ... blue = 3 + ... green = 2 + ... + Traceback (most recent call last): + ... + TypeError: member order does not match _order_ + +.. note:: + + In Python 2 code the :attr:`_order_` attribute is necessary as definition + order is lost during class creation. diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 5a71933..1747efe 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -228,9 +228,21 @@ The following exceptions are the exceptions that are usually raised. .. exception:: NotImplementedError This exception is derived from :exc:`RuntimeError`. In user defined base - classes, abstract methods should raise this exception when they require derived - classes to override the method. + classes, abstract methods should raise this exception when they require + derived classes to override the method, or while the class is being + developed to indicate that the real implementation still needs to be added. + .. note:: + + It should not be used to indicate that an operater or method is not + meant to be supported at all -- in that case either leave the operator / + method undefined or, if a subclass, set it to :data:`None`. + + .. note:: + + ``NotImplementedError`` and ``NotImplemented`` are not interchangeable, + even though they have similar names and purposes. See + :data:`NotImplemented` for details on when to use it. .. exception:: OSError([arg]) OSError(errno, strerror[, filename[, winerror[, filename2]]]) @@ -436,6 +448,15 @@ The following exceptions are the exceptions that are usually raised. Raised when an operation or function is applied to an object of inappropriate type. The associated value is a string giving details about the type mismatch. + This exception may be raised by user code to indicate that an attempted + operation on an object is not supported, and is not meant to be. If an object + is meant to support a given operation but has not yet provided an + implementation, :exc:`NotImplementedError` is the proper exception to raise. + + Passing arguments of the wrong type (e.g. passing a :class:`list` when an + :class:`int` is expected) should result in a :exc:`TypeError`, but passing + arguments with the wrong value (e.g. a number outside expected boundaries) + should result in a :exc:`ValueError`. .. exception:: UnboundLocalError diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst index deedea1..d0c4cd0 100644 --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -70,6 +70,9 @@ Fault handler state .. versionchanged:: 3.5 Added support for passing file descriptor to this function. + .. versionchanged:: 3.6 + On Windows, a handler for Windows exception is also installed. + .. function:: disable() Disable the fault handler: uninstall the signal handlers installed by diff --git a/Doc/library/fileinput.rst b/Doc/library/fileinput.rst index aa4c529..5881fef 100644 --- a/Doc/library/fileinput.rst +++ b/Doc/library/fileinput.rst @@ -72,9 +72,8 @@ The following function is the primary interface of this module: .. versionchanged:: 3.2 Can be used as a context manager. - .. versionchanged:: 3.5.2 - The *bufsize* parameter is no longer used. - + .. deprecated-removed:: 3.6 3.8 + The *bufsize* parameter. The following functions use the global state created by :func:`fileinput.input`; if there is no active state, :exc:`RuntimeError` is raised. @@ -167,8 +166,8 @@ available for subclassing as well: .. deprecated:: 3.4 The ``'rU'`` and ``'U'`` modes. - .. versionchanged:: 3.5.2 - The *bufsize* parameter is no longer used. + .. deprecated-removed:: 3.6 3.8 + The *bufsize* parameter. **Optional in-place filtering:** if the keyword argument ``inplace=True`` is @@ -195,10 +194,14 @@ The two following opening hooks are provided by this module: Usage example: ``fi = fileinput.FileInput(openhook=fileinput.hook_compressed)`` -.. function:: hook_encoded(encoding) +.. function:: hook_encoded(encoding, errors=None) Returns a hook which opens each file with :func:`open`, using the given - *encoding* to read the file. + *encoding* and *errors* to read the file. Usage example: ``fi = - fileinput.FileInput(openhook=fileinput.hook_encoded("iso-8859-1"))`` + fileinput.FileInput(openhook=fileinput.hook_encoded("utf-8", + "surrogateescape"))`` + + .. versionchanged:: 3.6 + Added the optional *errors* parameter. diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index efa5bd3..be6f2ec 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -878,11 +878,11 @@ are always available. They are listed here in alphabetical order. Open *file* and return a corresponding :term:`file object`. If the file cannot be opened, an :exc:`OSError` is raised. - *file* is either a string or bytes object giving the pathname (absolute or - relative to the current working directory) of the file to be opened or - an integer file descriptor of the file to be wrapped. (If a file descriptor - is given, it is closed when the returned I/O object is closed, unless - *closefd* is set to ``False``.) + *file* is either a string, bytes, or :class:`os.PathLike` object giving the + pathname (absolute or relative to the current working directory) of the file + to be opened or an integer file descriptor of the file to be wrapped. (If a + file descriptor is given, it is closed when the returned I/O object is + closed, unless *closefd* is set to ``False``.) *mode* is an optional string that specifies the mode in which the file is opened. It defaults to ``'r'`` which means open for reading in text mode. @@ -1077,6 +1077,9 @@ are always available. They are listed here in alphabetical order. .. versionchanged:: 3.5 The ``'namereplace'`` error handler was added. + .. versionchanged:: 3.6 + Support added to accept objects implementing :class:`os.PathLike`. + .. function:: ord(c) Given a string representing one Unicode character, return an integer @@ -1460,6 +1463,9 @@ are always available. They are listed here in alphabetical order. See also :ref:`bltin-type-objects`. + .. versionchanged:: 3.6 + Subclasses of :class:`type` which don't override ``type.__new__`` may no + longer use the one-argument form to get the type of an object. .. function:: vars([object]) diff --git a/Doc/library/grp.rst b/Doc/library/grp.rst index a30e622..74de3f9 100644 --- a/Doc/library/grp.rst +++ b/Doc/library/grp.rst @@ -43,6 +43,9 @@ It defines the following items: Return the group database entry for the given numeric group ID. :exc:`KeyError` is raised if the entry asked for cannot be found. + .. deprecated:: 3.6 + Since Python 3.6 the support of non-integer arguments like floats or + strings in :func:`getgrgid` is deprecated. .. function:: getgrnam(name) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index a2e96ca..f6d4808 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -44,8 +44,8 @@ Hash algorithms --------------- There is one constructor method named for each type of :dfn:`hash`. All return -a hash object with the same simple interface. For example: use :func:`sha1` to -create a SHA1 hash object. You can now feed this object with :term:`bytes-like +a hash object with the same simple interface. For example: use :func:`sha256` to +create a SHA-256 hash object. You can now feed this object with :term:`bytes-like objects <bytes-like object>` (normally :class:`bytes`) using the :meth:`update` method. At any point you can ask it for the :dfn:`digest` of the concatenation of the data fed to it so far using the :meth:`digest` or @@ -64,21 +64,23 @@ concatenation of the data fed to it so far using the :meth:`digest` or .. index:: single: OpenSSL; (use in module hashlib) Constructors for hash algorithms that are always present in this module are -:func:`md5`, :func:`sha1`, :func:`sha224`, :func:`sha256`, :func:`sha384`, -and :func:`sha512`. Additional algorithms may also be available depending upon -the OpenSSL library that Python uses on your platform. +:func:`sha1`, :func:`sha224`, :func:`sha256`, :func:`sha384`, +and :func:`sha512`. :func:`md5` is normally available as well, though it +may be missing if you are using a rare "FIPS compliant" build of Python. +Additional algorithms may also be available depending upon the OpenSSL +library that Python uses on your platform. For example, to obtain the digest of the byte string ``b'Nobody inspects the spammish repetition'``:: >>> import hashlib - >>> m = hashlib.md5() + >>> m = hashlib.sha256() >>> m.update(b"Nobody inspects") >>> m.update(b" the spammish repetition") >>> m.digest() - b'\xbbd\x9c\x83\xdd\x1e\xa5\xc9\xd9\xde\xc9\xa1\x8d\xf0\xff\xe9' + b'\x03\x1e\xdd}Ae\x15\x93\xc5\xfe\\\x00o\xa5u+7\xfd\xdf\xf7\xbcN\x84:\xa6\xaf\x0c\x95\x0fK\x94\x06' >>> m.digest_size - 16 + 32 >>> m.block_size 64 @@ -107,7 +109,9 @@ Hashlib provides the following constant attributes: .. data:: algorithms_guaranteed A set containing the names of the hash algorithms guaranteed to be supported - by this module on all platforms. + by this module on all platforms. Note that 'md5' is in this list despite + some upstream vendors offering an odd "FIPS compliant" Python build that + excludes it. .. versionadded:: 3.2 diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index a9ca4b0..9429fb6 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -219,39 +219,62 @@ HTTPConnection Objects :class:`HTTPConnection` instances have the following methods: -.. method:: HTTPConnection.request(method, url, body=None, headers={}) +.. method:: HTTPConnection.request(method, url, body=None, headers={}, *, \ + encode_chunked=False) This will send a request to the server using the HTTP request method *method* and the selector *url*. If *body* is specified, the specified data is sent after the headers are - finished. It may be a string, a :term:`bytes-like object`, an open - :term:`file object`, or an iterable of :term:`bytes-like object`\s. If - *body* is a string, it is encoded as ISO-8859-1, the default for HTTP. If - it is a bytes-like object the bytes are sent as is. If it is a :term:`file - object`, the contents of the file is sent; this file object should support - at least the ``read()`` method. If the file object has a ``mode`` - attribute, the data returned by the ``read()`` method will be encoded as - ISO-8859-1 unless the ``mode`` attribute contains the substring ``b``, - otherwise the data returned by ``read()`` is sent as is. If *body* is an - iterable, the elements of the iterable are sent as is until the iterable is - exhausted. - - The *headers* argument should be a mapping of extra HTTP - headers to send with the request. - - If *headers* does not contain a Content-Length item, one is added - automatically if possible. If *body* is ``None``, the Content-Length header - is set to ``0`` for methods that expect a body (``PUT``, ``POST``, and - ``PATCH``). If *body* is a string or bytes object, the Content-Length - header is set to its length. If *body* is a :term:`file object` and it - works to call :func:`~os.fstat` on the result of its ``fileno()`` method, - then the Content-Length header is set to the ``st_size`` reported by the - ``fstat`` call. Otherwise no Content-Length header is added. + finished. It may be a :class:`str`, a :term:`bytes-like object`, an + open :term:`file object`, or an iterable of :class:`bytes`. If *body* + is a string, it is encoded as ISO-8859-1, the default for HTTP. If it + is a bytes-like object, the bytes are sent as is. If it is a :term:`file + object`, the contents of the file is sent; this file object should + support at least the ``read()`` method. If the file object is an + instance of :class:`io.TextIOBase`, the data returned by the ``read()`` + method will be encoded as ISO-8859-1, otherwise the data returned by + ``read()`` is sent as is. If *body* is an iterable, the elements of the + iterable are sent as is until the iterable is exhausted. + + The *headers* argument should be a mapping of extra HTTP headers to send + with the request. + + If *headers* contains neither Content-Length nor Transfer-Encoding, a + Content-Length header will be added automatically if possible. If + *body* is ``None``, the Content-Length header is set to ``0`` for + methods that expect a body (``PUT``, ``POST``, and ``PATCH``). If + *body* is a string or bytes-like object, the Content-Length header is + set to its length. If *body* is a binary :term:`file object` + supporting :meth:`~io.IOBase.seek`, this will be used to determine + its size. Otherwise, the Content-Length header is not added + automatically. In cases where determining the Content-Length up + front is not possible, the body will be chunk-encoded and the + Transfer-Encoding header will automatically be set. + + The *encode_chunked* argument is only relevant if Transfer-Encoding is + specified in *headers*. If *encode_chunked* is ``False``, the + HTTPConnection object assumes that all encoding is handled by the + calling code. If it is ``True``, the body will be chunk-encoded. + + .. note:: + Chunked transfer encoding has been added to the HTTP protocol + version 1.1. Unless the HTTP server is known to handle HTTP 1.1, + the caller must either specify the Content-Length or must use a + body representation whose length can be determined automatically. .. versionadded:: 3.2 *body* can now be an iterable. + .. versionchanged:: 3.6 + If neither Content-Length nor Transfer-Encoding are set in + *headers* and Content-Length cannot be determined, *body* will now + be automatically chunk-encoded. The *encode_chunked* argument + was added. + The Content-Length for binary file objects is determined with seek. + No attempt is made to determine the Content-Length for text file + objects. + .. method:: HTTPConnection.getresponse() Should be called after a request is sent to get the response from the server. @@ -336,13 +359,32 @@ also send your request step by step, by using the four functions below. an argument. -.. method:: HTTPConnection.endheaders(message_body=None) +.. method:: HTTPConnection.endheaders(message_body=None, *, encode_chunked=False) Send a blank line to the server, signalling the end of the headers. The optional *message_body* argument can be used to pass a message body - associated with the request. The message body will be sent in the same - packet as the message headers if it is string, otherwise it is sent in a - separate packet. + associated with the request. + + If *encode_chunked* is ``True``, the result of each iteration of + *message_body* will be chunk-encoded as specified in :rfc:`7230`, + Section 3.3.1. How the data is encoded is dependent on the type of + *message_body*. If *message_body* implements the :ref:`buffer interface + <bufferobjects>` the encoding will result in a single chunk. + If *message_body* is a :class:`collections.Iterable`, each iteration + of *message_body* will result in a chunk. If *message_body* is a + :term:`file object`, each call to ``.read()`` will result in a chunk. + The method automatically signals the end of the chunk-encoded data + immediately after *message_body*. + + .. note:: Due to the chunked encoding specification, empty chunks + yielded by an iterator body will be ignored by the chunk-encoder. + This is to avoid premature termination of the read of the request by + the target server due to malformed encoding. + + .. versionadded:: 3.6 + Chunked encoding support. The *encode_chunked* parameter was + added. + .. method:: HTTPConnection.send(data) diff --git a/Doc/library/http.server.rst b/Doc/library/http.server.rst index 16c4fac..c1ea873 100644 --- a/Doc/library/http.server.rst +++ b/Doc/library/http.server.rst @@ -98,8 +98,8 @@ of which this module provides three different variants: .. attribute:: rfile - Contains an input stream, positioned at the start of the optional input - data. + An :class:`io.BufferedIOBase` input stream, ready to read from + the start of the optional input data. .. attribute:: wfile @@ -107,6 +107,9 @@ of which this module provides three different variants: client. Proper adherence to the HTTP protocol must be used when writing to this stream. + .. versionchanged:: 3.6 + This is an :class:`io.BufferedIOBase` stream. + :class:`BaseHTTPRequestHandler` has the following attributes: .. attribute:: server_version @@ -369,10 +372,9 @@ the current directory:: Handler = http.server.SimpleHTTPRequestHandler - httpd = socketserver.TCPServer(("", PORT), Handler) - - print("serving at port", PORT) - httpd.serve_forever() + with socketserver.TCPServer(("", PORT), Handler) as httpd: + print("serving at port", PORT) + httpd.serve_forever() .. _http-server-cli: diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index c25e7d8..b9b3b91 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -500,6 +500,17 @@ An :class:`IMAP4` instance has the following methods: M.store(num, '+FLAGS', '\\Deleted') M.expunge() + .. note:: + + Creating flags containing ']' (for example: "[test]") violates + :rfc:`3501` (the IMAP protocol). However, imaplib has historically + allowed creation of such tags, and popular IMAP servers, such as Gmail, + accept and produce such flags. There are non-Python programs which also + create such tags. Although it is an RFC violation and IMAP clients and + servers are supposed to be strict, imaplib nonetheless continues to allow + such tags to be created for backward compatibility reasons, and as of + python 3.6, handles them if they are sent from the server, since this + improves real-world compatibility. .. method:: IMAP4.subscribe(mailbox) diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst index 9828ba6..ccf5f92 100644 --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -85,7 +85,9 @@ This module provides an interface to the mechanisms used to implement the .. deprecated:: 3.3 Use :func:`importlib.util.find_spec` instead unless Python 3.3 compatibility is required, in which case use - :func:`importlib.find_loader`. + :func:`importlib.find_loader`. For example usage of the former case, + see the :ref:`importlib-examples` section of the :mod:`importlib` + documentation. .. function:: load_module(name, file, pathname, description) @@ -112,9 +114,12 @@ This module provides an interface to the mechanisms used to implement the If previously used in conjunction with :func:`imp.find_module` then consider using :func:`importlib.import_module`, otherwise use the loader returned by the replacement you chose for :func:`imp.find_module`. If you - called :func:`imp.load_module` and related functions directly then use the - classes in :mod:`importlib.machinery`, e.g. - ``importlib.machinery.SourceFileLoader(name, path).load_module()``. + called :func:`imp.load_module` and related functions directly with file + path arguments then use a combination of + :func:`importlib.util.spec_from_file_location` and + :func:`importlib.util.module_from_spec`. See the :ref:`importlib-examples` + section of the :mod:`importlib` documentation for details of the various + approaches. .. function:: new_module(name) @@ -123,7 +128,7 @@ This module provides an interface to the mechanisms used to implement the in ``sys.modules``. .. deprecated:: 3.4 - Use :class:`types.ModuleType` instead. + Use :func:`importlib.util.module_from_spec` instead. .. function:: reload(module) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 06e9ea3..24b8104 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -267,7 +267,7 @@ ABC hierarchy:: module and *path* will be the value of :attr:`__path__` from the parent package. If a spec cannot be found, ``None`` is returned. When passed in, ``target`` is a module object that the finder may - use to make a more educated about what spec to return. + use to make a more educated guess about what spec to return. .. versionadded:: 3.4 @@ -317,7 +317,7 @@ ABC hierarchy:: within the :term:`path entry` to which it is assigned. If a spec cannot be found, ``None`` is returned. When passed in, ``target`` is a module object that the finder may use to make a more educated - about what spec to return. + guess about what spec to return. .. versionadded:: 3.4 @@ -379,10 +379,14 @@ ABC hierarchy:: An abstract method that executes the module in its own namespace when a module is imported or reloaded. The module should already - be initialized when exec_module() is called. + be initialized when ``exec_module()`` is called. When this method exists, + :meth:`~importlib.abc.Loader.create_module` must be defined. .. versionadded:: 3.4 + .. versionchanged:: 3.6 + :meth:`~importlib.abc.Loader.create_module` must also be defined. + .. method:: load_module(fullname) A legacy method for loading a module. If the module cannot be @@ -936,6 +940,10 @@ find and load modules. Concrete implementation of :meth:`importlib.abc.Loader.load_module` where specifying the name of the module to load is optional. + .. deprecated:: 3.6 + + Use :meth:`importlib.abc.Loader.exec_module` instead. + .. class:: SourcelessFileLoader(fullname, path) @@ -975,6 +983,10 @@ find and load modules. Concrete implementation of :meth:`importlib.abc.Loader.load_module` where specifying the name of the module to load is optional. + .. deprecated:: 3.6 + + Use :meth:`importlib.abc.Loader.exec_module` instead. + .. class:: ExtensionFileLoader(fullname, path) @@ -1192,12 +1204,13 @@ an :term:`importer`. .. function:: module_from_spec(spec) - Create a new module based on **spec** and ``spec.loader.create_module()``. + Create a new module based on **spec** and + :meth:`spec.loader.create_module <importlib.abc.Loader.create_module>`. - If ``spec.loader.create_module()`` does not return ``None``, then any - pre-existing attributes will not be reset. Also, no :exc:`AttributeError` - will be raised if triggered while accessing **spec** or setting an attribute - on the module. + If :meth:`spec.loader.create_module <importlib.abc.Loader.create_module>` + does not return ``None``, then any pre-existing attributes will not be reset. + Also, no :exc:`AttributeError` will be raised if triggered while accessing + **spec** or setting an attribute on the module. This function is preferred over using :class:`types.ModuleType` to create a new module as **spec** is used to set as many import-controlled attributes on @@ -1259,7 +1272,8 @@ an :term:`importer`. .. decorator:: set_package - A :term:`decorator` for :meth:`importlib.abc.Loader.load_module` to set the :attr:`__package__` attribute on the returned module. If :attr:`__package__` + A :term:`decorator` for :meth:`importlib.abc.Loader.load_module` to set the + :attr:`__package__` attribute on the returned module. If :attr:`__package__` is set and has a value other than ``None`` it will not be changed. .. deprecated:: 3.4 @@ -1292,13 +1306,12 @@ an :term:`importer`. This class **only** works with loaders that define :meth:`~importlib.abc.Loader.exec_module` as control over what module type is used for the module is required. For those same reasons, the loader's - :meth:`~importlib.abc.Loader.create_module` method will be ignored (i.e., the - loader's method should only return ``None``; this excludes - :class:`BuiltinImporter` and :class:`ExtensionFileLoader`). Finally, - modules which substitute the object placed into :attr:`sys.modules` will - not work as there is no way to properly replace the module references - throughout the interpreter safely; :exc:`ValueError` is raised if such a - substitution is detected. + :meth:`~importlib.abc.Loader.create_module` method must return ``None`` or a + type for which its ``__class__`` attribute can be mutated along with not + using :term:`slots <__slots__>`. Finally, modules which substitute the object + placed into :attr:`sys.modules` will not work as there is no way to properly + replace the module references throughout the interpreter safely; + :exc:`ValueError` is raised if such a substitution is detected. .. note:: For projects where startup time is critical, this class allows for @@ -1309,6 +1322,11 @@ an :term:`importer`. .. versionadded:: 3.5 + .. versionchanged:: 3.6 + Began calling :meth:`~importlib.abc.Loader.create_module`, removing the + compatibility warning for :class:`importlib.machinery.BuiltinImporter` and + :class:`importlib.machinery.ExtensionFileLoader`. + .. classmethod:: factory(loader) A static method which returns a callable that creates a lazy loader. This @@ -1320,3 +1338,120 @@ an :term:`importer`. loader = importlib.machinery.SourceFileLoader lazy_loader = importlib.util.LazyLoader.factory(loader) finder = importlib.machinery.FileFinder(path, (lazy_loader, suffixes)) + +.. _importlib-examples: + +Examples +-------- + +To programmatically import a module, use :func:`importlib.import_module`. +:: + + import importlib + + itertools = importlib.import_module('itertools') + +If you need to find out if a module can be imported without actually doing the +import, then you should use :func:`importlib.util.find_spec`. +:: + + import importlib.util + import sys + + # For illustrative purposes. + name = 'itertools' + + spec = importlib.util.find_spec(name) + if spec is None: + print("can't find the itertools module") + else: + # If you chose to perform the actual import ... + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + # Adding the module to sys.modules is optional. + sys.modules[name] = module + +To import a Python source file directly, use the following recipe +(Python 3.4 and newer only):: + + import importlib.util + import sys + + # For illustrative purposes. + import tokenize + file_path = tokenize.__file__ + module_name = tokenize.__name__ + + spec = importlib.util.spec_from_file_location(module_name, file_path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + # Optional; only necessary if you want to be able to import the module + # by name later. + sys.modules[module_name] = module + +For deep customizations of import, you typically want to implement an +:term:`importer`. This means managing both the :term:`finder` and :term:`loader` +side of things. For finders there are two flavours to choose from depending on +your needs: a :term:`meta path finder` or a :term:`path entry finder`. The +former is what you would put on :attr:`sys.meta_path` while the latter is what +you create using a :term:`path entry hook` on :attr:`sys.path_hooks` which works +with :attr:`sys.path` entries to potentially create a finder. This example will +show you how to register your own importers so that import will use them (for +creating an importer for yourself, read the documentation for the appropriate +classes defined within this package):: + + import importlib.machinery + import sys + + # For illustrative purposes only. + SpamMetaPathFinder = importlib.machinery.PathFinder + SpamPathEntryFinder = importlib.machinery.FileFinder + loader_details = (importlib.machinery.SourceFileLoader, + importlib.machinery.SOURCE_SUFFIXES) + + # Setting up a meta path finder. + # Make sure to put the finder in the proper location in the list in terms of + # priority. + sys.meta_path.append(SpamMetaPathFinder) + + # Setting up a path entry finder. + # Make sure to put the path hook in the proper location in the list in terms + # of priority. + sys.path_hooks.append(SpamPathEntryFinder.path_hook(loader_details)) + +Import itself is implemented in Python code, making it possible to +expose most of the import machinery through importlib. The following +helps illustrate the various APIs that importlib exposes by providing an +approximate implementation of +:func:`importlib.import_module` (Python 3.4 and newer for the importlib usage, +Python 3.6 and newer for other parts of the code). +:: + + import importlib.util + import sys + + def import_module(name, package=None): + """An approximate implementation of import.""" + absolute_name = importlib.util.resolve_name(name, package) + try: + return sys.modules[absolute_name] + except KeyError: + pass + + path = None + if '.' in absolute_name: + parent_name, _, child_name = absolute_name.rpartition('.') + parent_module = import_module(parent_name) + path = parent_module.spec.submodule_search_locations + for finder in sys.meta_path: + spec = finder.find_spec(absolute_name, path) + if spec is not None: + break + else: + raise ImportError(f'No module named {absolute_name!r}') + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + sys.modules[absolute_name] = module + if path is not None: + setattr(parent_module, child_name, module) + return module diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 8e7ed19..5cb7c22 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -235,24 +235,6 @@ attributes: listed in the metaclass' custom :meth:`__dir__`. -.. function:: getmoduleinfo(path) - - Returns a :term:`named tuple` ``ModuleInfo(name, suffix, mode, module_type)`` - of values that describe how Python will interpret the file identified by - *path* if it is a module, or ``None`` if it would not be identified as a - module. In that tuple, *name* is the name of the module without the name of - any enclosing package, *suffix* is the trailing part of the file name (which - may not be a dot-delimited extension), *mode* is the :func:`open` mode that - would be used (``'r'`` or ``'rb'``), and *module_type* is an integer giving - the type of the module. *module_type* will have a value which can be - compared to the constants defined in the :mod:`imp` module; see the - documentation for that module for more information on module types. - - .. deprecated:: 3.3 - You may check the file path's suffix against the supported suffixes - listed in :mod:`importlib.machinery` to infer the same information. - - .. function:: getmodulename(path) Return the name of the module named by the file *path*, without including the @@ -266,8 +248,7 @@ attributes: still return ``None``. .. versionchanged:: 3.3 - This function is now based directly on :mod:`importlib` rather than the - deprecated :func:`getmoduleinfo`. + The function is based directly on :mod:`importlib`. .. function:: ismodule(object) @@ -646,6 +627,16 @@ function. The name of the parameter as a string. The name must be a valid Python identifier. + .. impl-detail:: + + CPython generates implicit parameter names of the form ``.0`` on the + code objects used to implement comprehensions and generator + expressions. + + .. versionchanged:: 3.6 + These parameter names are exposed by this module as names like + ``implicit0``. + .. attribute:: Parameter.default The default value for the parameter. If the parameter has no default @@ -854,8 +845,6 @@ Classes and functions from kwonlyargs to defaults. *annotations* is a dictionary mapping argument names to annotations. - The first four items in the tuple correspond to :func:`getargspec`. - .. versionchanged:: 3.4 This function is now based on :func:`signature`, but still ignores ``__wrapped__`` attributes and includes the already bound first @@ -884,7 +873,7 @@ Classes and functions .. function:: formatargspec(args[, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations[, formatarg, formatvarargs, formatvarkw, formatvalue, formatreturns, formatannotations]]) Format a pretty argument spec from the values returned by - :func:`getargspec` or :func:`getfullargspec`. + :func:`getfullargspec`. The first seven arguments are (``args``, ``varargs``, ``varkw``, ``defaults``, ``kwonlyargs``, ``kwonlydefaults``, ``annotations``). diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index dfc1ddc..b0d0a8c 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -591,7 +591,13 @@ loops that truncate the stream. .. function:: tee(iterable, n=2) - Return *n* independent iterators from a single iterable. Roughly equivalent to:: + Return *n* independent iterators from a single iterable. + + The following Python code helps explain what *tee* does (although the actual + implementation is more complex and uses only a single underlying + :abbr:`FIFO (first-in, first-out)` queue). + + Roughly equivalent to:: def tee(iterable, n=2): it = iter(iterable) diff --git a/Doc/library/json.rst b/Doc/library/json.rst index ee58266..73824f8 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -126,7 +126,7 @@ See :ref:`json-commandline` for detailed documentation. Basic Usage ----------- -.. function:: dump(obj, fp, skipkeys=False, ensure_ascii=True, \ +.. function:: dump(obj, fp, *, skipkeys=False, ensure_ascii=True, \ check_circular=True, allow_nan=True, cls=None, \ indent=None, separators=None, default=None, \ sort_keys=False, **kw) @@ -187,8 +187,11 @@ Basic Usage :meth:`default` method to serialize additional types), specify it with the *cls* kwarg; otherwise :class:`JSONEncoder` is used. + .. versionchanged:: 3.6 + All optional parameters are now :ref:`keyword-only <keyword-only_parameter>`. -.. function:: dumps(obj, skipkeys=False, ensure_ascii=True, \ + +.. function:: dumps(obj, *, skipkeys=False, ensure_ascii=True, \ check_circular=True, allow_nan=True, cls=None, \ indent=None, separators=None, default=None, \ sort_keys=False, **kw) @@ -212,7 +215,7 @@ Basic Usage the original one. That is, ``loads(dumps(x)) != x`` if x has non-string keys. -.. function:: load(fp, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) +.. function:: load(fp, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) Deserialize *fp* (a ``.read()``-supporting :term:`file-like object` containing a JSON document) to a Python object using this :ref:`conversion @@ -260,7 +263,10 @@ Basic Usage If the data being deserialized is not a valid JSON document, a :exc:`JSONDecodeError` will be raised. -.. function:: loads(s, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) + .. versionchanged:: 3.6 + All optional parameters are now :ref:`keyword-only <keyword-only_parameter>`. + +.. function:: loads(s, *, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) Deserialize *s* (a :class:`str` instance containing a JSON document) to a Python object using this :ref:`conversion table <json-to-py-table>`. @@ -274,7 +280,7 @@ Basic Usage Encoders and Decoders --------------------- -.. class:: JSONDecoder(object_hook=None, parse_float=None, parse_int=None, parse_constant=None, strict=True, object_pairs_hook=None) +.. class:: JSONDecoder(*, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, strict=True, object_pairs_hook=None) Simple JSON decoder. @@ -344,6 +350,9 @@ Encoders and Decoders If the data being deserialized is not a valid JSON document, a :exc:`JSONDecodeError` will be raised. + .. versionchanged:: 3.6 + All parameters are now :ref:`keyword-only <keyword-only_parameter>`. + .. method:: decode(s) Return the Python representation of *s* (a :class:`str` instance @@ -362,7 +371,7 @@ Encoders and Decoders extraneous data at the end. -.. class:: JSONEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None) +.. class:: JSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None) Extensible JSON encoder for Python data structures. @@ -442,6 +451,9 @@ Encoders and Decoders the object or raise a :exc:`TypeError`. If not specified, :exc:`TypeError` is raised. + .. versionchanged:: 3.6 + All parameters are now :ref:`keyword-only <keyword-only_parameter>`. + .. method:: default(o) diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst index 916b702..8e27ad0 100644 --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -84,6 +84,9 @@ sends logging output to a disk file. It inherits the output functionality from with that encoding. If *delay* is true, then file opening is deferred until the first call to :meth:`emit`. By default, the file grows indefinitely. + .. versionchanged:: 3.6 + As well as string values, :class:`~pathlib.Path` objects are also accepted + for the *filename* argument. .. method:: close() @@ -160,12 +163,23 @@ for this value. with that encoding. If *delay* is true, then file opening is deferred until the first call to :meth:`emit`. By default, the file grows indefinitely. + .. versionchanged:: 3.6 + As well as string values, :class:`~pathlib.Path` objects are also accepted + for the *filename* argument. + + .. method:: reopenIfNeeded() + + Checks to see if the file has changed. If it has, the existing stream is + flushed and closed and the file opened again, typically as a precursor to + outputting the record to the file. + + .. versionadded:: 3.6 + .. method:: emit(record) - Outputs the record to the file, but first checks to see if the file has - changed. If it has, the existing stream is flushed and closed and the - file opened again, before outputting the record to the file. + Outputs the record to the file, but first calls :meth:`reopenIfNeeded` to + reopen the file if it has changed. .. _base-rotating-handler: @@ -279,6 +293,9 @@ module, supports rotation of disk log files. :file:`app.log.2`, etc. exist, then they are renamed to :file:`app.log.2`, :file:`app.log.3` etc. respectively. + .. versionchanged:: 3.6 + As well as string values, :class:`~pathlib.Path` objects are also accepted + for the *filename* argument. .. method:: doRollover() @@ -310,21 +327,24 @@ timed intervals. You can use the *when* to specify the type of *interval*. The list of possible values is below. Note that they are not case sensitive. - +----------------+-----------------------+ - | Value | Type of interval | - +================+=======================+ - | ``'S'`` | Seconds | - +----------------+-----------------------+ - | ``'M'`` | Minutes | - +----------------+-----------------------+ - | ``'H'`` | Hours | - +----------------+-----------------------+ - | ``'D'`` | Days | - +----------------+-----------------------+ - | ``'W0'-'W6'`` | Weekday (0=Monday) | - +----------------+-----------------------+ - | ``'midnight'`` | Roll over at midnight | - +----------------+-----------------------+ + +----------------+----------------------------+-------------------------+ + | Value | Type of interval | If/how *atTime* is used | + +================+============================+=========================+ + | ``'S'`` | Seconds | Ignored | + +----------------+----------------------------+-------------------------+ + | ``'M'`` | Minutes | Ignored | + +----------------+----------------------------+-------------------------+ + | ``'H'`` | Hours | Ignored | + +----------------+----------------------------+-------------------------+ + | ``'D'`` | Days | Ignored | + +----------------+----------------------------+-------------------------+ + | ``'W0'-'W6'`` | Weekday (0=Monday) | Used to compute initial | + | | | rollover time | + +----------------+----------------------------+-------------------------+ + | ``'midnight'`` | Roll over at midnight, if | Used to compute initial | + | | *atTime* not specified, | rollover time | + | | else at time *atTime* | | + +----------------+----------------------------+-------------------------+ When using weekday-based rotation, specify 'W0' for Monday, 'W1' for Tuesday, and so on up to 'W6' for Sunday. In this case, the value passed for @@ -352,16 +372,35 @@ timed intervals. If *atTime* is not ``None``, it must be a ``datetime.time`` instance which specifies the time of day when rollover occurs, for the cases where rollover - is set to happen "at midnight" or "on a particular weekday". + is set to happen "at midnight" or "on a particular weekday". Note that in + these cases, the *atTime* value is effectively used to compute the *initial* + rollover, and subsequent rollovers would be calculated via the normal + interval calculation. + + .. note:: Calculation of the initial rollover time is done when the handler + is initialised. Calculation of subsequent rollover times is done only + when rollover occurs, and rollover occurs only when emitting output. If + this is not kept in mind, it might lead to some confusion. For example, + if an interval of "every minute" is set, that does not mean you will + always see log files with times (in the filename) separated by a minute; + if, during application execution, logging output is generated more + frequently than once a minute, *then* you can expect to see log files + with times separated by a minute. If, on the other hand, logging messages + are only output once every five minutes (say), then there will be gaps in + the file times corresponding to the minutes where no output (and hence no + rollover) occurred. .. versionchanged:: 3.4 *atTime* parameter was added. + .. versionchanged:: 3.6 + As well as string values, :class:`~pathlib.Path` objects are also accepted + for the *filename* argument. + .. method:: doRollover() Does a rollover, as described above. - .. method:: emit(record) Outputs the record to the file, catering for rollover as described above. @@ -798,12 +837,18 @@ should, then :meth:`flush` is expected to do the flushing. overridden to implement custom flushing strategies. -.. class:: MemoryHandler(capacity, flushLevel=ERROR, target=None) +.. class:: MemoryHandler(capacity, flushLevel=ERROR, target=None, flushOnClose=True) Returns a new instance of the :class:`MemoryHandler` class. The instance is initialized with a buffer size of *capacity*. If *flushLevel* is not specified, :const:`ERROR` is used. If no *target* is specified, the target will need to be - set using :meth:`setTarget` before this handler does anything useful. + set using :meth:`setTarget` before this handler does anything useful. If + *flushOnClose* is specified as ``False``, then the buffer is *not* flushed when + the handler is closed. If not specified or specified as ``True``, the previous + behaviour of flushing the buffer will occur when the handler is closed. + + .. versionchanged:: 3.6 + The *flushOnClose* parameter was added. .. method:: close() diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 3fdea18..da2b8cc 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -426,6 +426,15 @@ Constants The mathematical constant e = 2.718281..., to available precision. +.. data:: tau + + The mathematical constant τ = 6.283185..., to available precision. + Tau is a circle constant equal to 2π, the ratio of a circle's circumference to + its radius. To learn more about Tau, check out Vi Hart's video `Pi is (still) + Wrong <https://www.youtube.com/watch?v=jG7vhMMXagQ>`_, and start celebrating + `Tau day <http://tauday.com/>`_ by eating twice as much pie! + + .. versionadded:: 3.6 .. data:: inf diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index 8f53833..c544c80 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -264,13 +264,18 @@ To map anonymous memory, -1 should be passed as the fileno along with the length .. method:: write(bytes) Write the bytes in *bytes* into memory at the current position of the - file pointer; the file position is updated to point after the bytes that - were written. If the mmap was created with :const:`ACCESS_READ`, then + file pointer and return the number of bytes written (never less than + ``len(bytes)``, since if the write fails, a :exc:`ValueError` will be + raised). The file position is updated to point after the bytes that + were written. If the mmap was created with :const:`ACCESS_READ`, then writing to it will raise a :exc:`TypeError` exception. .. versionchanged:: 3.5 Writable :term:`bytes-like object` is now accepted. + .. versionchanged:: 3.6 + The number of bytes written is now returned. + .. method:: write_byte(byte) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index d20098f..f886ecb 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -647,8 +647,9 @@ primitives like locks. For passing messages one can use :func:`Pipe` (for a connection between two processes) or a queue (which allows multiple producers and consumers). -The :class:`Queue`, :class:`SimpleQueue` and :class:`JoinableQueue` types are multi-producer, -multi-consumer FIFO queues modelled on the :class:`queue.Queue` class in the +The :class:`Queue`, :class:`SimpleQueue` and :class:`JoinableQueue` types +are multi-producer, multi-consumer :abbr:`FIFO (first-in, first-out)` +queues modelled on the :class:`queue.Queue` class in the standard library. They differ in that :class:`Queue` lacks the :meth:`~queue.Queue.task_done` and :meth:`~queue.Queue.join` methods introduced into Python 2.5's :class:`queue.Queue` class. @@ -886,8 +887,13 @@ Miscellaneous .. function:: cpu_count() - Return the number of CPUs in the system. May raise - :exc:`NotImplementedError`. + Return the number of CPUs in the system. + + This number is not equivalent to the number of CPUs the current process can + use. The number of usable CPUs can be obtained with + ``len(os.sched_getaffinity(0))`` + + May raise :exc:`NotImplementedError`. .. seealso:: :func:`os.cpu_count` diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 4265bc2..9456733 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -171,23 +171,60 @@ process and user. .. function:: fsencode(filename) - Encode *filename* to the filesystem encoding with ``'surrogateescape'`` - error handler, or ``'strict'`` on Windows; return :class:`bytes` unchanged. + Encode :term:`path-like <path-like object>` *filename* to the filesystem + encoding with ``'surrogateescape'`` error handler, or ``'strict'`` on + Windows; return :class:`bytes` unchanged. :func:`fsdecode` is the reverse function. .. versionadded:: 3.2 + .. versionchanged:: 3.6 + Support added to accept objects implementing the :class:`os.PathLike` + interface. + .. function:: fsdecode(filename) - Decode *filename* from the filesystem encoding with ``'surrogateescape'`` - error handler, or ``'strict'`` on Windows; return :class:`str` unchanged. + Decode the :term:`path-like <path-like object>` *filename* from the + filesystem encoding with ``'surrogateescape'`` error handler, or ``'strict'`` + on Windows; return :class:`str` unchanged. :func:`fsencode` is the reverse function. .. versionadded:: 3.2 + .. versionchanged:: 3.6 + Support added to accept objects implementing the :class:`os.PathLike` + interface. + + +.. function:: fspath(path) + + Return the file system representation of the path. + + If :class:`str` or :class:`bytes` is passed in, it is returned unchanged. + Otherwise :meth:`~os.PathLike.__fspath__` is called and its value is + returned as long as it is a :class:`str` or :class:`bytes` object. + In all other cases, :exc:`TypeError` is raised. + + .. versionadded:: 3.6 + + +.. class:: PathLike + + An :term:`abstract base class` for objects representing a file system path, + e.g. :class:`pathlib.PurePath`. + + .. versionadded:: 3.6 + + .. abstractmethod:: __fspath__() + + Return the file system path representation of the object. + + The method should only return a :class:`str` or :class:`bytes` object, + with the preference being for :class:`str`. + .. function:: getenv(key, default=None) @@ -1883,35 +1920,51 @@ features: .. function:: scandir(path='.') - Return an iterator of :class:`DirEntry` objects corresponding to the entries - in the directory given by *path*. The entries are yielded in arbitrary - order, and the special entries ``'.'`` and ``'..'`` are not included. + Return an iterator of :class:`os.DirEntry` objects corresponding to the + entries in the directory given by *path*. The entries are yielded in + arbitrary order, and the special entries ``'.'`` and ``'..'`` are not + included. Using :func:`scandir` instead of :func:`listdir` can significantly increase the performance of code that also needs file type or file - attribute information, because :class:`DirEntry` objects expose this + attribute information, because :class:`os.DirEntry` objects expose this information if the operating system provides it when scanning a directory. - All :class:`DirEntry` methods may perform a system call, but - :func:`~DirEntry.is_dir` and :func:`~DirEntry.is_file` usually only - require a system call for symbolic links; :func:`DirEntry.stat` + All :class:`os.DirEntry` methods may perform a system call, but + :func:`~os.DirEntry.is_dir` and :func:`~os.DirEntry.is_file` usually only + require a system call for symbolic links; :func:`os.DirEntry.stat` always requires a system call on Unix but only requires one for symbolic links on Windows. On Unix, *path* can be of type :class:`str` or :class:`bytes` (use :func:`~os.fsencode` and :func:`~os.fsdecode` to encode and decode :class:`bytes` paths). On Windows, *path* must be of type :class:`str`. - On both systems, the type of the :attr:`~DirEntry.name` and - :attr:`~DirEntry.path` attributes of each :class:`DirEntry` will be of + On both systems, the type of the :attr:`~os.DirEntry.name` and + :attr:`~os.DirEntry.path` attributes of each :class:`os.DirEntry` will be of the same type as *path*. + The :func:`scandir` iterator supports the :term:`context manager` protocol + and has the following method: + + .. method:: scandir.close() + + Close the iterator and free acquired resources. + + This is called automatically when the iterator is exhausted or garbage + collected, or when an error happens during iterating. However it + is advisable to call it explicitly or use the :keyword:`with` + statement. + + .. versionadded:: 3.6 + The following example shows a simple use of :func:`scandir` to display all the files (excluding directories) in the given *path* that don't start with ``'.'``. The ``entry.is_file()`` call will generally not make an additional system call:: - for entry in os.scandir(path): - if not entry.name.startswith('.') and entry.is_file(): - print(entry.name) + with os.scandir(path) as it: + for entry in it: + if not entry.name.startswith('.') and entry.is_file(): + print(entry.name) .. note:: @@ -1927,6 +1980,12 @@ features: .. versionadded:: 3.5 + .. versionadded:: 3.6 + Added support for the :term:`context manager` protocol and the + :func:`~scandir.close()` method. If a :func:`scandir` iterator is neither + exhausted nor explicitly closed a :exc:`ResourceWarning` will be emitted + in its destructor. + .. class:: DirEntry @@ -1935,19 +1994,22 @@ features: :func:`scandir` will provide as much of this information as possible without making additional system calls. When a ``stat()`` or ``lstat()`` system call - is made, the ``DirEntry`` object will cache the result. + is made, the ``os.DirEntry`` object will cache the result. - ``DirEntry`` instances are not intended to be stored in long-lived data + ``os.DirEntry`` instances are not intended to be stored in long-lived data structures; if you know the file metadata has changed or if a long time has elapsed since calling :func:`scandir`, call ``os.stat(entry.path)`` to fetch up-to-date information. - Because the ``DirEntry`` methods can make operating system calls, they may + Because the ``os.DirEntry`` methods can make operating system calls, they may also raise :exc:`OSError`. If you need very fine-grained control over errors, you can catch :exc:`OSError` when calling one of the - ``DirEntry`` methods and handle as appropriate. + ``os.DirEntry`` methods and handle as appropriate. + + To be directly usable as a :term:`path-like object`, ``os.DirEntry`` + implements the :class:`os.PathLike` interface. - Attributes and methods on a ``DirEntry`` instance are as follows: + Attributes and methods on a ``os.DirEntry`` instance are as follows: .. attribute:: name @@ -1973,8 +2035,9 @@ features: Return the inode number of the entry. - The result is cached on the ``DirEntry`` object. Use ``os.stat(entry.path, - follow_symlinks=False).st_ino`` to fetch up-to-date information. + The result is cached on the ``os.DirEntry`` object. Use + ``os.stat(entry.path, follow_symlinks=False).st_ino`` to fetch up-to-date + information. On the first, uncached call, a system call is required on Windows but not on Unix. @@ -1989,7 +2052,7 @@ features: is a directory (without following symlinks); return ``False`` if the entry is any other kind of file or if it doesn't exist anymore. - The result is cached on the ``DirEntry`` object, with a separate cache + The result is cached on the ``os.DirEntry`` object, with a separate cache for *follow_symlinks* ``True`` and ``False``. Call :func:`os.stat` along with :func:`stat.S_ISDIR` to fetch up-to-date information. @@ -2013,8 +2076,8 @@ features: is a file (without following symlinks); return ``False`` if the entry is a directory or other non-file entry, or if it doesn't exist anymore. - The result is cached on the ``DirEntry`` object. Caching, system calls - made, and exceptions raised are as per :func:`~DirEntry.is_dir`. + The result is cached on the ``os.DirEntry`` object. Caching, system calls + made, and exceptions raised are as per :func:`~os.DirEntry.is_dir`. .. method:: is_symlink() @@ -2022,7 +2085,7 @@ features: return ``False`` if the entry points to a directory or any kind of file, or if it doesn't exist anymore. - The result is cached on the ``DirEntry`` object. Call + The result is cached on the ``os.DirEntry`` object. Call :func:`os.path.islink` to fetch up-to-date information. On the first, uncached call, no system call is required in most cases. @@ -2047,17 +2110,21 @@ features: :class:`stat_result` are always set to zero. Call :func:`os.stat` to get these attributes. - The result is cached on the ``DirEntry`` object, with a separate cache + The result is cached on the ``os.DirEntry`` object, with a separate cache for *follow_symlinks* ``True`` and ``False``. Call :func:`os.stat` to fetch up-to-date information. Note that there is a nice correspondence between several attributes - and methods of ``DirEntry`` and of :class:`pathlib.Path`. In - particular, the ``name`` attribute has the same meaning, as do the - ``is_dir()``, ``is_file()``, ``is_symlink()`` and ``stat()`` methods. + and methods of ``os.DirEntry`` and of :class:`pathlib.Path`. In + particular, the ``name`` attribute has the same + meaning, as do the ``is_dir()``, ``is_file()``, ``is_symlink()`` + and ``stat()`` methods. .. versionadded:: 3.5 + .. versionchanged:: 3.6 + Added support for the :class:`os.PathLike` interface. + .. function:: stat(path, \*, dir_fd=None, follow_symlinks=True) @@ -3618,6 +3685,11 @@ Miscellaneous System Information Return the number of CPUs in the system. Returns None if undetermined. + This number is not equivalent to the number of CPUs the current process can + use. The number of usable CPUs can be obtained with + ``len(os.sched_getaffinity(0))`` + + .. versionadded:: 3.4 diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 384611c..9257d2d 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -35,12 +35,6 @@ Pure paths are useful in some special cases; for example: accessing the OS. In this case, instantiating one of the pure classes may be useful since those simply don't have any OS-accessing operations. -.. note:: - This module has been included in the standard library on a - :term:`provisional basis <provisional package>`. Backwards incompatible - changes (up to and including removal of the package) may occur if deemed - necessary by the core developers. - .. seealso:: :pep:`428`: The pathlib module -- object-oriented filesystem paths. @@ -111,7 +105,8 @@ we also call *flavours*: PurePosixPath('setup.py') Each element of *pathsegments* can be either a string representing a - path segment, or another path object:: + path segment, an object implementing the :class:`os.PathLike` interface + which returns a string, or another path object:: >>> PurePath('foo', 'some/path', 'bar') PurePosixPath('foo/some/path/bar') @@ -152,6 +147,12 @@ we also call *flavours*: to ``PurePosixPath('bar')``, which is wrong if ``foo`` is a symbolic link to another directory) + Pure path objects implement the :class:`os.PathLike` interface, allowing them + to be used anywhere the interface is accepted. + + .. versionchanged:: 3.6 + Added support for the :class:`os.PathLike` interface. + .. class:: PurePosixPath(*pathsegments) A subclass of :class:`PurePath`, this path flavour represents non-Windows @@ -199,7 +200,7 @@ Paths of a different flavour compare unequal and cannot be ordered:: >>> PureWindowsPath('foo') < PurePosixPath('foo') Traceback (most recent call last): File "<stdin>", line 1, in <module> - TypeError: unorderable types: PureWindowsPath() < PurePosixPath() + TypeError: '<' not supported between instances of 'PureWindowsPath' and 'PurePosixPath' Operators @@ -216,6 +217,14 @@ The slash operator helps create child paths, similarly to :func:`os.path.join`:: >>> '/usr' / q PurePosixPath('/usr/bin') +A path object can be used anywhere an object implementing :class:`os.PathLike` +is accepted:: + + >>> import os + >>> p = PurePath('/etc') + >>> os.fspath(p) + '/etc' + The string representation of a path is the raw filesystem path itself (in native form, e.g. with backslashes under Windows), which you can pass to any function taking a file path as a string:: diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index 0d64191..6e8430f 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -492,7 +492,7 @@ methods: .. method:: object.__getnewargs_ex__() - In protocols 4 and newer, classes that implements the + In protocols 2 and newer, classes that implements the :meth:`__getnewargs_ex__` method can dictate the values passed to the :meth:`__new__` method upon unpickling. The method must return a pair ``(args, kwargs)`` where *args* is a tuple of positional arguments @@ -504,15 +504,22 @@ methods: class requires keyword-only arguments. Otherwise, it is recommended for compatibility to implement :meth:`__getnewargs__`. + .. versionchanged:: 3.6 + :meth:`__getnewargs_ex__` is now used in protocols 2 and 3. + .. method:: object.__getnewargs__() - This method serve a similar purpose as :meth:`__getnewargs_ex__` but - for protocols 2 and newer. It must return a tuple of arguments ``args`` - which will be passed to the :meth:`__new__` method upon unpickling. + This method serve a similar purpose as :meth:`__getnewargs_ex__`, but + supports only positional arguments. It must return a tuple of arguments + ``args`` which will be passed to the :meth:`__new__` method upon unpickling. + + :meth:`__getnewargs__` will not be called if :meth:`__getnewargs_ex__` is + defined. - In protocols 4 and newer, :meth:`__getnewargs__` will not be called if - :meth:`__getnewargs_ex__` is defined. + .. versionchanged:: 3.6 + Before Python 3.6, :meth:`__getnewargs__` was called instead of + :meth:`__getnewargs_ex__` in protocols 2 and 3. .. method:: object.__getstate__() diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 26c5ac0..1f11a2d 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -46,10 +46,10 @@ support. .. class:: ImpImporter(dirname=None) - :pep:`302` Importer that wraps Python's "classic" import algorithm. + :pep:`302` Finder that wraps Python's "classic" import algorithm. - If *dirname* is a string, a :pep:`302` importer is created that searches that - directory. If *dirname* is ``None``, a :pep:`302` importer is created that + If *dirname* is a string, a :pep:`302` finder is created that searches that + directory. If *dirname* is ``None``, a :pep:`302` finder is created that searches the current :data:`sys.path`, plus any modules that are frozen or built-in. @@ -88,9 +88,9 @@ support. .. function:: get_importer(path_item) - Retrieve a :pep:`302` importer for the given *path_item*. + Retrieve a :pep:`302` finder for the given *path_item*. - The returned importer is cached in :data:`sys.path_importer_cache` if it was + The returned finder is cached in :data:`sys.path_importer_cache` if it was newly created by a path hook. The cache (or part of it) can be cleared manually if a rescan of @@ -121,16 +121,16 @@ support. .. function:: iter_importers(fullname='') - Yield :pep:`302` importers for the given module name. + Yield :pep:`302` finders for the given module name. - If fullname contains a '.', the importers will be for the package + If fullname contains a '.', the finders will be for the package containing fullname, otherwise they will be all registered top level - importers (i.e. those on both sys.meta_path and sys.path_hooks). + finders (i.e. those on both sys.meta_path and sys.path_hooks). If the named module is in a package, that package is imported as a side effect of invoking this function. - If no module name is specified, all top level importers are produced. + If no module name is specified, all top level finders are produced. .. versionchanged:: 3.3 Updated to be based directly on :mod:`importlib` rather than relying diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 1cb0935..e3eaa3f 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -16,8 +16,9 @@ availability of thread support in Python; see the :mod:`threading` module. The module implements three types of queue, which differ only in the order in -which the entries are retrieved. In a FIFO queue, the first tasks added are -the first retrieved. In a LIFO queue, the most recently added entry is +which the entries are retrieved. In a :abbr:`FIFO (first-in, first-out)` +queue, the first tasks added are the first retrieved. In a +:abbr:`LIFO (last-in, first-out)` queue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the :mod:`heapq` module) and the lowest valued entry is retrieved first. @@ -27,14 +28,16 @@ The :mod:`queue` module defines the following classes and exceptions: .. class:: Queue(maxsize=0) - Constructor for a FIFO queue. *maxsize* is an integer that sets the upperbound + Constructor for a :abbr:`FIFO (first-in, first-out)` queue. *maxsize* is + an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If *maxsize* is less than or equal to zero, the queue size is infinite. .. class:: LifoQueue(maxsize=0) - Constructor for a LIFO queue. *maxsize* is an integer that sets the upperbound + Constructor for a :abbr:`LIFO (last-in, first-out)` queue. *maxsize* is + an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If *maxsize* is less than or equal to zero, the queue size is infinite. diff --git a/Doc/library/random.rst b/Doc/library/random.rst index df502a0..e7b81ad 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -46,7 +46,8 @@ from sources provided by the operating system. .. warning:: The pseudo-random generators of this module should not be used for - security purposes. + security purposes. For security or cryptographic uses, see the + :mod:`secrets` module. Bookkeeping functions: diff --git a/Doc/library/re.rst b/Doc/library/re.rst index 569b522..dfbedd4 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -321,8 +321,9 @@ The special characters are: The special sequences consist of ``'\'`` and a character from the list below. -If the ordinary character is not on the list, then the resulting RE will match -the second character. For example, ``\$`` matches the character ``'$'``. +If the ordinary character is not an ASCII digit or an ASCII letter, then the +resulting RE will match the second character. For example, ``\$`` matches the +character ``'$'``. ``\number`` Matches the contents of the group of the same number. Groups are numbered @@ -442,9 +443,8 @@ three digits in length. .. versionchanged:: 3.3 The ``'\u'`` and ``'\U'`` escape sequences have been added. -.. deprecated-removed:: 3.5 3.6 - Unknown escapes consisting of ``'\'`` and ASCII letter now raise a - deprecation warning and will be forbidden in Python 3.6. +.. versionchanged:: 3.6 + Unknown escapes consisting of ``'\'`` and an ASCII letter now are errors. .. seealso:: @@ -532,11 +532,11 @@ form. current locale. The use of this flag is discouraged as the locale mechanism is very unreliable, and it only handles one "culture" at a time anyway; you should use Unicode matching instead, which is the default in Python 3 - for Unicode (str) patterns. This flag makes sense only with bytes patterns. + for Unicode (str) patterns. This flag can be used only with bytes patterns. - .. deprecated-removed:: 3.5 3.6 - Deprecated the use of :const:`re.LOCALE` with string patterns or - :const:`re.ASCII`. + .. versionchanged:: 3.6 + :const:`re.LOCALE` can be used only with bytes patterns and is + not compatible with :const:`re.ASCII`. .. data:: M @@ -742,9 +742,8 @@ form. .. versionchanged:: 3.5 Unmatched groups are replaced with an empty string. - .. deprecated-removed:: 3.5 3.6 - Unknown escapes consist of ``'\'`` and ASCII letter now raise a - deprecation warning and will be forbidden in Python 3.6. + .. versionchanged:: 3.6 + Unknown escapes consisting of ``'\'`` and an ASCII letter now are errors. .. function:: subn(pattern, repl, string, count=0, flags=0) diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index 4d3c099..37e400e 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -167,6 +167,20 @@ The following functions operate on a global history list: This calls :c:func:`add_history` in the underlying library. +.. function:: set_auto_history(enabled) + + Enable or disable automatic calls to :c:func:`add_history` when reading + input via readline. The *enabled* argument should be a Boolean value + that when true, enables auto history, and that when False, disables + auto history. + + .. versionadded:: 3.6 + + .. impl-detail:: + Auto history is enabled by default, and changes to this do not persist + across multiple sessions. + + Startup hooks ------------- diff --git a/Doc/library/secrets.rst b/Doc/library/secrets.rst new file mode 100644 index 0000000..9bf848f --- /dev/null +++ b/Doc/library/secrets.rst @@ -0,0 +1,198 @@ +:mod:`secrets` --- Generate secure random numbers for managing secrets +====================================================================== + +.. module:: secrets + :synopsis: Generate secure random numbers for managing secrets. + +.. moduleauthor:: Steven D'Aprano <steve+python@pearwood.info> +.. sectionauthor:: Steven D'Aprano <steve+python@pearwood.info> +.. versionadded:: 3.6 + +.. testsetup:: + + from secrets import * + __name__ = '<doctest>' + +**Source code:** :source:`Lib/secrets.py` + +------------- + +The :mod:`secrets` module is used for generating cryptographically strong +random numbers suitable for managing data such as passwords, account +authentication, security tokens, and related secrets. + +In particularly, :mod:`secrets` should be used in preference to the +default pseudo-random number generator in the :mod:`random` module, which +is designed for modelling and simulation, not security or cryptography. + +.. seealso:: + + :pep:`506` + + +Random numbers +-------------- + +The :mod:`secrets` module provides access to the most secure source of +randomness that your operating system provides. + +.. class:: SystemRandom + + A class for generating random numbers using the highest-quality + sources provided by the operating system. See + :class:`random.SystemRandom` for additional details. + +.. function:: choice(sequence) + + Return a randomly-chosen element from a non-empty sequence. + +.. function:: randbelow(n) + + Return a random int in the range [0, *n*). + +.. function:: randbits(k) + + Return an int with *k* random bits. + + +Generating tokens +----------------- + +The :mod:`secrets` module provides functions for generating secure +tokens, suitable for applications such as password resets, +hard-to-guess URLs, and similar. + +.. function:: token_bytes([nbytes=None]) + + Return a random byte string containing *nbytes* number of bytes. + If *nbytes* is ``None`` or not supplied, a reasonable default is + used. + + .. doctest:: + + >>> token_bytes(16) #doctest:+SKIP + b'\xebr\x17D*t\xae\xd4\xe3S\xb6\xe2\xebP1\x8b' + + +.. function:: token_hex([nbytes=None]) + + Return a random text string, in hexadecimal. The string has *nbytes* + random bytes, each byte converted to two hex digits. If *nbytes* is + ``None`` or not supplied, a reasonable default is used. + + .. doctest:: + + >>> token_hex(16) #doctest:+SKIP + 'f9bf78b9a18ce6d46a0cd2b0b86df9da' + +.. function:: token_urlsafe([nbytes=None]) + + Return a random URL-safe text string, containing *nbytes* random + bytes. The text is Base64 encoded, so on average each byte results + in approximately 1.3 characters. If *nbytes* is ``None`` or not + supplied, a reasonable default is used. + + .. doctest:: + + >>> token_urlsafe(16) #doctest:+SKIP + 'Drmhze6EPcv0fN_81Bj-nA' + + +How many bytes should tokens use? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To be secure against +`brute-force attacks <https://en.wikipedia.org/wiki/Brute-force_attack>`_, +tokens need to have sufficient randomness. Unfortunately, what is +considered sufficient will necessarily increase as computers get more +powerful and able to make more guesses in a shorter period. As of 2015, +it is believed that 32 bytes (256 bits) of randomness is sufficient for +the typical use-case expected for the :mod:`secrets` module. + +For those who want to manage their own token length, you can explicitly +specify how much randomness is used for tokens by giving an :class:`int` +argument to the various ``token_*`` functions. That argument is taken +as the number of bytes of randomness to use. + +Otherwise, if no argument is provided, or if the argument is ``None``, +the ``token_*`` functions will use a reasonable default instead. + +.. note:: + + That default is subject to change at any time, including during + maintenance releases. + + +Other functions +--------------- + +.. function:: compare_digest(a, b) + + Return ``True`` if strings *a* and *b* are equal, otherwise ``False``, + in such a way as to reduce the risk of + `timing attacks <http://codahale.com/a-lesson-in-timing-attacks/>`_. + See :func:`hmac.compare_digest` for additional details. + + +Recipes and best practices +-------------------------- + +This section shows recipes and best practices for using :mod:`secrets` +to manage a basic level of security. + +Generate an eight-character alphanumeric password: + +.. testcode:: + + import string + alphabet = string.ascii_letters + string.digits + password = ''.join(choice(alphabet) for i in range(8)) + + +.. note:: + + Applications should not + `store passwords in a recoverable format <http://cwe.mitre.org/data/definitions/257.html>`_, + whether plain text or encrypted. They should be salted and hashed + using a cryptographically-strong one-way (irreversible) hash function. + + +Generate a ten-character alphanumeric password with at least one +lowercase character, at least one uppercase character, and at least +three digits: + +.. testcode:: + + import string + alphabet = string.ascii_letters + string.digits + while True: + password = ''.join(choice(alphabet) for i in range(10)) + if (any(c.islower() for c in password) + and any(c.isupper() for c in password) + and sum(c.isdigit() for c in password) >= 3): + break + + +Generate an `XKCD-style passphrase <http://xkcd.com/936/>`_: + +.. testcode:: + + # On standard Linux systems, use a convenient dictionary file. + # Other platforms may need to provide their own word-list. + with open('/usr/share/dict/words') as f: + words = [word.strip() for word in f] + password = ' '.join(choice(words) for i in range(4)) + + +Generate a hard-to-guess temporary URL containing a security token +suitable for password recovery applications: + +.. testcode:: + + url = 'https://mydomain.com/reset=' + token_urlsafe() + + + +.. + # This modeline must appear within the last ten lines of the file. + kate: indent-width 3; remove-trailing-space on; replace-tabs on; encoding utf-8; diff --git a/Doc/library/select.rst b/Doc/library/select.rst index 6cec9f7..5494eef 100644 --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -266,35 +266,43 @@ Edge and Level Trigger Polling (epoll) Objects *eventmask* - +-----------------------+-----------------------------------------------+ - | Constant | Meaning | - +=======================+===============================================+ - | :const:`EPOLLIN` | Available for read | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLOUT` | Available for write | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLPRI` | Urgent data for read | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLERR` | Error condition happened on the assoc. fd | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLHUP` | Hang up happened on the assoc. fd | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLET` | Set Edge Trigger behavior, the default is | - | | Level Trigger behavior | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLONESHOT` | Set one-shot behavior. After one event is | - | | pulled out, the fd is internally disabled | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLRDNORM` | Equivalent to :const:`EPOLLIN` | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLRDBAND` | Priority data band can be read. | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLWRNORM` | Equivalent to :const:`EPOLLOUT` | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLWRBAND` | Priority data may be written. | - +-----------------------+-----------------------------------------------+ - | :const:`EPOLLMSG` | Ignored. | - +-----------------------+-----------------------------------------------+ + +-------------------------+-----------------------------------------------+ + | Constant | Meaning | + +=========================+===============================================+ + | :const:`EPOLLIN` | Available for read | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLOUT` | Available for write | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLPRI` | Urgent data for read | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLERR` | Error condition happened on the assoc. fd | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLHUP` | Hang up happened on the assoc. fd | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLET` | Set Edge Trigger behavior, the default is | + | | Level Trigger behavior | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLONESHOT` | Set one-shot behavior. After one event is | + | | pulled out, the fd is internally disabled | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLEXCLUSIVE` | Wake only one epoll object when the | + | | associated fd has an event. The default (if | + | | this flag is not set) is to wake all epoll | + | | objects polling on on a fd. | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLRDHUP` | Stream socket peer closed connection or shut | + | | down writing half of connection. | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLRDNORM` | Equivalent to :const:`EPOLLIN` | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLRDBAND` | Priority data band can be read. | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLWRNORM` | Equivalent to :const:`EPOLLOUT` | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLWRBAND` | Priority data may be written. | + +-------------------------+-----------------------------------------------+ + | :const:`EPOLLMSG` | Ignored. | + +-------------------------+-----------------------------------------------+ .. method:: epoll.close() @@ -383,6 +391,9 @@ linearly scanned again. :c:func:`select` is O(highest file descriptor), while +-------------------+------------------------------------------+ | :const:`POLLHUP` | Hung up | +-------------------+------------------------------------------+ + | :const:`POLLRDHUP`| Stream socket peer closed connection, or | + | | shut down writing half of connection | + +-------------------+------------------------------------------+ | :const:`POLLNVAL` | Invalid request: descriptor not open | +-------------------+------------------------------------------+ diff --git a/Doc/library/shlex.rst b/Doc/library/shlex.rst index e81f982..1a89bf6 100644 --- a/Doc/library/shlex.rst +++ b/Doc/library/shlex.rst @@ -73,11 +73,11 @@ The :mod:`shlex` module defines the following functions: The :mod:`shlex` module defines the following class: -.. class:: shlex(instream=None, infile=None, posix=False) +.. class:: shlex(instream=None, infile=None, posix=False, punctuation_chars=False) A :class:`~shlex.shlex` instance or subclass instance is a lexical analyzer object. The initialization argument, if present, specifies where to read - characters from. It must be a file-/stream-like object with + characters from. It must be a file-/stream-like object with :meth:`~io.TextIOBase.read` and :meth:`~io.TextIOBase.readline` methods, or a string. If no argument is given, input will be taken from ``sys.stdin``. The second optional argument is a filename string, which sets the initial @@ -87,8 +87,19 @@ The :mod:`shlex` module defines the following class: when *posix* is not true (default), the :class:`~shlex.shlex` instance will operate in compatibility mode. When operating in POSIX mode, :class:`~shlex.shlex` will try to be as close as possible to the POSIX shell - parsing rules. - + parsing rules. The *punctuation_chars* argument provides a way to make the + behaviour even closer to how real shells parse. This can take a number of + values: the default value, ``False``, preserves the behaviour seen under + Python 3.5 and earlier. If set to ``True``, then parsing of the characters + ``();<>|&`` is changed: any run of these characters (considered punctuation + characters) is returned as a single token. If set to a non-empty string of + characters, those characters will be used as the punctuation characters. Any + characters in the :attr:`wordchars` attribute that appear in + *punctuation_chars* will be removed from :attr:`wordchars`. See + :ref:`improved-shell-compatibility` for more information. + + .. versionchanged:: 3.6 + The *punctuation_chars* parameter was added. .. seealso:: @@ -191,7 +202,13 @@ variables which either control lexical analysis or can be used for debugging: .. attribute:: shlex.wordchars The string of characters that will accumulate into multi-character tokens. By - default, includes all ASCII alphanumerics and underscore. + default, includes all ASCII alphanumerics and underscore. In POSIX mode, the + accented characters in the Latin-1 set are also included. If + :attr:`punctuation_chars` is not empty, the characters ``~-./*?=``, which can + appear in filename specifications and command line parameters, will also be + included in this attribute, and any characters which appear in + ``punctuation_chars`` will be removed from ``wordchars`` if they are present + there. .. attribute:: shlex.whitespace @@ -222,9 +239,13 @@ variables which either control lexical analysis or can be used for debugging: .. attribute:: shlex.whitespace_split - If ``True``, tokens will only be split in whitespaces. This is useful, for + If ``True``, tokens will only be split in whitespaces. This is useful, for example, for parsing command lines with :class:`~shlex.shlex`, getting - tokens in a similar way to shell arguments. + tokens in a similar way to shell arguments. If this attribute is ``True``, + :attr:`punctuation_chars` will have no effect, and splitting will happen + only on whitespaces. When using :attr:`punctuation_chars`, which is + intended to provide parsing closer to that implemented by shells, it is + advisable to leave ``whitespace_split`` as ``False`` (the default value). .. attribute:: shlex.infile @@ -245,10 +266,9 @@ variables which either control lexical analysis or can be used for debugging: This attribute is ``None`` by default. If you assign a string to it, that string will be recognized as a lexical-level inclusion request similar to the ``source`` keyword in various shells. That is, the immediately following token - will be opened as a filename and input will - be taken from that stream until EOF, at which - point the :meth:`~io.IOBase.close` method of that stream will be called and - the input source will again become the original input stream. Source + will be opened as a filename and input will be taken from that stream until + EOF, at which point the :meth:`~io.IOBase.close` method of that stream will be + called and the input source will again become the original input stream. Source requests may be stacked any number of levels deep. @@ -275,6 +295,16 @@ variables which either control lexical analysis or can be used for debugging: (``''``), in non-POSIX mode, and to ``None`` in POSIX mode. +.. attribute:: shlex.punctuation_chars + + Characters that will be considered punctuation. Runs of punctuation + characters will be returned as a single token. However, note that no + semantic validity checking will be performed: for example, '>>>' could be + returned as a token, even though it may not be recognised as such by shells. + + .. versionadded:: 3.6 + + .. _shlex-parsing-rules: Parsing Rules @@ -327,3 +357,62 @@ following parsing rules. * EOF is signaled with a :const:`None` value; * Quoted empty strings (``''``) are allowed. + +.. _improved-shell-compatibility: + +Improved Compatibility with Shells +---------------------------------- + +.. versionadded:: 3.6 + +The :class:`shlex` class provides compatibility with the parsing performed by +common Unix shells like ``bash``, ``dash``, and ``sh``. To take advantage of +this compatibility, specify the ``punctuation_chars`` argument in the +constructor. This defaults to ``False``, which preserves pre-3.6 behaviour. +However, if it is set to ``True``, then parsing of the characters ``();<>|&`` +is changed: any run of these characters is returned as a single token. While +this is short of a full parser for shells (which would be out of scope for the +standard library, given the multiplicity of shells out there), it does allow +you to perform processing of command lines more easily than you could +otherwise. To illustrate, you can see the difference in the following snippet:: + + import shlex + + for punct in (False, True): + if punct: + message = 'Old' + else: + message = 'New' + text = "a && b; c && d || e; f >'abc'; (def \"ghi\")" + s = shlex.shlex(text, punctuation_chars=punct) + print('%s: %s' % (message, list(s))) + +which prints out:: + + Old: ['a', '&', '&', 'b', ';', 'c', '&', '&', 'd', '|', '|', 'e', ';', 'f', '>', "'abc'", ';', '(', 'def', '"ghi"', ')'] + New: ['a', '&&', 'b', ';', 'c', '&&', 'd', '||', 'e', ';', 'f', '>', "'abc'", ';', '(', 'def', '"ghi"', ')'] + +Of course, tokens will be returned which are not valid for shells, and you'll +need to implement your own error checks on the returned tokens. + +Instead of passing ``True`` as the value for the punctuation_chars parameter, +you can pass a string with specific characters, which will be used to determine +which characters constitute punctuation. For example:: + + >>> import shlex + >>> s = shlex.shlex("a && b || c", punctuation_chars="|") + >>> list(s) + ['a', '&', '&', 'b', '||', 'c'] + +.. note:: When ``punctuation_chars`` is specified, the :attr:`~shlex.wordchars` + attribute is augmented with the characters ``~-./*?=``. That is because these + characters can appear in file names (including wildcards) and command-line + arguments (e.g. ``--color=auto``). Hence:: + + >>> import shlex + >>> s = shlex.shlex('~/a && b-c --color=auto || d *.py?', + ... punctuation_chars=True) + >>> list(s) + ['~/a', '&&', 'b-c', '--color=auto', '||', 'd', '*.py?'] + + diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 0a73f5a..43daf79 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -52,7 +52,8 @@ searched for site-packages; otherwise they won't. A path configuration file is a file whose name has the form :file:`{name}.pth` and exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added to ``sys.path``. Non-existing items -are never added to ``sys.path``. No item is added to ``sys.path`` more than +are never added to ``sys.path``, and no check is made that the item refers to a +directory rather than a file. No item is added to ``sys.path`` more than once. Blank lines and lines beginning with ``#`` are skipped. Lines starting with ``import`` (followed by space or tab) are executed. diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst index 977f9a8..ad6bd3c 100644 --- a/Doc/library/smtpd.rst +++ b/Doc/library/smtpd.rst @@ -29,7 +29,7 @@ SMTPServer Objects .. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432,\ - map=None, enable_SMTPUTF8=False, decode_data=True) + map=None, enable_SMTPUTF8=False, decode_data=False) Create a new :class:`SMTPServer` object, which binds to local address *localaddr*. It will treat *remoteaddr* as an upstream SMTP relayer. It @@ -45,20 +45,19 @@ SMTPServer Objects global socket map is used. *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined - in :RFC:`6531`) should be enabled. The default is ``False``. If set to - ``True``, *decode_data* must be ``False`` (otherwise an error is raised). + in :RFC:`6531`) should be enabled. The default is ``False``. When ``True``, ``SMTPUTF8`` is accepted as a parameter to the ``MAIL`` command and when present is passed to :meth:`process_message` in the - ``kwargs['mail_options']`` list. + ``kwargs['mail_options']`` list. *decode_data* and *enable_SMTPUTF8* + cannot be set to ``True`` at the same time. *decode_data* specifies whether the data portion of the SMTP transaction - should be decoded using UTF-8. The default is ``True`` for backward - compatibility reasons, but will change to ``False`` in Python 3.6; specify - the keyword value explicitly to avoid the :exc:`DeprecationWarning`. When - *decode_data* is set to ``False`` the server advertises the ``8BITMIME`` + should be decoded using UTF-8. When *decode_data* is ``False`` (the + default), the server advertises the ``8BITMIME`` extension (:rfc:`6152`), accepts the ``BODY=8BITMIME`` parameter to the ``MAIL`` command, and when present passes it to :meth:`process_message` - in the ``kwargs['mail_options']`` list. + in the ``kwargs['mail_options']`` list. *decode_data* and *enable_SMTPUTF8* + cannot be set to ``True`` at the same time. .. method:: process_message(peer, mailfrom, rcpttos, data, **kwargs) @@ -75,9 +74,8 @@ SMTPServer Objects will be a bytes object. *kwargs* is a dictionary containing additional information. It is empty - unless at least one of ``decode_data=False`` or ``enable_SMTPUTF8=True`` - was given as an init parameter, in which case it contains the following - keys: + if ``decode_data=True`` was given as an init argument, otherwise + it contains the following keys: *mail_options*: a list of all received parameters to the ``MAIL`` @@ -108,9 +106,12 @@ SMTPServer Objects *localaddr* and *remoteaddr* may now contain IPv6 addresses. .. versionadded:: 3.5 - the *decode_data* and *enable_SMTPUTF8* constructor arguments, and the - *kwargs* argument to :meth:`process_message` when one or more of these is - specified. + The *decode_data* and *enable_SMTPUTF8* constructor parameters, and the + *kwargs* parameter to :meth:`process_message` when *decode_data* is + ``False``. + + .. versionchanged:: 3.6 + *decode_data* is now ``False`` by default. DebuggingServer Objects @@ -150,7 +151,7 @@ SMTPChannel Objects ------------------- .. class:: SMTPChannel(server, conn, addr, data_size_limit=33554432,\ - map=None, enable_SMTPUTF8=False, decode_data=True) + map=None, enable_SMTPUTF8=False, decode_data=False) Create a new :class:`SMTPChannel` object which manages the communication between the server and a single SMTP client. @@ -162,22 +163,25 @@ SMTPChannel Objects limit. *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined - in :RFC:`6531`) should be enabled. The default is ``False``. A - :exc:`ValueError` is raised if both *enable_SMTPUTF8* and *decode_data* are - set to ``True`` at the same time. + in :RFC:`6531`) should be enabled. The default is ``False``. + *decode_data* and *enable_SMTPUTF8* cannot be set to ``True`` at the same + time. A dictionary can be specified in *map* to avoid using a global socket map. *decode_data* specifies whether the data portion of the SMTP transaction - should be decoded using UTF-8. The default is ``True`` for backward - compatibility reasons, but will change to ``False`` in Python 3.6. Specify - the keyword value explicitly to avoid the :exc:`DeprecationWarning`. + should be decoded using UTF-8. The default is ``False``. + *decode_data* and *enable_SMTPUTF8* cannot be set to ``True`` at the same + time. To use a custom SMTPChannel implementation you need to override the :attr:`SMTPServer.channel_class` of your :class:`SMTPServer`. .. versionchanged:: 3.5 - the *decode_data* and *enable_SMTPUTF8* arguments were added. + The *decode_data* and *enable_SMTPUTF8* parameters were added. + + .. versionchanged:: 3.6 + *decode_data* is now ``False`` by default. The :class:`SMTPChannel` has the following instance variables: diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 02f2350..52c8f7f 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -281,6 +281,10 @@ Constants in the Unix header files are defined; for a few symbols, default values are provided. + .. versionchanged:: 3.6 + ``SO_DOMAIN``, ``SO_PROTOCOL``, ``SO_PEERSEC``, ``SO_PASSSEC`` + were added. + .. data:: AF_CAN PF_CAN SOL_CAN_* @@ -329,12 +333,17 @@ Constants .. versionadded:: 3.3 -.. data:: SIO_* +.. data:: SIO_RCVALL + SIO_KEEPALIVE_VALS + SIO_LOOPBACK_FAST_PATH RCVALL_* Constants for Windows' WSAIoctl(). The constants are used as arguments to the :meth:`~socket.socket.ioctl` method of socket objects. + .. versionchanged:: 3.6 + ``SIO_LOOPBACK_FAST_PATH`` was added. + .. data:: TIPC_* @@ -872,6 +881,10 @@ to sockets. it is recommended to :meth:`close` them explicitly, or to use a :keyword:`with` statement around them. + .. versionchanged:: 3.6 + :exc:`OSError` is now raised if an error occurs when the underlying + :c:func:`close` call is made. + .. note:: :meth:`close()` releases the resource associated with a connection but @@ -992,6 +1005,12 @@ to sockets. On other platforms, the generic :func:`fcntl.fcntl` and :func:`fcntl.ioctl` functions may be used; they accept a socket object as their first argument. + Currently only the following control codes are supported: + ``SIO_RCVALL``, ``SIO_KEEPALIVE_VALS``, and ``SIO_LOOPBACK_FAST_PATH``. + + .. versionchanged:: 3.6 + ``SIO_LOOPBACK_FAST_PATH`` was added. + .. method:: socket.listen([backlog]) Enable a server to accept connections. If *backlog* is specified, it must diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index 087f4e0..218a31c 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -52,11 +52,12 @@ handler class by subclassing the :class:`BaseRequestHandler` class and overriding its :meth:`~BaseRequestHandler.handle` method; this method will process incoming requests. Second, you must instantiate one of the server classes, passing it -the server's address and the request handler class. Then call the +the server's address and the request handler class. It is recommended to use +the server in a :keyword:`with` statement. Then call the :meth:`~BaseServer.handle_request` or :meth:`~BaseServer.serve_forever` method of the server object to process one or many requests. Finally, call :meth:`~BaseServer.server_close` -to close the socket. +to close the socket (unless you used a :keyword:`with` statement). When inheriting from :class:`ThreadingMixIn` for threaded connection behavior, you should explicitly declare how you want your threads to behave on an abrupt @@ -111,6 +112,8 @@ server classes. :class:`UDPServer`. Setting the various attributes also changes the behavior of the underlying server mechanism. + :class:`ForkingMixIn` and the Forking classes mentioned below are + only available on POSIX platforms that support :func:`~os.fork`. .. class:: ForkingTCPServer ForkingUDPServer @@ -304,7 +307,11 @@ Server Objects This function is called if the :meth:`~BaseRequestHandler.handle` method of a :attr:`RequestHandlerClass` instance raises an exception. The default action is to print the traceback to - standard output and continue handling further requests. + standard error and continue handling further requests. + + .. versionchanged:: 3.6 + Now only called for exceptions derived from the :exc:`Exception` + class. .. method:: handle_timeout() @@ -349,6 +356,11 @@ Server Objects default implementation always returns :const:`True`. + .. versionchanged:: 3.6 + Support for the :term:`context manager` protocol was added. Exiting the + context manager is equivalent to calling :meth:`server_close`. + + Request Handler Objects ----------------------- @@ -397,6 +409,15 @@ Request Handler Objects read or written, respectively, to get the request data or return data to the client. + The :attr:`rfile` attributes of both classes support the + :class:`io.BufferedIOBase` readable interface, and + :attr:`DatagramRequestHandler.wfile` supports the + :class:`io.BufferedIOBase` writable interface. + + .. versionchanged:: 3.6 + :attr:`StreamRequestHandler.wfile` also supports the + :class:`io.BufferedIOBase` writable interface. + Examples -------- @@ -429,11 +450,10 @@ This is the server side:: HOST, PORT = "localhost", 9999 # Create the server, binding to localhost on port 9999 - server = socketserver.TCPServer((HOST, PORT), MyTCPHandler) - - # Activate the server; this will keep running until you - # interrupt the program with Ctrl-C - server.serve_forever() + with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server: + # Activate the server; this will keep running until you + # interrupt the program with Ctrl-C + server.serve_forever() An alternative request handler class that makes use of streams (file-like objects that simplify communication by providing the standard file interface):: @@ -525,8 +545,8 @@ This is the server side:: if __name__ == "__main__": HOST, PORT = "localhost", 9999 - server = socketserver.UDPServer((HOST, PORT), MyUDPHandler) - server.serve_forever() + with socketserver.UDPServer((HOST, PORT), MyUDPHandler) as server: + server.serve_forever() This is the client side:: @@ -585,22 +605,22 @@ An example for the :class:`ThreadingMixIn` class:: HOST, PORT = "localhost", 0 server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler) - ip, port = server.server_address + with server: + ip, port = server.server_address - # Start a thread with the server -- that thread will then start one - # more thread for each request - server_thread = threading.Thread(target=server.serve_forever) - # Exit the server thread when the main thread terminates - server_thread.daemon = True - server_thread.start() - print("Server loop running in thread:", server_thread.name) + # Start a thread with the server -- that thread will then start one + # more thread for each request + server_thread = threading.Thread(target=server.serve_forever) + # Exit the server thread when the main thread terminates + server_thread.daemon = True + server_thread.start() + print("Server loop running in thread:", server_thread.name) - client(ip, port, "Hello World 1") - client(ip, port, "Hello World 2") - client(ip, port, "Hello World 3") + client(ip, port, "Hello World 1") + client(ip, port, "Hello World 2") + client(ip, port, "Hello World 3") - server.shutdown() - server.server_close() + server.shutdown() The output of the example should look something like this: @@ -616,3 +636,5 @@ The output of the example should look something like this: The :class:`ForkingMixIn` class is used in the same way, except that the server will spawn a new process for each request. +Available only on POSIX platforms that support :func:`~os.fork`. + diff --git a/Doc/library/spwd.rst b/Doc/library/spwd.rst index fd3c9ad..c6cad2a 100644 --- a/Doc/library/spwd.rst +++ b/Doc/library/spwd.rst @@ -55,6 +55,9 @@ The following functions are defined: Return the shadow password database entry for the given user name. + .. versionchanged:: 3.6 + Raises a :exc:`PermissionError` instead of :exc:`KeyError` if the user + doesn't have privileges. .. function:: getspall() diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 8009971..e53efda 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -629,9 +629,16 @@ Cursor Objects .. attribute:: lastrowid This read-only attribute provides the rowid of the last modified row. It is - only set if you issued an ``INSERT`` statement using the :meth:`execute` - method. For operations other than ``INSERT`` or when :meth:`executemany` is - called, :attr:`lastrowid` is set to :const:`None`. + only set if you issued an ``INSERT`` or a ``REPLACE`` statement using the + :meth:`execute` method. For operations other than ``INSERT`` or + ``REPLACE`` or when :meth:`executemany` is called, :attr:`lastrowid` is + set to :const:`None`. + + If the ``INSERT`` or ``REPLACE`` statement failed to insert the previous + successful rowid is returned. + + .. versionchanged:: 3.6 + Added support for the ``REPLACE`` statement. .. attribute:: description diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst index ea3d7da..232fb752 100644 --- a/Doc/library/statistics.rst +++ b/Doc/library/statistics.rst @@ -39,6 +39,8 @@ or sample. ======================= ============================================= :func:`mean` Arithmetic mean ("average") of data. +:func:`geometric_mean` Geometric mean of data. +:func:`harmonic_mean` Harmonic mean of data. :func:`median` Median (middle value) of data. :func:`median_low` Low median of data. :func:`median_high` High median of data. @@ -111,6 +113,66 @@ However, for reading convenience, most of the examples show sorted sequences. ``mean(data)`` is equivalent to calculating the true population mean μ. +.. function:: geometric_mean(data) + + Return the geometric mean of *data*, a sequence or iterator of + real-valued numbers. + + The geometric mean is the *n*-th root of the product of *n* data points. + It is a type of average, a measure of the central location of the data. + + The geometric mean is appropriate when averaging quantities which + are multiplied together rather than added, for example growth rates. + Suppose an investment grows by 10% in the first year, falls by 5% in + the second, then grows by 12% in the third, what is the average rate + of growth over the three years? + + .. doctest:: + + >>> geometric_mean([1.10, 0.95, 1.12]) + 1.0538483123382172 + + giving an average growth of 5.385%. Using the arithmetic mean will + give approximately 5.667%, which is too high. + + :exc:`StatisticsError` is raised if *data* is empty, or any + element is less than zero. + + .. versionadded:: 3.6 + + +.. function:: harmonic_mean(data) + + Return the harmonic mean of *data*, a sequence or iterator of + real-valued numbers. + + The harmonic mean, sometimes called the subcontrary mean, is the + reciprocal of the arithmetic :func:`mean` of the reciprocals of the + data. For example, the harmonic mean of three values *a*, *b* and *c* + will be equivalent to ``3/(1/a + 1/b + 1/c)``. + + The harmonic mean is a type of average, a measure of the central + location of the data. It is often appropriate when averaging quantities + which are rates or ratios, for example speeds. For example: + + Suppose an investor purchases an equal value of shares in each of + three companies, with P/E (price/earning) ratios of 2.5, 3 and 10. + What is the average P/E ratio for the investor's portfolio? + + .. doctest:: + + >>> harmonic_mean([2.5, 3, 10]) # For an equal investment portfolio. + 3.6 + + Using the arithmetic mean would give an average of about 5.167, which + is too high. + + :exc:`StatisticsError` is raised if *data* is empty, or any element + is less than zero. + + .. versionadded:: 3.6 + + .. function:: median(data) Return the median (middle value) of numeric data, using the common "mean of diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 2996eef..76ecd01 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1453,8 +1453,8 @@ multiple fragments. For more information on the ``str`` class and its methods, see :ref:`textseq` and the :ref:`string-methods` section below. To output - formatted strings, see the :ref:`formatstrings` section. In addition, - see the :ref:`stringservices` section. + formatted strings, see the :ref:`f-strings` and :ref:`formatstrings` + sections. In addition, see the :ref:`stringservices` section. .. index:: @@ -2056,8 +2056,8 @@ expression support in the :mod:`re` module). .. index:: single: formatting, string (%) single: interpolation, string (%) - single: string; formatting - single: string; interpolation + single: string; formatting, printf + single: string; interpolation, printf single: printf-style formatting single: sprintf-style formatting single: % formatting @@ -2067,9 +2067,10 @@ expression support in the :mod:`re` module). The formatting operations described here exhibit a variety of quirks that lead to a number of common errors (such as failing to display tuples and - dictionaries correctly). Using the newer :meth:`str.format` interface - helps avoid these errors, and also provides a generally more powerful, - flexible and extensible approach to formatting text. + dictionaries correctly). Using the newer :ref:`formatted + string literals <f-strings>` or the :meth:`str.format` interface + helps avoid these errors. These alternatives also provide more powerful, + flexible and extensible approaches to formatting text. String objects have one unique built-in operation: the ``%`` operator (modulo). This is also known as the string *formatting* or *interpolation* operator. diff --git a/Doc/library/string.rst b/Doc/library/string.rst index d5d2430..c421c72 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -188,7 +188,9 @@ Format String Syntax The :meth:`str.format` method and the :class:`Formatter` class share the same syntax for format strings (although in the case of :class:`Formatter`, -subclasses can define their own format string syntax). +subclasses can define their own format string syntax). The syntax is +related to that of :ref:`formatted string literals <f-strings>`, but +there are differences. Format strings contain "replacement fields" surrounded by curly braces ``{}``. Anything that is not contained in braces is considered literal text, which is @@ -283,7 +285,8 @@ Format Specification Mini-Language "Format specifications" are used within replacement fields contained within a format string to define how individual values are presented (see -:ref:`formatstrings`). They can also be passed directly to the built-in +:ref:`formatstrings` and :ref:`f-strings`). +They can also be passed directly to the built-in :func:`format` function. Each formattable type may define how the format specification is to be interpreted. @@ -308,7 +311,8 @@ The general form of a *standard format specifier* is: If a valid *align* value is specified, it can be preceded by a *fill* character that can be any character and defaults to a space if omitted. It is not possible to use a literal curly brace ("``{``" or "``}``") as -the *fill* character when using the :meth:`str.format` +the *fill* character in a :ref:`formatted string literal +<f-strings>` or when using the :meth:`str.format` method. However, it is possible to insert a curly brace with a nested replacement field. This limitation doesn't affect the :func:`format` function. diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 356605f..ab20889 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -502,6 +502,10 @@ functions. .. versionchanged:: 3.2 Added context manager support. + .. versionchanged:: 3.6 + Popen destructor now emits a :exc:`ResourceWarning` warning if the child + process is still running. + Exceptions ^^^^^^^^^^ diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index ed5db05..8c9ca2a 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -256,7 +256,7 @@ always available. (defaulting to zero), or another type of object. If it is an integer, zero is considered "successful termination" and any nonzero value is considered "abnormal termination" by shells and the like. Most systems require it to be - in the range 0-127, and produce undefined results otherwise. Some systems + in the range 0--127, and produce undefined results otherwise. Some systems have a convention for assigning specific meanings to specific exit codes, but these are generally underdeveloped; Unix programs generally use 2 for command line syntax errors and 1 for all other kind of errors. If another type of @@ -269,6 +269,11 @@ always available. the process when called from the main thread, and the exception is not intercepted. + .. versionchanged:: 3.6 + If an error occurs in the cleanup after the Python interpreter + has caught :exc:`SystemExit` (such as an error flushing buffered data + in the standard streams), the exit status is changed to 120. + .. data:: flags diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst index c51567a..ab248e7 100644 --- a/Doc/library/sysconfig.rst +++ b/Doc/library/sysconfig.rst @@ -164,7 +164,7 @@ Other functions .. function:: get_python_version() Return the ``MAJOR.MINOR`` Python version number as a string. Similar to - ``sys.version[:3]``. + ``'%d.%d' % sys.version_info[:2]``. .. function:: get_platform() diff --git a/Doc/library/telnetlib.rst b/Doc/library/telnetlib.rst index b950e41..f9c5153 100644 --- a/Doc/library/telnetlib.rst +++ b/Doc/library/telnetlib.rst @@ -43,6 +43,17 @@ Character), EL (Erase Line), GA (Go Ahead), SB (Subnegotiation Begin). :exc:`EOFError` when the end of the connection is read, because they can return an empty string for other reasons. See the individual descriptions below. + A :class:`Telnet` object is a context manager and can be used in a + :keyword:`with` statement. When the :keyword:`with` block ends, the + :meth:`close` method is called:: + + >>> from telnetlib import Telnet + >>> with Telnet('localhost', 23) as tn: + ... tn.interact() + ... + + .. versionchanged:: 3.6 Context manager support added + .. seealso:: diff --git a/Doc/library/test.rst b/Doc/library/test.rst index 2ea9c27..7114cdf 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -582,6 +582,48 @@ The :mod:`test.support` module defines the following functions: .. versionadded:: 3.5 +.. function:: check__all__(test_case, module, name_of_module=None, extra=(), blacklist=()) + + Assert that the ``__all__`` variable of *module* contains all public names. + + The module's public names (its API) are detected automatically + based on whether they match the public name convention and were defined in + *module*. + + The *name_of_module* argument can specify (as a string or tuple thereof) what + module(s) an API could be defined in in order to be detected as a public + API. One case for this is when *module* imports part of its public API from + other modules, possibly a C backend (like ``csv`` and its ``_csv``). + + The *extra* argument can be a set of names that wouldn't otherwise be automatically + detected as "public", like objects without a proper ``__module__`` + attribute. If provided, it will be added to the automatically detected ones. + + The *blacklist* argument can be a set of names that must not be treated as part of + the public API even though their names indicate otherwise. + + Example use:: + + import bar + import foo + import unittest + from test import support + + class MiscTestCase(unittest.TestCase): + def test__all__(self): + support.check__all__(self, foo) + + class OtherTestCase(unittest.TestCase): + def test__all__(self): + extra = {'BAR_CONST', 'FOO_CONST'} + blacklist = {'baz'} # Undocumented name. + # bar imports part of its API from _bar. + support.check__all__(self, bar, ('bar', '_bar'), + extra=extra, blacklist=blacklist) + + .. versionadded:: 3.6 + + The :mod:`test.support` module defines the following classes: .. class:: TransientResource(exc, **kwargs) diff --git a/Doc/library/time.rst b/Doc/library/time.rst index e6626f2..7c81ce7 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -637,11 +637,11 @@ The module defines the following functions and data items: it is possible to refer to February 29. :samp:`M{m}.{n}.{d}` - The *d*'th day (0 <= *d* <= 6) or week *n* of month *m* of the year (1 + The *d*'th day (0 <= *d* <= 6) of week *n* of month *m* of the year (1 <= *n* <= 5, 1 <= *m* <= 12, where week 5 means "the last *d* day in month *m*" which may occur in either the fourth or the fifth week). Week 1 is the first week in which the *d*'th day occurs. Day - zero is Sunday. + zero is a Sunday. ``time`` has the same format as ``offset`` except that no leading sign ('-' or '+') is allowed. The default, if time is not given, is 02:00:00. diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst index 57a4834..5bae33b 100644 --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -100,8 +100,8 @@ The module defines three convenience functions and a public class: can be controlled by passing a namespace to *globals*. To measure the execution time of the first statement, use the :meth:`.timeit` - method. The :meth:`.repeat` method is a convenience to call :meth:`.timeit` - multiple times and return a list of results. + method. The :meth:`.repeat` and :meth:`.autorange` methods are convenience + methods to call :meth:`.timeit` multiple times. The execution time of *setup* is excluded from the overall timed execution run. @@ -134,6 +134,23 @@ The module defines three convenience functions and a public class: timeit.Timer('for i in range(10): oct(i)', 'gc.enable()').timeit() + .. method:: Timer.autorange(callback=None) + + Automatically determine how many times to call :meth:`.timeit`. + + This is a convenience function that calls :meth:`.timeit` repeatedly + so that the total time >= 0.2 second, returning the eventual + (number of loops, time taken for that number of loops). It calls + :meth:`.timeit` with *number* set to successive powers of ten (10, + 100, 1000, ...) up to a maximum of one billion, until the time taken + is at least 0.2 second, or the maximum is reached. + + If *callback* is given and is not *None*, it will be called after + each trial with two arguments: ``callback(number, time_taken)``. + + .. versionadded:: 3.6 + + .. method:: Timer.repeat(repeat=3, number=1000000) Call :meth:`.timeit` a few times. diff --git a/Doc/library/traceback.rst b/Doc/library/traceback.rst index 3c1d9bb..066ee96 100644 --- a/Doc/library/traceback.rst +++ b/Doc/library/traceback.rst @@ -291,6 +291,20 @@ capture data for later printing in a lightweight fashion. of tuples. Each tuple should be a 4-tuple with filename, lineno, name, line as the elements. + .. method:: format() + + Returns a list of strings ready for printing. Each string in the + resulting list corresponds to a single frame from the stack. + Each string ends in a newline; the strings may contain internal + newlines as well, for those items with source text lines. + + For long sequences of the same frame and line, the first few + repetitions are shown, followed by a summary line stating the exact + number of further repetitions. + + .. versionchanged:: 3.6 + Long sequences of repeated frames are now abbreviated. + :class:`FrameSummary` Objects ----------------------------- diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst index 3a0b1e0..f56f27b 100644 --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -359,10 +359,32 @@ Functions See also the :func:`get_object_traceback` function. +DomainFilter +^^^^^^^^^^^^ + +.. class:: DomainFilter(inclusive: bool, domain: int) + + Filter traces of memory blocks by their address space (domain). + + .. versionadded:: 3.6 + + .. attribute:: inclusive + + If *inclusive* is ``True`` (include), match memory blocks allocated + in the address space :attr:`domain`. + + If *inclusive* is ``False`` (exclude), match memory blocks not allocated + in the address space :attr:`domain`. + + .. attribute:: domain + + Address space of a memory block (``int``). Read-only property. + + Filter ^^^^^^ -.. class:: Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False) +.. class:: Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False, domain: int=None) Filter on traces of memory blocks. @@ -382,9 +404,17 @@ Filter .. versionchanged:: 3.5 The ``'.pyo'`` file extension is no longer replaced with ``'.py'``. + .. versionchanged:: 3.6 + Added the :attr:`domain` attribute. + + + .. attribute:: domain + + Address space of a memory block (``int`` or ``None``). + .. attribute:: inclusive - If *inclusive* is ``True`` (include), only trace memory blocks allocated + If *inclusive* is ``True`` (include), only match memory blocks allocated in a file with a name matching :attr:`filename_pattern` at line number :attr:`lineno`. @@ -399,7 +429,7 @@ Filter .. attribute:: filename_pattern - Filename pattern of the filter (``str``). + Filename pattern of the filter (``str``). Read-only property. .. attribute:: all_frames @@ -462,14 +492,17 @@ Snapshot .. method:: filter_traces(filters) Create a new :class:`Snapshot` instance with a filtered :attr:`traces` - sequence, *filters* is a list of :class:`Filter` instances. If *filters* - is an empty list, return a new :class:`Snapshot` instance with a copy of - the traces. + sequence, *filters* is a list of :class:`DomainFilter` and + :class:`Filter` instances. If *filters* is an empty list, return a new + :class:`Snapshot` instance with a copy of the traces. All inclusive filters are applied at once, a trace is ignored if no inclusive filters match it. A trace is ignored if at least one exclusive filter matches it. + .. versionchanged:: 3.6 + :class:`DomainFilter` instances are now also accepted in *filters*. + .. classmethod:: load(filename) diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index bc71e1e..3eaf166 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -504,11 +504,15 @@ The module defines the following classes, functions and decorators: .. class:: Iterable(Generic[T_co]) - A generic version of the :class:`collections.abc.Iterable`. + A generic version of :class:`collections.abc.Iterable`. .. class:: Iterator(Iterable[T_co]) - A generic version of the :class:`collections.abc.Iterator`. + A generic version of :class:`collections.abc.Iterator`. + +.. class:: Reversible(Iterable[T_co]) + + A generic version of :class:`collections.abc.Reversible`. .. class:: SupportsInt @@ -528,11 +532,6 @@ The module defines the following classes, functions and decorators: An ABC with one abstract method ``__round__`` that is covariant in its return type. -.. class:: Reversible - - An ABC with one abstract method ``__reversed__`` returning - an ``Iterator[T_co]``. - .. class:: Container(Generic[T_co]) A generic version of :class:`collections.abc.Container`. @@ -553,7 +552,7 @@ The module defines the following classes, functions and decorators: A generic version of :class:`collections.abc.MutableMapping`. -.. class:: Sequence(Sized, Iterable[T_co], Container[T_co]) +.. class:: Sequence(Sized, Reversible[T_co], Container[T_co]) A generic version of :class:`collections.abc.Sequence`. @@ -608,6 +607,12 @@ The module defines the following classes, functions and decorators: A generic version of :class:`collections.abc.ValuesView`. +.. class:: ContextManager(Generic[T_co]) + + A generic version of :class:`contextlib.AbstractContextManager`. + + .. versionadded:: 3.6 + .. class:: Dict(dict, MutableMapping[KT, VT]) A generic version of :class:`dict`. diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index c13f095..1bc1edb 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -262,6 +262,34 @@ the *new_callable* argument to :func:`patch`. used to set attributes on the mock after it is created. See the :meth:`configure_mock` method for details. + .. method:: assert_called(*args, **kwargs) + + Assert that the mock was called at least once. + + >>> mock = Mock() + >>> mock.method() + <Mock name='mock.method()' id='...'> + >>> mock.method.assert_called() + + .. versionadded:: 3.6 + + .. method:: assert_called_once(*args, **kwargs) + + Assert that the mock was called exactly once. + + >>> mock = Mock() + >>> mock.method() + <Mock name='mock.method()' id='...'> + >>> mock.method.assert_called_once() + >>> mock.method() + <Mock name='mock.method()' id='...'> + >>> mock.method.assert_called_once() + Traceback (most recent call last): + ... + AssertionError: Expected 'method' to have been called once. Called 2 times. + + .. versionadded:: 3.6 + .. method:: assert_called_with(*args, **kwargs) @@ -339,7 +367,7 @@ the *new_callable* argument to :func:`patch`. .. versionadded:: 3.5 - .. method:: reset_mock() + .. method:: reset_mock(*, return_value=False, side_effect=False) The reset_mock method resets all the call attributes on a mock object: @@ -351,12 +379,20 @@ the *new_callable* argument to :func:`patch`. >>> mock.called False + .. versionchanged:: 3.6 + Added two keyword only argument to the reset_mock function. + This can be useful where you want to make a series of assertions that reuse the same object. Note that :meth:`reset_mock` *doesn't* clear the return value, :attr:`side_effect` or any child attributes you have - set using normal assignment. Child mocks and the return value mock + set using normal assignment by default. In case you want to reset + *return_value* or :attr:`side_effect`, then pass the corresponding + parameter as ``True``. Child mocks and the return value mock (if any) are reset as well. + .. note:: *return_value*, and :attr:`side_effect` are keyword only + argument. + .. method:: mock_add_spec(spec, spec_set=False) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index e5646ae..9f7fc57 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -1393,9 +1393,9 @@ Test cases Add a function to be called after :meth:`tearDown` to cleanup resources used during the test. Functions will be called in reverse order to the - order they are added (LIFO). They are called with any arguments and - keyword arguments passed into :meth:`addCleanup` when they are - added. + order they are added (:abbr:`LIFO (last-in, first-out)`). They + are called with any arguments and keyword arguments passed into + :meth:`addCleanup` when they are added. If :meth:`setUp` fails, meaning that :meth:`tearDown` is not called, then any cleanup functions added will still be called. diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index c6de230..d79d8f0 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -114,8 +114,9 @@ or on combining URL components into a URL string. | | | if present | | +------------------+-------+--------------------------+----------------------+ - See section :ref:`urlparse-result-object` for more information on the result - object. + Reading the :attr:`port` attribute will raise a :exc:`ValueError` if + an invalid port is specified in the URL. See section + :ref:`urlparse-result-object` for more information on the result object. .. versionchanged:: 3.2 Added IPv6 URL parsing capabilities. @@ -125,6 +126,10 @@ or on combining URL components into a URL string. false), in accordance with :rfc:`3986`. Previously, a whitelist of schemes that support fragments existed. + .. versionchanged:: 3.6 + Out-of-range port numbers now raise :exc:`ValueError`, instead of + returning :const:`None`. + .. function:: parse_qs(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace') @@ -227,8 +232,13 @@ or on combining URL components into a URL string. | | | if present | | +------------------+-------+-------------------------+----------------------+ - See section :ref:`urlparse-result-object` for more information on the result - object. + Reading the :attr:`port` attribute will raise a :exc:`ValueError` if + an invalid port is specified in the URL. See section + :ref:`urlparse-result-object` for more information on the result object. + + .. versionchanged:: 3.6 + Out-of-range port numbers now raise :exc:`ValueError`, instead of + returning :const:`None`. .. function:: urlunsplit(parts) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 1291aeb..e619cc1 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -30,18 +30,9 @@ The :mod:`urllib.request` module defines the following functions: Open the URL *url*, which can be either a string or a :class:`Request` object. - *data* must be a bytes object specifying additional data to be sent to the - server, or ``None`` if no such data is needed. *data* may also be an - iterable object and in that case Content-Length value must be specified in - the headers. Currently HTTP requests are the only ones that use *data*; the - HTTP request will be a POST instead of a GET when the *data* parameter is - provided. - - *data* should be a buffer in the standard - :mimetype:`application/x-www-form-urlencoded` format. The - :func:`urllib.parse.urlencode` function takes a mapping or sequence of - 2-tuples and returns an ASCII text string in this format. It should - be encoded to bytes before being used as the *data* parameter. + *data* must be an object specifying additional data to be sent to the + server, or ``None`` if no such data is needed. See :class:`Request` + for details. urllib.request module uses HTTP/1.1 and includes ``Connection:close`` header in its HTTP requests. @@ -192,14 +183,22 @@ The following classes are provided: *url* should be a string containing a valid URL. - *data* must be a bytes object specifying additional data to send to the - server, or ``None`` if no such data is needed. Currently HTTP requests are - the only ones that use *data*; the HTTP request will be a POST instead of a - GET when the *data* parameter is provided. *data* should be a buffer in the - standard :mimetype:`application/x-www-form-urlencoded` format. - The :func:`urllib.parse.urlencode` function takes a mapping or sequence of - 2-tuples and returns an ASCII string in this format. It should be - encoded to bytes before being used as the *data* parameter. + *data* must be an object specifying additional data to send to the + server, or ``None`` if no such data is needed. Currently HTTP + requests are the only ones that use *data*. The supported object + types include bytes, file-like objects, and iterables. If no + ``Content-Length`` header has been provided, :class:`HTTPHandler` will + try to determine the length of *data* and set this header accordingly. + If this fails, ``Transfer-Encoding: chunked`` as specified in + :rfc:`7230`, Section 3.3.1 will be used to send the data. See + :meth:`http.client.HTTPConnection.request` for details on the + supported object types and on how the content length is determined. + + For an HTTP POST request method, *data* should be a buffer in the + standard :mimetype:`application/x-www-form-urlencoded` format. The + :func:`urllib.parse.urlencode` function takes a mapping or sequence + of 2-tuples and returns an ASCII string in this format. It should + be encoded to bytes before being used as the *data* parameter. *headers* should be a dictionary, and will be treated as if :meth:`add_header` was called with each key and value as arguments. @@ -211,8 +210,10 @@ The following classes are provided: :mod:`urllib`'s default user agent string is ``"Python-urllib/2.6"`` (on Python 2.6). - An example of using ``Content-Type`` header with *data* argument would be - sending a dictionary like ``{"Content-Type": "application/x-www-form-urlencoded"}``. + An appropriate ``Content-Type`` header should be included if the *data* + argument is present. If this header has not been provided and *data* + is not None, ``Content-Type: application/x-www-form-urlencoded`` will + be added as a default. The final two arguments are only of interest for correct handling of third-party HTTP cookies: @@ -235,15 +236,28 @@ The following classes are provided: *method* should be a string that indicates the HTTP request method that will be used (e.g. ``'HEAD'``). If provided, its value is stored in the :attr:`~Request.method` attribute and is used by :meth:`get_method()`. - Subclasses may indicate a default method by setting the + The default is ``'GET'`` if *data* is ``None`` or ``'POST'`` otherwise. + Subclasses may indicate a different default method by setting the :attr:`~Request.method` attribute in the class itself. + .. note:: + The request will not work as expected if the data object is unable + to deliver its content more than once (e.g. a file or an iterable + that can produce the content only once) and the request is retried + for HTTP redirects or authentication. The *data* is sent to the + HTTP server right away after the headers. There is no support for + a 100-continue expectation in the library. + .. versionchanged:: 3.3 :attr:`Request.method` argument is added to the Request class. .. versionchanged:: 3.4 Default :attr:`Request.method` may be indicated at the class level. + .. versionchanged:: 3.6 + Do not raise an error if the ``Content-Length`` has not been + provided and could not be determined. Fall back to use chunked + transfer encoding instead. .. class:: OpenerDirector() diff --git a/Doc/library/urllib.robotparser.rst b/Doc/library/urllib.robotparser.rst index ba701c3..7d31932 100644 --- a/Doc/library/urllib.robotparser.rst +++ b/Doc/library/urllib.robotparser.rst @@ -57,15 +57,41 @@ structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html. Sets the time the ``robots.txt`` file was last fetched to the current time. + .. method:: crawl_delay(useragent) -The following example demonstrates basic use of the RobotFileParser class. + Returns the value of the ``Crawl-delay`` parameter from ``robots.txt`` + for the *useragent* in question. If there is no such parameter or it + doesn't apply to the *useragent* specified or the ``robots.txt`` entry + for this parameter has invalid syntax, return ``None``. + + .. versionadded:: 3.6 + + .. method:: request_rate(useragent) + + Returns the contents of the ``Request-rate`` parameter from + ``robots.txt`` in the form of a :func:`~collections.namedtuple` + ``(requests, seconds)``. If there is no such parameter or it doesn't + apply to the *useragent* specified or the ``robots.txt`` entry for this + parameter has invalid syntax, return ``None``. + + .. versionadded:: 3.6 + + +The following example demonstrates basic use of the :class:`RobotFileParser` +class:: >>> import urllib.robotparser >>> rp = urllib.robotparser.RobotFileParser() >>> rp.set_url("http://www.musi-cal.com/robots.txt") >>> rp.read() + >>> rrate = rp.request_rate("*") + >>> rrate.requests + 3 + >>> rrate.seconds + 20 + >>> rp.crawl_delay("*") + 6 >>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco") False >>> rp.can_fetch("*", "http://www.musi-cal.com/") True - diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst index af4a6d1..6bf26ff 100644 --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -31,44 +31,50 @@ Creating virtual environments .. _venv-def: -.. note:: A virtual environment (also called a ``venv``) is a Python - environment such that the Python interpreter, libraries and scripts - installed into it are isolated from those installed in other virtual - environments, and (by default) any libraries installed in a "system" Python, - i.e. one which is installed as part of your operating system. +.. note:: A virtual environment is a Python environment such that the Python + interpreter, libraries and scripts installed into it are isolated from those + installed in other virtual environments, and (by default) any libraries + installed in a "system" Python, i.e., one which is installed as part of your + operating system. - A venv is a directory tree which contains Python executable files and - other files which indicate that it is a venv. + A virtual environment is a directory tree which contains Python executable + files and other files which indicate that it is a virtual environment. Common installation tools such as ``Setuptools`` and ``pip`` work as - expected with venvs - i.e. when a venv is active, they install Python - packages into the venv without needing to be told to do so explicitly. - - When a venv is active (i.e. the venv's Python interpreter is running), the - attributes :attr:`sys.prefix` and :attr:`sys.exec_prefix` point to the base - directory of the venv, whereas :attr:`sys.base_prefix` and - :attr:`sys.base_exec_prefix` point to the non-venv Python installation - which was used to create the venv. If a venv is not active, then - :attr:`sys.prefix` is the same as :attr:`sys.base_prefix` and - :attr:`sys.exec_prefix` is the same as :attr:`sys.base_exec_prefix` (they - all point to a non-venv Python installation). - - When a venv is active, any options that change the installation path will be - ignored from all distutils configuration files to prevent projects being - inadvertently installed outside of the virtual environment. - - When working in a command shell, users can make a venv active by running an - ``activate`` script in the venv's executables directory (the precise filename - is shell-dependent), which prepends the venv's directory for executables to - the ``PATH`` environment variable for the running shell. There should be no - need in other circumstances to activate a venv -- scripts installed into - venvs have a shebang line which points to the venv's Python interpreter. This - means that the script will run with that interpreter regardless of the value - of ``PATH``. On Windows, shebang line processing is supported if you have the - Python Launcher for Windows installed (this was added to Python in 3.3 - see - :pep:`397` for more details). Thus, double-clicking an installed script in - a Windows Explorer window should run the script with the correct interpreter - without there needing to be any reference to its venv in ``PATH``. + expected with virtual environments. In other words, when a virtual + environment is active, they install Python packages into the virtual + environment without needing to be told to do so explicitly. + + When a virtual environment is active (i.e., the virtual environment's Python + interpreter is running), the attributes :attr:`sys.prefix` and + :attr:`sys.exec_prefix` point to the base directory of the virtual + environment, whereas :attr:`sys.base_prefix` and + :attr:`sys.base_exec_prefix` point to the non-virtual environment Python + installation which was used to create the virtual environment. If a virtual + environment is not active, then :attr:`sys.prefix` is the same as + :attr:`sys.base_prefix` and :attr:`sys.exec_prefix` is the same as + :attr:`sys.base_exec_prefix` (they all point to a non-virtual environment + Python installation). + + When a virtual environment is active, any options that change the + installation path will be ignored from all distutils configuration files to + prevent projects being inadvertently installed outside of the virtual + environment. + + When working in a command shell, users can make a virtual environment active + by running an ``activate`` script in the virtual environment's executables + directory (the precise filename is shell-dependent), which prepends the + virtual environment's directory for executables to the ``PATH`` environment + variable for the running shell. There should be no need in other + circumstances to activate a virtual environment—scripts installed into + virtual environments have a "shebang" line which points to the virtual + environment's Python interpreter. This means that the script will run with + that interpreter regardless of the value of ``PATH``. On Windows, "shebang" + line processing is supported if you have the Python Launcher for Windows + installed (this was added to Python in 3.3 - see :pep:`397` for more + details). Thus, double-clicking an installed script in a Windows Explorer + window should run the script with the correct interpreter without there + needing to be any reference to its virtual environment in ``PATH``. .. _venv-api: @@ -83,7 +89,8 @@ mechanisms for third-party virtual environment creators to customize environment creation according to their needs, the :class:`EnvBuilder` class. .. class:: EnvBuilder(system_site_packages=False, clear=False, \ - symlinks=False, upgrade=False, with_pip=False) + symlinks=False, upgrade=False, with_pip=False, \ + prompt=None) The :class:`EnvBuilder` class accepts the following keyword arguments on instantiation: @@ -107,9 +114,16 @@ creation according to their needs, the :class:`EnvBuilder` class. installed in the virtual environment. This uses :mod:`ensurepip` with the ``--default-pip`` option. + * ``prompt`` -- a String to be used after virtual environment is activated + (defaults to ``None`` which means directory name of the environment would + be used). + .. versionchanged:: 3.4 Added the ``with_pip`` parameter + .. versionadded:: 3.6 + Added the ``prompt`` parameter + Creators of third-party virtual environment tools will be free to use the provided ``EnvBuilder`` class as a base class. @@ -219,7 +233,7 @@ An example of extending ``EnvBuilder`` -------------------------------------- The following script shows how to extend :class:`EnvBuilder` by implementing a -subclass which installs setuptools and pip into a created venv:: +subclass which installs setuptools and pip into a created virtual environment:: import os import os.path @@ -233,12 +247,12 @@ subclass which installs setuptools and pip into a created venv:: class ExtendedEnvBuilder(venv.EnvBuilder): """ This builder installs setuptools and pip so that you can pip or - easy_install other packages into the created environment. + easy_install other packages into the created virtual environment. :param nodist: If True, setuptools and pip are not installed into the - created environment. + created virtual environment. :param nopip: If True, pip is not installed into the created - environment. + virtual environment. :param progress: If setuptools or pip are installed, the progress of the installation can be monitored by passing a progress callable. If specified, it is called with two @@ -264,10 +278,10 @@ subclass which installs setuptools and pip into a created venv:: def post_setup(self, context): """ Set up any packages which need to be pre-installed into the - environment being created. + virtual environment being created. - :param context: The information for the environment creation request - being processed. + :param context: The information for the virtual environment + creation request being processed. """ os.environ['VIRTUAL_ENV'] = context.env_dir if not self.nodist: @@ -301,7 +315,7 @@ subclass which installs setuptools and pip into a created venv:: fn = os.path.split(path)[-1] binpath = context.bin_path distpath = os.path.join(binpath, fn) - # Download script into the env's binaries folder + # Download script into the virtual environment's binaries folder urlretrieve(url, distpath) progress = self.progress if self.verbose: @@ -313,7 +327,7 @@ subclass which installs setuptools and pip into a created venv:: else: sys.stderr.write('Installing %s ...%s' % (name, term)) sys.stderr.flush() - # Install in the env + # Install in the virtual environment args = [context.env_exe, fn] p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) t1 = Thread(target=self.reader, args=(p.stdout, 'stdout')) @@ -332,10 +346,10 @@ subclass which installs setuptools and pip into a created venv:: def install_setuptools(self, context): """ - Install setuptools in the environment. + Install setuptools in the virtual environment. - :param context: The information for the environment creation request - being processed. + :param context: The information for the virtual environment + creation request being processed. """ url = 'https://bitbucket.org/pypa/setuptools/downloads/ez_setup.py' self.install_script(context, 'setuptools', url) @@ -348,10 +362,10 @@ subclass which installs setuptools and pip into a created venv:: def install_pip(self, context): """ - Install pip in the environment. + Install pip in the virtual environment. - :param context: The information for the environment creation request - being processed. + :param context: The information for the virtual environment + creation request being processed. """ url = 'https://raw.github.com/pypa/pip/master/contrib/get-pip.py' self.install_script(context, 'pip', url) @@ -374,7 +388,8 @@ subclass which installs setuptools and pip into a created venv:: 'more target ' 'directories.') parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', - help='A directory to create the environment in.') + help='A directory in which to create the + 'virtual environment.') parser.add_argument('--no-setuptools', default=False, action='store_true', dest='nodist', help="Don't install setuptools or pip in the " @@ -398,14 +413,17 @@ subclass which installs setuptools and pip into a created venv:: 'the platform.') parser.add_argument('--clear', default=False, action='store_true', dest='clear', help='Delete the contents of the ' - 'environment directory if it ' - 'already exists, before ' + 'virtual environment ' + 'directory if it already ' + 'exists, before virtual ' 'environment creation.') parser.add_argument('--upgrade', default=False, action='store_true', - dest='upgrade', help='Upgrade the environment ' - 'directory to use this version ' - 'of Python, assuming Python ' - 'has been upgraded in-place.') + dest='upgrade', help='Upgrade the virtual ' + 'environment directory to ' + 'use this version of ' + 'Python, assuming Python ' + 'has been upgraded ' + 'in-place.') parser.add_argument('--verbose', default=False, action='store_true', dest='verbose', help='Display the output ' 'from the scripts which ' diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst index 37f6874..5a42cc6 100644 --- a/Doc/library/warnings.rst +++ b/Doc/library/warnings.rst @@ -301,7 +301,7 @@ Available Functions ------------------- -.. function:: warn(message, category=None, stacklevel=1) +.. function:: warn(message, category=None, stacklevel=1, source=None) Issue a warning, or maybe ignore it or raise an exception. The *category* argument, if given, must be a warning category class (see above); it defaults to @@ -319,8 +319,14 @@ Available Functions source of :func:`deprecation` itself (since the latter would defeat the purpose of the warning message). + *source*, if supplied, is the destroyed object which emitted a + :exc:`ResourceWarning`. -.. function:: warn_explicit(message, category, filename, lineno, module=None, registry=None, module_globals=None) + .. versionchanged:: 3.6 + Added *source* parameter. + + +.. function:: warn_explicit(message, category, filename, lineno, module=None, registry=None, module_globals=None, source=None) This is a low-level interface to the functionality of :func:`warn`, passing in explicitly the message, category, filename and line number, and optionally the @@ -336,6 +342,12 @@ Available Functions source for modules found in zipfiles or other non-filesystem import sources). + *source*, if supplied, is the destroyed object which emitted a + :exc:`ResourceWarning`. + + .. versionchanged:: 3.6 + Add the *source* parameter. + .. function:: showwarning(message, category, filename, lineno, file=None, line=None) diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst index 52d591a..48bdf14 100644 --- a/Doc/library/winreg.rst +++ b/Doc/library/winreg.rst @@ -635,7 +635,7 @@ For more information, see `Registry Value Types .. data:: REG_DWORD_LITTLE_ENDIAN - A 32-bit number in little-endian format. + A 32-bit number in little-endian format. Equivalent to :const:`REG_DWORD`. .. data:: REG_DWORD_BIG_ENDIAN @@ -659,6 +659,18 @@ For more information, see `Registry Value Types No defined value type. +.. data:: REG_QWORD + + A 64-bit number. + + .. versionadded:: 3.6 + +.. data:: REG_QWORD_LITTLE_ENDIAN + + A 64-bit number in little-endian format. Equivalent to :const:`REG_QWORD`. + + .. versionadded:: 3.6 + .. data:: REG_RESOURCE_LIST A device-driver resource list. diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index aad27a8..a1d4469 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -133,9 +133,9 @@ parameter expect a WSGI-compliant dictionary to be supplied; please see for key, value in environ.items()] return ret - httpd = make_server('', 8000, simple_app) - print("Serving on port 8000...") - httpd.serve_forever() + with make_server('', 8000, simple_app) as httpd: + print("Serving on port 8000...") + httpd.serve_forever() In addition to the environment functions above, the :mod:`wsgiref.util` module @@ -285,14 +285,14 @@ request. (E.g., using the :func:`shift_path_info` function from from wsgiref.simple_server import make_server, demo_app - httpd = make_server('', 8000, demo_app) - print("Serving HTTP on port 8000...") + with make_server('', 8000, demo_app) as httpd: + print("Serving HTTP on port 8000...") - # Respond to requests until process is killed - httpd.serve_forever() + # Respond to requests until process is killed + httpd.serve_forever() - # Alternative: serve one request, then exit - httpd.handle_request() + # Alternative: serve one request, then exit + httpd.handle_request() .. function:: demo_app(environ, start_response) @@ -432,9 +432,9 @@ Paste" library. # This is the application wrapped in a validator validator_app = validator(simple_app) - httpd = make_server('', 8000, validator_app) - print("Listening on port 8000....") - httpd.serve_forever() + with make_server('', 8000, validator_app) as httpd: + print("Listening on port 8000....") + httpd.serve_forever() :mod:`wsgiref.handlers` -- server/gateway base classes @@ -774,8 +774,8 @@ This is a working "Hello World" WSGI application:: # The returned object is going to be printed return [b"Hello World"] - httpd = make_server('', 8000, hello_world_app) - print("Serving on port 8000...") + with make_server('', 8000, hello_world_app) as httpd: + print("Serving on port 8000...") - # Serve until process is killed - httpd.serve_forever() + # Serve until process is killed + httpd.serve_forever() diff --git a/Doc/library/xmlrpc.server.rst b/Doc/library/xmlrpc.server.rst index 1c77e84..0511ddf 100644 --- a/Doc/library/xmlrpc.server.rst +++ b/Doc/library/xmlrpc.server.rst @@ -148,29 +148,29 @@ Server code:: rpc_paths = ('/RPC2',) # Create server - server = SimpleXMLRPCServer(("localhost", 8000), - requestHandler=RequestHandler) - server.register_introspection_functions() + with SimpleXMLRPCServer(("localhost", 8000), + requestHandler=RequestHandler) as server: + server.register_introspection_functions() - # Register pow() function; this will use the value of - # pow.__name__ as the name, which is just 'pow'. - server.register_function(pow) + # Register pow() function; this will use the value of + # pow.__name__ as the name, which is just 'pow'. + server.register_function(pow) - # Register a function under a different name - def adder_function(x,y): - return x + y - server.register_function(adder_function, 'add') + # Register a function under a different name + def adder_function(x,y): + return x + y + server.register_function(adder_function, 'add') - # Register an instance; all the methods of the instance are - # published as XML-RPC methods (in this case, just 'mul'). - class MyFuncs: - def mul(self, x, y): - return x * y + # Register an instance; all the methods of the instance are + # published as XML-RPC methods (in this case, just 'mul'). + class MyFuncs: + def mul(self, x, y): + return x * y - server.register_instance(MyFuncs()) + server.register_instance(MyFuncs()) - # Run the server's main loop - server.serve_forever() + # Run the server's main loop + server.serve_forever() The following client code will call the methods made available by the preceding server:: @@ -207,18 +207,17 @@ a server allowing dotted names and registering a multicall function. def getCurrentTime(): return datetime.datetime.now() - server = SimpleXMLRPCServer(("localhost", 8000)) - server.register_function(pow) - server.register_function(lambda x,y: x+y, 'add') - server.register_instance(ExampleService(), allow_dotted_names=True) - server.register_multicall_functions() - print('Serving XML-RPC on localhost port 8000') - try: - server.serve_forever() - except KeyboardInterrupt: - print("\nKeyboard interrupt received, exiting.") - server.server_close() - sys.exit(0) + with SimpleXMLRPCServer(("localhost", 8000)) as server: + server.register_function(pow) + server.register_function(lambda x,y: x+y, 'add') + server.register_instance(ExampleService(), allow_dotted_names=True) + server.register_multicall_functions() + print('Serving XML-RPC on localhost port 8000') + try: + server.serve_forever() + except KeyboardInterrupt: + print("\nKeyboard interrupt received, exiting.") + sys.exit(0) This ExampleService demo can be invoked from the command line:: diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index abe38c4..6c9d207 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -205,18 +205,13 @@ ZipFile Objects Return a list of archive members by name. -.. index:: - single: universal newlines; zipfile.ZipFile.open method +.. method:: ZipFile.open(name, mode='r', pwd=None, *, force_zip64=False) -.. method:: ZipFile.open(name, mode='r', pwd=None) - - Extract a member from the archive as a file-like object (ZipExtFile). *name* - is the name of the file in the archive, or a :class:`ZipInfo` object. The - *mode* parameter, if included, must be one of the following: ``'r'`` (the - default), ``'U'``, or ``'rU'``. Choosing ``'U'`` or ``'rU'`` will enable - :term:`universal newlines` support in the read-only object. *pwd* is the - password used for encrypted files. Calling :meth:`.open` on a closed - ZipFile will raise a :exc:`RuntimeError`. + Access a member of the archive as a binary file-like object. *name* + can be either the name of a file within the archive or a :class:`ZipInfo` + object. The *mode* parameter, if included, must be ``'r'`` (the default) + or ``'w'``. *pwd* is the password used to decrypt encrypted ZIP files. + Calling :meth:`.open` on a closed ZipFile will raise a :exc:`RuntimeError`. :meth:`~ZipFile.open` is also a context manager and therefore supports the :keyword:`with` statement:: @@ -225,17 +220,23 @@ ZipFile Objects with myzip.open('eggs.txt') as myfile: print(myfile.read()) - .. note:: - - The file-like object is read-only and provides the following methods: - :meth:`~io.BufferedIOBase.read`, :meth:`~io.IOBase.readline`, - :meth:`~io.IOBase.readlines`, :meth:`__iter__`, - :meth:`~iterator.__next__`. + With *mode* ``'r'`` the file-like object + (``ZipExtFile``) is read-only and provides the following methods: + :meth:`~io.BufferedIOBase.read`, :meth:`~io.IOBase.readline`, + :meth:`~io.IOBase.readlines`, :meth:`__iter__`, + :meth:`~iterator.__next__`. These objects can operate independently of + the ZipFile. - .. note:: + With ``mode='w'``, a writable file handle is returned, which supports the + :meth:`~io.BufferedIOBase.write` method. While a writable file handle is open, + attempting to read or write other files in the ZIP file will raise a + :exc:`RuntimeError`. - Objects returned by :meth:`.open` can operate independently of the - ZipFile. + When writing a file, if the file size is not known in advance but may exceed + 2 GiB, pass ``force_zip64=True`` to ensure that the header format is + capable of supporting large files. If the file size is known in advance, + construct a :class:`ZipInfo` object with :attr:`~ZipInfo.file_size` set, and + use that as the *name* parameter. .. note:: @@ -243,10 +244,14 @@ ZipFile Objects or a :class:`ZipInfo` object. You will appreciate this when trying to read a ZIP file that contains members with duplicate names. - .. deprecated-removed:: 3.4 3.6 - The ``'U'`` or ``'rU'`` mode. Use :class:`io.TextIOWrapper` for reading + .. versionchanged:: 3.6 + Removed support of ``mode='U'``. Use :class:`io.TextIOWrapper` for reading compressed text files in :term:`universal newlines` mode. + .. versionchanged:: 3.6 + :meth:`open` can now be used to write files into the archive with the + ``mode='w'`` option. + .. method:: ZipFile.extract(member, path=None, pwd=None) Extract a member from the archive to the current working directory; *member* @@ -465,7 +470,31 @@ Instances of the :class:`ZipInfo` class are returned by the :meth:`.getinfo` and :meth:`.infolist` methods of :class:`ZipFile` objects. Each object stores information about a single member of the ZIP archive. -Instances have the following attributes: +There is one classmethod to make a :class:`ZipInfo` instance for a filesystem +file: + +.. classmethod:: ZipInfo.from_file(filename, arcname=None) + + Construct a :class:`ZipInfo` instance for a file on the filesystem, in + preparation for adding it to a zip file. + + *filename* should be the path to a file or directory on the filesystem. + + If *arcname* is specified, it is used as the name within the archive. + If *arcname* is not specified, the name will be the same as *filename*, but + with any drive letter and leading path separators removed. + + .. versionadded:: 3.6 + +Instances have the following methods and attributes: + +.. method:: ZipInfo.is_dir() + + Return True if this archive member is a directory. + + This uses the entry's name: directories should always end with ``/``. + + .. versionadded:: 3.6 .. attribute:: ZipInfo.filename @@ -574,4 +603,5 @@ Instances have the following attributes: Size of the uncompressed file. + .. _PKZIP Application Note: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst index 1de7bae..3d742ab 100644 --- a/Doc/library/zlib.rst +++ b/Doc/library/zlib.rst @@ -47,14 +47,19 @@ The available exception and functions in this module are: platforms, use ``adler32(data) & 0xffffffff``. -.. function:: compress(data[, level]) +.. function:: compress(data, level=-1) Compresses the bytes in *data*, returning a bytes object containing compressed data. - *level* is an integer from ``0`` to ``9`` controlling the level of compression; + *level* is an integer from ``0`` to ``9`` or ``-1`` controlling the level of compression; ``1`` is fastest and produces the least compression, ``9`` is slowest and - produces the most. ``0`` is no compression. The default value is ``6``. + produces the most. ``0`` is no compression. The default value is ``-1`` + (Z_DEFAULT_COMPRESSION). Z_DEFAULT_COMPRESSION represents a default + compromise between speed and compression (currently equivalent to level 6). Raises the :exc:`error` exception if any error occurs. + .. versionchanged:: 3.6 + *level* can now be used as a keyword parameter. + .. function:: compressobj(level=-1, method=DEFLATED, wbits=15, memLevel=8, strategy=Z_DEFAULT_STRATEGY[, zdict]) @@ -124,7 +129,7 @@ The available exception and functions in this module are: platforms, use ``crc32(data) & 0xffffffff``. -.. function:: decompress(data[, wbits[, bufsize]]) +.. function:: decompress(data, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE) Decompresses the bytes in *data*, returning a bytes object containing the uncompressed data. The *wbits* parameter depends on @@ -159,14 +164,16 @@ The available exception and functions in this module are: When decompressing a stream, the window size must not be smaller than the size originally used to compress the stream; using a too-small value may result in an :exc:`error` exception. The default *wbits* value - is 15, which corresponds to the largest window size and requires a zlib - header and trailer to be included. + corresponds to the largest window size and requires a zlib header and + trailer to be included. *bufsize* is the initial size of the buffer used to hold decompressed data. If more space is required, the buffer size will be increased as needed, so you don't have to get this value exactly right; tuning it will only save a few calls - to :c:func:`malloc`. The default size is 16384. + to :c:func:`malloc`. + .. versionchanged:: 3.6 + *wbits* and *bufsize* can be used as keyword arguments. .. function:: decompressobj(wbits=15[, zdict]) @@ -252,7 +259,7 @@ Decompression objects support the following methods and attributes: .. versionadded:: 3.3 -.. method:: Decompress.decompress(data[, max_length]) +.. method:: Decompress.decompress(data, max_length=0) Decompress *data*, returning a bytes object containing the uncompressed data corresponding to at least part of the data in *string*. This data should be @@ -264,9 +271,11 @@ Decompression objects support the following methods and attributes: no longer than *max_length*. This may mean that not all of the compressed input can be processed; and unconsumed data will be stored in the attribute :attr:`unconsumed_tail`. This bytestring must be passed to a subsequent call to - :meth:`decompress` if decompression is to continue. If *max_length* is not - supplied then the whole input is decompressed, and :attr:`unconsumed_tail` is - empty. + :meth:`decompress` if decompression is to continue. If *max_length* is zero + then the whole input is decompressed, and :attr:`unconsumed_tail` is empty. + + .. versionchanged:: 3.6 + *max_length* can be used as a keyword argument. .. method:: Decompress.flush([length]) diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 88b94ea..1c5bbdf 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -471,10 +471,10 @@ A function definition defines a user-defined function object (see section decorators: `decorator`+ decorator: "@" `dotted_name` ["(" [`argument_list` [","]] ")"] NEWLINE dotted_name: `identifier` ("." `identifier`)* - parameter_list: (`defparameter` ",")* - : | "*" [`parameter`] ("," `defparameter`)* ["," "**" `parameter`] - : | "**" `parameter` - : | `defparameter` [","] ) + parameter_list: `defparameter` ("," `defparameter`)* ["," [`parameter_list_starargs`]] + : | `parameter_list_starargs` + parameter_list_starargs: "*" [`parameter`] ("," `defparameter`)* ["," ["**" `parameter` [","]]] + : | "**" `parameter` [","] parameter: `identifier` [":" `expression`] defparameter: `parameter` ["=" `expression`] funcname: `identifier` diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index f97eb08..08d87cc 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1063,6 +1063,12 @@ to ``type(x).__getitem__(x, i)``. Except where mentioned, attempts to execute a operation raise an exception when no appropriate method is defined (typically :exc:`AttributeError` or :exc:`TypeError`). +Setting a special method to ``None`` indicates that the corresponding +operation is not available. For example, if a class sets +:meth:`__iter__` to ``None``, the class is not iterable, so calling +:func:`iter` on its instances will raise a :exc:`TypeError` (without +falling back to :meth:`__getitem__`). [#]_ + When implementing a class that emulates any built-in type, it is important that the emulation only be implemented to the degree that it makes sense for the object being modelled. For example, some sequences may work well with retrieval @@ -1233,8 +1239,9 @@ Basic customization .. method:: object.__format__(self, format_spec) - Called by the :func:`format` built-in function (and by extension, the - :meth:`str.format` method of class :class:`str`) to produce a "formatted" + Called by the :func:`format` built-in function, + and by extension, evaluation of :ref:`formatted string literals + <f-strings>` and the :meth:`str.format` method, to produce a "formatted" string representation of an object. The ``format_spec`` argument is a string that contains a description of the formatting options desired. The interpretation of the ``format_spec`` argument is up to the type @@ -1491,6 +1498,14 @@ class' :attr:`~object.__dict__`. Called to delete the attribute on an instance *instance* of the owner class. +.. method:: object.__set_name__(self, owner, name) + + Called at the time the owning class *owner* is created. The + descriptor has been assigned to *name*. + + .. versionadded:: 3.6 + + The attribute :attr:`__objclass__` is interpreted by the :mod:`inspect` module as specifying the class where this object was defined (setting this appropriately can assist in runtime introspection of dynamic class attributes). @@ -1628,11 +1643,56 @@ Notes on using *__slots__* * *__class__* assignment works only if both classes have the same *__slots__*. -.. _metaclasses: +.. _class-customization: Customizing class creation -------------------------- +Whenever a class inherits from another class, *__init_subclass__* is +called on that class. This way, it is possible to write classes which +change the behavior of subclasses. This is closely related to class +decorators, but where class decorators only affect the specific class they're +applied to, ``__init_subclass__`` solely applies to future subclasses of the +class defining the method. + +.. classmethod:: object.__init_subclass__(cls) + + This method is called whenever the containing class is subclassed. + *cls* is then the new subclass. If defined as a normal instance method, + this method is implicitly converted to a class method. + + Keyword arguments which are given to a new class are passed to + the parent's class ``__init_subclass__``. For compatibility with + other classes using ``__init_subclass__``, one should take out the + needed keyword arguments and pass the others over to the base + class, as in:: + + class Philosopher: + def __init_subclass__(cls, default_name, **kwargs): + super().__init_subclass__(**kwargs) + cls.default_name = default_name + + class AustralianPhilosopher(Philosopher, default_name="Bruce"): + pass + + The default implementation ``object.__init_subclass__`` does + nothing, but raises an error if it is called with any arguments. + + .. note:: + + The metaclass hint ``metaclass`` is consumed by the rest of the type + machinery, and is never passed to ``__init_subclass__`` implementations. + The actual metaclass (rather than the explicit hint) can be accessed as + ``type(cls)``. + + .. versionadded:: 3.6 + + +.. _metaclasses: + +Metaclasses +^^^^^^^^^^^ + By default, classes are constructed using :func:`type`. The class body is executed in a new namespace and the class name is bound locally to the result of ``type(name, bases, namespace)``. @@ -2059,7 +2119,7 @@ left undefined. (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected (swapped) operands. These functions are only called if the left operand does - not support the corresponding operation and the operands are of different + not support the corresponding operation [#]_ and the operands are of different types. [#]_ For instance, to evaluate the expression ``x - y``, where *y* is an instance of a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` is called if ``x.__sub__(y)`` returns *NotImplemented*. @@ -2475,6 +2535,17 @@ An example of an asynchronous context manager class:: controlled conditions. It generally isn't a good idea though, since it can lead to some very strange behaviour if it is handled incorrectly. +.. [#] The :meth:`__hash__`, :meth:`__iter__`, :meth:`__reversed__`, and + :meth:`__contains__` methods have special handling for this; others + will still raise a :exc:`TypeError`, but may do so by relying on + the behavior that ``None`` is not callable. + +.. [#] "Does not support" here means that the class has no such method, or + the method returns ``NotImplemented``. Do not set the method to + ``None`` if you want to force fallback to the right operand's reflected + method--that will instead have the opposite effect of explicitly + *blocking* such fallback. + .. [#] For operands of the same type, it is assumed that if the non-reflected method (such as :meth:`__add__`) fails the operation is not supported, which is why the reflected method is not called. diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 64302b8..fcc707b 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -554,19 +554,30 @@ the module. details. This attribute is used instead of ``__name__`` to calculate explicit - relative imports for main modules, as defined in :pep:`366`. + relative imports for main modules, as defined in :pep:`366`. It is + expected to have the same value as ``__spec__.parent``. + + .. versionchanged:: 3.6 + The value of ``__package__`` is expected to be the same as + ``__spec__.parent``. .. attribute:: __spec__ The ``__spec__`` attribute must be set to the module spec that was - used when importing the module. This is used primarily for - introspection and during reloading. Setting ``__spec__`` + used when importing the module. Setting ``__spec__`` appropriately applies equally to :ref:`modules initialized during interpreter startup <programs>`. The one exception is ``__main__``, where ``__spec__`` is :ref:`set to None in some cases <main_spec>`. + When ``__package__`` is not defined, ``__spec__.parent`` is used as + a fallback. + .. versionadded:: 3.4 + .. versionchanged:: 3.6 + ``__spec__.parent`` is used as a fallback when ``__package__`` is + not defined. + .. attribute:: __path__ If the module is a package (either regular or namespace), the module diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index 37f25f1..b3b71af 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -405,7 +405,8 @@ String literals are described by the following lexical definitions: .. productionlist:: stringliteral: [`stringprefix`](`shortstring` | `longstring`) - stringprefix: "r" | "u" | "R" | "U" + stringprefix: "r" | "u" | "R" | "U" | "f" | "F" + : | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | "Rf" | "RF" shortstring: "'" `shortstringitem`* "'" | '"' `shortstringitem`* '"' longstring: "'''" `longstringitem`* "'''" | '"""' `longstringitem`* '"""' shortstringitem: `shortstringchar` | `stringescapeseq` @@ -464,6 +465,11 @@ is not supported. to simplify the maintenance of dual Python 2.x and 3.x codebases. See :pep:`414` for more information. +A string literal with ``'f'`` or ``'F'`` in its prefix is a +:dfn:`formatted string literal`; see :ref:`f-strings`. The ``'f'`` may be +combined with ``'r'``, but not with ``'b'`` or ``'u'``, therefore raw +formatted strings are possible, but formatted bytes literals are not. + In triple-quoted literals, unescaped newlines and quotes are allowed (and are retained), except that three unescaped quotes in a row terminate the literal. (A "quote" is the character used to open the literal, i.e. either ``'`` or ``"``.) @@ -583,7 +589,106 @@ comments to parts of strings, for example:: Note that this feature is defined at the syntactical level, but implemented at compile time. The '+' operator must be used to concatenate string expressions at run time. Also note that literal concatenation can use different quoting -styles for each component (even mixing raw strings and triple quoted strings). +styles for each component (even mixing raw strings and triple quoted strings), +and formatted string literals may be concatenated with plain string literals. + + +.. index:: + single: formatted string literal + single: interpolated string literal + single: string; formatted literal + single: string; interpolated literal + single: f-string +.. _f-strings: + +Formatted string literals +------------------------- + +.. versionadded:: 3.6 + +A :dfn:`formatted string literal` or :dfn:`f-string` is a string literal +that is prefixed with ``'f'`` or ``'F'``. These strings may contain +replacement fields, which are expressions delimited by curly braces ``{}``. +While other string literals always have a constant value, formatted strings +are really expressions evaluated at run time. + +Escape sequences are decoded like in ordinary string literals (except when +a literal is also marked as a raw string). After decoding, the grammar +for the contents of the string is: + +.. productionlist:: + f_string: (`literal_char` | "{{" | "}}" | `replacement_field`)* + replacement_field: "{" `f_expression` ["!" `conversion`] [":" `format_spec`] "}" + f_expression: (`conditional_expression` | "*" `or_expr`) + : ("," `conditional_expression` | "," "*" `or_expr`)* [","] + : | `yield_expression` + conversion: "s" | "r" | "a" + format_spec: (`literal_char` | NULL | `replacement_field`)* + literal_char: <any code point except "{", "}" or NULL> + +The parts of the string outside curly braces are treated literally, +except that any doubled curly braces ``'{{'`` or ``'}}'`` are replaced +with the corresponding single curly brace. A single opening curly +bracket ``'{'`` marks a replacement field, which starts with a +Python expression. After the expression, there may be a conversion field, +introduced by an exclamation point ``'!'``. A format specifier may also +be appended, introduced by a colon ``':'``. A replacement field ends +with a closing curly bracket ``'}'``. + +Expressions in formatted string literals are treated like regular +Python expressions surrounded by parentheses, with a few exceptions. +An empty expression is not allowed, and a :keyword:`lambda` expression +must be surrounded by explicit parentheses. Replacement expressions +can contain line breaks (e.g. in triple-quoted strings), but they +cannot contain comments. Each expression is evaluated in the context +where the formatted string literal appears, in order from left to right. + +If a conversion is specified, the result of evaluating the expression +is converted before formatting. Conversion ``'!s'`` calls :func:`str` on +the result, ``'!r'`` calls :func:`repr`, and ``'!a'`` calls :func:`ascii`. + +The result is then formatted using the :func:`format` protocol. The +format specifier is passed to the :meth:`__format__` method of the +expression or conversion result. An empty string is passed when the +format specifier is omitted. The formatted result is then included in +the final value of the whole string. + +Top-level format specifiers may include nested replacement fields. +These nested fields may include their own conversion fields and +format specifiers, but may not include more deeply-nested replacement fields. + +Formatted string literals may be concatenated, but replacement fields +cannot be split across literals. + +Some examples of formatted string literals:: + + >>> name = "Fred" + >>> f"He said his name is {name!r}." + "He said his name is 'Fred'." + >>> f"He said his name is {repr(name)}." # repr() is equivalent to !r + "He said his name is 'Fred'." + >>> width = 10 + >>> precision = 4 + >>> value = decimal.Decimal("12.34567") + >>> f"result: {value:{width}.{precision}}" # nested fields + 'result: 12.35' + +A consequence of sharing the same syntax as regular string literals is +that characters in the replacement fields must not conflict with the +quoting used in the outer formatted string literal. Also, escape +sequences normally apply to the outer formatted string literal, +rather than inner string literals:: + + f"abc {a["x"]} def" # error: outer string literal ended prematurely + f"abc {a[\"x\"]} def" # workaround: escape the inner quotes + f"abc {a['x']} def" # workaround: use different quoting + + f"newline: {ord('\n')}" # error: literal line break in inner string + f"newline: {ord('\\n')}" # workaround: double escaping + fr"newline: {ord('\n')}" # workaround: raw outer string + +See also :pep:`498` for the proposal that added formatted string literals, +and :meth:`str.format`, which uses a related format string mechanism. .. _numbers: diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index d403c4d..eee3f43 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -84,7 +84,7 @@ attributes or items of mutable objects: assignment_stmt: (`target_list` "=")+ (`starred_expression` | `yield_expression`) target_list: `target` ("," `target`)* [","] target: `identifier` - : | "(" `target_list` ")" + : | "(" [`target_list`] ")" : | "[" [`target_list`] "]" : | `attributeref` : | `subscription` diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 6311283..7986017 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -34,7 +34,7 @@ import suspicious ISSUE_URI = 'https://bugs.python.org/issue%s' -SOURCE_URI = 'https://hg.python.org/cpython/file/3.5/%s' +SOURCE_URI = 'https://hg.python.org/cpython/file/default/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists from docutils.parsers.rst.states import Body diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index dba93bf..4aee2d6 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -205,7 +205,7 @@ library/uuid,,:uuid,urn:uuid:12345678-1234-5678-1234-567812345678 library/venv,,:param,":param nodist: If True, setuptools and pip are not installed into the" library/venv,,:param,":param progress: If setuptools or pip are installed, the progress of the" library/venv,,:param,":param nopip: If True, pip is not installed into the created" -library/venv,,:param,:param context: The information for the environment creation request +library/venv,,:param,:param context: The information for the virtual environment library/xmlrpc.client,,:pass,http://user:pass@host:port/path library/xmlrpc.client,,:pass,user:pass library/xmlrpc.client,,:port,http://user:pass@host:port/path @@ -272,6 +272,7 @@ whatsnew/3.2,,:feed,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe: whatsnew/3.2,,:gz,">>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:" whatsnew/3.2,,:location,zope9-location = ${zope9:location} whatsnew/3.2,,:prefix,zope-conf = ${custom:prefix}/etc/zope.conf +whatsnew/changelog,,:version,import sys; I = version[:version.index(' ')] whatsnew/changelog,,:gz,": TarFile opened with external fileobj and ""w:gz"" mode didn't" whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" library/tarfile,149,:xz,'x:xz' @@ -290,7 +291,6 @@ library/zipapp,82,:fn,"argument should have the form ""pkg.mod:fn"", where ""pkg library/zipapp,155,:callable,"""pkg.module:callable"" and the archive will be run by importing" library/stdtypes,,::,>>> m[::2].tolist() library/sys,,`,# ``wrapper`` creates a ``wrap(coro)`` coroutine: -tutorial/venv,77,:c7b9645a6f35,"Python 3.4.3+ (3.4:c7b9645a6f35+, May 22 2015, 09:31:25)" whatsnew/3.5,,:root,'WARNING:root:warning\n' whatsnew/3.5,,:warning,'WARNING:root:warning\n' whatsnew/3.5,,::,>>> addr6 = ipaddress.IPv6Address('::1') diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst index 12989b2..d434618 100644 --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -78,6 +78,9 @@ slice notation makes this especially convenient:: >>> words ['defenestrate', 'cat', 'window', 'defenestrate'] +With ``for w in words:``, the example would attempt to create an infinite list, +inserting ``defenestrate`` over and over again. + .. _tut-range: diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst index dd9c7cd..beeaac3 100644 --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -25,7 +25,8 @@ first way is to do all the string handling yourself; using string slicing and concatenation operations you can create any layout you can imagine. The string type has some methods that perform useful operations for padding strings to a given column width; these will be discussed shortly. The second -way is to use the :meth:`str.format` method. +way is to use :ref:`formatted string literals <f-strings>`, or the +:meth:`str.format` method. The :mod:`string` module contains a :class:`~string.Template` class which offers yet another way to substitute values into strings. diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst index e8d8e2b..faf57a3 100644 --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -10,13 +10,13 @@ Using the Python Interpreter Invoking the Interpreter ======================== -The Python interpreter is usually installed as :file:`/usr/local/bin/python3.5` +The Python interpreter is usually installed as :file:`/usr/local/bin/python3.6` on those machines where it is available; putting :file:`/usr/local/bin` in your Unix shell's search path makes it possible to start it by typing the command: .. code-block:: text - python3.5 + python3.6 to the shell. [#]_ Since the choice of the directory where the interpreter lives is an installation option, other places are possible; check with your local @@ -24,11 +24,11 @@ Python guru or system administrator. (E.g., :file:`/usr/local/python` is a popular alternative location.) On Windows machines, the Python installation is usually placed in -:file:`C:\\Python35`, though you can change this when you're running the +:file:`C:\\Python36`, though you can change this when you're running the installer. To add this directory to your path, you can type the following command into the command prompt in a DOS box:: - set path=%path%;C:\python35 + set path=%path%;C:\python36 Typing an end-of-file character (:kbd:`Control-D` on Unix, :kbd:`Control-Z` on Windows) at the primary prompt causes the interpreter to exit with a zero exit @@ -98,8 +98,8 @@ before printing the first prompt: .. code-block:: shell-session - $ python3.5 - Python 3.5 (default, Sep 16 2015, 09:25:04) + $ python3.6 + Python 3.6 (default, Sep 16 2015, 09:25:04) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index 2140329..7e8ee3e 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -352,6 +352,9 @@ The built-in function :func:`len` returns the length of a string:: Strings support a large number of methods for basic transformations and searching. + :ref:`f-strings` + String literals that have embedded expressions. + :ref:`formatstrings` Information about string formatting with :meth:`str.format`. diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index 52ffdbe..1dd06c2 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -15,7 +15,7 @@ operating system:: >>> import os >>> os.getcwd() # Return the current working directory - 'C:\\Python35' + 'C:\\Python36' >>> os.chdir('/server/accesslogs') # Change current working directory >>> os.system('mkdir today') # Run the command mkdir in the system shell 0 diff --git a/Doc/tutorial/stdlib2.rst b/Doc/tutorial/stdlib2.rst index 3714384..bf4cf87 100644 --- a/Doc/tutorial/stdlib2.rst +++ b/Doc/tutorial/stdlib2.rst @@ -278,7 +278,7 @@ applications include caching objects that are expensive to create:: Traceback (most recent call last): File "<stdin>", line 1, in <module> d['primary'] # entry was automatically removed - File "C:/python35/lib/weakref.py", line 46, in __getitem__ + File "C:/python36/lib/weakref.py", line 46, in __getitem__ o = self.data[key]() KeyError: 'primary' diff --git a/Doc/tutorial/venv.rst b/Doc/tutorial/venv.rst index 3b2ee2e..e2dd57d 100644 --- a/Doc/tutorial/venv.rst +++ b/Doc/tutorial/venv.rst @@ -20,15 +20,14 @@ the requirements of every application. If application A needs version the requirements are in conflict and installing either version 1.0 or 2.0 will leave one application unable to run. -The solution for this problem is to create a :term:`virtual -environment` (often shortened to "virtualenv"), a self-contained -directory tree that contains a Python installation for a particular -version of Python, plus a number of additional packages. +The solution for this problem is to create a :term:`virtual environment`, a +self-contained directory tree that contains a Python installation for a +particular version of Python, plus a number of additional packages. Different applications can then use different virtual environments. To resolve the earlier example of conflicting requirements, application A can have its own virtual environment with version 1.0 -installed while application B has another virtualenv with version 2.0. +installed while application B has another virtual environment with version 2.0. If application B requires a library be upgraded to version 3.0, this will not affect application A's environment. @@ -36,29 +35,26 @@ not affect application A's environment. Creating Virtual Environments ============================= -The script used to create and manage virtual environments is called -:program:`pyvenv`. :program:`pyvenv` will usually install the most -recent version of Python that you have available; the script is also -installed with a version number, so if you have multiple versions of -Python on your system you can select a specific Python version by -running ``pyvenv-3.4`` or whichever version you want. +The module used to create and manage virtual environments is called +:mod:`venv`. :mod:`venv` will usually install the most recent version of +Python that you have available. If you have multiple versions of Python on your +system, you can select a specific Python version by running ``python3`` or +whichever version you want. -To create a virtualenv, decide upon a directory -where you want to place it and run :program:`pyvenv` with the -directory path:: +To create a virtual environment, decide upon a directory where you want to +place it, and run the :mod:`venv` module as a script with the directory path:: - pyvenv tutorial-env + python3 -m venv tutorial-env This will create the ``tutorial-env`` directory if it doesn't exist, and also create directories inside it containing a copy of the Python interpreter, the standard library, and various supporting files. -Once you've created a virtual environment, you need to -activate it. +Once you've created a virtual environment, you may activate it. On Windows, run:: - tutorial-env/Scripts/activate + tutorial-env\Scripts\activate.bat On Unix or MacOS, run:: @@ -69,33 +65,36 @@ On Unix or MacOS, run:: ``activate.csh`` and ``activate.fish`` scripts you should use instead.) -Activating the virtualenv will change your shell's prompt to show what -virtualenv you're using, and modify the environment so that running -``python`` will get you that particular version and installation of -Python. For example:: +Activating the virtual environment will change your shell's prompt to show what +virtual environment you're using, and modify the environment so that running +``python`` will get you that particular version and installation of Python. +For example: - -> source ~/envs/tutorial-env/bin/activate - (tutorial-env) -> python - Python 3.4.3+ (3.4:c7b9645a6f35+, May 22 2015, 09:31:25) +.. code-block:: bash + + $ source ~/envs/tutorial-env/bin/activate + (tutorial-env) $ python + Python 3.5.1 (default, May 6 2016, 10:59:36) ... >>> import sys >>> sys.path - ['', '/usr/local/lib/python34.zip', ..., - '~/envs/tutorial-env/lib/python3.4/site-packages'] + ['', '/usr/local/lib/python35.zip', ..., + '~/envs/tutorial-env/lib/python3.5/site-packages'] >>> Managing Packages with pip ========================== -Once you've activated a virtual environment, you can install, upgrade, -and remove packages using a program called :program:`pip`. By default -``pip`` will install packages from the Python Package Index, -<https://pypi.python.org/pypi>. You can browse the Python Package Index -by going to it in your web browser, or you can use ``pip``'s -limited search feature:: +You can install, upgrade, and remove packages using a program called +:program:`pip`. By default ``pip`` will install packages from the Python +Package Index, <https://pypi.python.org/pypi>. You can browse the Python +Package Index by going to it in your web browser, or you can use ``pip``'s +limited search feature: + +.. code-block:: bash - (tutorial-env) -> pip search astronomy + (tutorial-env) $ pip search astronomy skyfield - Elegant astronomy for Python gary - Galactic astronomy and gravitational dynamics. novas - The United States Naval Observatory NOVAS astronomy library @@ -107,9 +106,11 @@ limited search feature:: "freeze", etc. (Consult the :ref:`installing-index` guide for complete documentation for ``pip``.) -You can install the latest version of a package by specifying a package's name:: +You can install the latest version of a package by specifying a package's name: + +.. code-block:: bash - -> pip install novas + (tutorial-env) $ pip install novas Collecting novas Downloading novas-3.1.1.3.tar.gz (136kB) Installing collected packages: novas @@ -117,9 +118,11 @@ You can install the latest version of a package by specifying a package's name:: Successfully installed novas-3.1.1.3 You can also install a specific version of a package by giving the -package name followed by ``==`` and the version number:: +package name followed by ``==`` and the version number: - -> pip install requests==2.6.0 +.. code-block:: bash + + (tutorial-env) $ pip install requests==2.6.0 Collecting requests==2.6.0 Using cached requests-2.6.0-py2.py3-none-any.whl Installing collected packages: requests @@ -128,9 +131,11 @@ package name followed by ``==`` and the version number:: If you re-run this command, ``pip`` will notice that the requested version is already installed and do nothing. You can supply a different version number to get that version, or you can run ``pip -install --upgrade`` to upgrade the package to the latest version:: +install --upgrade`` to upgrade the package to the latest version: + +.. code-block:: bash - -> pip install --upgrade requests + (tutorial-env) $ pip install --upgrade requests Collecting requests Installing collected packages: requests Found existing installation: requests 2.6.0 @@ -141,9 +146,11 @@ install --upgrade`` to upgrade the package to the latest version:: ``pip uninstall`` followed by one or more package names will remove the packages from the virtual environment. -``pip show`` will display information about a particular package:: +``pip show`` will display information about a particular package: - (tutorial-env) -> pip show requests +.. code-block:: bash + + (tutorial-env) $ pip show requests --- Metadata-Version: 2.0 Name: requests @@ -157,9 +164,11 @@ packages from the virtual environment. Requires: ``pip list`` will display all of the packages installed in the virtual -environment:: +environment: + +.. code-block:: bash - (tutorial-env) -> pip list + (tutorial-env) $ pip list novas (3.1.1.3) numpy (1.9.2) pip (7.0.3) @@ -168,19 +177,23 @@ environment:: ``pip freeze`` will produce a similar list of the installed packages, but the output uses the format that ``pip install`` expects. -A common convention is to put this list in a ``requirements.txt`` file:: +A common convention is to put this list in a ``requirements.txt`` file: - (tutorial-env) -> pip freeze > requirements.txt - (tutorial-env) -> cat requirements.txt +.. code-block:: bash + + (tutorial-env) $ pip freeze > requirements.txt + (tutorial-env) $ cat requirements.txt novas==3.1.1.3 numpy==1.9.2 requests==2.7.0 The ``requirements.txt`` can then be committed to version control and shipped as part of an application. Users can then install all the -necessary packages with ``install -r``:: +necessary packages with ``install -r``: + +.. code-block:: bash - -> pip install -r requirements.txt + (tutorial-env) $ pip install -r requirements.txt Collecting novas==3.1.1.3 (from -r requirements.txt (line 1)) ... Collecting numpy==1.9.2 (from -r requirements.txt (line 2)) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 57faf89..37a9e14 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -395,6 +395,8 @@ Miscellaneous options stored in a traceback of a trace. Use ``-X tracemalloc=NFRAME`` to start tracing with a traceback limit of *NFRAME* frames. See the :func:`tracemalloc.start` for more information. + * ``-X showalloccount`` to enable the output of the total count of allocated + objects for each type (only works when built with ``COUNT_ALLOCS`` defined); It also allows passing arbitrary values and retrieving them through the :data:`sys._xoptions` dictionary. @@ -408,6 +410,9 @@ Miscellaneous options .. versionadded:: 3.4 The ``-X showrefcount`` and ``-X tracemalloc`` options. + .. versionadded:: 3.6 + The ``-X showalloccount`` option. + Options you shouldn't use ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -619,6 +624,54 @@ conflict. .. versionadded:: 3.4 +.. envvar:: PYTHONMALLOC + + Set the Python memory allocators and/or install debug hooks. + + Set the family of memory allocators used by Python: + + * ``malloc``: use the :c:func:`malloc` function of the C library + for all domains (:c:data:`PYMEM_DOMAIN_RAW`, :c:data:`PYMEM_DOMAIN_MEM`, + :c:data:`PYMEM_DOMAIN_OBJ`). + * ``pymalloc``: use the :ref:`pymalloc allocator <pymalloc>` for + :c:data:`PYMEM_DOMAIN_MEM` and :c:data:`PYMEM_DOMAIN_OBJ` domains and use + the :c:func:`malloc` function for the :c:data:`PYMEM_DOMAIN_RAW` domain. + + Install debug hooks: + + * ``debug``: install debug hooks on top of the default memory allocator + * ``malloc_debug``: same as ``malloc`` but also install debug hooks + * ``pymalloc_debug``: same as ``pymalloc`` but also install debug hooks + + When Python is compiled in release mode, the default is ``pymalloc``. When + compiled in debug mode, the default is ``pymalloc_debug`` and the debug hooks + are used automatically. + + If Python is configured without ``pymalloc`` support, ``pymalloc`` and + ``pymalloc_debug`` are not available, the default is ``malloc`` in release + mode and ``malloc_debug`` in debug mode. + + See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python + memory allocators. + + .. versionadded:: 3.6 + + +.. envvar:: PYTHONMALLOCSTATS + + If set to a non-empty string, Python will print statistics of the + :ref:`pymalloc memory allocator <pymalloc>` every time a new pymalloc object + arena is created, and on shutdown. + + This variable is ignored if the :envvar:`PYTHONMALLOC` environment variable + is used to force the :c:func:`malloc` allocator of the C library, or if + Python is configured without ``pymalloc`` support. + + .. versionchanged:: 3.6 + This variable can now also be used on Python compiled in release mode. + It now has no effect if set to an empty string. + + Debug-mode variables ~~~~~~~~~~~~~~~~~~~~ @@ -634,9 +687,3 @@ if Python was configured with the ``--with-pydebug`` build option. If set, Python will dump objects and reference counts still alive after shutting down the interpreter. - - -.. envvar:: PYTHONMALLOCSTATS - - If set, Python will print memory allocation statistics every time a new - object arena is created, and on shutdown. diff --git a/Doc/using/index.rst b/Doc/using/index.rst index 502afa9..a5df713 100644 --- a/Doc/using/index.rst +++ b/Doc/using/index.rst @@ -17,4 +17,3 @@ interpreter and things that make working with Python easier. unix.rst windows.rst mac.rst - scripts.rst diff --git a/Doc/using/mac.rst b/Doc/using/mac.rst index 05c91bb..8f1ac3f 100644 --- a/Doc/using/mac.rst +++ b/Doc/using/mac.rst @@ -25,7 +25,7 @@ there. What you get after installing is a number of things: -* A :file:`MacPython 3.5` folder in your :file:`Applications` folder. In here +* A :file:`MacPython 3.6` folder in your :file:`Applications` folder. In here you find IDLE, the development environment that is a standard part of official Python distributions; PythonLauncher, which handles double-clicking Python scripts from the Finder; and the "Build Applet" tool, which allows you to @@ -93,7 +93,7 @@ aware of: programs that talk to the Aqua window manager (in other words, anything that has a GUI) need to be run in a special way. Use :program:`pythonw` instead of :program:`python` to start such scripts. -With Python 3.5, you can use either :program:`python` or :program:`pythonw`. +With Python 3.6, you can use either :program:`python` or :program:`pythonw`. Configuration @@ -159,7 +159,7 @@ https://riverbankcomputing.com/software/pyqt/intro. Distributing Python Applications on the Mac =========================================== -The "Build Applet" tool that is placed in the MacPython 3.5 folder is fine for +The "Build Applet" tool that is placed in the MacPython 3.6 folder is fine for packaging small Python scripts on your own machine to run as a standard Mac application. This tool, however, is not robust enough to distribute Python applications to other users. diff --git a/Doc/using/scripts.rst b/Doc/using/scripts.rst deleted file mode 100644 index 2c87416..0000000 --- a/Doc/using/scripts.rst +++ /dev/null @@ -1,12 +0,0 @@ -.. _tools-and-scripts: - -Additional Tools and Scripts -============================ - -.. _scripts-pyvenv: - -pyvenv - Creating virtual environments --------------------------------------- - -.. include:: venv-create.inc - diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc index 7ad3008..53f431b 100644 --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -1,31 +1,39 @@ Creation of :ref:`virtual environments <venv-def>` is done by executing the -``pyvenv`` script:: +command ``venv``:: - pyvenv /path/to/new/virtual/environment + python3 -m venv /path/to/new/virtual/environment Running this command creates the target directory (creating any parent directories that don't exist already) and places a ``pyvenv.cfg`` file in it -with a ``home`` key pointing to the Python installation the command was run -from. It also creates a ``bin`` (or ``Scripts`` on Windows) subdirectory +with a ``home`` key pointing to the Python installation from which the command +was run. It also creates a ``bin`` (or ``Scripts`` on Windows) subdirectory containing a copy of the ``python`` binary (or binaries, in the case of Windows). It also creates an (initially empty) ``lib/pythonX.Y/site-packages`` subdirectory (on Windows, this is ``Lib\site-packages``). +.. deprecated:: 3.6 + ``pyvenv`` was the recommended tool for creating virtual environments for + Python 3.3 and 3.4, and is `deprecated in Python 3.6 + <https://docs.python.org/dev/whatsnew/3.6.html#deprecated-features>`_. + +.. versionchanged:: 3.5 + The use of ``venv`` is now recommended for creating virtual environments. + .. seealso:: `Python Packaging User Guide: Creating and using virtual environments - <https://packaging.python.org/en/latest/installing/#creating-virtual-environments>`__ + <https://packaging.python.org/installing/#creating-virtual-environments>`__ .. highlight:: none -On Windows, you may have to invoke the ``pyvenv`` script as follows, if you -don't have the relevant PATH and PATHEXT settings:: +On Windows, invoke the ``venv`` command as follows:: - c:\Temp>c:\Python35\python c:\Python35\Tools\Scripts\pyvenv.py myenv + c:\>c:\Python35\python -m venv c:\path\to\myenv -or equivalently:: +Alternatively, if you configured the ``PATH`` and ``PATHEXT`` variables for +your :ref:`Python installation <using-on-windows>`:: - c:\Temp>c:\Python35\python -m venv myenv + c:\>python -m venv myenv c:\path\to\myenv The command, if run with ``-h``, will show the available options:: @@ -36,25 +44,26 @@ The command, if run with ``-h``, will show the available options:: Creates virtual Python environments in one or more target directories. positional arguments: - ENV_DIR A directory to create the environment in. + ENV_DIR A directory to create the environment in. optional arguments: - -h, --help show this help message and exit - --system-site-packages Give the virtual environment access to the system - site-packages dir. - --symlinks Try to use symlinks rather than copies, when symlinks - are not the default for the platform. - --copies Try to use copies rather than symlinks, even when - symlinks are the default for the platform. - --clear Delete the contents of the environment directory if it - already exists, before environment creation. - --upgrade Upgrade the environment directory to use this version - of Python, assuming Python has been upgraded in-place. - --without-pip Skips installing or upgrading pip in the virtual - environment (pip is bootstrapped by default) - -Depending on how the ``venv`` functionality has been invoked, the usage message -may vary slightly, e.g. referencing ``pyvenv`` rather than ``venv``. + -h, --help show this help message and exit + --system-site-packages + Give the virtual environment access to the system + site-packages dir. + --symlinks Try to use symlinks rather than copies, when symlinks + are not the default for the platform. + --copies Try to use copies rather than symlinks, even when + symlinks are the default for the platform. + --clear Delete the contents of the environment directory if it + already exists, before environment creation. + --upgrade Upgrade the environment directory to use this version + of Python, assuming Python has been upgraded in-place. + --without-pip Skips installing or upgrading pip in the virtual + environment (pip is bootstrapped by default) + + Once an environment has been created, you may wish to activate it, e.g. by + sourcing an activate script in its bin directory. .. versionchanged:: 3.4 Installs pip by default, added the ``--without-pip`` and ``--copies`` @@ -73,12 +82,13 @@ run with the ``--system-site-packages`` option, ``false`` otherwise. Unless the ``--without-pip`` option is given, :mod:`ensurepip` will be invoked to bootstrap ``pip`` into the virtual environment. -Multiple paths can be given to ``pyvenv``, in which case an identical -virtualenv will be created, according to the given options, at each -provided path. +Multiple paths can be given to ``venv``, in which case an identical virtual +environment will be created, according to the given options, at each provided +path. -Once a venv has been created, it can be "activated" using a script in the -venv's binary directory. The invocation of the script is platform-specific: +Once a virtual environment has been created, it can be "activated" using a +script in the virtual environment's binary directory. The invocation of the +script is platform-specific: +-------------+-----------------+-----------------------------------------+ | Platform | Shell | Command to activate virtual environment | @@ -95,16 +105,17 @@ venv's binary directory. The invocation of the script is platform-specific: +-------------+-----------------+-----------------------------------------+ You don't specifically *need* to activate an environment; activation just -prepends the venv's binary directory to your path, so that "python" invokes the -venv's Python interpreter and you can run installed scripts without having to -use their full path. However, all scripts installed in a venv should be -runnable without activating it, and run with the venv's Python automatically. - -You can deactivate a venv by typing "deactivate" in your shell. The exact -mechanism is platform-specific: for example, the Bash activation script defines -a "deactivate" function, whereas on Windows there are separate scripts called -``deactivate.bat`` and ``Deactivate.ps1`` which are installed when the venv is -created. +prepends the virtual environment's binary directory to your path, so that +"python" invokes the virtual environment's Python interpreter and you can run +installed scripts without having to use their full path. However, all scripts +installed in a virtual environment should be runnable without activating it, +and run with the virtual environment's Python automatically. + +You can deactivate a virtual environment by typing "deactivate" in your shell. +The exact mechanism is platform-specific: for example, the Bash activation +script defines a "deactivate" function, whereas on Windows there are separate +scripts called ``deactivate.bat`` and ``Deactivate.ps1`` which are installed +when the virtual environment is created. .. versionadded:: 3.4 ``fish`` and ``csh`` activation scripts. diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index a4a6a30..3f6b68d 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -418,6 +418,8 @@ Getting started From the command-line ^^^^^^^^^^^^^^^^^^^^^ +.. versionchanged:: 3.6 + System-wide installations of Python 3.3 and later will put the launcher on your :envvar:`PATH`. The launcher is compatible with all available versions of Python, so it does not matter which version is installed. To check that the @@ -427,25 +429,26 @@ launcher is available, execute the following command in Command Prompt: py -You should find that the latest version of Python 2.x you have installed is +You should find that the latest version of Python you have installed is started - it can be exited as normal, and any additional command-line arguments specified will be sent directly to Python. -If you have multiple versions of Python 2.x installed (e.g., 2.6 and 2.7) you -will have noticed that Python 2.7 was started - to launch Python 2.6, try the +If you have multiple versions of Python installed (e.g., 2.7 and 3.6) you +will have noticed that Python 3.6 was started - to launch Python 2.7, try the command: :: - py -2.6 + py -2.7 -If you have a Python 3.x installed, try the command: +If you want the latest version of Python 2.x you have installed, try the +command: :: - py -3 + py -2 -You should find the latest version of Python 3.x starts. +You should find the latest version of Python 2.x starts. If you see the following error, you do not have the launcher installed: @@ -500,6 +503,11 @@ version qualifier. Assuming you have Python 2.6 installed, try changing the first line to ``#! python2.6`` and you should find the 2.6 version information printed. +Note that unlike interactive use, a bare "python" will use the latest +version of Python 2.x that you have installed. This is for backward +compatibility and for compatibility with Unix, where the command ``python`` +typically refers to Python 2. + From file associations ^^^^^^^^^^^^^^^^^^^^^^ diff --git a/Doc/whatsnew/2.1.rst b/Doc/whatsnew/2.1.rst index 06366b8..6aae726 100644 --- a/Doc/whatsnew/2.1.rst +++ b/Doc/whatsnew/2.1.rst @@ -367,7 +367,7 @@ dictionary:: This version works for simple things such as integers, but it has a side effect; the ``_cache`` dictionary holds a reference to the return values, so they'll -never be deallocated until the Python process exits and cleans up This isn't +never be deallocated until the Python process exits and cleans up. This isn't very noticeable for integers, but if :func:`f` returns an object, or a data structure that takes up a lot of memory, this can be a problem. diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst index 3e48c82..44c71e0 100644 --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -108,7 +108,7 @@ packages. Their concept and implementation are inspired by the popular with the interpreter core. This PEP adds the :mod:`venv` module for programmatic access, and the -:ref:`pyvenv <scripts-pyvenv>` script for command-line access and +``pyvenv`` script for command-line access and administration. The Python interpreter checks for a ``pyvenv.cfg``, file whose existence signals the base of a virtual environment's directory tree. diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst index 1e5c9d1..2a23cbc 100644 --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -197,7 +197,7 @@ will also be installed. On other platforms, the system wide unversioned ``pip`` command typically refers to the separately installed Python 2 version. -The :ref:`pyvenv <scripts-pyvenv>` command line utility and the :mod:`venv` +The ``pyvenv`` command line utility and the :mod:`venv` module make use of the :mod:`ensurepip` module to make ``pip`` readily available in virtual environments. When using the command line utility, ``pip`` is installed by default, while when using the :mod:`venv` module @@ -1989,11 +1989,11 @@ Other Improvements Stinner using his :pep:`445`-based ``pyfailmalloc`` tool (:issue:`18408`, :issue:`18520`). -* The :ref:`pyvenv <scripts-pyvenv>` command now accepts a ``--copies`` option +* The ``pyvenv`` command now accepts a ``--copies`` option to use copies rather than symlinks even on systems where symlinks are the default. (Contributed by Vinay Sajip in :issue:`18807`.) -* The :ref:`pyvenv <scripts-pyvenv>` command also accepts a ``--without-pip`` +* The ``pyvenv`` command also accepts a ``--without-pip`` option to suppress the otherwise-automatic bootstrapping of pip into the virtual environment. (Contributed by Nick Coghlan in :issue:`19552` as part of the :pep:`453` implementation.) @@ -2459,7 +2459,7 @@ Changes in the Python API stream in :mod:`~io.TextIOWrapper` to use its *newline* argument (:issue:`15204`). -* If you use :ref:`pyvenv <scripts-pyvenv>` in a script and desire that pip +* If you use ``pyvenv`` in a script and desire that pip *not* be installed, you must add ``--without-pip`` to your command invocation. diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst new file mode 100644 index 0000000..6d5bbc0 --- /dev/null +++ b/Doc/whatsnew/3.6.rst @@ -0,0 +1,881 @@ +**************************** + What's New In Python 3.6 +**************************** + +:Release: |release| +:Date: |today| + +.. Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten to some degree. + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. + + * It's helpful to add the bug/patch number as a comment: + + XXX Describe the transmogrify() function added to the socket + module. + (Contributed by P.Y. Developer in :issue:`12345`.) + + This saves the maintainer the effort of going through the Mercurial log + when researching a change. + +This article explains the new features in Python 3.6, compared to 3.5. + +For full details, see the :source:`Misc/NEWS` file. + +.. note:: + + Prerelease users should be aware that this document is currently in draft + form. It will be updated substantially as Python 3.6 moves towards release, + so it's worth checking back even after reading earlier versions. + + +Summary -- Release highlights +============================= + +.. This section singles out the most important changes in Python 3.6. + Brevity is key. + +New syntax features: + +* PEP 498: :ref:`Formatted string literals <whatsnew-fstrings>` + +Windows improvements: + +* The ``py.exe`` launcher, when used interactively, no longer prefers + Python 2 over Python 3 when the user doesn't specify a version (via + command line arguments or a config file). Handling of shebang lines + remains unchanged - "python" refers to Python 2 in that case. + +.. PEP-sized items next. + +.. _pep-4XX: + +.. PEP 4XX: Virtual Environments +.. ============================= + + +.. (Implemented by Foo Bar.) + +.. .. seealso:: + + :pep:`4XX` - Python Virtual Environments + PEP written by Carl Meyer + + +New Features +============ + +.. _whatsnew-fstrings: + +PEP 498: Formatted string literals +---------------------------------- + +Formatted string literals are a new kind of string literal, prefixed +with ``'f'``. They are similar to the format strings accepted by +:meth:`str.format`. They contain replacement fields surrounded by +curly braces. The replacement fields are expressions, which are +evaluated at run time, and then formatted using the :func:`format` protocol. + + >>> name = "Fred" + >>> f"He said his name is {name}." + 'He said his name is Fred.' + +See :pep:`498` and the main documentation at :ref:`f-strings`. + + +PEP 487: Simpler customization of class creation +------------------------------------------------ + +Upon subclassing a class, the ``__init_subclass__`` classmethod (if defined) is +called on the base class. This makes it straightforward to write classes that +customize initialization of future subclasses without introducing the +complexity of a full custom metaclass. + +The descriptor protocol has also been expanded to include a new optional method, +``__set_name__``. Whenever a new class is defined, the new method will be called +on all descriptors included in the definition, providing them with a reference +to the class being defined and the name given to the descriptor within the +class namespace. + +Also see :pep:`487` and the updated class customization documentation at +:ref:`class-customization` and :ref:`descriptors`. + +(Contributed by Martin Teichmann in :issue:`27366`) + + +PYTHONMALLOC environment variable +--------------------------------- + +The new :envvar:`PYTHONMALLOC` environment variable allows setting the Python +memory allocators and/or install debug hooks. + +It is now possible to install debug hooks on Python memory allocators on Python +compiled in release mode using ``PYTHONMALLOC=debug``. Effects of debug hooks: + +* Newly allocated memory is filled with the byte ``0xCB`` +* Freed memory is filled with the byte ``0xDB`` +* Detect violations of Python memory allocator API. For example, + :c:func:`PyObject_Free` called on a memory block allocated by + :c:func:`PyMem_Malloc`. +* Detect write before the start of the buffer (buffer underflow) +* Detect write after the end of the buffer (buffer overflow) +* Check that the :term:`GIL <global interpreter lock>` is held when allocator + functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex: :c:func:`PyObject_Malloc`) and + :c:data:`PYMEM_DOMAIN_MEM` (ex: :c:func:`PyMem_Malloc`) domains are called. + +Checking if the GIL is held is also a new feature of Python 3.6. + +See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python +memory allocators. + +It is now also possible to force the usage of the :c:func:`malloc` allocator of +the C library for all Python memory allocations using ``PYTHONMALLOC=malloc``. +It helps to use external memory debuggers like Valgrind on a Python compiled in +release mode. + +On error, the debug hooks on Python memory allocators now use the +:mod:`tracemalloc` module to get the traceback where a memory block was +allocated. + +Example of fatal error on buffer overflow using +``python3.6 -X tracemalloc=5`` (store 5 frames in traces):: + + Debug memory block at address p=0x7fbcd41666f8: API 'o' + 4 bytes originally requested + The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. + The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb): + at tail+0: 0x02 *** OUCH + at tail+1: 0xfb + at tail+2: 0xfb + at tail+3: 0xfb + at tail+4: 0xfb + at tail+5: 0xfb + at tail+6: 0xfb + at tail+7: 0xfb + The block was made by call #1233329 to debug malloc/realloc. + Data at p: 1a 2b 30 00 + + Memory block allocated at (most recent call first): + File "test/test_bytes.py", line 323 + File "unittest/case.py", line 600 + File "unittest/case.py", line 648 + File "unittest/suite.py", line 122 + File "unittest/suite.py", line 84 + + Fatal Python error: bad trailing pad byte + + Current thread 0x00007fbcdbd32700 (most recent call first): + File "test/test_bytes.py", line 323 in test_hex + File "unittest/case.py", line 600 in run + File "unittest/case.py", line 648 in __call__ + File "unittest/suite.py", line 122 in run + File "unittest/suite.py", line 84 in __call__ + File "unittest/suite.py", line 122 in run + File "unittest/suite.py", line 84 in __call__ + ... + +(Contributed by Victor Stinner in :issue:`26516` and :issue:`26564`.) + + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* Long sequences of repeated traceback lines are now abbreviated as + ``"[Previous line repeated {count} more times]"`` (see + :ref:`py36-traceback` for an example). + (Contributed by Emanuel Barry in :issue:`26823`.) + + +New Modules +=========== + +* None yet. + + +Improved Modules +================ + + +asyncio +------- + +Since the :mod:`asyncio` module is :term:`provisional <provisional api>`, +all changes introduced in Python 3.6 have also been backported to Python +3.5.x. + +Notable changes in the :mod:`asyncio` module since Python 3.5.0: + +* The :func:`~asyncio.ensure_future` function and all functions that + use it, such as :meth:`loop.run_until_complete() <asyncio.BaseEventLoop.run_until_complete>`, + now accept all kinds of :term:`awaitable objects <awaitable>`. + (Contributed by Yury Selivanov.) + +* New :func:`~asyncio.run_coroutine_threadsafe` function to submit + coroutines to event loops from other threads. + (Contributed by Vincent Michel.) + +* New :meth:`Transport.is_closing() <asyncio.BaseTransport.is_closing>` + method to check if the transport is closing or closed. + (Contributed by Yury Selivanov.) + +* The :meth:`loop.create_server() <asyncio.BaseEventLoop.create_server>` + method can now accept a list of hosts. + (Contributed by Yann Sionneau.) + +* New :meth:`loop.create_future() <asyncio.BaseEventLoop.create_future>` + method to create Future objects. This allows alternative event + loop implementations, such as + `uvloop <https://github.com/MagicStack/uvloop>`_, to provide a faster + :class:`asyncio.Future` implementation. + (Contributed by Yury Selivanov.) + +* New :meth:`loop.get_exception_handler() <asyncio.BaseEventLoop.get_exception_handler>` + method to get the current exception handler. + (Contributed by Yury Selivanov.) + +* New :func:`~asyncio.timeout` context manager to simplify timeouts + handling code. + (Contributed by Andrew Svetlov.) + +* New :meth:`StreamReader.readuntil() <asyncio.StreamReader.readuntil>` + method to read data from the stream until a separator bytes + sequence appears. + (Contributed by Mark Korenberg.) + +* The :meth:`loop.getaddrinfo() <asyncio.BaseEventLoop.getaddrinfo>` + method is optimized to avoid calling the system ``getaddrinfo`` + function if the address is already resolved. + (Contributed by A. Jesse Jiryu Davis.) + + +contextlib +---------- + +The :class:`contextlib.AbstractContextManager` class has been added to +provide an abstract base class for context managers. It provides a +sensible default implementation for `__enter__()` which returns +``self`` and leaves `__exit__()` an abstract method. A matching +class has been added to the :mod:`typing` module as +:class:`typing.ContextManager`. +(Contributed by Brett Cannon in :issue:`25609`.) + + +venv +---- + +:mod:`venv` accepts a new parameter ``--prompt``. This parameter provides an +alternative prefix for the virtual environment. (Proposed by Łukasz.Balcerzak +and ported to 3.6 by Stéphane Wirtel in :issue:`22829`.) + + +datetime +-------- + +The :meth:`datetime.strftime() <datetime.datetime.strftime>` and +:meth:`date.strftime() <datetime.date.strftime>` methods now support ISO 8601 date +directives ``%G``, ``%u`` and ``%V``. +(Contributed by Ashley Anderson in :issue:`12006`.) + + +distutils.command.sdist +----------------------- + +The ``default_format`` attribute has been removed from +:class:`distutils.command.sdist.sdist` and the ``formats`` +attribute defaults to ``['gztar']``. Although not anticipated, +Any code relying on the presence of ``default_format`` may +need to be adapted. See :issue:`27819` for more details. + + +faulthandler +------------ + +On Windows, the :mod:`faulthandler` module now installs a handler for Windows +exceptions: see :func:`faulthandler.enable`. (Contributed by Victor Stinner in +:issue:`23848`.) + + +http.client +----------- + +:meth:`HTTPConnection.request() <http.client.HTTPConnection.request>` and +:meth:`~http.client.HTTPConnection.endheaders` both now support +chunked encoding request bodies. +(Contributed by Demian Brecht and Rolf Krahl in :issue:`12319`.) + + +idlelib and IDLE +---------------- + +The idlelib package is being modernized and refactored to make IDLE look and work better and to make the code easier to understand, test, and improve. Part of making IDLE look better, especially on Linux and Mac, is using ttk widgets, mostly in the dialogs. As a result, IDLE no longer runs with tcl/tk 8.4. It now requires tcl/tk 8.5 or 8.6. We recommend running the latest release of either. + +'Modernizing' includes renaming and consolidation of idlelib modules. The renaming of files with partial uppercase names is similar to the renaming of, for instance, Tkinter and TkFont to tkinter and tkinter.font in 3.0. As a result, imports of idlelib files that worked in 3.5 will usually not work in 3.6. At least a module name change will be needed (see idlelib/README.txt), sometimes more. (Name changes contributed by Al Swiegart and Terry Reedy in :issue:`24225`. Most idlelib patches since have been and will be part of the process.) + +In compensation, the eventual result with be that some idlelib classes will be easier to use, with better APIs and docstrings explaining them. Additional useful information will be added to idlelib when available. + + +importlib +--------- + +:class:`importlib.util.LazyLoader` now calls +:meth:`~importlib.abc.Loader.create_module` on the wrapped loader, removing the +restriction that :class:`importlib.machinery.BuiltinImporter` and +:class:`importlib.machinery.ExtensionFileLoader` couldn't be used with +:class:`importlib.util.LazyLoader`. + + +os +-- + +A new :meth:`~os.scandir.close` method allows explicitly closing a +:func:`~os.scandir` iterator. The :func:`~os.scandir` iterator now +supports the :term:`context manager` protocol. If a :func:`scandir` +iterator is neither exhausted nor explicitly closed a :exc:`ResourceWarning` +will be emitted in its destructor. +(Contributed by Serhiy Storchaka in :issue:`25994`.) + + +pickle +------ + +Objects that need calling ``__new__`` with keyword arguments can now be pickled +using :ref:`pickle protocols <pickle-protocols>` older than protocol version 4. +Protocol version 4 already supports this case. (Contributed by Serhiy +Storchaka in :issue:`24164`.) + + +readline +-------- + +Added :func:`~readline.set_auto_history` to enable or disable +automatic addition of input to the history list. (Contributed by +Tyler Crompton in :issue:`26870`.) + + +rlcompleter +----------- + +Private and special attribute names now are omitted unless the prefix starts +with underscores. A space or a colon is added after some completed keywords. +(Contributed by Serhiy Storchaka in :issue:`25011` and :issue:`25209`.) + +Names of most attributes listed by :func:`dir` are now completed. +Previously, names of properties and slots which were not yet created on +an instance were excluded. (Contributed by Martin Panter in :issue:`25590`.) + + +site +---- + +When specifying paths to add to :attr:`sys.path` in a `.pth` file, +you may now specify file paths on top of directories (e.g. zip files). +(Contributed by Wolfgang Langner in :issue:`26587`). + + +sqlite3 +------- + +* :attr:`sqlite3.Cursor.lastrowid` now supports the ``REPLACE`` statement. + (Contributed by Alex LordThorsen in :issue:`16864`.) + + +socket +------ + +The :func:`~socket.socket.ioctl` function now supports the :data:`~socket.SIO_LOOPBACK_FAST_PATH` +control code. +(Contributed by Daniel Stokes in :issue:`26536`.) + +The :meth:`~socket.socket.getsockopt` constants ``SO_DOMAIN``, +``SO_PROTOCOL``, ``SO_PEERSEC``, and ``SO_PASSSEC`` are now supported. +(Contributed by Christian Heimes in :issue:`26907`.) + + +socketserver +------------ + +Servers based on the :mod:`socketserver` module, including those +defined in :mod:`http.server`, :mod:`xmlrpc.server` and +:mod:`wsgiref.simple_server`, now support the :term:`context manager` +protocol. +(Contributed by Aviv Palivoda in :issue:`26404`.) + +The :attr:`~socketserver.StreamRequestHandler.wfile` attribute of +:class:`~socketserver.StreamRequestHandler` classes now implements +the :class:`io.BufferedIOBase` writable interface. In particular, +calling :meth:`~io.BufferedIOBase.write` is now guaranteed to send the +data in full. (Contributed by Martin Panter in :issue:`26721`.) + + +subprocess +---------- + +:class:`subprocess.Popen` destructor now emits a :exc:`ResourceWarning` warning +if the child process is still running. Use the context manager protocol (``with +proc: ...``) or call explicitly the :meth:`~subprocess.Popen.wait` method to +read the exit status of the child process (Contributed by Victor Stinner in +:issue:`26741`). + + +telnetlib +--------- + +:class:`~telnetlib.Telnet` is now a context manager (contributed by +Stéphane Wirtel in :issue:`25485`). + + +tkinter +------- + +Added methods :meth:`~tkinter.Variable.trace_add`, +:meth:`~tkinter.Variable.trace_remove` and :meth:`~tkinter.Variable.trace_info` +in the :class:`tkinter.Variable` class. They replace old methods +:meth:`~tkinter.Variable.trace_variable`, :meth:`~tkinter.Variable.trace`, +:meth:`~tkinter.Variable.trace_vdelete` and +:meth:`~tkinter.Variable.trace_vinfo` that use obsolete Tcl commands and might +not work in future versions of Tcl. +(Contributed by Serhiy Storchaka in :issue:`22115`). + + +.. _py36-traceback: + +traceback +--------- + +Both the traceback module and the interpreter's builtin exception display now +abbreviate long sequences of repeated lines in tracebacks as shown in the +following example:: + + >>> def f(): f() + ... + >>> f() + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + File "<stdin>", line 1, in f + File "<stdin>", line 1, in f + File "<stdin>", line 1, in f + [Previous line repeated 995 more times] + RecursionError: maximum recursion depth exceeded + +(Contributed by Emanuel Barry in :issue:`26823`.) + + +typing +------ + +The :class:`typing.ContextManager` class has been added for +representing :class:`contextlib.AbstractContextManager`. +(Contributed by Brett Cannon in :issue:`25609`.) + + +unittest.mock +------------- + +The :class:`~unittest.mock.Mock` class has the following improvements: + +* Two new methods, :meth:`Mock.assert_called() + <unittest.mock.Mock.assert_called>` and :meth:`Mock.assert_called_once() + <unittest.mock.Mock.assert_called_once>` to check if the mock object + was called. + (Contributed by Amit Saha in :issue:`26323`.) + + +urllib.request +-------------- + +If a HTTP request has a non-empty body but no Content-Length header +and the content length cannot be determined up front, rather than +throwing an error, :class:`~urllib.request.AbstractHTTPHandler` now +falls back to use chunked transfer encoding. +(Contributed by Demian Brecht and Rolf Krahl in :issue:`12319`.) + + +urllib.robotparser +------------------ + +:class:`~urllib.robotparser.RobotFileParser` now supports the ``Crawl-delay`` and +``Request-rate`` extensions. +(Contributed by Nikolay Bogoychev in :issue:`16099`.) + + +warnings +-------- + +A new optional *source* parameter has been added to the +:func:`warnings.warn_explicit` function: the destroyed object which emitted a +:exc:`ResourceWarning`. A *source* attribute has also been added to +:class:`warnings.WarningMessage` (contributed by Victor Stinner in +:issue:`26568` and :issue:`26567`). + +When a :exc:`ResourceWarning` warning is logged, the :mod:`tracemalloc` is now +used to try to retrieve the traceback where the detroyed object was allocated. + +Example with the script ``example.py``:: + + import warnings + + def func(): + return open(__file__) + + f = func() + f = None + +Output of the command ``python3.6 -Wd -X tracemalloc=5 example.py``:: + + example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper name='example.py' mode='r' encoding='UTF-8'> + f = None + Object allocated at (most recent call first): + File "example.py", lineno 4 + return open(__file__) + File "example.py", lineno 6 + f = func() + +The "Object allocated at" traceback is new and only displayed if +:mod:`tracemalloc` is tracing Python memory allocations and if the +:mod:`warnings` was already imported. + + +winreg +------ + +Added the 64-bit integer type :data:`REG_QWORD <winreg.REG_QWORD>`. +(Contributed by Clement Rouault in :issue:`23026`.) + + +zipfile +------- + +A new :meth:`ZipInfo.from_file() <zipfile.ZipInfo.from_file>` class method +allows making a :class:`~zipfile.ZipInfo` instance from a filesystem file. +A new :meth:`ZipInfo.is_dir() <zipfile.ZipInfo.is_dir>` method can be used +to check if the :class:`~zipfile.ZipInfo` instance represents a directory. +(Contributed by Thomas Kluyver in :issue:`26039`.) + +The :meth:`ZipFile.open() <zipfile.ZipFile.open>` method can now be used to +write data into a ZIP file, as well as for extracting data. +(Contributed by Thomas Kluyver in :issue:`26039`.) + + +zlib +---- + +The :func:`~zlib.compress` function now accepts keyword arguments. +(Contributed by Aviv Palivoda in :issue:`26243`.) + + +fileinput +--------- + +:func:`~fileinput.hook_encoded` now supports the *errors* argument. +(Contributed by Joseph Hackman in :issue:`25788`.) + + +Optimizations +============= + +* The ASCII decoder is now up to 60 times as fast for error handlers + ``surrogateescape``, ``ignore`` and ``replace`` (Contributed + by Victor Stinner in :issue:`24870`). + +* The ASCII and the Latin1 encoders are now up to 3 times as fast for the + error handler ``surrogateescape`` (Contributed by Victor Stinner in :issue:`25227`). + +* The UTF-8 encoder is now up to 75 times as fast for error handlers + ``ignore``, ``replace``, ``surrogateescape``, ``surrogatepass`` (Contributed + by Victor Stinner in :issue:`25267`). + +* The UTF-8 decoder is now up to 15 times as fast for error handlers + ``ignore``, ``replace`` and ``surrogateescape`` (Contributed + by Victor Stinner in :issue:`25301`). + +* ``bytes % args`` is now up to 2 times faster. (Contributed by Victor Stinner + in :issue:`25349`). + +* ``bytearray % args`` is now between 2.5 and 5 times faster. (Contributed by + Victor Stinner in :issue:`25399`). + +* Optimize :meth:`bytes.fromhex` and :meth:`bytearray.fromhex`: they are now + between 2x and 3.5x faster. (Contributed by Victor Stinner in :issue:`25401`). + +* Optimize ``bytes.replace(b'', b'.')`` and ``bytearray.replace(b'', b'.')``: + up to 80% faster. (Contributed by Josh Snider in :issue:`26574`). + +* Allocator functions of the :c:func:`PyMem_Malloc` domain + (:c:data:`PYMEM_DOMAIN_MEM`) now use the :ref:`pymalloc memory allocator + <pymalloc>` instead of :c:func:`malloc` function of the C library. The + pymalloc allocator is optimized for objects smaller or equal to 512 bytes + with a short lifetime, and use :c:func:`malloc` for larger memory blocks. + (Contributed by Victor Stinner in :issue:`26249`). + +* :func:`pickle.load` and :func:`pickle.loads` are now up to 10% faster when + deserializing many small objects (Contributed by Victor Stinner in + :issue:`27056`). + +- Passing :term:`keyword arguments <keyword argument>` to a function has an + overhead in comparison with passing :term:`positional arguments + <positional argument>`. Now in extension functions implemented with using + Argument Clinic this overhead is significantly decreased. + (Contributed by Serhiy Storchaka in :issue:`27574`). + + +Build and C API Changes +======================= + +* New :c:func:`Py_FinalizeEx` API which indicates if flushing buffered data + failed (:issue:`5319`). + +* :c:func:`PyArg_ParseTupleAndKeywords` now supports :ref:`positional-only + parameters <positional-only_parameter>`. Positional-only parameters are + defined by empty names. + (Contributed by Serhiy Storchaka in :issue:`26282`). + +* ``PyTraceback_Print`` method now abbreviates long sequences of repeated lines + as ``"[Previous line repeated {count} more times]"``. + (Contributed by Emanuel Barry in :issue:`26823`.) + + +Deprecated +========== + +New Keywords +------------ + +``async`` and ``await`` are not recommended to be used as variable, class, +function or module names. Introduced by :pep:`492` in Python 3.5, they will +become proper keywords in Python 3.7. + + +Deprecated Python modules, functions and methods +------------------------------------------------ + +* :meth:`importlib.machinery.SourceFileLoader.load_module` and + :meth:`importlib.machinery.SourcelessFileLoader.load_module` are now + deprecated. They were the only remaining implementations of + :meth:`importlib.abc.Loader.load_module` in :mod:`importlib` that had not + been deprecated in previous versions of Python in favour of + :meth:`importlib.abc.Loader.exec_module`. + + +Deprecated functions and types of the C API +------------------------------------------- + +* None yet. + + +Deprecated features +------------------- + +* The ``pyvenv`` script has been deprecated in favour of ``python3 -m venv``. + This prevents confusion as to what Python interpreter ``pyvenv`` is + connected to and thus what Python interpreter will be used by the virtual + environment. (Contributed by Brett Cannon in :issue:`25154`.) + +* When performing a relative import, falling back on ``__name__`` and + ``__path__`` from the calling module when ``__spec__`` or + ``__package__`` are not defined now raises an :exc:`ImportWarning`. + (Contributed by Rose Ames in :issue:`25791`.) + +* Unlike to other :mod:`dbm` implementations, the :mod:`dbm.dumb` module + creates database in ``'r'`` and ``'w'`` modes if it doesn't exist and + allows modifying database in ``'r'`` mode. This behavior is now deprecated + and will be removed in 3.8. + (Contributed by Serhiy Storchaka in :issue:`21708`.) + +* Undocumented support of general :term:`bytes-like objects <bytes-like object>` + as paths in :mod:`os` functions, :func:`compile` and similar functions is + now deprecated. + (Contributed by Serhiy Storchaka in :issue:`25791` and :issue:`26754`.) + + +Deprecated Python behavior +-------------------------- + +* Raising the :exc:`StopIteration` exception inside a generator will now generate a + :exc:`DeprecationWarning`, and will trigger a :exc:`RuntimeError` in Python 3.7. + See :ref:`whatsnew-pep-479` for details. + + +Removed +======= + +API and Feature Removals +------------------------ + +* ``inspect.getmoduleinfo()`` was removed (was deprecated since CPython 3.3). + :func:`inspect.getmodulename` should be used for obtaining the module + name for a given path. + +* ``traceback.Ignore`` class and ``traceback.usage``, ``traceback.modname``, + ``traceback.fullmodname``, ``traceback.find_lines_from_code``, + ``traceback.find_lines``, ``traceback.find_strings``, + ``traceback.find_executable_lines`` methods were removed from the + :mod:`traceback` module. They were undocumented methods deprecated since + Python 3.2 and equivalent functionality is available from private methods. + +* The ``tk_menuBar()`` and ``tk_bindForTraversal()`` dummy methods in + :mod:`tkinter` widget classes were removed (corresponding Tk commands + were obsolete since Tk 4.0). + +* The :meth:`~zipfile.ZipFile.open` method of the :class:`zipfile.ZipFile` + class no longer supports the ``'U'`` mode (was deprecated since Python 3.4). + Use :class:`io.TextIOWrapper` for reading compressed text files in + :term:`universal newlines` mode. + + +Porting to Python 3.6 +===================== + +This section lists previously described changes and other bugfixes +that may require changes to your code. + +Changes in 'python' Command Behavior +------------------------------------ + +* The output of a special Python build with defined ``COUNT_ALLOCS``, + ``SHOW_ALLOC_COUNT`` or ``SHOW_TRACK_COUNT`` macros is now off by + default. It can be re-enabled using the ``-X showalloccount`` option. + It now outputs to ``stderr`` instead of ``stdout``. + (Contributed by Serhiy Storchaka in :issue:`23034`.) + + +Changes in the Python API +------------------------- + +* When :meth:`importlib.abc.Loader.exec_module` is defined, + :meth:`importlib.abc.Loader.create_module` must also be defined. + +* The format of the ``co_lnotab`` attribute of code objects changed to support + negative line number delta. By default, Python does not emit bytecode with + negative line number delta. Functions using ``frame.f_lineno``, + ``PyFrame_GetLineNumber()`` or ``PyCode_Addr2Line()`` are not affected. + Functions decoding directly ``co_lnotab`` should be updated to use a signed + 8-bit integer type for the line number delta, but it's only required to + support applications using negative line number delta. See + ``Objects/lnotab_notes.txt`` for the ``co_lnotab`` format and how to decode + it, and see the :pep:`511` for the rationale. + +* The functions in the :mod:`compileall` module now return booleans instead + of ``1`` or ``0`` to represent success or failure, respectively. Thanks to + booleans being a subclass of integers, this should only be an issue if you + were doing identity checks for ``1`` or ``0``. See :issue:`25768`. + +* Reading the :attr:`~urllib.parse.SplitResult.port` attribute of + :func:`urllib.parse.urlsplit` and :func:`~urllib.parse.urlparse` results + now raises :exc:`ValueError` for out-of-range values, rather than + returning :const:`None`. See :issue:`20059`. + +* The :mod:`imp` module now raises a :exc:`DeprecationWarning` instead of + :exc:`PendingDeprecationWarning`. + +* The following modules have had missing APIs added to their :attr:`__all__` + attributes to match the documented APIs: + :mod:`calendar`, :mod:`cgi`, :mod:`csv`, + :mod:`~xml.etree.ElementTree`, :mod:`enum`, + :mod:`fileinput`, :mod:`ftplib`, :mod:`logging`, :mod:`mailbox`, + :mod:`mimetypes`, :mod:`optparse`, :mod:`plistlib`, :mod:`smtpd`, + :mod:`subprocess`, :mod:`tarfile`, :mod:`threading` and + :mod:`wave`. This means they will export new symbols when ``import *`` + is used. See :issue:`23883`. + +* When performing a relative import, if ``__package__`` does not compare equal + to ``__spec__.parent`` then :exc:`ImportWarning` is raised. + (Contributed by Brett Cannon in :issue:`25791`.) + +* When a relative import is performed and no parent package is known, then + :exc:`ImportError` will be raised. Previously, :exc:`SystemError` could be + raised. (Contributed by Brett Cannon in :issue:`18018`.) + +* Servers based on the :mod:`socketserver` module, including those + defined in :mod:`http.server`, :mod:`xmlrpc.server` and + :mod:`wsgiref.simple_server`, now only catch exceptions derived + from :exc:`Exception`. Therefore if a request handler raises + an exception like :exc:`SystemExit` or :exc:`KeyboardInterrupt`, + :meth:`~socketserver.BaseServer.handle_error` is no longer called, and + the exception will stop a single-threaded server. (Contributed by + Martin Panter in :issue:`23430`.) + +* :func:`spwd.getspnam` now raises a :exc:`PermissionError` instead of + :exc:`KeyError` if the user doesn't have privileges. + +* The :meth:`socket.socket.close` method now raises an exception if + an error (e.g. EBADF) was reported by the underlying system call. + See :issue:`26685`. + +* The *decode_data* argument for :class:`smtpd.SMTPChannel` and + :class:`smtpd.SMTPServer` constructors is now ``False`` by default. + This means that the argument passed to + :meth:`~smtpd.SMTPServer.process_message` is now a bytes object by + default, and ``process_message()`` will be passed keyword arguments. + Code that has already been updated in accordance with the deprecation + warning generated by 3.5 will not be affected. + +* All optional parameters of the :func:`~json.dump`, :func:`~json.dumps`, + :func:`~json.load` and :func:`~json.loads` functions and + :class:`~json.JSONEncoder` and :class:`~json.JSONDecoder` class + constructors in the :mod:`json` module are now :ref:`keyword-only + <keyword-only_parameter>`. + (Contributed by Serhiy Storchaka in :issue:`18726`.) + +* As part of :pep:`487`, the handling of keyword arguments passed to + :class:`type` (other than the metaclass hint, ``metaclass``) is now + consistently delegated to :meth:`object.__init_subclass__`. This means that + :meth:`type.__new__` and :meth:`type.__init__` both now accept arbitrary + keyword arguments, but :meth:`object.__init_subclass__` (which is called from + :meth:`type.__new__`) will reject them by default. Custom metaclasses + accepting additional keyword arguments will need to adjust their calls to + :meth:`type.__new__` (whether direct or via :class:`super`) accordingly. + +* In :class:`distutils.command.sdist.sdist`, the ``default_format`` + attribute has been removed and is no longer honored. Instead, the + gzipped tarfile format is the default on all platforms and no + platform-specific selection is made. + In environments where distributions are + built on Windows and zip distributions are required, configure + the project with a ``setup.cfg`` file containing the following:: + + [sdist] + formats=zip + + This behavior has also been backported to earlier Python versions + by Setuptools 26.0.0. + +Changes in the C API +-------------------- + +* :c:func:`PyMem_Malloc` allocator family now uses the :ref:`pymalloc allocator + <pymalloc>` rather than system :c:func:`malloc`. Applications calling + :c:func:`PyMem_Malloc` without holding the GIL can now crash. Set the + :envvar:`PYTHONMALLOC` environment variable to ``debug`` to validate the + usage of memory allocators in your application. See :issue:`26249`. + +* :c:func:`Py_Exit` (and the main interpreter) now override the exit status + with 120 if flushing buffered data failed. See :issue:`5319`. diff --git a/Doc/whatsnew/index.rst b/Doc/whatsnew/index.rst index edb5502..7c92524 100644 --- a/Doc/whatsnew/index.rst +++ b/Doc/whatsnew/index.rst @@ -11,6 +11,7 @@ anyone wishing to stay up-to-date after a new release. .. toctree:: :maxdepth: 2 + 3.6.rst 3.5.rst 3.4.rst 3.3.rst |