diff options
Diffstat (limited to 'Doc/whatsnew')
| -rw-r--r-- | Doc/whatsnew/2.0.rst | 32 | ||||
| -rw-r--r-- | Doc/whatsnew/2.1.rst | 28 | ||||
| -rw-r--r-- | Doc/whatsnew/2.2.rst | 42 | ||||
| -rw-r--r-- | Doc/whatsnew/2.3.rst | 28 | ||||
| -rw-r--r-- | Doc/whatsnew/2.4.rst | 6 | ||||
| -rw-r--r-- | Doc/whatsnew/2.5.rst | 33 | ||||
| -rw-r--r-- | Doc/whatsnew/2.6.rst | 58 | ||||
| -rw-r--r-- | Doc/whatsnew/2.7.rst | 48 | ||||
| -rw-r--r-- | Doc/whatsnew/3.0.rst | 929 | ||||
| -rw-r--r-- | Doc/whatsnew/3.1.rst | 552 | ||||
| -rw-r--r-- | Doc/whatsnew/3.2.rst | 2739 | ||||
| -rw-r--r-- | Doc/whatsnew/3.3.rst | 2517 | ||||
| -rw-r--r-- | Doc/whatsnew/3.4.rst | 2545 | ||||
| -rw-r--r-- | Doc/whatsnew/3.5.rst | 2574 | ||||
| -rw-r--r-- | Doc/whatsnew/3.6.rst | 2435 | ||||
| -rw-r--r-- | Doc/whatsnew/3.7.rst | 2543 | ||||
| -rw-r--r-- | Doc/whatsnew/3.8.rst | 2206 | ||||
| -rw-r--r-- | Doc/whatsnew/3.9.rst | 374 | ||||
| -rw-r--r-- | Doc/whatsnew/changelog.rst | 7 | ||||
| -rw-r--r-- | Doc/whatsnew/index.rst | 20 |
20 files changed, 120 insertions, 19596 deletions
diff --git a/Doc/whatsnew/2.0.rst b/Doc/whatsnew/2.0.rst index ace396b..a35728a 100644 --- a/Doc/whatsnew/2.0.rst +++ b/Doc/whatsnew/2.0.rst @@ -111,7 +111,7 @@ python-dev. Therefore, a relatively formal process has been set up to write Python Enhancement Proposals (PEPs), modelled on the Internet RFC process. PEPs are draft documents that describe a proposed new feature, and are continually revised until the community reaches a consensus, either accepting or rejecting -the proposal. Quoting from the introduction to :pep:`1`, "PEP Purpose and +the proposal. Quoting from the introduction to PEP 1, "PEP Purpose and Guidelines": @@ -127,11 +127,11 @@ Guidelines": that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. -Read the rest of :pep:`1` for the details of the PEP editorial process, style, and +Read the rest of PEP 1 for the details of the PEP editorial process, style, and format. PEPs are kept in the Python CVS tree on SourceForge, though they're not part of the Python 2.0 distribution, and are also available in HTML form from https://www.python.org/dev/peps/. As of September 2000, there are 25 PEPS, ranging -from :pep:`201`, "Lockstep Iteration", to PEP 225, "Elementwise/Objectwise +from PEP 201, "Lockstep Iteration", to PEP 225, "Elementwise/Objectwise Operators". .. ====================================================================== @@ -166,7 +166,7 @@ encoding. Encodings are named by strings, such as ``'ascii'``, ``'utf-8'``, registering new encodings that are then available throughout a Python program. If an encoding isn't specified, the default encoding is usually 7-bit ASCII, though it can be changed for your Python installation by calling the -``sys.setdefaultencoding(encoding)`` function in a customized version of +``sys.setdefaultencoding(encoding)`` function in a customised version of :file:`site.py`. Combining 8-bit and Unicode strings always coerces to Unicode, using the default @@ -189,7 +189,7 @@ support Unicode: ignored and ``'replace'`` uses U+FFFD, the official replacement character, in case of any problems. -* The ``exec`` statement, and various built-ins such as ``eval()``, +* The :keyword:`exec` statement, and various built-ins such as ``eval()``, ``getattr()``, and ``setattr()`` will also accept Unicode strings as well as regular strings. (It's possible that the process of fixing this missed some built-ins; if you find a built-in function that accepts strings but doesn't @@ -284,7 +284,7 @@ write the following to do it:: L) Because of Python's scoping rules, a default argument is used so that the -anonymous function created by the :keyword:`lambda` expression knows what +anonymous function created by the :keyword:`lambda` statement knows what substring is being searched for. List comprehensions make this cleaner:: sublist = [ s for s in L if string.find(s, S) != -1 ] @@ -296,11 +296,11 @@ List comprehensions have the form:: for exprN in sequenceN if condition ] -The :keyword:`!for`...\ :keyword:`!in` clauses contain the sequences to be +The :keyword:`for`...\ :keyword:`in` clauses contain the sequences to be iterated over. The sequences do not have to be the same length, because they are *not* iterated over in parallel, but from left to right; this is explained more clearly in the following paragraphs. The elements of the generated list -will be the successive values of *expression*. The final :keyword:`!if` clause +will be the successive values of *expression*. The final :keyword:`if` clause is optional; if present, *expression* is only evaluated and added to the result if *condition* is true. @@ -316,7 +316,7 @@ following Python code:: # the expression to the # resulting list. -This means that when there are multiple :keyword:`!for`...\ :keyword:`!in` +This means that when there are multiple :keyword:`for`...\ :keyword:`in` clauses, the resulting list will be equal to the product of the lengths of all the sequences. If you have two lists of length 3, the output list is 9 elements long:: @@ -515,11 +515,11 @@ functions:: # kw is a dictionary of keyword args ... -The ``print`` statement can now have its output directed to a file-like -object by following the ``print`` with ``>> file``, similar to the +The :keyword:`print` statement can now have its output directed to a file-like +object by following the :keyword:`print` with ``>> file``, similar to the redirection operator in Unix shells. Previously you'd either have to use the :meth:`write` method of the file-like object, which lacks the convenience and -simplicity of ``print``, or you could assign a new value to +simplicity of :keyword:`print`, or you could assign a new value to ``sys.stdout`` and then restore the old value. For sending output to standard error, it's much easier to write this:: @@ -541,8 +541,8 @@ true if *obj* is present in the sequence *seq*; Python computes this by simply trying every index of the sequence until either *obj* is found or an :exc:`IndexError` is encountered. Moshe Zadka contributed a patch which adds a :meth:`__contains__` magic method for providing a custom implementation for -:keyword:`!in`. Additionally, new built-in objects written in C can define what -:keyword:`!in` means for them via a new slot in the sequence protocol. +:keyword:`in`. Additionally, new built-in objects written in C can define what +:keyword:`in` means for them via a new slot in the sequence protocol. Earlier versions of Python used a recursive algorithm for deleting objects. Deeply nested data structures could cause the interpreter to fill up the C stack @@ -581,7 +581,7 @@ Consult the README in the Python source distribution for more instructions. An attempt has been made to alleviate one of Python's warts, the often-confusing :exc:`NameError` exception when code refers to a local variable before the variable has been assigned a value. For example, the following code raises an -exception on the ``print`` statement in both 1.5.2 and 2.0; in 1.5.2 a +exception on the :keyword:`print` statement in both 1.5.2 and 2.0; in 1.5.2 a :exc:`NameError` exception is raised, while 2.0 raises a new :exc:`UnboundLocalError` exception. :exc:`UnboundLocalError` is a subclass of :exc:`NameError`, so any existing code that expects :exc:`NameError` to be @@ -1139,7 +1139,7 @@ module. Unix, not to be confused with :program:`gzip`\ -format files (which are supported by the :mod:`gzip` module) (Contributed by James C. Ahlstrom.) -* :mod:`imputil`: A module that provides a simpler way for writing customized +* :mod:`imputil`: A module that provides a simpler way for writing customised import hooks, in comparison to the existing :mod:`ihooks` module. (Implemented by Greg Stein, with much discussion on python-dev along the way.) diff --git a/Doc/whatsnew/2.1.rst b/Doc/whatsnew/2.1.rst index 8b1eac9..f13d55b 100644 --- a/Doc/whatsnew/2.1.rst +++ b/Doc/whatsnew/2.1.rst @@ -52,7 +52,7 @@ The function :func:`g` will always raise a :exc:`NameError` exception, because the binding of the name ``g`` isn't in either its local namespace or in the module-level namespace. This isn't much of a problem in practice (how often do you recursively define interior functions like this?), but this also made using -the :keyword:`lambda` expression clumsier, and this was a problem in practice. +the :keyword:`lambda` statement clumsier, and this was a problem in practice. In code which uses :keyword:`lambda` you can often find local variables being copied by passing them as the default values of arguments. :: @@ -81,13 +81,13 @@ though, since such code would have been pretty confusing to read in the first place. One side effect of the change is that the ``from module import *`` and -``exec`` statements have been made illegal inside a function scope under +:keyword:`exec` statements have been made illegal inside a function scope under certain conditions. The Python reference manual has said all along that ``from module import *`` is only legal at the top level of a module, but the CPython interpreter has never enforced this before. As part of the implementation of nested scopes, the compiler which turns Python source into bytecodes has to generate different code to access variables in a containing scope. ``from -module import *`` and ``exec`` make it impossible for the compiler to +module import *`` and :keyword:`exec` make it impossible for the compiler to figure this out, because they add names to the local namespace that are unknowable at compile time. Therefore, if a function contains function definitions or :keyword:`lambda` expressions with free variables, the compiler @@ -102,18 +102,18 @@ To make the preceding explanation a bit clearer, here's an example:: def g(): return x -Line 4 containing the ``exec`` statement is a syntax error, since -``exec`` would define a new local variable named ``x`` whose value should +Line 4 containing the :keyword:`exec` statement is a syntax error, since +:keyword:`exec` would define a new local variable named ``x`` whose value should be accessed by :func:`g`. -This shouldn't be much of a limitation, since ``exec`` is rarely used in +This shouldn't be much of a limitation, since :keyword:`exec` is rarely used in most Python code (and when it is used, it's often a sign of a poor design anyway). Compatibility concerns have led to nested scopes being introduced gradually; in Python 2.1, they aren't enabled by default, but can be turned on within a module -by using a future statement as described in :pep:`236`. (See the following section -for further discussion of :pep:`236`.) In Python 2.2, nested scopes will become +by using a future statement as described in PEP 236. (See the following section +for further discussion of PEP 236.) In Python 2.2, nested scopes will become the default and there will be no way to turn them off, but users will have had all of 2.1's lifetime to fix any breakage resulting from their introduction. @@ -143,7 +143,7 @@ The syntax uses a ``from...import`` statement using the reserved module name While it looks like a normal :keyword:`import` statement, it's not; there are strict rules on where such a future statement can be put. They can only be at the top of a module, and must precede any Python code or regular -:keyword:`!import` statements. This is because such statements can affect how +:keyword:`import` statements. This is because such statements can affect how the Python bytecode compiler parses code and generates bytecode, so they must precede any statement that will result in bytecodes being produced. @@ -213,7 +213,7 @@ otherwise it will call the appropriate method and can return any Python object. There are also corresponding changes of interest to C programmers; there's a new slot ``tp_richcmp`` in type objects and an API for performing a given rich -comparison. I won't cover the C API here, but will refer you to :pep:`207`, or to +comparison. I won't cover the C API here, but will refer you to PEP 207, or to 2.1's C API documentation, for the full list of related functions. @@ -367,7 +367,7 @@ dictionary:: This version works for simple things such as integers, but it has a side effect; the ``_cache`` dictionary holds a reference to the return values, so they'll -never be deallocated until the Python process exits and cleans up. This isn't +never be deallocated until the Python process exits and cleans up This isn't very noticeable for integers, but if :func:`f` returns an object, or a data structure that takes up a lot of memory, this can be a problem. @@ -445,7 +445,7 @@ The dictionary containing attributes can be accessed as the function's :attr:`~object.__dict__`. Unlike the :attr:`~object.__dict__` attribute of class instances, in functions you can actually assign a new dictionary to :attr:`~object.__dict__`, though the new value is restricted to a regular Python dictionary; you *can't* be -tricky and set it to a :class:`UserDict` instance, or any other random object +tricky and set it to a :class:`~UserDict.UserDict` instance, or any other random object that behaves like a mapping. @@ -548,7 +548,7 @@ registering software at the Vaults is optional, and many people don't bother. As a first small step toward fixing the problem, Python software packaged using the Distutils :command:`sdist` command will include a file named :file:`PKG-INFO` containing information about the package such as its name, -version, and author (metadata, in cataloguing terminology). :pep:`241` contains +version, and author (metadata, in cataloguing terminology). PEP 241 contains the full list of fields that can be present in the :file:`PKG-INFO` file. As people began to package their software using Python 2.1, more and more packages will include metadata, making it possible to build automated cataloguing systems @@ -561,7 +561,7 @@ package to a catalog server. You can start creating packages containing :file:`PKG-INFO` even if you're not using Python 2.1, since a new release of the Distutils will be made for users of earlier Python versions. Version 1.0.2 of the Distutils includes the changes -described in :pep:`241`, as well as various bugfixes and enhancements. It will be +described in PEP 241, as well as various bugfixes and enhancements. It will be available from the Distutils SIG at https://www.python.org/community/sigs/current/distutils-sig/. diff --git a/Doc/whatsnew/2.2.rst b/Doc/whatsnew/2.2.rst index b4cd434..b231967 100644 --- a/Doc/whatsnew/2.2.rst +++ b/Doc/whatsnew/2.2.rst @@ -55,7 +55,7 @@ implemented in C. In particular, it's not possible to subclass built-in types, so you can't just subclass, say, lists in order to add a single useful method to them. The :mod:`UserList` module provides a class that supports all of the methods of lists and that can be subclassed further, but there's lots of C code -that expects a regular Python list and won't accept a :class:`UserList` +that expects a regular Python list and won't accept a :class:`~UserList.UserList` instance. Python 2.2 fixes this, and in the process adds some exciting new capabilities. @@ -121,7 +121,7 @@ added so if no built-in type is suitable, you can just subclass This means that :keyword:`class` statements that don't have any base classes are always classic classes in Python 2.2. (Actually you can also change this by setting a module-level variable named :attr:`__metaclass__` --- see :pep:`253` -for the details --- but it's easier to just subclass :class:`object`.) +for the details --- but it's easier to just subclass :keyword:`object`.) The type objects for the built-in types are available as built-ins, named using a clever trick. Python has always had built-in functions named :func:`int`, @@ -560,7 +560,7 @@ Here's the simplest example of a generator function:: yield i A new keyword, :keyword:`yield`, was introduced for generators. Any function -containing a :keyword:`!yield` statement is a generator function; this is +containing a :keyword:`yield` statement is a generator function; this is detected by Python's bytecode compiler which compiles the function specially as a result. Because a new keyword was introduced, generators must be explicitly enabled in a module by including a ``from __future__ import generators`` @@ -571,14 +571,14 @@ When you call a generator function, it doesn't return a single value; instead it returns a generator object that supports the iterator protocol. On executing the :keyword:`yield` statement, the generator outputs the value of ``i``, similar to a :keyword:`return` statement. The big difference between -:keyword:`!yield` and a :keyword:`!return` statement is that on reaching a -:keyword:`!yield` the generator's state of execution is suspended and local +:keyword:`yield` and a :keyword:`return` statement is that on reaching a +:keyword:`yield` the generator's state of execution is suspended and local variables are preserved. On the next call to the generator's ``next()`` method, -the function will resume executing immediately after the :keyword:`!yield` -statement. (For complicated reasons, the :keyword:`!yield` statement isn't -allowed inside the :keyword:`!try` block of a -:keyword:`try`...\ :keyword:`finally` statement; read :pep:`255` for a full -explanation of the interaction between :keyword:`!yield` and exceptions.) +the function will resume executing immediately after the :keyword:`yield` +statement. (For complicated reasons, the :keyword:`yield` statement isn't +allowed inside the :keyword:`try` block of a :keyword:`try`...\ +:keyword:`finally` statement; read :pep:`255` for a full explanation of the +interaction between :keyword:`yield` and exceptions.) Here's a sample usage of the :func:`generate_ints` generator:: @@ -602,7 +602,7 @@ generate_ints(3)``. Inside a generator function, the :keyword:`return` statement can only be used without a value, and signals the end of the procession of values; afterwards the -generator cannot return any further values. :keyword:`!return` with a value, such +generator cannot return any further values. :keyword:`return` with a value, such as ``return 5``, is a syntax error inside a generator function. The end of the generator's results can also be indicated by raising :exc:`StopIteration` manually, or by just letting the flow of execution fall off the bottom of the @@ -758,8 +758,8 @@ Here are the changes 2.2 introduces: operators. * Python 2.2 supports some command-line arguments for testing whether code will - work with the changed division semantics. Running python with :option:`!-Q - warn` will cause a warning to be issued whenever division is applied to two + work with the changed division semantics. Running python with :option:`-Q + warn <-Q>` will cause a warning to be issued whenever division is applied to two integers. You can use this to find code that's affected by the change and fix it. By default, Python 2.2 will simply perform classic division without a warning; the warning will be turned on by default in Python 2.3. @@ -863,8 +863,8 @@ The function :func:`g` will always raise a :exc:`NameError` exception, because the binding of the name ``g`` isn't in either its local namespace or in the module-level namespace. This isn't much of a problem in practice (how often do you recursively define interior functions like this?), but this also made using -the :keyword:`lambda` expression clumsier, and this was a problem in practice. -In code which uses :keyword:`!lambda` you can often find local variables being +the :keyword:`lambda` statement clumsier, and this was a problem in practice. +In code which uses :keyword:`lambda` you can often find local variables being copied by passing them as the default values of arguments. :: def find(self, name): @@ -892,13 +892,13 @@ though, since such code would have been pretty confusing to read in the first place. One side effect of the change is that the ``from module import *`` and -``exec`` statements have been made illegal inside a function scope under +:keyword:`exec` statements have been made illegal inside a function scope under certain conditions. The Python reference manual has said all along that ``from module import *`` is only legal at the top level of a module, but the CPython interpreter has never enforced this before. As part of the implementation of nested scopes, the compiler which turns Python source into bytecodes has to generate different code to access variables in a containing scope. ``from -module import *`` and ``exec`` make it impossible for the compiler to +module import *`` and :keyword:`exec` make it impossible for the compiler to figure this out, because they add names to the local namespace that are unknowable at compile time. Therefore, if a function contains function definitions or :keyword:`lambda` expressions with free variables, the compiler @@ -913,11 +913,11 @@ To make the preceding explanation a bit clearer, here's an example:: def g(): return x -Line 4 containing the ``exec`` statement is a syntax error, since -``exec`` would define a new local variable named ``x`` whose value should +Line 4 containing the :keyword:`exec` statement is a syntax error, since +:keyword:`exec` would define a new local variable named ``x`` whose value should be accessed by :func:`g`. -This shouldn't be much of a limitation, since ``exec`` is rarely used in +This shouldn't be much of a limitation, since :keyword:`exec` is rarely used in most Python code (and when it is used, it's often a sign of a poor design anyway). @@ -957,7 +957,7 @@ New and Improved Modules # 'title': 'html2fo 0.3 (Default)'}, ... ] The :mod:`SimpleXMLRPCServer` module makes it easy to create straightforward - XML-RPC servers. See http://xmlrpc.scripting.com/ for more information about XML-RPC. + XML-RPC servers. See http://www.xmlrpc.com/ for more information about XML-RPC. * The new :mod:`hmac` module implements the HMAC algorithm described by :rfc:`2104`. (Contributed by Gerhard Häring.) diff --git a/Doc/whatsnew/2.3.rst b/Doc/whatsnew/2.3.rst index dac0e63..1ef2cb5 100644 --- a/Doc/whatsnew/2.3.rst +++ b/Doc/whatsnew/2.3.rst @@ -149,7 +149,7 @@ Here's the simplest example of a generator function:: yield i A new keyword, :keyword:`yield`, was introduced for generators. Any function -containing a :keyword:`!yield` statement is a generator function; this is +containing a :keyword:`yield` statement is a generator function; this is detected by Python's bytecode compiler which compiles the function specially as a result. @@ -157,14 +157,14 @@ When you call a generator function, it doesn't return a single value; instead it returns a generator object that supports the iterator protocol. On executing the :keyword:`yield` statement, the generator outputs the value of ``i``, similar to a :keyword:`return` statement. The big difference between -:keyword:`!yield` and a :keyword:`!return` statement is that on reaching a -:keyword:`!yield` the generator's state of execution is suspended and local +:keyword:`yield` and a :keyword:`return` statement is that on reaching a +:keyword:`yield` the generator's state of execution is suspended and local variables are preserved. On the next call to the generator's ``.next()`` method, the function will resume executing immediately after the -:keyword:`!yield` statement. (For complicated reasons, the :keyword:`!yield` -statement isn't allowed inside the :keyword:`try` block of a -:keyword:`!try`...\ :keyword:`!finally` statement; read :pep:`255` for a full -explanation of the interaction between :keyword:`!yield` and exceptions.) +:keyword:`yield` statement. (For complicated reasons, the :keyword:`yield` +statement isn't allowed inside the :keyword:`try` block of a :keyword:`try`...\ +:keyword:`finally` statement; read :pep:`255` for a full explanation of the +interaction between :keyword:`yield` and exceptions.) Here's a sample usage of the :func:`generate_ints` generator:: @@ -188,7 +188,7 @@ generate_ints(3)``. Inside a generator function, the :keyword:`return` statement can only be used without a value, and signals the end of the procession of values; afterwards the -generator cannot return any further values. :keyword:`!return` with a value, such +generator cannot return any further values. :keyword:`return` with a value, such as ``return 5``, is a syntax error inside a generator function. The end of the generator's results can also be indicated by raising :exc:`StopIteration` manually, or by just letting the flow of execution fall off the bottom of the @@ -589,7 +589,7 @@ strict language such as Pascal would also prevent you performing arithmetic with Booleans, and would require that the expression in an :keyword:`if` statement always evaluate to a Boolean result. Python is not this strict and never will be, as :pep:`285` explicitly says. This means you can still use any expression -in an :keyword:`!if` statement, even ones that evaluate to a list or tuple or +in an :keyword:`if` statement, even ones that evaluate to a list or tuple or some random object. The Boolean type is a subclass of the :class:`int` class so that arithmetic using a Boolean still works. :: @@ -1201,7 +1201,7 @@ Optimizations * The ``SET_LINENO`` opcode is now gone. This may provide a small speed increase, depending on your compiler's idiosyncrasies. See section - :ref:`23section-other` for a longer explanation. (Removed by Michael Hudson.) + :ref:`section-other` for a longer explanation. (Removed by Michael Hudson.) * :func:`xrange` objects now have their own iterator, making ``for i in xrange(n)`` slightly faster than ``for i in range(n)``. (Patch by Raymond @@ -1247,8 +1247,8 @@ complete list of changes, or look through the CVS logs for all the details. will have to change your ``import`` statements to import it as :mod:`bsddb`. * The new :mod:`bz2` module is an interface to the bz2 data compression library. - bz2-compressed data is usually smaller than corresponding - :mod:`zlib`\ -compressed data. (Contributed by Gustavo Niemeyer.) + bz2-compressed data is usually smaller than corresponding :mod:`zlib`\ + -compressed data. (Contributed by Gustavo Niemeyer.) * A set of standard date/time types has been added in the new :mod:`datetime` module. See the following section for more details. @@ -1955,12 +1955,12 @@ The RPM spec files, found in the :file:`Misc/RPM/` directory in the Python source distribution, were updated for 2.3. (Contributed by Sean Reifschneider.) Other new platforms now supported by Python include AtheOS -(http://www.atheos.cx/), GNU/Hurd, and OpenVMS. +(http://atheos.cx/), GNU/Hurd, and OpenVMS. .. ====================================================================== -.. _23section-other: +.. _section-other: Other Changes and Fixes ======================= diff --git a/Doc/whatsnew/2.4.rst b/Doc/whatsnew/2.4.rst index 7e11c98..421dc0c 100644 --- a/Doc/whatsnew/2.4.rst +++ b/Doc/whatsnew/2.4.rst @@ -899,7 +899,7 @@ Here are all of the changes that Python 2.4 makes to the core Python language. (Contributed by Nick Coghlan.) * The ``eval(expr, globals, locals)`` and ``execfile(filename, globals, - locals)`` functions and the ``exec`` statement now accept any mapping type + locals)`` functions and the :keyword:`exec` statement now accept any mapping type for the *locals* parameter. Previously this had to be a regular Python dictionary. (Contributed by Raymond Hettinger.) @@ -998,7 +998,7 @@ complete list of changes, or look through the CVS logs for all the details. that lets you perform a limited number of passes through the polling loop. The default is still to loop forever. -* The :mod:`base64` module now has more complete :rfc:`3548` support for Base64, +* The :mod:`base64` module now has more complete RFC 3548 support for Base64, Base32, and Base16 encoding and decoding, including optional case folding and optional alternative alphabets. (Contributed by Barry Warsaw.) @@ -1282,7 +1282,7 @@ complete list of changes, or look through the CVS logs for all the details. interface, accessed only by :mod:`atexit`. * The :mod:`tarfile` module now generates GNU-format tar files by default. - (Contributed by Lars Gustäbel.) + (Contributed by Lars Gustaebel.) * The :mod:`threading` module now has an elegantly simple way to support thread-local data. The module contains a :class:`local` class whose attribute diff --git a/Doc/whatsnew/2.5.rst b/Doc/whatsnew/2.5.rst index 4e85aba..6400bb1 100644 --- a/Doc/whatsnew/2.5.rst +++ b/Doc/whatsnew/2.5.rst @@ -256,7 +256,7 @@ Package uploading was implemented by Martin von Löwis and Richard Jones. PEP 328: Absolute and Relative Imports ====================================== -The simpler part of :pep:`328` was implemented in Python 2.4: parentheses could now +The simpler part of PEP 328 was implemented in Python 2.4: parentheses could now be used to enclose the names imported from a module using the ``from ... import ...`` statement, making it easier to import many different names. @@ -330,7 +330,7 @@ statement, only the ``from ... import`` form. :pep:`328` - Imports: Multi-Line and Absolute/Relative PEP written by Aahz; implemented by Thomas Wouters. - https://pylib.readthedocs.io/ + https://pylib.readthedocs.org/ The py library by Holger Krekel, which contains the :mod:`py.std` package. .. ====================================================================== @@ -370,7 +370,7 @@ PEP 341: Unified try/except/finally Until Python 2.5, the :keyword:`try` statement came in two flavours. You could use a :keyword:`finally` block to ensure that code is always executed, or one or more :keyword:`except` blocks to catch specific exceptions. You couldn't -combine both :keyword:`!except` blocks and a :keyword:`!finally` block, because +combine both :keyword:`except` blocks and a :keyword:`finally` block, because generating the right bytecode for the combined version was complicated and it wasn't clear what the semantics of the combined statement should be. @@ -435,10 +435,10 @@ When you call ``counter(10)``, the result is an iterator that returns the values from 0 up to 9. On encountering the :keyword:`yield` statement, the iterator returns the provided value and suspends the function's execution, preserving the local variables. Execution resumes on the following call to the iterator's -:meth:`next` method, picking up after the :keyword:`!yield` statement. +:meth:`next` method, picking up after the :keyword:`yield` statement. In Python 2.3, :keyword:`yield` was a statement; it didn't return any value. In -2.5, :keyword:`!yield` is now an expression, returning a value that can be +2.5, :keyword:`yield` is now an expression, returning a value that can be assigned to a variable or otherwise operated on:: val = (yield i) @@ -448,9 +448,8 @@ when you're doing something with the returned value, as in the above example. The parentheses aren't always necessary, but it's easier to always add them instead of having to remember when they're needed. -(:pep:`342` explains the exact rules, which are that a -:keyword:`yield`\ -expression must always be parenthesized except when it -occurs at the top-level +(:pep:`342` explains the exact rules, which are that a :keyword:`yield`\ +-expression must always be parenthesized except when it occurs at the top-level expression on the right-hand side of an assignment. This means you can write ``val = yield i`` but have to use parentheses when there's an operation, as in ``val = (yield i) + 12``.) @@ -458,7 +457,7 @@ expression on the right-hand side of an assignment. This means you can write Values are sent into a generator by calling its ``send(value)`` method. The generator's code is then resumed and the :keyword:`yield` expression returns the specified *value*. If the regular :meth:`next` method is called, the -:keyword:`!yield` returns :const:`None`. +:keyword:`yield` returns :const:`None`. Here's the previous example, modified to allow changing the value of the internal counter. :: @@ -530,7 +529,7 @@ chance to run. The syntactic restriction that you couldn't mix :keyword:`yield` statements with a ``try...finally`` suite has therefore been removed. This seems like a minor bit of language trivia, but using generators and ``try...finally`` is actually necessary in order to implement the -:keyword:`with` statement described by :pep:`343`. I'll look at this new statement +:keyword:`with` statement described by PEP 343. I'll look at this new statement in the following section. Another even more esoteric effect of this change: previously, the @@ -644,7 +643,7 @@ Writing Context Managers ------------------------ Under the hood, the ':keyword:`with`' statement is fairly complicated. Most -people will only use ':keyword:`!with`' in company with existing objects and +people will only use ':keyword:`with`' in company with existing objects and don't need to know these details, so you can skip the rest of this section if you like. Authors of new objects will need to understand the details of the underlying implementation and should keep reading. @@ -750,9 +749,9 @@ generator function instead of defining a new class. The generator should yield exactly one value. The code up to the :keyword:`yield` will be executed as the :meth:`__enter__` method, and the value yielded will be the method's return value that will get bound to the variable in the ':keyword:`with`' statement's -:keyword:`!as` clause, if any. The code after the :keyword:`yield` will be +:keyword:`as` clause, if any. The code after the :keyword:`yield` will be executed in the :meth:`__exit__` method. Any exception raised in the block will -be raised by the :keyword:`!yield` statement. +be raised by the :keyword:`yield` statement. Our database example from the previous section could be written using this decorator as:: @@ -776,7 +775,7 @@ decorator as:: The :mod:`contextlib` module also has a ``nested(mgr1, mgr2, ...)`` function that combines a number of context managers so you don't need to write nested -':keyword:`with`' statements. In this example, the single ':keyword:`!with`' +':keyword:`with`' statements. In this example, the single ':keyword:`with`' statement both starts a database transaction and acquires a thread lock:: lock = threading.Lock() @@ -1484,7 +1483,7 @@ complete list of changes, or look through the SVN logs for all the details. 2.4, so this completes the removal of the feature. * The :mod:`pkgutil` module, containing various utility functions for finding - packages, was enhanced to support :pep:`302`'s import hooks and now also works for + packages, was enhanced to support PEP 302's import hooks and now also works for packages stored in ZIP-format archives. (Contributed by Phillip J. Eby.) * The pybench benchmark suite by Marc-André Lemburg is now included in the @@ -1498,7 +1497,7 @@ complete list of changes, or look through the SVN logs for all the details. * The :mod:`pyexpat` module now uses version 2.0 of the Expat parser. (Contributed by Trent Mick.) -* The :class:`~queue.Queue` class provided by the :mod:`Queue` module gained two new +* The :class:`~Queue.Queue` class provided by the :mod:`Queue` module gained two new methods. :meth:`join` blocks until all items in the queue have been retrieved and all processing work on the items have been completed. Worker threads call the other new method, :meth:`task_done`, to signal that processing for an item @@ -2044,7 +2043,7 @@ between web servers and Python web applications and is described in :pep:`333`. The :mod:`wsgiref` package is a reference implementation of the WSGI specification. -.. XXX should this be in a :pep:`333` section instead? +.. XXX should this be in a PEP 333 section instead? The package includes a basic HTTP server that will run a WSGI application; this server is useful for debugging but isn't intended for production use. Setting diff --git a/Doc/whatsnew/2.6.rst b/Doc/whatsnew/2.6.rst index b6174a1..f4c742e 100644 --- a/Doc/whatsnew/2.6.rst +++ b/Doc/whatsnew/2.6.rst @@ -1,5 +1,3 @@ -.. _whats-new-in-2.6: - **************************** What's New in Python 2.6 **************************** @@ -58,7 +56,7 @@ Python 2.6 incorporates new features and syntax from 3.0 while remaining compatible with existing code by not removing older features or syntax. When it's not possible to do that, Python 2.6 tries to do what it can, adding compatibility functions in a -:mod:`future_builtins` module and a :option:`!-3` switch to warn about +:mod:`future_builtins` module and a :option:`-3` switch to warn about usages that will become unsupported in 3.0. Some significant new packages have been added to the standard library, @@ -116,7 +114,7 @@ module has versions with the new 3.0 semantics. Code written to be compatible with 3.0 can do ``from future_builtins import hex, map`` as necessary. -A new command-line switch, :option:`!-3`, enables warnings +A new command-line switch, :option:`-3`, enables warnings about features that will be removed in Python 3.0. You can run code with this switch to see how much work will be necessary to port code to 3.0. The value of this switch is available @@ -172,8 +170,8 @@ this edition of "What's New in Python" links to the bug/patch item for each change. Hosting of the Python bug tracker is kindly provided by -`Upfront Systems <http://www.upfrontsoftware.co.za>`__ -of Stellenbosch, South Africa. Martin von Löwis put a +`Upfront Systems <http://www.upfrontsystems.co.za/>`__ +of Stellenbosch, South Africa. Martin von Loewis put a lot of effort into importing existing bugs and patches from SourceForge; his scripts for this import operation are at http://svn.python.org/view/tracker/importer/ and may be useful to @@ -191,7 +189,7 @@ other projects wishing to move from SourceForge to Roundup. Roundup downloads and documentation. http://svn.python.org/view/tracker/importer/ - Martin von Löwis's conversion scripts. + Martin von Loewis's conversion scripts. New Documentation Format: reStructuredText Using Sphinx ----------------------------------------------------------- @@ -227,12 +225,12 @@ the Python community. Sphinx is a standalone package that can be used for writing, and almost two dozen other projects -(`listed on the Sphinx web site <https://www.sphinx-doc.org/en/master/examples.html>`__) +(`listed on the Sphinx web site <http://sphinx-doc.org/examples.html>`__) have adopted Sphinx as their documentation tool. .. seealso:: - `Documenting Python <https://devguide.python.org/documenting/>`__ + `Documenting Python <https://docs.python.org/devguide/documenting.html>`__ Describes how to write for Python's documentation. `Sphinx <http://sphinx-doc.org/>`__ @@ -242,18 +240,16 @@ have adopted Sphinx as their documentation tool. The underlying reStructuredText parser and toolset. -.. _pep-0343: - PEP 343: The 'with' statement ============================= The previous version, Python 2.5, added the ':keyword:`with`' statement as an optional feature, to be enabled by a ``from __future__ import with_statement`` directive. In 2.6 the statement no longer needs to -be specially enabled; this means that :keyword:`!with` is now always a +be specially enabled; this means that :keyword:`with` is now always a keyword. The rest of this section is a copy of the corresponding section from the "What's New in Python 2.5" document; if you're -familiar with the ':keyword:`!with`' statement +familiar with the ':keyword:`with`' statement from Python 2.5, you can skip this section. The ':keyword:`with`' statement clarifies code that previously would use @@ -331,7 +327,7 @@ Writing Context Managers ------------------------ Under the hood, the ':keyword:`with`' statement is fairly complicated. Most -people will only use ':keyword:`!with`' in company with existing objects and +people will only use ':keyword:`with`' in company with existing objects and don't need to know these details, so you can skip the rest of this section if you like. Authors of new objects will need to understand the details of the underlying implementation and should keep reading. @@ -425,7 +421,7 @@ add a :keyword:`return` statement at the marked location. :: # return False -.. _new-module-contextlib: +.. _module-contextlib: The contextlib module --------------------- @@ -438,9 +434,9 @@ generator function instead of defining a new class. The generator should yield exactly one value. The code up to the :keyword:`yield` will be executed as the :meth:`__enter__` method, and the value yielded will be the method's return value that will get bound to the variable in the ':keyword:`with`' statement's -:keyword:`!as` clause, if any. The code after the :keyword:`!yield` will be +:keyword:`as` clause, if any. The code after the :keyword:`yield` will be executed in the :meth:`__exit__` method. Any exception raised in the block will -be raised by the :keyword:`!yield` statement. +be raised by the :keyword:`yield` statement. Using this decorator, our database example from the previous section could be written as:: @@ -464,7 +460,7 @@ could be written as:: The :mod:`contextlib` module also has a ``nested(mgr1, mgr2, ...)`` function that combines a number of context managers so you don't need to write nested -':keyword:`with`' statements. In this example, the single ':keyword:`!with`' +':keyword:`with`' statements. In this example, the single ':keyword:`with`' statement both starts a database transaction and acquires a thread lock:: lock = threading.Lock() @@ -613,10 +609,10 @@ multiple of 4. result = queue.get() print 'Factorial', N, '=', result -A :class:`~queue.Queue` is used to communicate the result of the factorial. -The :class:`~queue.Queue` object is stored in a global variable. +A :class:`~Queue.Queue` is used to communicate the result of the factorial. +The :class:`~Queue.Queue` object is stored in a global variable. The child process will use the value of the variable when the child -was created; because it's a :class:`~queue.Queue`, parent and child can use +was created; because it's a :class:`~Queue.Queue`, parent and child can use the object to communicate. (If the parent were to change the value of the global variable, the child's value would be unaffected, and vice versa.) @@ -1081,7 +1077,7 @@ the :mod:`io` module: There are two concrete implementations. :class:`TextIOWrapper` wraps a buffered I/O object, supporting all of the methods for text I/O and adding a :attr:`buffer` attribute for access - to the underlying object. :class:`StringIO` simply buffers + to the underlying object. :class:`~StringIO.StringIO` simply buffers everything in memory without ever writing anything to disk. (In Python 2.6, :class:`io.StringIO` is implemented in @@ -1101,7 +1097,7 @@ their own implementations of buffering and text I/O. :pep:`3116` - New I/O PEP written by Daniel Stutzbach, Mike Verdone, and Guido van Rossum. Code by Guido van Rossum, Georg Brandl, Walter Doerwald, - Jeremy Hylton, Martin von Löwis, Tony Lownds, and others. + Jeremy Hylton, Martin von Loewis, Tony Lownds, and others. .. ====================================================================== @@ -1684,7 +1680,7 @@ Some smaller changes made to the core Python language are: * An obscure change: when you use the :func:`locals` function inside a :keyword:`class` statement, the resulting dictionary no longer returns free variables. (Free variables, in this case, are variables referenced in the - :keyword:`!class` statement that aren't attributes of the class.) + :keyword:`class` statement that aren't attributes of the class.) .. ====================================================================== @@ -1775,7 +1771,7 @@ The *encoding* part specifies the encoding's name, e.g. ``utf-8`` or ``latin-1``; the optional *errorhandler* part specifies what to do with characters that can't be handled by the encoding, and should be one of "error", "ignore", or "replace". (Contributed -by Martin von Löwis.) +by Martin von Loewis.) .. ====================================================================== @@ -1793,7 +1789,7 @@ changes, or look through the Subversion logs for all the details. were applied. (Maintained by Josiah Carlson; see :issue:`1736190` for one patch.) -* The :mod:`bsddb` module also has a new maintainer, Jesús Cea Avión, and the package +* The :mod:`bsddb` module also has a new maintainer, Jesús Cea Avion, and the package is now available as a standalone package. The web page for the package is `www.jcea.es/programacion/pybsddb.htm <https://www.jcea.es/programacion/pybsddb.htm>`__. @@ -2385,7 +2381,7 @@ changes, or look through the Subversion logs for all the details. (Contributed by Pedro Werneck and Jeffrey Yasskin; :issue:`742598`, :issue:`1193577`.) -* The :mod:`sqlite3` module, maintained by Gerhard Häring, +* The :mod:`sqlite3` module, maintained by Gerhard Haering, has been updated from version 2.3.2 in Python 2.5 to version 2.4.1. @@ -2598,7 +2594,7 @@ changes, or look through the Subversion logs for all the details. * The Unicode database provided by the :mod:`unicodedata` module has been updated to version 5.1.0. (Updated by - Martin von Löwis; :issue:`3811`.) + Martin von Loewis; :issue:`3811`.) * The :mod:`warnings` module's :func:`formatwarning` and :func:`showwarning` gained an optional *line* argument that can be used to supply the @@ -2609,7 +2605,7 @@ changes, or look through the Subversion logs for all the details. intended for testing purposes that lets you temporarily modify the warning filters and then restore their original values (:issue:`3781`). -* The XML-RPC :class:`SimpleXMLRPCServer` and :class:`DocXMLRPCServer` +* The XML-RPC :class:`~SimpleXMLRPCServer.SimpleXMLRPCServer` and :class:`~DocXMLRPCServer.DocXMLRPCServer` classes can now be prevented from immediately opening and binding to their socket by passing ``False`` as the *bind_and_activate* constructor parameter. This can be used to modify the instance's @@ -2618,7 +2614,7 @@ changes, or look through the Subversion logs for all the details. open the socket and begin listening for connections. (Contributed by Peter Parente; :issue:`1599845`.) - :class:`SimpleXMLRPCServer` also has a :attr:`_send_traceback_header` + :class:`~SimpleXMLRPCServer.SimpleXMLRPCServer` also has a :attr:`_send_traceback_header` attribute; if true, the exception and formatted traceback are returned as HTTP headers "X-Exception" and "X-Traceback". This feature is for debugging purposes only and should not be used on production servers @@ -3105,7 +3101,7 @@ Port-Specific Changes: Windows :file:`PCbuild` directory supports cross compilation for X64, debug builds and Profile Guided Optimization (PGO). PGO builds are roughly 10% faster than normal builds. (Contributed by Christian Heimes - with help from Amaury Forgeot d'Arc and Martin von Löwis.) + with help from Amaury Forgeot d'Arc and Martin von Loewis.) * The :mod:`msvcrt` module now supports both the normal and wide char variants of the console I/O diff --git a/Doc/whatsnew/2.7.rst b/Doc/whatsnew/2.7.rst index d19c8e0..992658e 100644 --- a/Doc/whatsnew/2.7.rst +++ b/Doc/whatsnew/2.7.rst @@ -198,8 +198,8 @@ Other new Python3-mode warnings include: * :func:`operator.isCallable` and :func:`operator.sequenceIncludes`, which are not supported in 3.x, now trigger warnings. -* The :option:`!-3` switch now automatically - enables the :option:`!-Qwarn` switch that causes warnings +* The :option:`-3` switch now automatically + enables the :option:`-Qwarn <-Q>` switch that causes warnings about using classic division with integers and long integers. @@ -437,7 +437,7 @@ standard input or output. :mod:`argparse` documentation The documentation page of the argparse module. - :ref:`upgrading-optparse-code` + :ref:`argparse-from-optparse` Part of the Python documentation, describing how to convert code that uses :mod:`optparse`. @@ -613,9 +613,6 @@ PEP 3137: The memoryview Object The :class:`memoryview` object provides a view of another object's memory content that matches the :class:`bytes` type's interface. -.. doctest:: - :options: +SKIP - >>> import string >>> m = memoryview(string.letters) >>> m @@ -631,9 +628,6 @@ memory content that matches the :class:`bytes` type's interface. The content of the view can be converted to a string of bytes or a list of integers: -.. doctest:: - :options: +SKIP - >>> m2.tobytes() 'abcdefghijklmnopqrstuvwxyz' >>> m2.tolist() @@ -643,9 +637,6 @@ a list of integers: :class:`memoryview` objects allow modifying the underlying object if it's a mutable object. -.. doctest:: - :options: +SKIP - >>> m2[0] = 75 Traceback (most recent call last): File "<stdin>", line 1, in <module> @@ -680,9 +671,6 @@ Some smaller changes made to the core Python language are: ``{}`` continues to represent an empty dictionary; use ``set()`` for an empty set. - .. doctest:: - :options: +SKIP - >>> {1, 2, 3, 4, 5} set([1, 2, 3, 4, 5]) >>> set() # empty set @@ -696,9 +684,6 @@ Some smaller changes made to the core Python language are: 3.x, generalizing list/generator comprehensions to use the literal syntax for sets and dictionaries. - .. doctest:: - :options: +SKIP - >>> {x: x*x for x in range(6)} {0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25} >>> {('a'*x) for x in range(6)} @@ -708,7 +693,7 @@ Some smaller changes made to the core Python language are: * The :keyword:`with` statement can now use multiple context managers in one statement. Context managers are processed from left to right - and each one is treated as beginning a new :keyword:`!with` statement. + and each one is treated as beginning a new :keyword:`with` statement. This means that:: with A() as a, B() as b: @@ -844,7 +829,7 @@ Some smaller changes made to the core Python language are: * The :keyword:`import` statement will no longer try an absolute import if a relative import (e.g. ``from .os import sep``) fails. This - fixes a bug, but could possibly break certain :keyword:`!import` + fixes a bug, but could possibly break certain :keyword:`import` statements that were only working by accident. (Fixed by Meador Inge; :issue:`7902`.) @@ -1067,7 +1052,7 @@ changes, or look through the Subversion logs for all the details. >>> for letter in 'here is a sample of english text': ... c[letter] += 1 ... - >>> c # doctest: +SKIP + >>> c Counter({' ': 6, 'e': 5, 's': 3, 'a': 2, 'i': 2, 'h': 2, 'l': 2, 't': 2, 'g': 1, 'f': 1, 'm': 1, 'o': 1, 'n': 1, 'p': 1, 'r': 1, 'x': 1}) @@ -1158,7 +1143,7 @@ changes, or look through the Subversion logs for all the details. * Deprecated function: :func:`contextlib.nested`, which allows handling more than one context manager with a single :keyword:`with` - statement, has been deprecated, because the :keyword:`!with` statement + statement, has been deprecated, because the :keyword:`with` statement now supports multiple context managers. * The :mod:`cookielib` module now ignores cookies that have an invalid @@ -1169,7 +1154,7 @@ changes, or look through the Subversion logs for all the details. correctly copy bound instance methods. (Implemented by Robert Collins; :issue:`1515`.) -* The :mod:`ctypes` module now always converts ``None`` to a C ``NULL`` +* The :mod:`ctypes` module now always converts ``None`` to a C NULL pointer for arguments declared as pointers. (Changed by Thomas Heller; :issue:`4606`.) The underlying `libffi library <https://sourceware.org/libffi/>`__ has been updated to version @@ -1653,18 +1638,12 @@ changes, or look through the Subversion logs for all the details. worked around the old behaviour. For example, Python 2.6.4 or 2.5 will return the following: - .. doctest:: - :options: +SKIP - >>> import urlparse >>> urlparse.urlsplit('invented://host/filename?query') ('invented', '', '//host/filename?query', '', '') Python 2.7 (and Python 2.6.5) will return: - .. doctest:: - :options: +SKIP - >>> import urlparse >>> urlparse.urlsplit('invented://host/filename?query') ('invented', 'host', '/filename?query', '', '') @@ -1673,10 +1652,7 @@ changes, or look through the Subversion logs for all the details. returns a named tuple instead of a standard tuple.) The :mod:`urlparse` module also supports IPv6 literal addresses as defined by - :rfc:`2732` (contributed by Senthil Kumaran; :issue:`2987`). - - .. doctest:: - :options: +SKIP + :rfc:`2732` (contributed by Senthil Kumaran; :issue:`2987`). :: >>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]', @@ -2513,18 +2489,12 @@ In the standard library: worked around the old behaviour. For example, Python 2.6.4 or 2.5 will return the following: - .. doctest:: - :options: +SKIP - >>> import urlparse >>> urlparse.urlsplit('invented://host/filename?query') ('invented', '', '//host/filename?query', '', '') Python 2.7 (and Python 2.6.5) will return: - .. doctest:: - :options: +SKIP - >>> import urlparse >>> urlparse.urlsplit('invented://host/filename?query') ('invented', 'host', '/filename?query', '', '') diff --git a/Doc/whatsnew/3.0.rst b/Doc/whatsnew/3.0.rst deleted file mode 100644 index 880958d..0000000 --- a/Doc/whatsnew/3.0.rst +++ /dev/null @@ -1,929 +0,0 @@ -**************************** - What's New In Python 3.0 -**************************** - -.. XXX Add trademark info for Apple, Microsoft. - -:Author: Guido van Rossum - -.. $Id$ - Rules for maintenance: - - * Anyone can add text to this document. Do not spend very much time - on the wording of your changes, because your text will probably - get rewritten to some degree. - - * The maintainer will go through Misc/NEWS periodically and add - changes; it's therefore more important to add your changes to - Misc/NEWS than to this file. (Note: I didn't get to this for 3.0. - GvR.) - - * This is not a complete list of every single change; completeness - is the purpose of Misc/NEWS. Some changes I consider too small - or esoteric to include. If such a change is added to the text, - I'll just remove it. (This is another reason you shouldn't spend - too much time on writing your addition.) - - * If you want to draw your new text to the attention of the - maintainer, add 'XXX' to the beginning of the paragraph or - section. - - * It's OK to just add a fragmentary note about a change. For - example: "XXX Describe the transmogrify() function added to the - socket module." The maintainer will research the change and - write the necessary text. - - * You can comment out your additions if you like, but it's not - necessary (especially when a final release is some months away). - - * Credit the author of a patch or bugfix. Just the name is - sufficient; the e-mail address isn't necessary. (Due to time - constraints I haven't managed to do this for 3.0. GvR.) - - * It's helpful to add the bug/patch number as a comment: - - % Patch 12345 - XXX Describe the transmogrify() function added to the socket - module. - (Contributed by P.Y. Developer.) - - This saves the maintainer the effort of going through the SVN log - when researching a change. (Again, I didn't get to this for 3.0. - GvR.) - -This article explains the new features in Python 3.0, compared to 2.6. -Python 3.0, also known as "Python 3000" or "Py3K", is the first ever -*intentionally backwards incompatible* Python release. There are more -changes than in a typical release, and more that are important for all -Python users. Nevertheless, after digesting the changes, you'll find -that Python really hasn't changed all that much -- by and large, we're -mostly fixing well-known annoyances and warts, and removing a lot of -old cruft. - -This article doesn't attempt to provide a complete specification of -all new features, but instead tries to give a convenient overview. -For full details, you should refer to the documentation for Python -3.0, and/or the many PEPs referenced in the text. If you want to -understand the complete implementation and design rationale for a -particular feature, PEPs usually have more details than the regular -documentation; but note that PEPs usually are not kept up-to-date once -a feature has been fully implemented. - -Due to time constraints this document is not as complete as it should -have been. As always for a new release, the ``Misc/NEWS`` file in the -source distribution contains a wealth of detailed information about -every small thing that was changed. - -.. Compare with previous release in 2 - 3 sentences here. -.. add hyperlink when the documentation becomes available online. - -.. ====================================================================== -.. Large, PEP-level features and changes should be described here. -.. Should there be a new section here for 3k migration? -.. Or perhaps a more general section describing module changes/deprecation? -.. sets module deprecated -.. ====================================================================== - - -Common Stumbling Blocks -======================= - -This section lists those few changes that are most likely to trip you -up if you're used to Python 2.5. - -Print Is A Function -------------------- - -The ``print`` statement has been replaced with a :func:`print` -function, with keyword arguments to replace most of the special syntax -of the old ``print`` statement (:pep:`3105`). Examples:: - - Old: print "The answer is", 2*2 - New: print("The answer is", 2*2) - - Old: print x, # Trailing comma suppresses newline - New: print(x, end=" ") # Appends a space instead of a newline - - Old: print # Prints a newline - New: print() # You must call the function! - - Old: print >>sys.stderr, "fatal error" - New: print("fatal error", file=sys.stderr) - - Old: print (x, y) # prints repr((x, y)) - New: print((x, y)) # Not the same as print(x, y)! - -You can also customize the separator between items, e.g.:: - - print("There are <", 2**32, "> possibilities!", sep="") - -which produces: - -.. code-block:: none - - There are <4294967296> possibilities! - -Note: - -* The :func:`print` function doesn't support the "softspace" feature of - the old ``print`` statement. For example, in Python 2.x, - ``print "A\n", "B"`` would write ``"A\nB\n"``; but in Python 3.0, - ``print("A\n", "B")`` writes ``"A\n B\n"``. - -* Initially, you'll be finding yourself typing the old ``print x`` - a lot in interactive mode. Time to retrain your fingers to type - ``print(x)`` instead! - -* When using the ``2to3`` source-to-source conversion tool, all - ``print`` statements are automatically converted to - :func:`print` function calls, so this is mostly a non-issue for - larger projects. - -Views And Iterators Instead Of Lists -------------------------------------- - -Some well-known APIs no longer return lists: - -* :class:`dict` methods :meth:`dict.keys`, :meth:`dict.items` and - :meth:`dict.values` return "views" instead of lists. For example, - this no longer works: ``k = d.keys(); k.sort()``. Use ``k = - sorted(d)`` instead (this works in Python 2.5 too and is just - as efficient). - -* Also, the :meth:`dict.iterkeys`, :meth:`dict.iteritems` and - :meth:`dict.itervalues` methods are no longer supported. - -* :func:`map` and :func:`filter` return iterators. If you really need - a list and the input sequences are all of equal length, a quick - fix is to wrap :func:`map` in :func:`list`, e.g. ``list(map(...))``, - but a better fix is - often to use a list comprehension (especially when the original code - uses :keyword:`lambda`), or rewriting the code so it doesn't need a - list at all. Particularly tricky is :func:`map` invoked for the - side effects of the function; the correct transformation is to use a - regular :keyword:`for` loop (since creating a list would just be - wasteful). - - If the input sequences are not of equal length, :func:`map` will - stop at the termination of the shortest of the sequences. For full - compatibility with :func:`map` from Python 2.x, also wrap the sequences in - :func:`itertools.zip_longest`, e.g. ``map(func, *sequences)`` becomes - ``list(map(func, itertools.zip_longest(*sequences)))``. - -* :func:`range` now behaves like :func:`xrange` used to behave, except - it works with values of arbitrary size. The latter no longer - exists. - -* :func:`zip` now returns an iterator. - -Ordering Comparisons --------------------- - -Python 3.0 has simplified the rules for ordering comparisons: - -* The ordering comparison operators (``<``, ``<=``, ``>=``, ``>``) - raise a TypeError exception when the operands don't have a - meaningful natural ordering. Thus, expressions like ``1 < ''``, ``0 - > None`` or ``len <= len`` are no longer valid, and e.g. ``None < - None`` raises :exc:`TypeError` instead of returning - ``False``. A corollary is that sorting a heterogeneous list - no longer makes sense -- all the elements must be comparable to each - other. Note that this does not apply to the ``==`` and ``!=`` - operators: objects of different incomparable types always compare - unequal to each other. - -* :meth:`builtin.sorted` and :meth:`list.sort` no longer accept the - *cmp* argument providing a comparison function. Use the *key* - argument instead. N.B. the *key* and *reverse* arguments are now - "keyword-only". - -* The :func:`cmp` function should be treated as gone, and the :meth:`__cmp__` - special method is no longer supported. Use :meth:`__lt__` for sorting, - :meth:`__eq__` with :meth:`__hash__`, and other rich comparisons as needed. - (If you really need the :func:`cmp` functionality, you could use the - expression ``(a > b) - (a < b)`` as the equivalent for ``cmp(a, b)``.) - -Integers --------- - -* :pep:`237`: Essentially, :class:`long` renamed to :class:`int`. - That is, there is only one built-in integral type, named - :class:`int`; but it behaves mostly like the old :class:`long` type. - -* :pep:`238`: An expression like ``1/2`` returns a float. Use - ``1//2`` to get the truncating behavior. (The latter syntax has - existed for years, at least since Python 2.2.) - -* The :data:`sys.maxint` constant was removed, since there is no - longer a limit to the value of integers. However, :data:`sys.maxsize` - can be used as an integer larger than any practical list or string - index. It conforms to the implementation's "natural" integer size - and is typically the same as :data:`sys.maxint` in previous releases - on the same platform (assuming the same build options). - -* The :func:`repr` of a long integer doesn't include the trailing ``L`` - anymore, so code that unconditionally strips that character will - chop off the last digit instead. (Use :func:`str` instead.) - -* Octal literals are no longer of the form ``0720``; use ``0o720`` - instead. - -Text Vs. Data Instead Of Unicode Vs. 8-bit ------------------------------------------- - -Everything you thought you knew about binary data and Unicode has -changed. - -* Python 3.0 uses the concepts of *text* and (binary) *data* instead - of Unicode strings and 8-bit strings. All text is Unicode; however - *encoded* Unicode is represented as binary data. The type used to - hold text is :class:`str`, the type used to hold data is - :class:`bytes`. The biggest difference with the 2.x situation is - that any attempt to mix text and data in Python 3.0 raises - :exc:`TypeError`, whereas if you were to mix Unicode and 8-bit - strings in Python 2.x, it would work if the 8-bit string happened to - contain only 7-bit (ASCII) bytes, but you would get - :exc:`UnicodeDecodeError` if it contained non-ASCII values. This - value-specific behavior has caused numerous sad faces over the - years. - -* As a consequence of this change in philosophy, pretty much all code - that uses Unicode, encodings or binary data most likely has to - change. The change is for the better, as in the 2.x world there - were numerous bugs having to do with mixing encoded and unencoded - text. To be prepared in Python 2.x, start using :class:`unicode` - for all unencoded text, and :class:`str` for binary or encoded data - only. Then the ``2to3`` tool will do most of the work for you. - -* You can no longer use ``u"..."`` literals for Unicode text. - However, you must use ``b"..."`` literals for binary data. - -* As the :class:`str` and :class:`bytes` types cannot be mixed, you - must always explicitly convert between them. Use :meth:`str.encode` - to go from :class:`str` to :class:`bytes`, and :meth:`bytes.decode` - to go from :class:`bytes` to :class:`str`. You can also use - ``bytes(s, encoding=...)`` and ``str(b, encoding=...)``, - respectively. - -* Like :class:`str`, the :class:`bytes` type is immutable. There is a - separate *mutable* type to hold buffered binary data, - :class:`bytearray`. Nearly all APIs that accept :class:`bytes` also - accept :class:`bytearray`. The mutable API is based on - :class:`collections.MutableSequence`. - -* All backslashes in raw string literals are interpreted literally. - This means that ``'\U'`` and ``'\u'`` escapes in raw strings are not - treated specially. For example, ``r'\u20ac'`` is a string of 6 - characters in Python 3.0, whereas in 2.6, ``ur'\u20ac'`` was the - single "euro" character. (Of course, this change only affects raw - string literals; the euro character is ``'\u20ac'`` in Python 3.0.) - -* The built-in :class:`basestring` abstract type was removed. Use - :class:`str` instead. The :class:`str` and :class:`bytes` types - don't have functionality enough in common to warrant a shared base - class. The ``2to3`` tool (see below) replaces every occurrence of - :class:`basestring` with :class:`str`. - -* Files opened as text files (still the default mode for :func:`open`) - always use an encoding to map between strings (in memory) and bytes - (on disk). Binary files (opened with a ``b`` in the mode argument) - always use bytes in memory. This means that if a file is opened - using an incorrect mode or encoding, I/O will likely fail loudly, - instead of silently producing incorrect data. It also means that - even Unix users will have to specify the correct mode (text or - binary) when opening a file. There is a platform-dependent default - encoding, which on Unixy platforms can be set with the ``LANG`` - environment variable (and sometimes also with some other - platform-specific locale-related environment variables). In many - cases, but not all, the system default is UTF-8; you should never - count on this default. Any application reading or writing more than - pure ASCII text should probably have a way to override the encoding. - There is no longer any need for using the encoding-aware streams - in the :mod:`codecs` module. - -* The initial values of :data:`sys.stdin`, :data:`sys.stdout` and - :data:`sys.stderr` are now unicode-only text files (i.e., they are - instances of :class:`io.TextIOBase`). To read and write bytes data - with these streams, you need to use their :data:`io.TextIOBase.buffer` - attribute. - -* Filenames are passed to and returned from APIs as (Unicode) strings. - This can present platform-specific problems because on some - platforms filenames are arbitrary byte strings. (On the other hand, - on Windows filenames are natively stored as Unicode.) As a - work-around, most APIs (e.g. :func:`open` and many functions in the - :mod:`os` module) that take filenames accept :class:`bytes` objects - as well as strings, and a few APIs have a way to ask for a - :class:`bytes` return value. Thus, :func:`os.listdir` returns a - list of :class:`bytes` instances if the argument is a :class:`bytes` - instance, and :func:`os.getcwdb` returns the current working - directory as a :class:`bytes` instance. Note that when - :func:`os.listdir` returns a list of strings, filenames that - cannot be decoded properly are omitted rather than raising - :exc:`UnicodeError`. - -* Some system APIs like :data:`os.environ` and :data:`sys.argv` can - also present problems when the bytes made available by the system is - not interpretable using the default encoding. Setting the ``LANG`` - variable and rerunning the program is probably the best approach. - -* :pep:`3138`: The :func:`repr` of a string no longer escapes - non-ASCII characters. It still escapes control characters and code - points with non-printable status in the Unicode standard, however. - -* :pep:`3120`: The default source encoding is now UTF-8. - -* :pep:`3131`: Non-ASCII letters are now allowed in identifiers. - (However, the standard library remains ASCII-only with the exception - of contributor names in comments.) - -* The :mod:`StringIO` and :mod:`cStringIO` modules are gone. Instead, - import the :mod:`io` module and use :class:`io.StringIO` or - :class:`io.BytesIO` for text and data respectively. - -* See also the :ref:`unicode-howto`, which was updated for Python 3.0. - - -Overview Of Syntax Changes -========================== - -This section gives a brief overview of every *syntactic* change in -Python 3.0. - -New Syntax ----------- - -* :pep:`3107`: Function argument and return value annotations. This - provides a standardized way of annotating a function's parameters - and return value. There are no semantics attached to such - annotations except that they can be introspected at runtime using - the :attr:`__annotations__` attribute. The intent is to encourage - experimentation through metaclasses, decorators or frameworks. - -* :pep:`3102`: Keyword-only arguments. Named parameters occurring - after ``*args`` in the parameter list *must* be specified using - keyword syntax in the call. You can also use a bare ``*`` in the - parameter list to indicate that you don't accept a variable-length - argument list, but you do have keyword-only arguments. - -* Keyword arguments are allowed after the list of base classes in a - class definition. This is used by the new convention for specifying - a metaclass (see next section), but can be used for other purposes - as well, as long as the metaclass supports it. - -* :pep:`3104`: :keyword:`nonlocal` statement. Using ``nonlocal x`` - you can now assign directly to a variable in an outer (but - non-global) scope. :keyword:`!nonlocal` is a new reserved word. - -* :pep:`3132`: Extended Iterable Unpacking. You can now write things - like ``a, b, *rest = some_sequence``. And even ``*rest, a = - stuff``. The ``rest`` object is always a (possibly empty) list; the - right-hand side may be any iterable. Example:: - - (a, *rest, b) = range(5) - - This sets *a* to ``0``, *b* to ``4``, and *rest* to ``[1, 2, 3]``. - -* Dictionary comprehensions: ``{k: v for k, v in stuff}`` means the - same thing as ``dict(stuff)`` but is more flexible. (This is - :pep:`274` vindicated. :-) - -* Set literals, e.g. ``{1, 2}``. Note that ``{}`` is an empty - dictionary; use ``set()`` for an empty set. Set comprehensions are - also supported; e.g., ``{x for x in stuff}`` means the same thing as - ``set(stuff)`` but is more flexible. - -* New octal literals, e.g. ``0o720`` (already in 2.6). The old octal - literals (``0720``) are gone. - -* New binary literals, e.g. ``0b1010`` (already in 2.6), and - there is a new corresponding built-in function, :func:`bin`. - -* Bytes literals are introduced with a leading ``b`` or ``B``, and - there is a new corresponding built-in function, :func:`bytes`. - -Changed Syntax --------------- - -* :pep:`3109` and :pep:`3134`: new :keyword:`raise` statement syntax: - :samp:`raise [{expr} [from {expr}]]`. See below. - -* :keyword:`!as` and :keyword:`with` are now reserved words. (Since - 2.6, actually.) - -* ``True``, ``False``, and ``None`` are reserved words. (2.6 partially enforced - the restrictions on ``None`` already.) - -* Change from :keyword:`except` *exc*, *var* to - :keyword:`!except` *exc* :keyword:`!as` *var*. See :pep:`3110`. - -* :pep:`3115`: New Metaclass Syntax. Instead of:: - - class C: - __metaclass__ = M - ... - - you must now use:: - - class C(metaclass=M): - ... - - The module-global :data:`__metaclass__` variable is no longer - supported. (It was a crutch to make it easier to default to - new-style classes without deriving every class from - :class:`object`.) - -* List comprehensions no longer support the syntactic form - :samp:`[... for {var} in {item1}, {item2}, ...]`. Use - :samp:`[... for {var} in ({item1}, {item2}, ...)]` instead. - Also note that list comprehensions have different semantics: they - are closer to syntactic sugar for a generator expression inside a - :func:`list` constructor, and in particular the loop control - variables are no longer leaked into the surrounding scope. - -* The *ellipsis* (``...``) can be used as an atomic expression - anywhere. (Previously it was only allowed in slices.) Also, it - *must* now be spelled as ``...``. (Previously it could also be - spelled as ``. . .``, by a mere accident of the grammar.) - -Removed Syntax --------------- - -* :pep:`3113`: Tuple parameter unpacking removed. You can no longer - write ``def foo(a, (b, c)): ...``. - Use ``def foo(a, b_c): b, c = b_c`` instead. - -* Removed backticks (use :func:`repr` instead). - -* Removed ``<>`` (use ``!=`` instead). - -* Removed keyword: :func:`exec` is no longer a keyword; it remains as - a function. (Fortunately the function syntax was also accepted in - 2.x.) Also note that :func:`exec` no longer takes a stream argument; - instead of ``exec(f)`` you can use ``exec(f.read())``. - -* Integer literals no longer support a trailing ``l`` or ``L``. - -* String literals no longer support a leading ``u`` or ``U``. - -* The :keyword:`from` *module* :keyword:`import` ``*`` syntax is only - allowed at the module level, no longer inside functions. - -* The only acceptable syntax for relative imports is :samp:`from .[{module}] - import {name}`. All :keyword:`import` forms not starting with ``.`` are - interpreted as absolute imports. (:pep:`328`) - -* Classic classes are gone. - - -Changes Already Present In Python 2.6 -===================================== - -Since many users presumably make the jump straight from Python 2.5 to -Python 3.0, this section reminds the reader of new features that were -originally designed for Python 3.0 but that were back-ported to Python -2.6. The corresponding sections in :ref:`whats-new-in-2.6` should be -consulted for longer descriptions. - -* :ref:`pep-0343`. The :keyword:`with` statement is now a standard - feature and no longer needs to be imported from the :mod:`__future__`. - Also check out :ref:`new-26-context-managers` and - :ref:`new-module-contextlib`. - -* :ref:`pep-0366`. This enhances the usefulness of the :option:`-m` - option when the referenced module lives in a package. - -* :ref:`pep-0370`. - -* :ref:`pep-0371`. - -* :ref:`pep-3101`. Note: the 2.6 description mentions the - :meth:`format` method for both 8-bit and Unicode strings. In 3.0, - only the :class:`str` type (text strings with Unicode support) - supports this method; the :class:`bytes` type does not. The plan is - to eventually make this the only API for string formatting, and to - start deprecating the ``%`` operator in Python 3.1. - -* :ref:`pep-3105`. This is now a standard feature and no longer needs - to be imported from :mod:`__future__`. More details were given above. - -* :ref:`pep-3110`. The :keyword:`except` *exc* :keyword:`!as` *var* - syntax is now standard and :keyword:`!except` *exc*, *var* is no - longer supported. (Of course, the :keyword:`!as` *var* part is still - optional.) - -* :ref:`pep-3112`. The ``b"..."`` string literal notation (and its - variants like ``b'...'``, ``b"""..."""``, and ``br"..."``) now - produces a literal of type :class:`bytes`. - -* :ref:`pep-3116`. The :mod:`io` module is now the standard way of - doing file I/O. The built-in :func:`open` function is now an - alias for :func:`io.open` and has additional keyword arguments - *encoding*, *errors*, *newline* and *closefd*. Also note that an - invalid *mode* argument now raises :exc:`ValueError`, not - :exc:`IOError`. The binary file object underlying a text file - object can be accessed as :attr:`f.buffer` (but beware that the - text object maintains a buffer of itself in order to speed up - the encoding and decoding operations). - -* :ref:`pep-3118`. The old builtin :func:`buffer` is now really gone; - the new builtin :func:`memoryview` provides (mostly) similar - functionality. - -* :ref:`pep-3119`. The :mod:`abc` module and the ABCs defined in the - :mod:`collections` module plays a somewhat more prominent role in - the language now, and built-in collection types like :class:`dict` - and :class:`list` conform to the :class:`collections.MutableMapping` - and :class:`collections.MutableSequence` ABCs, respectively. - -* :ref:`pep-3127`. As mentioned above, the new octal literal - notation is the only one supported, and binary literals have been - added. - -* :ref:`pep-3129`. - -* :ref:`pep-3141`. The :mod:`numbers` module is another new use of - ABCs, defining Python's "numeric tower". Also note the new - :mod:`fractions` module which implements :class:`numbers.Rational`. - - -Library Changes -=============== - -Due to time constraints, this document does not exhaustively cover the -very extensive changes to the standard library. :pep:`3108` is the -reference for the major changes to the library. Here's a capsule -review: - -* Many old modules were removed. Some, like :mod:`gopherlib` (no - longer used) and :mod:`md5` (replaced by :mod:`hashlib`), were - already deprecated by :pep:`4`. Others were removed as a result - of the removal of support for various platforms such as Irix, BeOS - and Mac OS 9 (see :pep:`11`). Some modules were also selected for - removal in Python 3.0 due to lack of use or because a better - replacement exists. See :pep:`3108` for an exhaustive list. - -* The :mod:`bsddb3` package was removed because its presence in the - core standard library has proved over time to be a particular burden - for the core developers due to testing instability and Berkeley DB's - release schedule. However, the package is alive and well, - externally maintained at https://www.jcea.es/programacion/pybsddb.htm. - -* Some modules were renamed because their old name disobeyed - :pep:`8`, or for various other reasons. Here's the list: - - ======================= ======================= - Old Name New Name - ======================= ======================= - _winreg winreg - ConfigParser configparser - copy_reg copyreg - Queue queue - SocketServer socketserver - markupbase _markupbase - repr reprlib - test.test_support test.support - ======================= ======================= - -* A common pattern in Python 2.x is to have one version of a module - implemented in pure Python, with an optional accelerated version - implemented as a C extension; for example, :mod:`pickle` and - :mod:`cPickle`. This places the burden of importing the accelerated - version and falling back on the pure Python version on each user of - these modules. In Python 3.0, the accelerated versions are - considered implementation details of the pure Python versions. - Users should always import the standard version, which attempts to - import the accelerated version and falls back to the pure Python - version. The :mod:`pickle` / :mod:`cPickle` pair received this - treatment. The :mod:`profile` module is on the list for 3.1. The - :mod:`StringIO` module has been turned into a class in the :mod:`io` - module. - -* Some related modules have been grouped into packages, and usually - the submodule names have been simplified. The resulting new - packages are: - - * :mod:`dbm` (:mod:`anydbm`, :mod:`dbhash`, :mod:`dbm`, - :mod:`dumbdbm`, :mod:`gdbm`, :mod:`whichdb`). - - * :mod:`html` (:mod:`HTMLParser`, :mod:`htmlentitydefs`). - - * :mod:`http` (:mod:`httplib`, :mod:`BaseHTTPServer`, - :mod:`CGIHTTPServer`, :mod:`SimpleHTTPServer`, :mod:`Cookie`, - :mod:`cookielib`). - - * :mod:`tkinter` (all :mod:`Tkinter`-related modules except - :mod:`turtle`). The target audience of :mod:`turtle` doesn't - really care about :mod:`tkinter`. Also note that as of Python - 2.6, the functionality of :mod:`turtle` has been greatly enhanced. - - * :mod:`urllib` (:mod:`urllib`, :mod:`urllib2`, :mod:`urlparse`, - :mod:`robotparse`). - - * :mod:`xmlrpc` (:mod:`xmlrpclib`, :mod:`DocXMLRPCServer`, - :mod:`SimpleXMLRPCServer`). - -Some other changes to standard library modules, not covered by -:pep:`3108`: - -* Killed :mod:`sets`. Use the built-in :func:`set` class. - -* Cleanup of the :mod:`sys` module: removed :func:`sys.exitfunc`, - :func:`sys.exc_clear`, :data:`sys.exc_type`, :data:`sys.exc_value`, - :data:`sys.exc_traceback`. (Note that :data:`sys.last_type` - etc. remain.) - -* Cleanup of the :class:`array.array` type: the :meth:`read` and - :meth:`write` methods are gone; use :meth:`fromfile` and - :meth:`tofile` instead. Also, the ``'c'`` typecode for array is - gone -- use either ``'b'`` for bytes or ``'u'`` for Unicode - characters. - -* Cleanup of the :mod:`operator` module: removed - :func:`sequenceIncludes` and :func:`isCallable`. - -* Cleanup of the :mod:`thread` module: :func:`acquire_lock` and - :func:`release_lock` are gone; use :func:`acquire` and - :func:`release` instead. - -* Cleanup of the :mod:`random` module: removed the :func:`jumpahead` API. - -* The :mod:`new` module is gone. - -* The functions :func:`os.tmpnam`, :func:`os.tempnam` and - :func:`os.tmpfile` have been removed in favor of the :mod:`tempfile` - module. - -* The :mod:`tokenize` module has been changed to work with bytes. The - main entry point is now :func:`tokenize.tokenize`, instead of - generate_tokens. - -* :data:`string.letters` and its friends (:data:`string.lowercase` and - :data:`string.uppercase`) are gone. Use - :data:`string.ascii_letters` etc. instead. (The reason for the - removal is that :data:`string.letters` and friends had - locale-specific behavior, which is a bad idea for such - attractively-named global "constants".) - -* Renamed module :mod:`__builtin__` to :mod:`builtins` (removing the - underscores, adding an 's'). The :data:`__builtins__` variable - found in most global namespaces is unchanged. To modify a builtin, - you should use :mod:`builtins`, not :data:`__builtins__`! - - -:pep:`3101`: A New Approach To String Formatting -================================================ - -* A new system for built-in string formatting operations replaces the - ``%`` string formatting operator. (However, the ``%`` operator is - still supported; it will be deprecated in Python 3.1 and removed - from the language at some later time.) Read :pep:`3101` for the full - scoop. - - -Changes To Exceptions -===================== - -The APIs for raising and catching exception have been cleaned up and -new powerful features added: - -* :pep:`352`: All exceptions must be derived (directly or indirectly) - from :exc:`BaseException`. This is the root of the exception - hierarchy. This is not new as a recommendation, but the - *requirement* to inherit from :exc:`BaseException` is new. (Python - 2.6 still allowed classic classes to be raised, and placed no - restriction on what you can catch.) As a consequence, string - exceptions are finally truly and utterly dead. - -* Almost all exceptions should actually derive from :exc:`Exception`; - :exc:`BaseException` should only be used as a base class for - exceptions that should only be handled at the top level, such as - :exc:`SystemExit` or :exc:`KeyboardInterrupt`. The recommended - idiom for handling all exceptions except for this latter category is - to use :keyword:`except` :exc:`Exception`. - -* :exc:`StandardError` was removed. - -* Exceptions no longer behave as sequences. Use the :attr:`args` - attribute instead. - -* :pep:`3109`: Raising exceptions. You must now use :samp:`raise - {Exception}({args})` instead of :samp:`raise {Exception}, {args}`. - Additionally, you can no longer explicitly specify a traceback; - instead, if you *have* to do this, you can assign directly to the - :attr:`__traceback__` attribute (see below). - -* :pep:`3110`: Catching exceptions. You must now use - :samp:`except {SomeException} as {variable}` instead - of :samp:`except {SomeException}, {variable}`. Moreover, the - *variable* is explicitly deleted when the :keyword:`except` block - is left. - -* :pep:`3134`: Exception chaining. There are two cases: implicit - chaining and explicit chaining. Implicit chaining happens when an - exception is raised in an :keyword:`except` or :keyword:`finally` - handler block. This usually happens due to a bug in the handler - block; we call this a *secondary* exception. In this case, the - original exception (that was being handled) is saved as the - :attr:`__context__` attribute of the secondary exception. - Explicit chaining is invoked with this syntax:: - - raise SecondaryException() from primary_exception - - (where *primary_exception* is any expression that produces an - exception object, probably an exception that was previously caught). - In this case, the primary exception is stored on the - :attr:`__cause__` attribute of the secondary exception. The - traceback printed when an unhandled exception occurs walks the chain - of :attr:`__cause__` and :attr:`__context__` attributes and prints a - separate traceback for each component of the chain, with the primary - exception at the top. (Java users may recognize this behavior.) - -* :pep:`3134`: Exception objects now store their traceback as the - :attr:`__traceback__` attribute. This means that an exception - object now contains all the information pertaining to an exception, - and there are fewer reasons to use :func:`sys.exc_info` (though the - latter is not removed). - -* A few exception messages are improved when Windows fails to load an - extension module. For example, ``error code 193`` is now ``%1 is - not a valid Win32 application``. Strings now deal with non-English - locales. - - -Miscellaneous Other Changes -=========================== - -Operators And Special Methods ------------------------------ - -* ``!=`` now returns the opposite of ``==``, unless ``==`` returns - :data:`NotImplemented`. - -* The concept of "unbound methods" has been removed from the language. - When referencing a method as a class attribute, you now get a plain - function object. - -* :meth:`__getslice__`, :meth:`__setslice__` and :meth:`__delslice__` - were killed. The syntax ``a[i:j]`` now translates to - ``a.__getitem__(slice(i, j))`` (or :meth:`__setitem__` or - :meth:`__delitem__`, when used as an assignment or deletion target, - respectively). - -* :pep:`3114`: the standard :meth:`next` method has been renamed to - :meth:`~iterator.__next__`. - -* The :meth:`__oct__` and :meth:`__hex__` special methods are removed - -- :func:`oct` and :func:`hex` use :meth:`__index__` now to convert - the argument to an integer. - -* Removed support for :attr:`__members__` and :attr:`__methods__`. - -* The function attributes named :attr:`func_X` have been renamed to - use the :data:`__X__` form, freeing up these names in the function - attribute namespace for user-defined attributes. To wit, - :attr:`func_closure`, :attr:`func_code`, :attr:`func_defaults`, - :attr:`func_dict`, :attr:`func_doc`, :attr:`func_globals`, - :attr:`func_name` were renamed to :attr:`__closure__`, - :attr:`__code__`, :attr:`__defaults__`, :attr:`~object.__dict__`, - :attr:`__doc__`, :attr:`__globals__`, :attr:`~definition.__name__`, - respectively. - -* :meth:`__nonzero__` is now :meth:`__bool__`. - -Builtins --------- - -* :pep:`3135`: New :func:`super`. You can now invoke :func:`super` - without arguments and (assuming this is in a regular instance method - defined inside a :keyword:`class` statement) the right class and - instance will automatically be chosen. With arguments, the behavior - of :func:`super` is unchanged. - -* :pep:`3111`: :func:`raw_input` was renamed to :func:`input`. That - is, the new :func:`input` function reads a line from - :data:`sys.stdin` and returns it with the trailing newline stripped. - It raises :exc:`EOFError` if the input is terminated prematurely. - To get the old behavior of :func:`input`, use ``eval(input())``. - -* A new built-in function :func:`next` was added to call the - :meth:`~iterator.__next__` method on an object. - -* The :func:`round` function rounding strategy and return type have - changed. Exact halfway cases are now rounded to the nearest even - result instead of away from zero. (For example, ``round(2.5)`` now - returns ``2`` rather than ``3``.) ``round(x[, n])`` now - delegates to ``x.__round__([n])`` instead of always returning a - float. It generally returns an integer when called with a single - argument and a value of the same type as ``x`` when called with two - arguments. - -* Moved :func:`intern` to :func:`sys.intern`. - -* Removed: :func:`apply`. Instead of ``apply(f, args)`` use - ``f(*args)``. - -* Removed :func:`callable`. Instead of ``callable(f)`` you can use - ``isinstance(f, collections.Callable)``. The :func:`operator.isCallable` - function is also gone. - -* Removed :func:`coerce`. This function no longer serves a purpose - now that classic classes are gone. - -* Removed :func:`execfile`. Instead of ``execfile(fn)`` use - ``exec(open(fn).read())``. - -* Removed the :class:`file` type. Use :func:`open`. There are now several - different kinds of streams that open can return in the :mod:`io` module. - -* Removed :func:`reduce`. Use :func:`functools.reduce` if you really - need it; however, 99 percent of the time an explicit :keyword:`for` - loop is more readable. - -* Removed :func:`reload`. Use :func:`imp.reload`. - -* Removed. :meth:`dict.has_key` -- use the :keyword:`in` operator - instead. - -.. ====================================================================== - - -Build and C API Changes -======================= - -Due to time constraints, here is a *very* incomplete list of changes -to the C API. - -* Support for several platforms was dropped, including but not limited - to Mac OS 9, BeOS, RISCOS, Irix, and Tru64. - -* :pep:`3118`: New Buffer API. - -* :pep:`3121`: Extension Module Initialization & Finalization. - -* :pep:`3123`: Making :c:macro:`PyObject_HEAD` conform to standard C. - -* No more C API support for restricted execution. - -* :c:func:`PyNumber_Coerce`, :c:func:`PyNumber_CoerceEx`, - :c:func:`PyMember_Get`, and :c:func:`PyMember_Set` C APIs are removed. - -* New C API :c:func:`PyImport_ImportModuleNoBlock`, works like - :c:func:`PyImport_ImportModule` but won't block on the import lock - (returning an error instead). - -* Renamed the boolean conversion C-level slot and method: - ``nb_nonzero`` is now ``nb_bool``. - -* Removed :c:macro:`METH_OLDARGS` and :c:macro:`WITH_CYCLE_GC` from the C API. - -.. ====================================================================== - - -Performance -=========== - -The net result of the 3.0 generalizations is that Python 3.0 runs the -pystone benchmark around 10% slower than Python 2.5. Most likely the -biggest cause is the removal of special-casing for small integers. -There's room for improvement, but it will happen after 3.0 is -released! - -.. ====================================================================== - - -Porting To Python 3.0 -===================== - -For porting existing Python 2.5 or 2.6 source code to Python 3.0, the -best strategy is the following: - -0. (Prerequisite:) Start with excellent test coverage. - -1. Port to Python 2.6. This should be no more work than the average - port from Python 2.x to Python 2.(x+1). Make sure all your tests - pass. - -2. (Still using 2.6:) Turn on the :option:`!-3` command line switch. - This enables warnings about features that will be removed (or - change) in 3.0. Run your test suite again, and fix code that you - get warnings about until there are no warnings left, and all your - tests still pass. - -3. Run the ``2to3`` source-to-source translator over your source code - tree. (See :ref:`2to3-reference` for more on this tool.) Run the - result of the translation under Python 3.0. Manually fix up any - remaining issues, fixing problems until all tests pass again. - -It is not recommended to try to write source code that runs unchanged -under both Python 2.6 and 3.0; you'd have to use a very contorted -coding style, e.g. avoiding ``print`` statements, metaclasses, -and much more. If you are maintaining a library that needs to support -both Python 2.6 and Python 3.0, the best approach is to modify step 3 -above by editing the 2.6 version of the source code and running the -``2to3`` translator again, rather than editing the 3.0 version of the -source code. - -For porting C extensions to Python 3.0, please see :ref:`cporting-howto`. - -.. ====================================================================== diff --git a/Doc/whatsnew/3.1.rst b/Doc/whatsnew/3.1.rst deleted file mode 100644 index 919fbee..0000000 --- a/Doc/whatsnew/3.1.rst +++ /dev/null @@ -1,552 +0,0 @@ -**************************** - What's New In Python 3.1 -**************************** - -:Author: Raymond Hettinger - -.. $Id$ - Rules for maintenance: - - * Anyone can add text to this document. Do not spend very much time - on the wording of your changes, because your text will probably - get rewritten to some degree. - - * The maintainer will go through Misc/NEWS periodically and add - changes; it's therefore more important to add your changes to - Misc/NEWS than to this file. - - * This is not a complete list of every single change; completeness - is the purpose of Misc/NEWS. Some changes I consider too small - or esoteric to include. If such a change is added to the text, - I'll just remove it. (This is another reason you shouldn't spend - too much time on writing your addition.) - - * If you want to draw your new text to the attention of the - maintainer, add 'XXX' to the beginning of the paragraph or - section. - - * It's OK to just add a fragmentary note about a change. For - example: "XXX Describe the transmogrify() function added to the - socket module." The maintainer will research the change and - write the necessary text. - - * You can comment out your additions if you like, but it's not - necessary (especially when a final release is some months away). - - * Credit the author of a patch or bugfix. Just the name is - sufficient; the e-mail address isn't necessary. - - * It's helpful to add the bug/patch number as a comment: - - % Patch 12345 - XXX Describe the transmogrify() function added to the socket - module. - (Contributed by P.Y. Developer.) - - This saves the maintainer the effort of going through the SVN log - when researching a change. - -This article explains the new features in Python 3.1, compared to 3.0. - - -PEP 372: Ordered Dictionaries -============================= - -Regular Python dictionaries iterate over key/value pairs in arbitrary order. -Over the years, a number of authors have written alternative implementations -that remember the order that the keys were originally inserted. Based on -the experiences from those implementations, a new -:class:`collections.OrderedDict` class has been introduced. - -The OrderedDict API is substantially the same as regular dictionaries -but will iterate over keys and values in a guaranteed order depending on -when a key was first inserted. If a new entry overwrites an existing entry, -the original insertion position is left unchanged. Deleting an entry and -reinserting it will move it to the end. - -The standard library now supports use of ordered dictionaries in several -modules. The :mod:`configparser` module uses them by default. This lets -configuration files be read, modified, and then written back in their original -order. The *_asdict()* method for :func:`collections.namedtuple` now -returns an ordered dictionary with the values appearing in the same order as -the underlying tuple indices. The :mod:`json` module is being built-out with -an *object_pairs_hook* to allow OrderedDicts to be built by the decoder. -Support was also added for third-party tools like `PyYAML <http://pyyaml.org/>`_. - -.. seealso:: - - :pep:`372` - Ordered Dictionaries - PEP written by Armin Ronacher and Raymond Hettinger. Implementation - written by Raymond Hettinger. - - -PEP 378: Format Specifier for Thousands Separator -================================================= - -The built-in :func:`format` function and the :meth:`str.format` method use -a mini-language that now includes a simple, non-locale aware way to format -a number with a thousands separator. That provides a way to humanize a -program's output, improving its professional appearance and readability:: - - >>> format(1234567, ',d') - '1,234,567' - >>> format(1234567.89, ',.2f') - '1,234,567.89' - >>> format(12345.6 + 8901234.12j, ',f') - '12,345.600000+8,901,234.120000j' - >>> format(Decimal('1234567.89'), ',f') - '1,234,567.89' - -The supported types are :class:`int`, :class:`float`, :class:`complex` -and :class:`decimal.Decimal`. - -Discussions are underway about how to specify alternative separators -like dots, spaces, apostrophes, or underscores. Locale-aware applications -should use the existing *n* format specifier which already has some support -for thousands separators. - -.. seealso:: - - :pep:`378` - Format Specifier for Thousands Separator - PEP written by Raymond Hettinger and implemented by Eric Smith and - Mark Dickinson. - - -Other Language Changes -====================== - -Some smaller changes made to the core Python language are: - -* Directories and zip archives containing a :file:`__main__.py` - file can now be executed directly by passing their name to the - interpreter. The directory/zipfile is automatically inserted as the - first entry in sys.path. (Suggestion and initial patch by Andy Chu; - revised patch by Phillip J. Eby and Nick Coghlan; :issue:`1739468`.) - -* The :func:`int` type gained a ``bit_length`` method that returns the - number of bits necessary to represent its argument in binary:: - - >>> n = 37 - >>> bin(37) - '0b100101' - >>> n.bit_length() - 6 - >>> n = 2**123-1 - >>> n.bit_length() - 123 - >>> (n+1).bit_length() - 124 - - (Contributed by Fredrik Johansson, Victor Stinner, Raymond Hettinger, - and Mark Dickinson; :issue:`3439`.) - -* The fields in :func:`format` strings can now be automatically - numbered:: - - >>> 'Sir {} of {}'.format('Gallahad', 'Camelot') - 'Sir Gallahad of Camelot' - - Formerly, the string would have required numbered fields such as: - ``'Sir {0} of {1}'``. - - (Contributed by Eric Smith; :issue:`5237`.) - -* The :func:`string.maketrans` function is deprecated and is replaced by new - static methods, :meth:`bytes.maketrans` and :meth:`bytearray.maketrans`. - This change solves the confusion around which types were supported by the - :mod:`string` module. Now, :class:`str`, :class:`bytes`, and - :class:`bytearray` each have their own **maketrans** and **translate** - methods with intermediate translation tables of the appropriate type. - - (Contributed by Georg Brandl; :issue:`5675`.) - -* The syntax of the :keyword:`with` statement now allows multiple context - managers in a single statement:: - - >>> with open('mylog.txt') as infile, open('a.out', 'w') as outfile: - ... for line in infile: - ... if '<critical>' in line: - ... outfile.write(line) - - With the new syntax, the :func:`contextlib.nested` function is no longer - needed and is now deprecated. - - (Contributed by Georg Brandl and Mattias Brändström; - `appspot issue 53094 <https://codereview.appspot.com/53094>`_.) - -* ``round(x, n)`` now returns an integer if *x* is an integer. - Previously it returned a float:: - - >>> round(1123, -2) - 1100 - - (Contributed by Mark Dickinson; :issue:`4707`.) - -* Python now uses David Gay's algorithm for finding the shortest floating - point representation that doesn't change its value. This should help - mitigate some of the confusion surrounding binary floating point - numbers. - - The significance is easily seen with a number like ``1.1`` which does not - have an exact equivalent in binary floating point. Since there is no exact - equivalent, an expression like ``float('1.1')`` evaluates to the nearest - representable value which is ``0x1.199999999999ap+0`` in hex or - ``1.100000000000000088817841970012523233890533447265625`` in decimal. That - nearest value was and still is used in subsequent floating point - calculations. - - What is new is how the number gets displayed. Formerly, Python used a - simple approach. The value of ``repr(1.1)`` was computed as ``format(1.1, - '.17g')`` which evaluated to ``'1.1000000000000001'``. The advantage of - using 17 digits was that it relied on IEEE-754 guarantees to assure that - ``eval(repr(1.1))`` would round-trip exactly to its original value. The - disadvantage is that many people found the output to be confusing (mistaking - intrinsic limitations of binary floating point representation as being a - problem with Python itself). - - The new algorithm for ``repr(1.1)`` is smarter and returns ``'1.1'``. - Effectively, it searches all equivalent string representations (ones that - get stored with the same underlying float value) and returns the shortest - representation. - - The new algorithm tends to emit cleaner representations when possible, but - it does not change the underlying values. So, it is still the case that - ``1.1 + 2.2 != 3.3`` even though the representations may suggest otherwise. - - The new algorithm depends on certain features in the underlying floating - point implementation. If the required features are not found, the old - algorithm will continue to be used. Also, the text pickle protocols - assure cross-platform portability by using the old algorithm. - - (Contributed by Eric Smith and Mark Dickinson; :issue:`1580`) - -New, Improved, and Deprecated Modules -===================================== - -* Added a :class:`collections.Counter` class to support convenient - counting of unique items in a sequence or iterable:: - - >>> Counter(['red', 'blue', 'red', 'green', 'blue', 'blue']) - Counter({'blue': 3, 'red': 2, 'green': 1}) - - (Contributed by Raymond Hettinger; :issue:`1696199`.) - -* Added a new module, :mod:`tkinter.ttk` for access to the Tk themed widget set. - The basic idea of ttk is to separate, to the extent possible, the code - implementing a widget's behavior from the code implementing its appearance. - - (Contributed by Guilherme Polo; :issue:`2983`.) - -* The :class:`gzip.GzipFile` and :class:`bz2.BZ2File` classes now support - the context management protocol:: - - >>> # Automatically close file after writing - >>> with gzip.GzipFile(filename, "wb") as f: - ... f.write(b"xxx") - - (Contributed by Antoine Pitrou.) - -* The :mod:`decimal` module now supports methods for creating a - decimal object from a binary :class:`float`. The conversion is - exact but can sometimes be surprising:: - - >>> Decimal.from_float(1.1) - Decimal('1.100000000000000088817841970012523233890533447265625') - - The long decimal result shows the actual binary fraction being - stored for *1.1*. The fraction has many digits because *1.1* cannot - be exactly represented in binary. - - (Contributed by Raymond Hettinger and Mark Dickinson.) - -* The :mod:`itertools` module grew two new functions. The - :func:`itertools.combinations_with_replacement` function is one of - four for generating combinatorics including permutations and Cartesian - products. The :func:`itertools.compress` function mimics its namesake - from APL. Also, the existing :func:`itertools.count` function now has - an optional *step* argument and can accept any type of counting - sequence including :class:`fractions.Fraction` and - :class:`decimal.Decimal`:: - - >>> [p+q for p,q in combinations_with_replacement('LOVE', 2)] - ['LL', 'LO', 'LV', 'LE', 'OO', 'OV', 'OE', 'VV', 'VE', 'EE'] - - >>> list(compress(data=range(10), selectors=[0,0,1,1,0,1,0,1,0,0])) - [2, 3, 5, 7] - - >>> c = count(start=Fraction(1,2), step=Fraction(1,6)) - >>> [next(c), next(c), next(c), next(c)] - [Fraction(1, 2), Fraction(2, 3), Fraction(5, 6), Fraction(1, 1)] - - (Contributed by Raymond Hettinger.) - -* :func:`collections.namedtuple` now supports a keyword argument - *rename* which lets invalid fieldnames be automatically converted to - positional names in the form _0, _1, etc. This is useful when - the field names are being created by an external source such as a - CSV header, SQL field list, or user input:: - - >>> query = input() - SELECT region, dept, count(*) FROM main GROUPBY region, dept - - >>> cursor.execute(query) - >>> query_fields = [desc[0] for desc in cursor.description] - >>> UserQuery = namedtuple('UserQuery', query_fields, rename=True) - >>> pprint.pprint([UserQuery(*row) for row in cursor]) - [UserQuery(region='South', dept='Shipping', _2=185), - UserQuery(region='North', dept='Accounting', _2=37), - UserQuery(region='West', dept='Sales', _2=419)] - - (Contributed by Raymond Hettinger; :issue:`1818`.) - -* The :func:`re.sub`, :func:`re.subn` and :func:`re.split` functions now - accept a flags parameter. - - (Contributed by Gregory Smith.) - -* The :mod:`logging` module now implements a simple :class:`logging.NullHandler` - class for applications that are not using logging but are calling - library code that does. Setting-up a null handler will suppress - spurious warnings such as "No handlers could be found for logger foo":: - - >>> h = logging.NullHandler() - >>> logging.getLogger("foo").addHandler(h) - - (Contributed by Vinay Sajip; :issue:`4384`). - -* The :mod:`runpy` module which supports the ``-m`` command line switch - now supports the execution of packages by looking for and executing - a ``__main__`` submodule when a package name is supplied. - - (Contributed by Andi Vajda; :issue:`4195`.) - -* The :mod:`pdb` module can now access and display source code loaded via - :mod:`zipimport` (or any other conformant :pep:`302` loader). - - (Contributed by Alexander Belopolsky; :issue:`4201`.) - -* :class:`functools.partial` objects can now be pickled. - - (Suggested by Antoine Pitrou and Jesse Noller. Implemented by - Jack Diederich; :issue:`5228`.) - -* Add :mod:`pydoc` help topics for symbols so that ``help('@')`` - works as expected in the interactive environment. - - (Contributed by David Laban; :issue:`4739`.) - -* The :mod:`unittest` module now supports skipping individual tests or classes - of tests. And it supports marking a test as an expected failure, a test that - is known to be broken, but shouldn't be counted as a failure on a - TestResult:: - - class TestGizmo(unittest.TestCase): - - @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") - def test_gizmo_on_windows(self): - ... - - @unittest.expectedFailure - def test_gimzo_without_required_library(self): - ... - - Also, tests for exceptions have been builtout to work with context managers - using the :keyword:`with` statement:: - - def test_division_by_zero(self): - with self.assertRaises(ZeroDivisionError): - x / 0 - - In addition, several new assertion methods were added including - :func:`assertSetEqual`, :func:`assertDictEqual`, - :func:`assertDictContainsSubset`, :func:`assertListEqual`, - :func:`assertTupleEqual`, :func:`assertSequenceEqual`, - :func:`assertRaisesRegexp`, :func:`assertIsNone`, - and :func:`assertIsNotNone`. - - (Contributed by Benjamin Peterson and Antoine Pitrou.) - -* The :mod:`io` module has three new constants for the :meth:`seek` - method :data:`SEEK_SET`, :data:`SEEK_CUR`, and :data:`SEEK_END`. - -* The :attr:`sys.version_info` tuple is now a named tuple:: - - >>> sys.version_info - sys.version_info(major=3, minor=1, micro=0, releaselevel='alpha', serial=2) - - (Contributed by Ross Light; :issue:`4285`.) - -* The :mod:`nntplib` and :mod:`imaplib` modules now support IPv6. - - (Contributed by Derek Morr; :issue:`1655` and :issue:`1664`.) - -* The :mod:`pickle` module has been adapted for better interoperability with - Python 2.x when used with protocol 2 or lower. The reorganization of the - standard library changed the formal reference for many objects. For - example, ``__builtin__.set`` in Python 2 is called ``builtins.set`` in Python - 3. This change confounded efforts to share data between different versions of - Python. But now when protocol 2 or lower is selected, the pickler will - automatically use the old Python 2 names for both loading and dumping. This - remapping is turned-on by default but can be disabled with the *fix_imports* - option:: - - >>> s = {1, 2, 3} - >>> pickle.dumps(s, protocol=0) - b'c__builtin__\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.' - >>> pickle.dumps(s, protocol=0, fix_imports=False) - b'cbuiltins\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.' - - An unfortunate but unavoidable side-effect of this change is that protocol 2 - pickles produced by Python 3.1 won't be readable with Python 3.0. The latest - pickle protocol, protocol 3, should be used when migrating data between - Python 3.x implementations, as it doesn't attempt to remain compatible with - Python 2.x. - - (Contributed by Alexandre Vassalotti and Antoine Pitrou, :issue:`6137`.) - -* A new module, :mod:`importlib` was added. It provides a complete, portable, - pure Python reference implementation of the :keyword:`import` statement and its - counterpart, the :func:`__import__` function. It represents a substantial - step forward in documenting and defining the actions that take place during - imports. - - (Contributed by Brett Cannon.) - -Optimizations -============= - -Major performance enhancements have been added: - -* The new I/O library (as defined in :pep:`3116`) was mostly written in - Python and quickly proved to be a problematic bottleneck in Python 3.0. - In Python 3.1, the I/O library has been entirely rewritten in C and is - 2 to 20 times faster depending on the task at hand. The pure Python - version is still available for experimentation purposes through - the ``_pyio`` module. - - (Contributed by Amaury Forgeot d'Arc and Antoine Pitrou.) - -* Added a heuristic so that tuples and dicts containing only untrackable objects - are not tracked by the garbage collector. This can reduce the size of - collections and therefore the garbage collection overhead on long-running - programs, depending on their particular use of datatypes. - - (Contributed by Antoine Pitrou, :issue:`4688`.) - -* Enabling a configure option named ``--with-computed-gotos`` - on compilers that support it (notably: gcc, SunPro, icc), the bytecode - evaluation loop is compiled with a new dispatch mechanism which gives - speedups of up to 20%, depending on the system, the compiler, and - the benchmark. - - (Contributed by Antoine Pitrou along with a number of other participants, - :issue:`4753`). - -* The decoding of UTF-8, UTF-16 and LATIN-1 is now two to four times - faster. - - (Contributed by Antoine Pitrou and Amaury Forgeot d'Arc, :issue:`4868`.) - -* The :mod:`json` module now has a C extension to substantially improve - its performance. In addition, the API was modified so that json works - only with :class:`str`, not with :class:`bytes`. That change makes the - module closely match the `JSON specification <http://json.org/>`_ - which is defined in terms of Unicode. - - (Contributed by Bob Ippolito and converted to Py3.1 by Antoine Pitrou - and Benjamin Peterson; :issue:`4136`.) - -* Unpickling now interns the attribute names of pickled objects. This saves - memory and allows pickles to be smaller. - - (Contributed by Jake McGuire and Antoine Pitrou; :issue:`5084`.) - -IDLE -==== - -* IDLE's format menu now provides an option to strip trailing whitespace - from a source file. - - (Contributed by Roger D. Serwy; :issue:`5150`.) - -Build and C API Changes -======================= - -Changes to Python's build process and to the C API include: - -* Integers are now stored internally either in base 2**15 or in base - 2**30, the base being determined at build time. Previously, they - were always stored in base 2**15. Using base 2**30 gives - significant performance improvements on 64-bit machines, but - benchmark results on 32-bit machines have been mixed. Therefore, - the default is to use base 2**30 on 64-bit machines and base 2**15 - on 32-bit machines; on Unix, there's a new configure option - ``--enable-big-digits`` that can be used to override this default. - - Apart from the performance improvements this change should be invisible to - end users, with one exception: for testing and debugging purposes there's a - new :attr:`sys.int_info` that provides information about the - internal format, giving the number of bits per digit and the size in bytes - of the C type used to store each digit:: - - >>> import sys - >>> sys.int_info - sys.int_info(bits_per_digit=30, sizeof_digit=4) - - (Contributed by Mark Dickinson; :issue:`4258`.) - -* The :c:func:`PyLong_AsUnsignedLongLong()` function now handles a negative - *pylong* by raising :exc:`OverflowError` instead of :exc:`TypeError`. - - (Contributed by Mark Dickinson and Lisandro Dalcrin; :issue:`5175`.) - -* Deprecated :c:func:`PyNumber_Int`. Use :c:func:`PyNumber_Long` instead. - - (Contributed by Mark Dickinson; :issue:`4910`.) - -* Added a new :c:func:`PyOS_string_to_double` function to replace the - deprecated functions :c:func:`PyOS_ascii_strtod` and :c:func:`PyOS_ascii_atof`. - - (Contributed by Mark Dickinson; :issue:`5914`.) - -* Added :c:type:`PyCapsule` as a replacement for the :c:type:`PyCObject` API. - The principal difference is that the new type has a well defined interface - for passing typing safety information and a less complicated signature - for calling a destructor. The old type had a problematic API and is now - deprecated. - - (Contributed by Larry Hastings; :issue:`5630`.) - -Porting to Python 3.1 -===================== - -This section lists previously described changes and other bugfixes -that may require changes to your code: - -* The new floating point string representations can break existing doctests. - For example:: - - def e(): - '''Compute the base of natural logarithms. - - >>> e() - 2.7182818284590451 - - ''' - return sum(1/math.factorial(x) for x in reversed(range(30))) - - doctest.testmod() - - ********************************************************************** - Failed example: - e() - Expected: - 2.7182818284590451 - Got: - 2.718281828459045 - ********************************************************************** - -* The automatic name remapping in the pickle module for protocol 2 or lower can - make Python 3.1 pickles unreadable in Python 3.0. One solution is to use - protocol 3. Another solution is to set the *fix_imports* option to ``False``. - See the discussion above for more details. diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst deleted file mode 100644 index ca3eda0..0000000 --- a/Doc/whatsnew/3.2.rst +++ /dev/null @@ -1,2739 +0,0 @@ -**************************** - What's New In Python 3.2 -**************************** - -:Author: Raymond Hettinger - -.. $Id$ - Rules for maintenance: - - * Anyone can add text to this document. Do not spend very much time - on the wording of your changes, because your text will probably - get rewritten. (Note, during release candidate phase or just before - a beta release, please use the tracker instead -- this helps avoid - merge conflicts. If you must add a suggested entry directly, - please put it in an XXX comment and the maintainer will take notice). - - * The maintainer will go through Misc/NEWS periodically and add - changes; it's therefore more important to add your changes to - Misc/NEWS than to this file. - - * This is not a complete list of every single change; completeness - is the purpose of Misc/NEWS. Some changes I consider too small - or esoteric to include. If such a change is added to the text, - I'll just remove it. (This is another reason you shouldn't spend - too much time on writing your addition.) - - * If you want to draw your new text to the attention of the - maintainer, add 'XXX' to the beginning of the paragraph or - section. - - * It's OK to just add a fragmentary note about a change. For - example: "XXX Describe the transmogrify() function added to the - socket module." The maintainer will research the change and - write the necessary text. - - * You can comment out your additions if you like, but it's not - necessary (especially when a final release is some months away). - - * Credit the author of a patch or bugfix. Just the name is - sufficient; the e-mail address isn't necessary. It's helpful to - add the issue number: - - XXX Describe the transmogrify() function added to the socket - module. - - (Contributed by P.Y. Developer; :issue:`12345`.) - - This saves the maintainer the effort of going through the SVN log - when researching a change. - -This article explains the new features in Python 3.2 as compared to 3.1. It -focuses on a few highlights and gives a few examples. For full details, see the -`Misc/NEWS -<https://github.com/python/cpython/blob/076ca6c3c8df3030307e548d9be792ce3c1c6eea/Misc/NEWS>`_ -file. - -.. seealso:: - - :pep:`392` - Python 3.2 Release Schedule - - -PEP 384: Defining a Stable ABI -============================== - -In the past, extension modules built for one Python version were often -not usable with other Python versions. Particularly on Windows, every -feature release of Python required rebuilding all extension modules that -one wanted to use. This requirement was the result of the free access to -Python interpreter internals that extension modules could use. - -With Python 3.2, an alternative approach becomes available: extension -modules which restrict themselves to a limited API (by defining -Py_LIMITED_API) cannot use many of the internals, but are constrained -to a set of API functions that are promised to be stable for several -releases. As a consequence, extension modules built for 3.2 in that -mode will also work with 3.3, 3.4, and so on. Extension modules that -make use of details of memory structures can still be built, but will -need to be recompiled for every feature release. - -.. seealso:: - - :pep:`384` - Defining a Stable ABI - PEP written by Martin von Löwis. - - -PEP 389: Argparse Command Line Parsing Module -============================================= - -A new module for command line parsing, :mod:`argparse`, was introduced to -overcome the limitations of :mod:`optparse` which did not provide support for -positional arguments (not just options), subcommands, required options and other -common patterns of specifying and validating options. - -This module has already had widespread success in the community as a -third-party module. Being more fully featured than its predecessor, the -:mod:`argparse` module is now the preferred module for command-line processing. -The older module is still being kept available because of the substantial amount -of legacy code that depends on it. - -Here's an annotated example parser showing features like limiting results to a -set of choices, specifying a *metavar* in the help screen, validating that one -or more positional arguments is present, and making a required option:: - - import argparse - parser = argparse.ArgumentParser( - description = 'Manage servers', # main description for help - epilog = 'Tested on Solaris and Linux') # displayed after help - parser.add_argument('action', # argument name - choices = ['deploy', 'start', 'stop'], # three allowed values - help = 'action on each target') # help msg - parser.add_argument('targets', - metavar = 'HOSTNAME', # var name used in help msg - nargs = '+', # require one or more targets - help = 'url for target machines') # help msg explanation - parser.add_argument('-u', '--user', # -u or --user option - required = True, # make it a required argument - help = 'login as user') - -Example of calling the parser on a command string:: - - >>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain' - >>> result = parser.parse_args(cmd.split()) - >>> result.action - 'deploy' - >>> result.targets - ['sneezy.example.com', 'sleepy.example.com'] - >>> result.user - 'skycaptain' - -Example of the parser's automatically generated help:: - - >>> parser.parse_args('-h'.split()) - - usage: manage_cloud.py [-h] -u USER - {deploy,start,stop} HOSTNAME [HOSTNAME ...] - - Manage servers - - positional arguments: - {deploy,start,stop} action on each target - HOSTNAME url for target machines - - optional arguments: - -h, --help show this help message and exit - -u USER, --user USER login as user - - Tested on Solaris and Linux - -An especially nice :mod:`argparse` feature is the ability to define subparsers, -each with their own argument patterns and help displays:: - - import argparse - parser = argparse.ArgumentParser(prog='HELM') - subparsers = parser.add_subparsers() - - parser_l = subparsers.add_parser('launch', help='Launch Control') # first subgroup - parser_l.add_argument('-m', '--missiles', action='store_true') - parser_l.add_argument('-t', '--torpedos', action='store_true') - - parser_m = subparsers.add_parser('move', help='Move Vessel', # second subgroup - aliases=('steer', 'turn')) # equivalent names - parser_m.add_argument('-c', '--course', type=int, required=True) - parser_m.add_argument('-s', '--speed', type=int, default=0) - -.. code-block:: shell-session - - $ ./helm.py --help # top level help (launch and move) - $ ./helm.py launch --help # help for launch options - $ ./helm.py launch --missiles # set missiles=True and torpedos=False - $ ./helm.py steer --course 180 --speed 5 # set movement parameters - -.. seealso:: - - :pep:`389` - New Command Line Parsing Module - PEP written by Steven Bethard. - - :ref:`upgrading-optparse-code` for details on the differences from :mod:`optparse`. - - -PEP 391: Dictionary Based Configuration for Logging -==================================================== - -The :mod:`logging` module provided two kinds of configuration, one style with -function calls for each option or another style driven by an external file saved -in a :mod:`ConfigParser` format. Those options did not provide the flexibility -to create configurations from JSON or YAML files, nor did they support -incremental configuration, which is needed for specifying logger options from a -command line. - -To support a more flexible style, the module now offers -:func:`logging.config.dictConfig` for specifying logging configuration with -plain Python dictionaries. The configuration options include formatters, -handlers, filters, and loggers. Here's a working example of a configuration -dictionary:: - - {"version": 1, - "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"}, - "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"} - }, - "handlers": {"console": { - "class": "logging.StreamHandler", - "formatter": "brief", - "level": "INFO", - "stream": "ext://sys.stdout"}, - "console_priority": { - "class": "logging.StreamHandler", - "formatter": "full", - "level": "ERROR", - "stream": "ext://sys.stderr"} - }, - "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}} - - -If that dictionary is stored in a file called :file:`conf.json`, it can be -loaded and called with code like this:: - - >>> import json, logging.config - >>> with open('conf.json') as f: - ... conf = json.load(f) - ... - >>> logging.config.dictConfig(conf) - >>> logging.info("Transaction completed normally") - INFO : root : Transaction completed normally - >>> logging.critical("Abnormal termination") - 2011-02-17 11:14:36,694 root CRITICAL Abnormal termination - -.. seealso:: - - :pep:`391` - Dictionary Based Configuration for Logging - PEP written by Vinay Sajip. - - -PEP 3148: The ``concurrent.futures`` module -============================================ - -Code for creating and managing concurrency is being collected in a new top-level -namespace, *concurrent*. Its first member is a *futures* package which provides -a uniform high-level interface for managing threads and processes. - -The design for :mod:`concurrent.futures` was inspired by the -*java.util.concurrent* package. In that model, a running call and its result -are represented by a :class:`~concurrent.futures.Future` object that abstracts -features common to threads, processes, and remote procedure calls. That object -supports status checks (running or done), timeouts, cancellations, adding -callbacks, and access to results or exceptions. - -The primary offering of the new module is a pair of executor classes for -launching and managing calls. The goal of the executors is to make it easier to -use existing tools for making parallel calls. They save the effort needed to -setup a pool of resources, launch the calls, create a results queue, add -time-out handling, and limit the total number of threads, processes, or remote -procedure calls. - -Ideally, each application should share a single executor across multiple -components so that process and thread limits can be centrally managed. This -solves the design challenge that arises when each component has its own -competing strategy for resource management. - -Both classes share a common interface with three methods: -:meth:`~concurrent.futures.Executor.submit` for scheduling a callable and -returning a :class:`~concurrent.futures.Future` object; -:meth:`~concurrent.futures.Executor.map` for scheduling many asynchronous calls -at a time, and :meth:`~concurrent.futures.Executor.shutdown` for freeing -resources. The class is a :term:`context manager` and can be used in a -:keyword:`with` statement to assure that resources are automatically released -when currently pending futures are done executing. - -A simple of example of :class:`~concurrent.futures.ThreadPoolExecutor` is a -launch of four parallel threads for copying files:: - - import concurrent.futures, shutil - with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e: - e.submit(shutil.copy, 'src1.txt', 'dest1.txt') - e.submit(shutil.copy, 'src2.txt', 'dest2.txt') - e.submit(shutil.copy, 'src3.txt', 'dest3.txt') - e.submit(shutil.copy, 'src3.txt', 'dest4.txt') - -.. seealso:: - - :pep:`3148` - Futures -- Execute Computations Asynchronously - PEP written by Brian Quinlan. - - :ref:`Code for Threaded Parallel URL reads<threadpoolexecutor-example>`, an - example using threads to fetch multiple web pages in parallel. - - :ref:`Code for computing prime numbers in - parallel<processpoolexecutor-example>`, an example demonstrating - :class:`~concurrent.futures.ProcessPoolExecutor`. - - -PEP 3147: PYC Repository Directories -===================================== - -Python's scheme for caching bytecode in *.pyc* files did not work well in -environments with multiple Python interpreters. If one interpreter encountered -a cached file created by another interpreter, it would recompile the source and -overwrite the cached file, thus losing the benefits of caching. - -The issue of "pyc fights" has become more pronounced as it has become -commonplace for Linux distributions to ship with multiple versions of Python. -These conflicts also arise with CPython alternatives such as Unladen Swallow. - -To solve this problem, Python's import machinery has been extended to use -distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and -Unladen Swallow each competing for a file called "mymodule.pyc", they will now -look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and -"mymodule.unladen10.pyc". And to prevent all of these new files from -cluttering source directories, the *pyc* files are now collected in a -"__pycache__" directory stored under the package directory. - -Aside from the filenames and target directories, the new scheme has a few -aspects that are visible to the programmer: - -* Imported modules now have a :attr:`__cached__` attribute which stores the name - of the actual file that was imported: - - >>> import collections - >>> collections.__cached__ # doctest: +SKIP - 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' - -* The tag that is unique to each interpreter is accessible from the :mod:`imp` - module: - - >>> import imp - >>> imp.get_tag() # doctest: +SKIP - 'cpython-32' - -* Scripts that try to deduce source filename from the imported file now need to - be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc" - filename. Instead, use the new functions in the :mod:`imp` module: - - >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc') - 'c:/py32/lib/collections.py' - >>> imp.cache_from_source('c:/py32/lib/collections.py') # doctest: +SKIP - 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' - -* The :mod:`py_compile` and :mod:`compileall` modules have been updated to - reflect the new naming convention and target directory. The command-line - invocation of *compileall* has new options: ``-i`` for - specifying a list of files and directories to compile and ``-b`` which causes - bytecode files to be written to their legacy location rather than - *__pycache__*. - -* The :mod:`importlib.abc` module has been updated with new :term:`abstract base - classes <abstract base class>` for loading bytecode files. The obsolete - ABCs, :class:`~importlib.abc.PyLoader` and - :class:`~importlib.abc.PyPycLoader`, have been deprecated (instructions on how - to stay Python 3.1 compatible are included with the documentation). - -.. seealso:: - - :pep:`3147` - PYC Repository Directories - PEP written by Barry Warsaw. - - -PEP 3149: ABI Version Tagged .so Files -====================================== - -The PYC repository directory allows multiple bytecode cache files to be -co-located. This PEP implements a similar mechanism for shared object files by -giving them a common directory and distinct names for each version. - -The common directory is "pyshared" and the file names are made distinct by -identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the -major and minor version numbers, and optional build flags (such as "d" for -debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo", -you may see these files when the distribution package is installed:: - - /usr/share/pyshared/foo.cpython-32m.so - /usr/share/pyshared/foo.cpython-33md.so - -In Python itself, the tags are accessible from functions in the :mod:`sysconfig` -module:: - - >>> import sysconfig - >>> sysconfig.get_config_var('SOABI') # find the version tag - 'cpython-32mu' - >>> sysconfig.get_config_var('EXT_SUFFIX') # find the full filename extension - '.cpython-32mu.so' - -.. seealso:: - - :pep:`3149` - ABI Version Tagged .so Files - PEP written by Barry Warsaw. - - -PEP 3333: Python Web Server Gateway Interface v1.0.1 -===================================================== - -This informational PEP clarifies how bytes/text issues are to be handled by the -WSGI protocol. The challenge is that string handling in Python 3 is most -conveniently handled with the :class:`str` type even though the HTTP protocol -is itself bytes oriented. - -The PEP differentiates so-called *native strings* that are used for -request/response headers and metadata versus *byte strings* which are used for -the bodies of requests and responses. - -The *native strings* are always of type :class:`str` but are restricted to code -points between *U+0000* through *U+00FF* which are translatable to bytes using -*Latin-1* encoding. These strings are used for the keys and values in the -environment dictionary and for response headers and statuses in the -:func:`start_response` function. They must follow :rfc:`2616` with respect to -encoding. That is, they must either be *ISO-8859-1* characters or use -:rfc:`2047` MIME encoding. - -For developers porting WSGI applications from Python 2, here are the salient -points: - -* If the app already used strings for headers in Python 2, no change is needed. - -* If instead, the app encoded output headers or decoded input headers, then the - headers will need to be re-encoded to Latin-1. For example, an output header - encoded in utf-8 was using ``h.encode('utf-8')`` now needs to convert from - bytes to native strings using ``h.encode('utf-8').decode('latin-1')``. - -* Values yielded by an application or sent using the :meth:`write` method - must be byte strings. The :func:`start_response` function and environ - must use native strings. The two cannot be mixed. - -For server implementers writing CGI-to-WSGI pathways or other CGI-style -protocols, the users must to be able access the environment using native strings -even though the underlying platform may have a different convention. To bridge -this gap, the :mod:`wsgiref` module has a new function, -:func:`wsgiref.handlers.read_environ` for transcoding CGI variables from -:attr:`os.environ` into native strings and returning a new dictionary. - -.. seealso:: - - :pep:`3333` - Python Web Server Gateway Interface v1.0.1 - PEP written by Phillip Eby. - - -Other Language Changes -====================== - -Some smaller changes made to the core Python language are: - -* String formatting for :func:`format` and :meth:`str.format` gained new - capabilities for the format character **#**. Previously, for integers in - binary, octal, or hexadecimal, it caused the output to be prefixed with '0b', - '0o', or '0x' respectively. Now it can also handle floats, complex, and - Decimal, causing the output to always have a decimal point even when no digits - follow it. - - >>> format(20, '#o') - '0o24' - >>> format(12.34, '#5.0f') - ' 12.' - - (Suggested by Mark Dickinson and implemented by Eric Smith in :issue:`7094`.) - -* There is also a new :meth:`str.format_map` method that extends the - capabilities of the existing :meth:`str.format` method by accepting arbitrary - :term:`mapping` objects. This new method makes it possible to use string - formatting with any of Python's many dictionary-like objects such as - :class:`~collections.defaultdict`, :class:`~shelve.Shelf`, - :class:`~configparser.ConfigParser`, or :mod:`dbm`. It is also useful with - custom :class:`dict` subclasses that normalize keys before look-up or that - supply a :meth:`__missing__` method for unknown keys:: - - >>> import shelve - >>> d = shelve.open('tmp.shl') - >>> 'The {project_name} status is {status} as of {date}'.format_map(d) - 'The testing project status is green as of February 15, 2011' - - >>> class LowerCasedDict(dict): - ... def __getitem__(self, key): - ... return dict.__getitem__(self, key.lower()) - >>> lcd = LowerCasedDict(part='widgets', quantity=10) - >>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd) - 'There are 10 widgets in stock' - - >>> class PlaceholderDict(dict): - ... def __missing__(self, key): - ... return '<{}>'.format(key) - >>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict()) - 'Hello <name>, welcome to <location>' - - (Suggested by Raymond Hettinger and implemented by Eric Smith in - :issue:`6081`.) - -* The interpreter can now be started with a quiet option, ``-q``, to prevent - the copyright and version information from being displayed in the interactive - mode. The option can be introspected using the :attr:`sys.flags` attribute: - - .. code-block:: shell-session - - $ python -q - >>> sys.flags - sys.flags(debug=0, division_warning=0, inspect=0, interactive=0, - optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, - ignore_environment=0, verbose=0, bytes_warning=0, quiet=1) - - (Contributed by Marcin Wojdyr in :issue:`1772833`). - -* The :func:`hasattr` function works by calling :func:`getattr` and detecting - whether an exception is raised. This technique allows it to detect methods - created dynamically by :meth:`__getattr__` or :meth:`__getattribute__` which - would otherwise be absent from the class dictionary. Formerly, *hasattr* - would catch any exception, possibly masking genuine errors. Now, *hasattr* - has been tightened to only catch :exc:`AttributeError` and let other - exceptions pass through:: - - >>> class A: - ... @property - ... def f(self): - ... return 1 // 0 - ... - >>> a = A() - >>> hasattr(a, 'f') - Traceback (most recent call last): - ... - ZeroDivisionError: integer division or modulo by zero - - (Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.) - -* The :func:`str` of a float or complex number is now the same as its - :func:`repr`. Previously, the :func:`str` form was shorter but that just - caused confusion and is no longer needed now that the shortest possible - :func:`repr` is displayed by default: - - >>> import math - >>> repr(math.pi) - '3.141592653589793' - >>> str(math.pi) - '3.141592653589793' - - (Proposed and implemented by Mark Dickinson; :issue:`9337`.) - -* :class:`memoryview` objects now have a :meth:`~memoryview.release()` method - and they also now support the context management protocol. This allows timely - release of any resources that were acquired when requesting a buffer from the - original object. - - >>> with memoryview(b'abcdefgh') as v: - ... print(v.tolist()) - [97, 98, 99, 100, 101, 102, 103, 104] - - (Added by Antoine Pitrou; :issue:`9757`.) - -* Previously it was illegal to delete a name from the local namespace if it - occurs as a free variable in a nested block:: - - def outer(x): - def inner(): - return x - inner() - del x - - This is now allowed. Remember that the target of an :keyword:`except` clause - is cleared, so this code which used to work with Python 2.6, raised a - :exc:`SyntaxError` with Python 3.1 and now works again:: - - def f(): - def print_error(): - print(e) - try: - something - except Exception as e: - print_error() - # implicit "del e" here - - (See :issue:`4617`.) - -* The internal :c:type:`structsequence` tool now creates subclasses of tuple. - This means that C structures like those returned by :func:`os.stat`, - :func:`time.gmtime`, and :attr:`sys.version_info` now work like a - :term:`named tuple` and now work with functions and methods that - expect a tuple as an argument. This is a big step forward in making the C - structures as flexible as their pure Python counterparts: - - >>> import sys - >>> isinstance(sys.version_info, tuple) - True - >>> 'Version %d.%d.%d %s(%d)' % sys.version_info # doctest: +SKIP - 'Version 3.2.0 final(0)' - - (Suggested by Arfrever Frehtes Taifersar Arahesis and implemented - by Benjamin Peterson in :issue:`8413`.) - -* Warnings are now easier to control using the :envvar:`PYTHONWARNINGS` - environment variable as an alternative to using ``-W`` at the command line: - - .. code-block:: shell-session - - $ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::' - - (Suggested by Barry Warsaw and implemented by Philip Jenvey in :issue:`7301`.) - -* A new warning category, :exc:`ResourceWarning`, has been added. It is - emitted when potential issues with resource consumption or cleanup - are detected. It is silenced by default in normal release builds but - can be enabled through the means provided by the :mod:`warnings` - module, or on the command line. - - A :exc:`ResourceWarning` is issued at interpreter shutdown if the - :data:`gc.garbage` list isn't empty, and if :attr:`gc.DEBUG_UNCOLLECTABLE` is - set, all uncollectable objects are printed. This is meant to make the - programmer aware that their code contains object finalization issues. - - A :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed - without having been explicitly closed. While the deallocator for such - object ensures it closes the underlying operating system resource - (usually, a file descriptor), the delay in deallocating the object could - produce various issues, especially under Windows. Here is an example - of enabling the warning from the command line: - - .. code-block:: shell-session - - $ python -q -Wdefault - >>> f = open("foo", "wb") - >>> del f - __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'> - - (Added by Antoine Pitrou and Georg Brandl in :issue:`10093` and :issue:`477863`.) - -* :class:`range` objects now support *index* and *count* methods. This is part - of an effort to make more objects fully implement the - :class:`collections.Sequence` :term:`abstract base class`. As a result, the - language will have a more uniform API. In addition, :class:`range` objects - now support slicing and negative indices, even with values larger than - :attr:`sys.maxsize`. This makes *range* more interoperable with lists:: - - >>> range(0, 100, 2).count(10) - 1 - >>> range(0, 100, 2).index(10) - 5 - >>> range(0, 100, 2)[5] - 10 - >>> range(0, 100, 2)[0:5] - range(0, 10, 2) - - (Contributed by Daniel Stutzbach in :issue:`9213`, by Alexander Belopolsky - in :issue:`2690`, and by Nick Coghlan in :issue:`10889`.) - -* The :func:`callable` builtin function from Py2.x was resurrected. It provides - a concise, readable alternative to using an :term:`abstract base class` in an - expression like ``isinstance(x, collections.Callable)``: - - >>> callable(max) - True - >>> callable(20) - False - - (See :issue:`10518`.) - -* Python's import mechanism can now load modules installed in directories with - non-ASCII characters in the path name. This solved an aggravating problem - with home directories for users with non-ASCII characters in their usernames. - - (Required extensive work by Victor Stinner in :issue:`9425`.) - - -New, Improved, and Deprecated Modules -===================================== - -Python's standard library has undergone significant maintenance efforts and -quality improvements. - -The biggest news for Python 3.2 is that the :mod:`email` package, :mod:`mailbox` -module, and :mod:`nntplib` modules now work correctly with the bytes/text model -in Python 3. For the first time, there is correct handling of messages with -mixed encodings. - -Throughout the standard library, there has been more careful attention to -encodings and text versus bytes issues. In particular, interactions with the -operating system are now better able to exchange non-ASCII data using the -Windows MBCS encoding, locale-aware encodings, or UTF-8. - -Another significant win is the addition of substantially better support for -*SSL* connections and security certificates. - -In addition, more classes now implement a :term:`context manager` to support -convenient and reliable resource clean-up using a :keyword:`with` statement. - -email ------ - -The usability of the :mod:`email` package in Python 3 has been mostly fixed by -the extensive efforts of R. David Murray. The problem was that emails are -typically read and stored in the form of :class:`bytes` rather than :class:`str` -text, and they may contain multiple encodings within a single email. So, the -email package had to be extended to parse and generate email messages in bytes -format. - -* New functions :func:`~email.message_from_bytes` and - :func:`~email.message_from_binary_file`, and new classes - :class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser` - allow binary message data to be parsed into model objects. - -* Given bytes input to the model, :meth:`~email.message.Message.get_payload` - will by default decode a message body that has a - :mailheader:`Content-Transfer-Encoding` of *8bit* using the charset - specified in the MIME headers and return the resulting string. - -* Given bytes input to the model, :class:`~email.generator.Generator` will - convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of - *8bit* to instead have a *7bit* :mailheader:`Content-Transfer-Encoding`. - - Headers with unencoded non-ASCII bytes are deemed to be :rfc:`2047`\ -encoded - using the *unknown-8bit* character set. - -* A new class :class:`~email.generator.BytesGenerator` produces bytes as output, - preserving any unchanged non-ASCII data that was present in the input used to - build the model, including message bodies with a - :mailheader:`Content-Transfer-Encoding` of *8bit*. - -* The :mod:`smtplib` :class:`~smtplib.SMTP` class now accepts a byte string - for the *msg* argument to the :meth:`~smtplib.SMTP.sendmail` method, - and a new method, :meth:`~smtplib.SMTP.send_message` accepts a - :class:`~email.message.Message` object and can optionally obtain the - *from_addr* and *to_addrs* addresses directly from the object. - -(Proposed and implemented by R. David Murray, :issue:`4661` and :issue:`10321`.) - -elementtree ------------ - -The :mod:`xml.etree.ElementTree` package and its :mod:`xml.etree.cElementTree` -counterpart have been updated to version 1.3. - -Several new and useful functions and methods have been added: - -* :func:`xml.etree.ElementTree.fromstringlist` which builds an XML document - from a sequence of fragments -* :func:`xml.etree.ElementTree.register_namespace` for registering a global - namespace prefix -* :func:`xml.etree.ElementTree.tostringlist` for string representation - including all sublists -* :meth:`xml.etree.ElementTree.Element.extend` for appending a sequence of zero - or more elements -* :meth:`xml.etree.ElementTree.Element.iterfind` searches an element and - subelements -* :meth:`xml.etree.ElementTree.Element.itertext` creates a text iterator over - an element and its subelements -* :meth:`xml.etree.ElementTree.TreeBuilder.end` closes the current element -* :meth:`xml.etree.ElementTree.TreeBuilder.doctype` handles a doctype - declaration - -Two methods have been deprecated: - -* :meth:`xml.etree.ElementTree.getchildren` use ``list(elem)`` instead. -* :meth:`xml.etree.ElementTree.getiterator` use ``Element.iter`` instead. - -For details of the update, see `Introducing ElementTree -<http://effbot.org/zone/elementtree-13-intro.htm>`_ on Fredrik Lundh's website. - -(Contributed by Florent Xicluna and Fredrik Lundh, :issue:`6472`.) - -functools ---------- - -* The :mod:`functools` module includes a new decorator for caching function - calls. :func:`functools.lru_cache` can save repeated queries to an external - resource whenever the results are expected to be the same. - - For example, adding a caching decorator to a database query function can save - database accesses for popular searches: - - >>> import functools - >>> @functools.lru_cache(maxsize=300) - ... def get_phone_number(name): - ... c = conn.cursor() - ... c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,)) - ... return c.fetchone()[0] - - >>> for name in user_requests: # doctest: +SKIP - ... get_phone_number(name) # cached lookup - - To help with choosing an effective cache size, the wrapped function is - instrumented for tracking cache statistics: - - >>> get_phone_number.cache_info() # doctest: +SKIP - CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300) - - If the phonelist table gets updated, the outdated contents of the cache can be - cleared with: - - >>> get_phone_number.cache_clear() - - (Contributed by Raymond Hettinger and incorporating design ideas from Jim - Baker, Miki Tebeka, and Nick Coghlan; see `recipe 498245 - <https://code.activestate.com/recipes/498245>`_\, `recipe 577479 - <https://code.activestate.com/recipes/577479>`_\, :issue:`10586`, and - :issue:`10593`.) - -* The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute - pointing to the original callable function. This allows wrapped functions to - be introspected. It also copies :attr:`__annotations__` if defined. And now - it also gracefully skips over missing attributes such as :attr:`__doc__` which - might not be defined for the wrapped callable. - - In the above example, the cache can be removed by recovering the original - function: - - >>> get_phone_number = get_phone_number.__wrapped__ # uncached function - - (By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and - :issue:`8814`.) - -* To help write classes with rich comparison methods, a new decorator - :func:`functools.total_ordering` will use existing equality and inequality - methods to fill in the remaining methods. - - For example, supplying *__eq__* and *__lt__* will enable - :func:`~functools.total_ordering` to fill-in *__le__*, *__gt__* and *__ge__*:: - - @total_ordering - class Student: - def __eq__(self, other): - return ((self.lastname.lower(), self.firstname.lower()) == - (other.lastname.lower(), other.firstname.lower())) - - def __lt__(self, other): - return ((self.lastname.lower(), self.firstname.lower()) < - (other.lastname.lower(), other.firstname.lower())) - - With the *total_ordering* decorator, the remaining comparison methods - are filled in automatically. - - (Contributed by Raymond Hettinger.) - -* To aid in porting programs from Python 2, the :func:`functools.cmp_to_key` - function converts an old-style comparison function to - modern :term:`key function`: - - >>> # locale-aware sort order - >>> sorted(iterable, key=cmp_to_key(locale.strcoll)) # doctest: +SKIP - - For sorting examples and a brief sorting tutorial, see the `Sorting HowTo - <https://wiki.python.org/moin/HowTo/Sorting/>`_ tutorial. - - (Contributed by Raymond Hettinger.) - -itertools ---------- - -* The :mod:`itertools` module has a new :func:`~itertools.accumulate` function - modeled on APL's *scan* operator and Numpy's *accumulate* function: - - >>> from itertools import accumulate - >>> list(accumulate([8, 2, 50])) - [8, 10, 60] - - >>> prob_dist = [0.1, 0.4, 0.2, 0.3] - >>> list(accumulate(prob_dist)) # cumulative probability distribution - [0.1, 0.5, 0.7, 1.0] - - For an example using :func:`~itertools.accumulate`, see the :ref:`examples for - the random module <random-examples>`. - - (Contributed by Raymond Hettinger and incorporating design suggestions - from Mark Dickinson.) - -collections ------------ - -* The :class:`collections.Counter` class now has two forms of in-place - subtraction, the existing *-=* operator for `saturating subtraction - <https://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new - :meth:`~collections.Counter.subtract` method for regular subtraction. The - former is suitable for `multisets <https://en.wikipedia.org/wiki/Multiset>`_ - which only have positive counts, and the latter is more suitable for use cases - that allow negative counts: - - >>> from collections import Counter - >>> tally = Counter(dogs=5, cats=3) - >>> tally -= Counter(dogs=2, cats=8) # saturating subtraction - >>> tally - Counter({'dogs': 3}) - - >>> tally = Counter(dogs=5, cats=3) - >>> tally.subtract(dogs=2, cats=8) # regular subtraction - >>> tally - Counter({'dogs': 3, 'cats': -5}) - - (Contributed by Raymond Hettinger.) - -* The :class:`collections.OrderedDict` class has a new method - :meth:`~collections.OrderedDict.move_to_end` which takes an existing key and - moves it to either the first or last position in the ordered sequence. - - The default is to move an item to the last position. This is equivalent of - renewing an entry with ``od[k] = od.pop(k)``. - - A fast move-to-end operation is useful for resequencing entries. For example, - an ordered dictionary can be used to track order of access by aging entries - from the oldest to the most recently accessed. - - >>> from collections import OrderedDict - >>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e']) - >>> list(d) - ['a', 'b', 'X', 'd', 'e'] - >>> d.move_to_end('X') - >>> list(d) - ['a', 'b', 'd', 'e', 'X'] - - (Contributed by Raymond Hettinger.) - -* The :class:`collections.deque` class grew two new methods - :meth:`~collections.deque.count` and :meth:`~collections.deque.reverse` that - make them more substitutable for :class:`list` objects: - - >>> from collections import deque - >>> d = deque('simsalabim') - >>> d.count('s') - 2 - >>> d.reverse() - >>> d - deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's']) - - (Contributed by Raymond Hettinger.) - -threading ---------- - -The :mod:`threading` module has a new :class:`~threading.Barrier` -synchronization class for making multiple threads wait until all of them have -reached a common barrier point. Barriers are useful for making sure that a task -with multiple preconditions does not run until all of the predecessor tasks are -complete. - -Barriers can work with an arbitrary number of threads. This is a generalization -of a `Rendezvous <https://en.wikipedia.org/wiki/Synchronous_rendezvous>`_ which -is defined for only two threads. - -Implemented as a two-phase cyclic barrier, :class:`~threading.Barrier` objects -are suitable for use in loops. The separate *filling* and *draining* phases -assure that all threads get released (drained) before any one of them can loop -back and re-enter the barrier. The barrier fully resets after each cycle. - -Example of using barriers:: - - from threading import Barrier, Thread - - def get_votes(site): - ballots = conduct_election(site) - all_polls_closed.wait() # do not count until all polls are closed - totals = summarize(ballots) - publish(site, totals) - - all_polls_closed = Barrier(len(sites)) - for site in sites: - Thread(target=get_votes, args=(site,)).start() - -In this example, the barrier enforces a rule that votes cannot be counted at any -polling site until all polls are closed. Notice how a solution with a barrier -is similar to one with :meth:`threading.Thread.join`, but the threads stay alive -and continue to do work (summarizing ballots) after the barrier point is -crossed. - -If any of the predecessor tasks can hang or be delayed, a barrier can be created -with an optional *timeout* parameter. Then if the timeout period elapses before -all the predecessor tasks reach the barrier point, all waiting threads are -released and a :exc:`~threading.BrokenBarrierError` exception is raised:: - - def get_votes(site): - ballots = conduct_election(site) - try: - all_polls_closed.wait(timeout=midnight - time.now()) - except BrokenBarrierError: - lockbox = seal_ballots(ballots) - queue.put(lockbox) - else: - totals = summarize(ballots) - publish(site, totals) - -In this example, the barrier enforces a more robust rule. If some election -sites do not finish before midnight, the barrier times-out and the ballots are -sealed and deposited in a queue for later handling. - -See `Barrier Synchronization Patterns -<http://osl.cs.illinois.edu/media/papers/karmani-2009-barrier_synchronization_pattern.pdf>`_ -for more examples of how barriers can be used in parallel computing. Also, there is -a simple but thorough explanation of barriers in `The Little Book of Semaphores -<https://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf>`_, *section 3.6*. - -(Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in -:issue:`8777`.) - -datetime and time ------------------ - -* The :mod:`datetime` module has a new type :class:`~datetime.timezone` that - implements the :class:`~datetime.tzinfo` interface by returning a fixed UTC - offset and timezone name. This makes it easier to create timezone-aware - datetime objects:: - - >>> from datetime import datetime, timezone - - >>> datetime.now(timezone.utc) - datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc) - - >>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z") - datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc) - -* Also, :class:`~datetime.timedelta` objects can now be multiplied by - :class:`float` and divided by :class:`float` and :class:`int` objects. - And :class:`~datetime.timedelta` objects can now divide one another. - -* The :meth:`datetime.date.strftime` method is no longer restricted to years - after 1900. The new supported year range is from 1000 to 9999 inclusive. - -* Whenever a two-digit year is used in a time tuple, the interpretation has been - governed by :attr:`time.accept2dyear`. The default is ``True`` which means that - for a two-digit year, the century is guessed according to the POSIX rules - governing the ``%y`` strptime format. - - Starting with Py3.2, use of the century guessing heuristic will emit a - :exc:`DeprecationWarning`. Instead, it is recommended that - :attr:`time.accept2dyear` be set to ``False`` so that large date ranges - can be used without guesswork:: - - >>> import time, warnings - >>> warnings.resetwarnings() # remove the default warning filters - - >>> time.accept2dyear = True # guess whether 11 means 11 or 2011 - >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0)) - Warning (from warnings module): - ... - DeprecationWarning: Century info guessed for a 2-digit year. - 'Fri Jan 1 12:34:56 2011' - - >>> time.accept2dyear = False # use the full range of allowable dates - >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0)) - 'Fri Jan 1 12:34:56 11' - - Several functions now have significantly expanded date ranges. When - :attr:`time.accept2dyear` is false, the :func:`time.asctime` function will - accept any year that fits in a C int, while the :func:`time.mktime` and - :func:`time.strftime` functions will accept the full range supported by the - corresponding operating system functions. - -(Contributed by Alexander Belopolsky and Victor Stinner in :issue:`1289118`, -:issue:`5094`, :issue:`6641`, :issue:`2706`, :issue:`1777412`, :issue:`8013`, -and :issue:`10827`.) - -.. XXX https://bugs.python.org/issue?%40search_text=datetime&%40sort=-activity - -math ----- - -The :mod:`math` module has been updated with six new functions inspired by the -C99 standard. - -The :func:`~math.isfinite` function provides a reliable and fast way to detect -special values. It returns ``True`` for regular numbers and ``False`` for *Nan* or -*Infinity*: - ->>> from math import isfinite ->>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))] -[True, True, False, False] - -The :func:`~math.expm1` function computes ``e**x-1`` for small values of *x* -without incurring the loss of precision that usually accompanies the subtraction -of nearly equal quantities: - ->>> from math import expm1 ->>> expm1(0.013671875) # more accurate way to compute e**x-1 for a small x -0.013765762467652909 - -The :func:`~math.erf` function computes a probability integral or `Gaussian -error function <https://en.wikipedia.org/wiki/Error_function>`_. The -complementary error function, :func:`~math.erfc`, is ``1 - erf(x)``: - -.. doctest:: - :options: +SKIP - - >>> from math import erf, erfc, sqrt - >>> erf(1.0/sqrt(2.0)) # portion of normal distribution within 1 standard deviation - 0.682689492137086 - >>> erfc(1.0/sqrt(2.0)) # portion of normal distribution outside 1 standard deviation - 0.31731050786291404 - >>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0)) - 1.0 - -The :func:`~math.gamma` function is a continuous extension of the factorial -function. See https://en.wikipedia.org/wiki/Gamma_function for details. Because -the function is related to factorials, it grows large even for small values of -*x*, so there is also a :func:`~math.lgamma` function for computing the natural -logarithm of the gamma function: - ->>> from math import gamma, lgamma ->>> gamma(7.0) # six factorial -720.0 ->>> lgamma(801.0) # log(800 factorial) -4551.950730698041 - -(Contributed by Mark Dickinson.) - -abc ---- - -The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and -:func:`~abc.abstractstaticmethod`. - -These tools make it possible to define an :term:`abstract base class` that -requires a particular :func:`classmethod` or :func:`staticmethod` to be -implemented:: - - class Temperature(metaclass=abc.ABCMeta): - @abc.abstractclassmethod - def from_fahrenheit(cls, t): - ... - @abc.abstractclassmethod - def from_celsius(cls, t): - ... - -(Patch submitted by Daniel Urban; :issue:`5867`.) - -io --- - -The :class:`io.BytesIO` has a new method, :meth:`~io.BytesIO.getbuffer`, which -provides functionality similar to :func:`memoryview`. It creates an editable -view of the data without making a copy. The buffer's random access and support -for slice notation are well-suited to in-place editing:: - - >>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11 - - >>> def change_location(buffer, record_number, location): - ... start = record_number * REC_LEN + LOC_START - ... buffer[start: start+LOC_LEN] = location - - >>> import io - - >>> byte_stream = io.BytesIO( - ... b'G3805 storeroom Main chassis ' - ... b'X7899 shipping Reserve cog ' - ... b'L6988 receiving Primary sprocket' - ... ) - >>> buffer = byte_stream.getbuffer() - >>> change_location(buffer, 1, b'warehouse ') - >>> change_location(buffer, 0, b'showroom ') - >>> print(byte_stream.getvalue()) - b'G3805 showroom Main chassis ' - b'X7899 warehouse Reserve cog ' - b'L6988 receiving Primary sprocket' - -(Contributed by Antoine Pitrou in :issue:`5506`.) - -reprlib -------- - -When writing a :meth:`__repr__` method for a custom container, it is easy to -forget to handle the case where a member refers back to the container itself. -Python's builtin objects such as :class:`list` and :class:`set` handle -self-reference by displaying "..." in the recursive part of the representation -string. - -To help write such :meth:`__repr__` methods, the :mod:`reprlib` module has a new -decorator, :func:`~reprlib.recursive_repr`, for detecting recursive calls to -:meth:`__repr__` and substituting a placeholder string instead:: - - >>> class MyList(list): - ... @recursive_repr() - ... def __repr__(self): - ... return '<' + '|'.join(map(repr, self)) + '>' - ... - >>> m = MyList('abc') - >>> m.append(m) - >>> m.append('x') - >>> print(m) - <'a'|'b'|'c'|...|'x'> - -(Contributed by Raymond Hettinger in :issue:`9826` and :issue:`9840`.) - -logging -------- - -In addition to dictionary-based configuration described above, the -:mod:`logging` package has many other improvements. - -The logging documentation has been augmented by a :ref:`basic tutorial -<logging-basic-tutorial>`\, an :ref:`advanced tutorial -<logging-advanced-tutorial>`\, and a :ref:`cookbook <logging-cookbook>` of -logging recipes. These documents are the fastest way to learn about logging. - -The :func:`logging.basicConfig` set-up function gained a *style* argument to -support three different types of string formatting. It defaults to "%" for -traditional %-formatting, can be set to "{" for the new :meth:`str.format` style, or -can be set to "$" for the shell-style formatting provided by -:class:`string.Template`. The following three configurations are equivalent:: - - >>> from logging import basicConfig - >>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s") - >>> basicConfig(style='{', format="{name} -> {levelname} {message}") - >>> basicConfig(style='$', format="$name -> $levelname: $message") - -If no configuration is set-up before a logging event occurs, there is now a -default configuration using a :class:`~logging.StreamHandler` directed to -:attr:`sys.stderr` for events of ``WARNING`` level or higher. Formerly, an -event occurring before a configuration was set-up would either raise an -exception or silently drop the event depending on the value of -:attr:`logging.raiseExceptions`. The new default handler is stored in -:attr:`logging.lastResort`. - -The use of filters has been simplified. Instead of creating a -:class:`~logging.Filter` object, the predicate can be any Python callable that -returns ``True`` or ``False``. - -There were a number of other improvements that add flexibility and simplify -configuration. See the module documentation for a full listing of changes in -Python 3.2. - -csv ---- - -The :mod:`csv` module now supports a new dialect, :class:`~csv.unix_dialect`, -which applies quoting for all fields and a traditional Unix style with ``'\n'`` as -the line terminator. The registered dialect name is ``unix``. - -The :class:`csv.DictWriter` has a new method, -:meth:`~csv.DictWriter.writeheader` for writing-out an initial row to document -the field names:: - - >>> import csv, sys - >>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix') - >>> w.writeheader() - "name","dept" - >>> w.writerows([ - ... {'name': 'tom', 'dept': 'accounting'}, - ... {'name': 'susan', 'dept': 'Salesl'}]) - "tom","accounting" - "susan","sales" - -(New dialect suggested by Jay Talbot in :issue:`5975`, and the new method -suggested by Ed Abraham in :issue:`1537721`.) - -contextlib ----------- - -There is a new and slightly mind-blowing tool -:class:`~contextlib.ContextDecorator` that is helpful for creating a -:term:`context manager` that does double duty as a function decorator. - -As a convenience, this new functionality is used by -:func:`~contextlib.contextmanager` so that no extra effort is needed to support -both roles. - -The basic idea is that both context managers and function decorators can be used -for pre-action and post-action wrappers. Context managers wrap a group of -statements using a :keyword:`with` statement, and function decorators wrap a -group of statements enclosed in a function. So, occasionally there is a need to -write a pre-action or post-action wrapper that can be used in either role. - -For example, it is sometimes useful to wrap functions or groups of statements -with a logger that can track the time of entry and time of exit. Rather than -writing both a function decorator and a context manager for the task, the -:func:`~contextlib.contextmanager` provides both capabilities in a single -definition:: - - from contextlib import contextmanager - import logging - - logging.basicConfig(level=logging.INFO) - - @contextmanager - def track_entry_and_exit(name): - logging.info('Entering: %s', name) - yield - logging.info('Exiting: %s', name) - -Formerly, this would have only been usable as a context manager:: - - with track_entry_and_exit('widget loader'): - print('Some time consuming activity goes here') - load_widget() - -Now, it can be used as a decorator as well:: - - @track_entry_and_exit('widget loader') - def activity(): - print('Some time consuming activity goes here') - load_widget() - -Trying to fulfill two roles at once places some limitations on the technique. -Context managers normally have the flexibility to return an argument usable by -a :keyword:`with` statement, but there is no parallel for function decorators. - -In the above example, there is not a clean way for the *track_entry_and_exit* -context manager to return a logging instance for use in the body of enclosed -statements. - -(Contributed by Michael Foord in :issue:`9110`.) - -decimal and fractions ---------------------- - -Mark Dickinson crafted an elegant and efficient scheme for assuring that -different numeric datatypes will have the same hash value whenever their actual -values are equal (:issue:`8188`):: - - assert hash(Fraction(3, 2)) == hash(1.5) == \ - hash(Decimal("1.5")) == hash(complex(1.5, 0)) - -Some of the hashing details are exposed through a new attribute, -:attr:`sys.hash_info`, which describes the bit width of the hash value, the -prime modulus, the hash values for *infinity* and *nan*, and the multiplier -used for the imaginary part of a number: - ->>> sys.hash_info # doctest: +SKIP -sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003) - -An early decision to limit the inter-operability of various numeric types has -been relaxed. It is still unsupported (and ill-advised) to have implicit -mixing in arithmetic expressions such as ``Decimal('1.1') + float('1.1')`` -because the latter loses information in the process of constructing the binary -float. However, since existing floating point value can be converted losslessly -to either a decimal or rational representation, it makes sense to add them to -the constructor and to support mixed-type comparisons. - -* The :class:`decimal.Decimal` constructor now accepts :class:`float` objects - directly so there in no longer a need to use the :meth:`~decimal.Decimal.from_float` - method (:issue:`8257`). - -* Mixed type comparisons are now fully supported so that - :class:`~decimal.Decimal` objects can be directly compared with :class:`float` - and :class:`fractions.Fraction` (:issue:`2531` and :issue:`8188`). - -Similar changes were made to :class:`fractions.Fraction` so that the -:meth:`~fractions.Fraction.from_float()` and :meth:`~fractions.Fraction.from_decimal` -methods are no longer needed (:issue:`8294`): - ->>> from decimal import Decimal ->>> from fractions import Fraction ->>> Decimal(1.1) -Decimal('1.100000000000000088817841970012523233890533447265625') ->>> Fraction(1.1) -Fraction(2476979795053773, 2251799813685248) - -Another useful change for the :mod:`decimal` module is that the -:attr:`Context.clamp` attribute is now public. This is useful in creating -contexts that correspond to the decimal interchange formats specified in IEEE -754 (see :issue:`8540`). - -(Contributed by Mark Dickinson and Raymond Hettinger.) - -ftp ---- - -The :class:`ftplib.FTP` class now supports the context management protocol to -unconditionally consume :exc:`socket.error` exceptions and to close the FTP -connection when done:: - - >>> from ftplib import FTP - >>> with FTP("ftp1.at.proftpd.org") as ftp: - ftp.login() - ftp.dir() - - '230 Anonymous login ok, restrictions apply.' - dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 . - dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .. - dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS - dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora - -Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input` -also grew auto-closing context managers:: - - with fileinput.input(files=('log1.txt', 'log2.txt')) as f: - for line in f: - process(line) - -(Contributed by Tarek Ziadé and Giampaolo Rodolà in :issue:`4972`, and -by Georg Brandl in :issue:`8046` and :issue:`1286`.) - -The :class:`~ftplib.FTP_TLS` class now accepts a *context* parameter, which is a -:class:`ssl.SSLContext` object allowing bundling SSL configuration options, -certificates and private keys into a single (potentially long-lived) structure. - -(Contributed by Giampaolo Rodolà; :issue:`8806`.) - -popen ------ - -The :func:`os.popen` and :func:`subprocess.Popen` functions now support -:keyword:`with` statements for auto-closing of the file descriptors. - -(Contributed by Antoine Pitrou and Brian Curtin in :issue:`7461` and -:issue:`10554`.) - -select ------- - -The :mod:`select` module now exposes a new, constant attribute, -:attr:`~select.PIPE_BUF`, which gives the minimum number of bytes which are -guaranteed not to block when :func:`select.select` says a pipe is ready -for writing. - ->>> import select ->>> select.PIPE_BUF # doctest: +SKIP -512 - -(Available on Unix systems. Patch by Sébastien Sablé in :issue:`9862`) - -gzip and zipfile ----------------- - -:class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase` -:term:`abstract base class` (except for ``truncate()``). It also has a -:meth:`~gzip.GzipFile.peek` method and supports unseekable as well as -zero-padded file objects. - -The :mod:`gzip` module also gains the :func:`~gzip.compress` and -:func:`~gzip.decompress` functions for easier in-memory compression and -decompression. Keep in mind that text needs to be encoded as :class:`bytes` -before compressing and decompressing: - ->>> import gzip ->>> s = 'Three shall be the number thou shalt count, ' ->>> s += 'and the number of the counting shall be three' ->>> b = s.encode() # convert to utf-8 ->>> len(b) -89 ->>> c = gzip.compress(b) ->>> len(c) -77 ->>> gzip.decompress(c).decode()[:42] # decompress and convert to text -'Three shall be the number thou shalt count' - -(Contributed by Anand B. Pillai in :issue:`3488`; and by Antoine Pitrou, Nir -Aides and Brian Curtin in :issue:`9962`, :issue:`1675951`, :issue:`7471` and -:issue:`2846`.) - -Also, the :class:`zipfile.ZipExtFile` class was reworked internally to represent -files stored inside an archive. The new implementation is significantly faster -and can be wrapped in an :class:`io.BufferedReader` object for more speedups. It -also solves an issue where interleaved calls to *read* and *readline* gave the -wrong results. - -(Patch submitted by Nir Aides in :issue:`7610`.) - -tarfile -------- - -The :class:`~tarfile.TarFile` class can now be used as a context manager. In -addition, its :meth:`~tarfile.TarFile.add` method has a new option, *filter*, -that controls which files are added to the archive and allows the file metadata -to be edited. - -The new *filter* option replaces the older, less flexible *exclude* parameter -which is now deprecated. If specified, the optional *filter* parameter needs to -be a :term:`keyword argument`. The user-supplied filter function accepts a -:class:`~tarfile.TarInfo` object and returns an updated -:class:`~tarfile.TarInfo` object, or if it wants the file to be excluded, the -function can return ``None``:: - - >>> import tarfile, glob - - >>> def myfilter(tarinfo): - ... if tarinfo.isfile(): # only save real files - ... tarinfo.uname = 'monty' # redact the user name - ... return tarinfo - - >>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf: - ... for filename in glob.glob('*.txt'): - ... tf.add(filename, filter=myfilter) - ... tf.list() - -rw-r--r-- monty/501 902 2011-01-26 17:59:11 annotations.txt - -rw-r--r-- monty/501 123 2011-01-26 17:59:11 general_questions.txt - -rw-r--r-- monty/501 3514 2011-01-26 17:59:11 prion.txt - -rw-r--r-- monty/501 124 2011-01-26 17:59:11 py_todo.txt - -rw-r--r-- monty/501 1399 2011-01-26 17:59:11 semaphore_notes.txt - -(Proposed by Tarek Ziadé and implemented by Lars Gustäbel in :issue:`6856`.) - -hashlib -------- - -The :mod:`hashlib` module has two new constant attributes listing the hashing -algorithms guaranteed to be present in all implementations and those available -on the current implementation:: - - >>> import hashlib - - >>> hashlib.algorithms_guaranteed - {'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'} - - >>> hashlib.algorithms_available - {'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256', - 'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2', - 'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224', - 'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'} - -(Suggested by Carl Chenet in :issue:`7418`.) - -ast ---- - -The :mod:`ast` module has a wonderful a general-purpose tool for safely -evaluating expression strings using the Python literal -syntax. The :func:`ast.literal_eval` function serves as a secure alternative to -the builtin :func:`eval` function which is easily abused. Python 3.2 adds -:class:`bytes` and :class:`set` literals to the list of supported types: -strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and ``None``. - -:: - - >>> from ast import literal_eval - - >>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}" - >>> literal_eval(request) - {'args': (2, 0.5), 'req': 3, 'func': 'pow'} - - >>> request = "os.system('do something harmful')" - >>> literal_eval(request) - Traceback (most recent call last): - ... - ValueError: malformed node or string: <_ast.Call object at 0x101739a10> - -(Implemented by Benjamin Peterson and Georg Brandl.) - -os --- - -Different operating systems use various encodings for filenames and environment -variables. The :mod:`os` module provides two new functions, -:func:`~os.fsencode` and :func:`~os.fsdecode`, for encoding and decoding -filenames: - ->>> import os ->>> filename = 'Sehenswürdigkeiten' ->>> os.fsencode(filename) -b'Sehensw\xc3\xbcrdigkeiten' - -Some operating systems allow direct access to encoded bytes in the -environment. If so, the :attr:`os.supports_bytes_environ` constant will be -true. - -For direct access to encoded environment variables (if available), -use the new :func:`os.getenvb` function or use :data:`os.environb` -which is a bytes version of :data:`os.environ`. - -(Contributed by Victor Stinner.) - -shutil ------- - -The :func:`shutil.copytree` function has two new options: - -* *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function - copies a file pointed to by a symlink, not the symlink itself. This option - will silence the error raised if the file doesn't exist. - -* *copy_function*: is a callable that will be used to copy files. - :func:`shutil.copy2` is used by default. - -(Contributed by Tarek Ziadé.) - -In addition, the :mod:`shutil` module now supports :ref:`archiving operations -<archiving-operations>` for zipfiles, uncompressed tarfiles, gzipped tarfiles, -and bzipped tarfiles. And there are functions for registering additional -archiving file formats (such as xz compressed tarfiles or custom formats). - -The principal functions are :func:`~shutil.make_archive` and -:func:`~shutil.unpack_archive`. By default, both operate on the current -directory (which can be set by :func:`os.chdir`) and on any sub-directories. -The archive filename needs to be specified with a full pathname. The archiving -step is non-destructive (the original files are left unchanged). - -:: - - >>> import shutil, pprint - - >>> os.chdir('mydata') # change to the source directory - >>> f = shutil.make_archive('/var/backup/mydata', - ... 'zip') # archive the current directory - >>> f # show the name of archive - '/var/backup/mydata.zip' - >>> os.chdir('tmp') # change to an unpacking - >>> shutil.unpack_archive('/var/backup/mydata.zip') # recover the data - - >>> pprint.pprint(shutil.get_archive_formats()) # display known formats - [('bztar', "bzip2'ed tar-file"), - ('gztar', "gzip'ed tar-file"), - ('tar', 'uncompressed tar file'), - ('zip', 'ZIP file')] - - >>> shutil.register_archive_format( # register a new archive format - ... name='xz', - ... function=xz.compress, # callable archiving function - ... extra_args=[('level', 8)], # arguments to the function - ... description='xz compression' - ... ) - -(Contributed by Tarek Ziadé.) - -sqlite3 -------- - -The :mod:`sqlite3` module was updated to pysqlite version 2.6.0. It has two new capabilities. - -* The :attr:`sqlite3.Connection.in_transit` attribute is true if there is an - active transaction for uncommitted changes. - -* The :meth:`sqlite3.Connection.enable_load_extension` and - :meth:`sqlite3.Connection.load_extension` methods allows you to load SQLite - extensions from ".so" files. One well-known extension is the fulltext-search - extension distributed with SQLite. - -(Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.) - -html ----- - -A new :mod:`html` module was introduced with only a single function, -:func:`~html.escape`, which is used for escaping reserved characters from HTML -markup: - ->>> import html ->>> html.escape('x > 2 && x < 7') -'x > 2 && x < 7' - -socket ------- - -The :mod:`socket` module has two new improvements. - -* Socket objects now have a :meth:`~socket.socket.detach()` method which puts - the socket into closed state without actually closing the underlying file - descriptor. The latter can then be reused for other purposes. - (Added by Antoine Pitrou; :issue:`8524`.) - -* :func:`socket.create_connection` now supports the context management protocol - to unconditionally consume :exc:`socket.error` exceptions and to close the - socket when done. - (Contributed by Giampaolo Rodolà; :issue:`9794`.) - -ssl ---- - -The :mod:`ssl` module added a number of features to satisfy common requirements -for secure (encrypted, authenticated) internet connections: - -* A new class, :class:`~ssl.SSLContext`, serves as a container for persistent - SSL data, such as protocol settings, certificates, private keys, and various - other options. It includes a :meth:`~ssl.SSLContext.wrap_socket` for creating - an SSL socket from an SSL context. - -* A new function, :func:`ssl.match_hostname`, supports server identity - verification for higher-level protocols by implementing the rules of HTTPS - (from :rfc:`2818`) which are also suitable for other protocols. - -* The :func:`ssl.wrap_socket` constructor function now takes a *ciphers* - argument. The *ciphers* string lists the allowed encryption algorithms using - the format described in the `OpenSSL documentation - <https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-LIST-FORMAT>`__. - -* When linked against recent versions of OpenSSL, the :mod:`ssl` module now - supports the Server Name Indication extension to the TLS protocol, allowing - multiple "virtual hosts" using different certificates on a single IP port. - This extension is only supported in client mode, and is activated by passing - the *server_hostname* argument to :meth:`ssl.SSLContext.wrap_socket`. - -* Various options have been added to the :mod:`ssl` module, such as - :data:`~ssl.OP_NO_SSLv2` which disables the insecure and obsolete SSLv2 - protocol. - -* The extension now loads all the OpenSSL ciphers and digest algorithms. If - some SSL certificates cannot be verified, they are reported as an "unknown - algorithm" error. - -* The version of OpenSSL being used is now accessible using the module - attributes :data:`ssl.OPENSSL_VERSION` (a string), - :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and - :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer). - -(Contributed by Antoine Pitrou in :issue:`8850`, :issue:`1589`, :issue:`8322`, -:issue:`5639`, :issue:`4870`, :issue:`8484`, and :issue:`8321`.) - -nntp ----- - -The :mod:`nntplib` module has a revamped implementation with better bytes and -text semantics as well as more practical APIs. These improvements break -compatibility with the nntplib version in Python 3.1, which was partly -dysfunctional in itself. - -Support for secure connections through both implicit (using -:class:`nntplib.NNTP_SSL`) and explicit (using :meth:`nntplib.NNTP.starttls`) -TLS has also been added. - -(Contributed by Antoine Pitrou in :issue:`9360` and Andrew Vant in :issue:`1926`.) - -certificates ------------- - -:class:`http.client.HTTPSConnection`, :class:`urllib.request.HTTPSHandler` -and :func:`urllib.request.urlopen` now take optional arguments to allow for -server certificate checking against a set of Certificate Authorities, -as recommended in public uses of HTTPS. - -(Added by Antoine Pitrou, :issue:`9003`.) - -imaplib -------- - -Support for explicit TLS on standard IMAP4 connections has been added through -the new :mod:`imaplib.IMAP4.starttls` method. - -(Contributed by Lorenzo M. Catucci and Antoine Pitrou, :issue:`4471`.) - -http.client ------------ - -There were a number of small API improvements in the :mod:`http.client` module. -The old-style HTTP 0.9 simple responses are no longer supported and the *strict* -parameter is deprecated in all classes. - -The :class:`~http.client.HTTPConnection` and -:class:`~http.client.HTTPSConnection` classes now have a *source_address* -parameter for a (host, port) tuple indicating where the HTTP connection is made -from. - -Support for certificate checking and HTTPS virtual hosts were added to -:class:`~http.client.HTTPSConnection`. - -The :meth:`~http.client.HTTPConnection.request` method on connection objects -allowed an optional *body* argument so that a :term:`file object` could be used -to supply the content of the request. Conveniently, the *body* argument now -also accepts an :term:`iterable` object so long as it includes an explicit -``Content-Length`` header. This extended interface is much more flexible than -before. - -To establish an HTTPS connection through a proxy server, there is a new -:meth:`~http.client.HTTPConnection.set_tunnel` method that sets the host and -port for HTTP Connect tunneling. - -To match the behavior of :mod:`http.server`, the HTTP client library now also -encodes headers with ISO-8859-1 (Latin-1) encoding. It was already doing that -for incoming headers, so now the behavior is consistent for both incoming and -outgoing traffic. (See work by Armin Ronacher in :issue:`10980`.) - -unittest --------- - -The unittest module has a number of improvements supporting test discovery for -packages, easier experimentation at the interactive prompt, new testcase -methods, improved diagnostic messages for test failures, and better method -names. - -* The command-line call ``python -m unittest`` can now accept file paths - instead of module names for running specific tests (:issue:`10620`). The new - test discovery can find tests within packages, locating any test importable - from the top-level directory. The top-level directory can be specified with - the `-t` option, a pattern for matching files with ``-p``, and a directory to - start discovery with ``-s``: - - .. code-block:: shell-session - - $ python -m unittest discover -s my_proj_dir -p _test.py - - (Contributed by Michael Foord.) - -* Experimentation at the interactive prompt is now easier because the - :class:`unittest.case.TestCase` class can now be instantiated without - arguments: - - >>> from unittest import TestCase - >>> TestCase().assertEqual(pow(2, 3), 8) - - (Contributed by Michael Foord.) - -* The :mod:`unittest` module has two new methods, - :meth:`~unittest.TestCase.assertWarns` and - :meth:`~unittest.TestCase.assertWarnsRegex` to verify that a given warning type - is triggered by the code under test:: - - with self.assertWarns(DeprecationWarning): - legacy_function('XYZ') - - (Contributed by Antoine Pitrou, :issue:`9754`.) - - Another new method, :meth:`~unittest.TestCase.assertCountEqual` is used to - compare two iterables to determine if their element counts are equal (whether - the same elements are present with the same number of occurrences regardless - of order):: - - def test_anagram(self): - self.assertCountEqual('algorithm', 'logarithm') - - (Contributed by Raymond Hettinger.) - -* A principal feature of the unittest module is an effort to produce meaningful - diagnostics when a test fails. When possible, the failure is recorded along - with a diff of the output. This is especially helpful for analyzing log files - of failed test runs. However, since diffs can sometime be voluminous, there is - a new :attr:`~unittest.TestCase.maxDiff` attribute that sets maximum length of - diffs displayed. - -* In addition, the method names in the module have undergone a number of clean-ups. - - For example, :meth:`~unittest.TestCase.assertRegex` is the new name for - :meth:`~unittest.TestCase.assertRegexpMatches` which was misnamed because the - test uses :func:`re.search`, not :func:`re.match`. Other methods using - regular expressions are now named using short form "Regex" in preference to - "Regexp" -- this matches the names used in other unittest implementations, - matches Python's old name for the :mod:`re` module, and it has unambiguous - camel-casing. - - (Contributed by Raymond Hettinger and implemented by Ezio Melotti.) - -* To improve consistency, some long-standing method aliases are being - deprecated in favor of the preferred names: - - =============================== ============================== - Old Name Preferred Name - =============================== ============================== - :meth:`assert_` :meth:`.assertTrue` - :meth:`assertEquals` :meth:`.assertEqual` - :meth:`assertNotEquals` :meth:`.assertNotEqual` - :meth:`assertAlmostEquals` :meth:`.assertAlmostEqual` - :meth:`assertNotAlmostEquals` :meth:`.assertNotAlmostEqual` - =============================== ============================== - - Likewise, the ``TestCase.fail*`` methods deprecated in Python 3.1 are expected - to be removed in Python 3.3. Also see the :ref:`deprecated-aliases` section in - the :mod:`unittest` documentation. - - (Contributed by Ezio Melotti; :issue:`9424`.) - -* The :meth:`~unittest.TestCase.assertDictContainsSubset` method was deprecated - because it was misimplemented with the arguments in the wrong order. This - created hard-to-debug optical illusions where tests like - ``TestCase().assertDictContainsSubset({'a':1, 'b':2}, {'a':1})`` would fail. - - (Contributed by Raymond Hettinger.) - -random ------- - -The integer methods in the :mod:`random` module now do a better job of producing -uniform distributions. Previously, they computed selections with -``int(n*random())`` which had a slight bias whenever *n* was not a power of two. -Now, multiple selections are made from a range up to the next power of two and a -selection is kept only when it falls within the range ``0 <= x < n``. The -functions and methods affected are :func:`~random.randrange`, -:func:`~random.randint`, :func:`~random.choice`, :func:`~random.shuffle` and -:func:`~random.sample`. - -(Contributed by Raymond Hettinger; :issue:`9025`.) - -poplib ------- - -:class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a -:class:`ssl.SSLContext` object allowing bundling SSL configuration options, -certificates and private keys into a single (potentially long-lived) -structure. - -(Contributed by Giampaolo Rodolà; :issue:`8807`.) - -asyncore --------- - -:class:`asyncore.dispatcher` now provides a -:meth:`~asyncore.dispatcher.handle_accepted()` method -returning a `(sock, addr)` pair which is called when a connection has actually -been established with a new remote endpoint. This is supposed to be used as a -replacement for old :meth:`~asyncore.dispatcher.handle_accept()` and avoids -the user to call :meth:`~asyncore.dispatcher.accept()` directly. - -(Contributed by Giampaolo Rodolà; :issue:`6706`.) - -tempfile --------- - -The :mod:`tempfile` module has a new context manager, -:class:`~tempfile.TemporaryDirectory` which provides easy deterministic -cleanup of temporary directories:: - - with tempfile.TemporaryDirectory() as tmpdirname: - print('created temporary dir:', tmpdirname) - -(Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.) - -inspect -------- - -* The :mod:`inspect` module has a new function - :func:`~inspect.getgeneratorstate` to easily identify the current state of a - generator-iterator:: - - >>> from inspect import getgeneratorstate - >>> def gen(): - ... yield 'demo' - >>> g = gen() - >>> getgeneratorstate(g) - 'GEN_CREATED' - >>> next(g) - 'demo' - >>> getgeneratorstate(g) - 'GEN_SUSPENDED' - >>> next(g, None) - >>> getgeneratorstate(g) - 'GEN_CLOSED' - - (Contributed by Rodolpho Eckhardt and Nick Coghlan, :issue:`10220`.) - -* To support lookups without the possibility of activating a dynamic attribute, - the :mod:`inspect` module has a new function, :func:`~inspect.getattr_static`. - Unlike :func:`hasattr`, this is a true read-only search, guaranteed not to - change state while it is searching:: - - >>> class A: - ... @property - ... def f(self): - ... print('Running') - ... return 10 - ... - >>> a = A() - >>> getattr(a, 'f') - Running - 10 - >>> inspect.getattr_static(a, 'f') - <property object at 0x1022bd788> - - (Contributed by Michael Foord.) - -pydoc ------ - -The :mod:`pydoc` module now provides a much-improved Web server interface, as -well as a new command-line option ``-b`` to automatically open a browser window -to display that server: - -.. code-block:: shell-session - - $ pydoc3.2 -b - -(Contributed by Ron Adam; :issue:`2001`.) - -dis ---- - -The :mod:`dis` module gained two new functions for inspecting code, -:func:`~dis.code_info` and :func:`~dis.show_code`. Both provide detailed code -object information for the supplied function, method, source code string or code -object. The former returns a string and the latter prints it:: - - >>> import dis, random - >>> dis.show_code(random.choice) - Name: choice - Filename: /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py - Argument count: 2 - Kw-only arguments: 0 - Number of locals: 3 - Stack size: 11 - Flags: OPTIMIZED, NEWLOCALS, NOFREE - Constants: - 0: 'Choose a random element from a non-empty sequence.' - 1: 'Cannot choose from an empty sequence' - Names: - 0: _randbelow - 1: len - 2: ValueError - 3: IndexError - Variable names: - 0: self - 1: seq - 2: i - -In addition, the :func:`~dis.dis` function now accepts string arguments -so that the common idiom ``dis(compile(s, '', 'eval'))`` can be shortened -to ``dis(s)``:: - - >>> dis('3*x+1 if x%2==1 else x//2') - 1 0 LOAD_NAME 0 (x) - 3 LOAD_CONST 0 (2) - 6 BINARY_MODULO - 7 LOAD_CONST 1 (1) - 10 COMPARE_OP 2 (==) - 13 POP_JUMP_IF_FALSE 28 - 16 LOAD_CONST 2 (3) - 19 LOAD_NAME 0 (x) - 22 BINARY_MULTIPLY - 23 LOAD_CONST 1 (1) - 26 BINARY_ADD - 27 RETURN_VALUE - >> 28 LOAD_NAME 0 (x) - 31 LOAD_CONST 0 (2) - 34 BINARY_FLOOR_DIVIDE - 35 RETURN_VALUE - -Taken together, these improvements make it easier to explore how CPython is -implemented and to see for yourself what the language syntax does -under-the-hood. - -(Contributed by Nick Coghlan in :issue:`9147`.) - -dbm ---- - -All database modules now support the :meth:`get` and :meth:`setdefault` methods. - -(Suggested by Ray Allen in :issue:`9523`.) - -ctypes ------- - -A new type, :class:`ctypes.c_ssize_t` represents the C :c:type:`ssize_t` datatype. - -site ----- - -The :mod:`site` module has three new functions useful for reporting on the -details of a given Python installation. - -* :func:`~site.getsitepackages` lists all global site-packages directories. - -* :func:`~site.getuserbase` reports on the user's base directory where data can - be stored. - -* :func:`~site.getusersitepackages` reveals the user-specific site-packages - directory path. - -:: - - >>> import site - >>> site.getsitepackages() - ['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages', - '/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python', - '/Library/Python/3.2/site-packages'] - >>> site.getuserbase() - '/Users/raymondhettinger/Library/Python/3.2' - >>> site.getusersitepackages() - '/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages' - -Conveniently, some of site's functionality is accessible directly from the -command-line: - -.. code-block:: shell-session - - $ python -m site --user-base - /Users/raymondhettinger/.local - $ python -m site --user-site - /Users/raymondhettinger/.local/lib/python3.2/site-packages - -(Contributed by Tarek Ziadé in :issue:`6693`.) - -sysconfig ---------- - -The new :mod:`sysconfig` module makes it straightforward to discover -installation paths and configuration variables that vary across platforms and -installations. - -The module offers access simple access functions for platform and version -information: - -* :func:`~sysconfig.get_platform` returning values like *linux-i586* or - *macosx-10.6-ppc*. -* :func:`~sysconfig.get_python_version` returns a Python version string - such as "3.2". - -It also provides access to the paths and variables corresponding to one of -seven named schemes used by :mod:`distutils`. Those include *posix_prefix*, -*posix_home*, *posix_user*, *nt*, *nt_user*, *os2*, *os2_home*: - -* :func:`~sysconfig.get_paths` makes a dictionary containing installation paths - for the current installation scheme. -* :func:`~sysconfig.get_config_vars` returns a dictionary of platform specific - variables. - -There is also a convenient command-line interface: - -.. code-block:: doscon - - C:\Python32>python -m sysconfig - Platform: "win32" - Python version: "3.2" - Current installation scheme: "nt" - - Paths: - data = "C:\Python32" - include = "C:\Python32\Include" - platinclude = "C:\Python32\Include" - platlib = "C:\Python32\Lib\site-packages" - platstdlib = "C:\Python32\Lib" - purelib = "C:\Python32\Lib\site-packages" - scripts = "C:\Python32\Scripts" - stdlib = "C:\Python32\Lib" - - Variables: - BINDIR = "C:\Python32" - BINLIBDEST = "C:\Python32\Lib" - EXE = ".exe" - INCLUDEPY = "C:\Python32\Include" - LIBDEST = "C:\Python32\Lib" - SO = ".pyd" - VERSION = "32" - abiflags = "" - base = "C:\Python32" - exec_prefix = "C:\Python32" - platbase = "C:\Python32" - prefix = "C:\Python32" - projectbase = "C:\Python32" - py_version = "3.2" - py_version_nodot = "32" - py_version_short = "3.2" - srcdir = "C:\Python32" - userbase = "C:\Documents and Settings\Raymond\Application Data\Python" - -(Moved out of Distutils by Tarek Ziadé.) - -pdb ---- - -The :mod:`pdb` debugger module gained a number of usability improvements: - -* :file:`pdb.py` now has a ``-c`` option that executes commands as given in a - :file:`.pdbrc` script file. -* A :file:`.pdbrc` script file can contain ``continue`` and ``next`` commands - that continue debugging. -* The :class:`Pdb` class constructor now accepts a *nosigint* argument. -* New commands: ``l(list)``, ``ll(long list)`` and ``source`` for - listing source code. -* New commands: ``display`` and ``undisplay`` for showing or hiding - the value of an expression if it has changed. -* New command: ``interact`` for starting an interactive interpreter containing - the global and local names found in the current scope. -* Breakpoints can be cleared by breakpoint number. - -(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.) - -configparser ------------- - -The :mod:`configparser` module was modified to improve usability and -predictability of the default parser and its supported INI syntax. The old -:class:`ConfigParser` class was removed in favor of :class:`SafeConfigParser` -which has in turn been renamed to :class:`~configparser.ConfigParser`. Support -for inline comments is now turned off by default and section or option -duplicates are not allowed in a single configuration source. - -Config parsers gained a new API based on the mapping protocol:: - - >>> parser = ConfigParser() - >>> parser.read_string(""" - ... [DEFAULT] - ... location = upper left - ... visible = yes - ... editable = no - ... color = blue - ... - ... [main] - ... title = Main Menu - ... color = green - ... - ... [options] - ... title = Options - ... """) - >>> parser['main']['color'] - 'green' - >>> parser['main']['editable'] - 'no' - >>> section = parser['options'] - >>> section['title'] - 'Options' - >>> section['title'] = 'Options (editable: %(editable)s)' - >>> section['title'] - 'Options (editable: no)' - -The new API is implemented on top of the classical API, so custom parser -subclasses should be able to use it without modifications. - -The INI file structure accepted by config parsers can now be customized. Users -can specify alternative option/value delimiters and comment prefixes, change the -name of the *DEFAULT* section or switch the interpolation syntax. - -There is support for pluggable interpolation including an additional interpolation -handler :class:`~configparser.ExtendedInterpolation`:: - - >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) - >>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'}, - ... 'custom': {'prefix': '/usr/local'}}) - >>> parser.read_string(""" - ... [buildout] - ... parts = - ... zope9 - ... instance - ... find-links = - ... ${buildout:directory}/downloads/dist - ... - ... [zope9] - ... recipe = plone.recipe.zope9install - ... location = /opt/zope - ... - ... [instance] - ... recipe = plone.recipe.zope9instance - ... zope9-location = ${zope9:location} - ... zope-conf = ${custom:prefix}/etc/zope.conf - ... """) - >>> parser['buildout']['find-links'] - '\n/home/ambv/zope9/downloads/dist' - >>> parser['instance']['zope-conf'] - '/usr/local/etc/zope.conf' - >>> instance = parser['instance'] - >>> instance['zope-conf'] - '/usr/local/etc/zope.conf' - >>> instance['zope9-location'] - '/opt/zope' - -A number of smaller features were also introduced, like support for specifying -encoding in read operations, specifying fallback values for get-functions, or -reading directly from dictionaries and strings. - -(All changes contributed by Łukasz Langa.) - -.. XXX consider showing a difflib example - -urllib.parse ------------- - -A number of usability improvements were made for the :mod:`urllib.parse` module. - -The :func:`~urllib.parse.urlparse` function now supports `IPv6 -<https://en.wikipedia.org/wiki/IPv6>`_ addresses as described in :rfc:`2732`: - - >>> import urllib.parse - >>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') # doctest: +NORMALIZE_WHITESPACE - ParseResult(scheme='http', - netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]', - path='/foo/', - params='', - query='', - fragment='') - -The :func:`~urllib.parse.urldefrag` function now returns a :term:`named tuple`:: - - >>> r = urllib.parse.urldefrag('http://python.org/about/#target') - >>> r - DefragResult(url='http://python.org/about/', fragment='target') - >>> r[0] - 'http://python.org/about/' - >>> r.fragment - 'target' - -And, the :func:`~urllib.parse.urlencode` function is now much more flexible, -accepting either a string or bytes type for the *query* argument. If it is a -string, then the *safe*, *encoding*, and *error* parameters are sent to -:func:`~urllib.parse.quote_plus` for encoding:: - - >>> urllib.parse.urlencode([ - ... ('type', 'telenovela'), - ... ('name', '¿Dónde Está Elisa?')], - ... encoding='latin-1') - 'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F' - -As detailed in :ref:`parsing-ascii-encoded-bytes`, all the :mod:`urllib.parse` -functions now accept ASCII-encoded byte strings as input, so long as they are -not mixed with regular strings. If ASCII-encoded byte strings are given as -parameters, the return types will also be an ASCII-encoded byte strings: - - >>> urllib.parse.urlparse(b'http://www.python.org:80/about/') # doctest: +NORMALIZE_WHITESPACE - ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80', - path=b'/about/', params=b'', query=b'', fragment=b'') - -(Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in :issue:`2987`, -:issue:`5468`, and :issue:`9873`.) - -mailbox -------- - -Thanks to a concerted effort by R. David Murray, the :mod:`mailbox` module has -been fixed for Python 3.2. The challenge was that mailbox had been originally -designed with a text interface, but email messages are best represented with -:class:`bytes` because various parts of a message may have different encodings. - -The solution harnessed the :mod:`email` package's binary support for parsing -arbitrary email messages. In addition, the solution required a number of API -changes. - -As expected, the :meth:`~mailbox.Mailbox.add` method for -:class:`mailbox.Mailbox` objects now accepts binary input. - -:class:`~io.StringIO` and text file input are deprecated. Also, string input -will fail early if non-ASCII characters are used. Previously it would fail when -the email was processed in a later step. - -There is also support for binary output. The :meth:`~mailbox.Mailbox.get_file` -method now returns a file in the binary mode (where it used to incorrectly set -the file to text-mode). There is also a new :meth:`~mailbox.Mailbox.get_bytes` -method that returns a :class:`bytes` representation of a message corresponding -to a given *key*. - -It is still possible to get non-binary output using the old API's -:meth:`~mailbox.Mailbox.get_string` method, but that approach -is not very useful. Instead, it is best to extract messages from -a :class:`~mailbox.Message` object or to load them from binary input. - -(Contributed by R. David Murray, with efforts from Steffen Daode Nurpmeso and an -initial patch by Victor Stinner in :issue:`9124`.) - -turtledemo ----------- - -The demonstration code for the :mod:`turtle` module was moved from the *Demo* -directory to main library. It includes over a dozen sample scripts with -lively displays. Being on :attr:`sys.path`, it can now be run directly -from the command-line: - -.. code-block:: shell-session - - $ python -m turtledemo - -(Moved from the Demo directory by Alexander Belopolsky in :issue:`10199`.) - -Multi-threading -=============== - -* The mechanism for serializing execution of concurrently running Python threads - (generally known as the :term:`GIL` or :term:`Global Interpreter Lock`) has - been rewritten. Among the objectives were more predictable switching - intervals and reduced overhead due to lock contention and the number of - ensuing system calls. The notion of a "check interval" to allow thread - switches has been abandoned and replaced by an absolute duration expressed in - seconds. This parameter is tunable through :func:`sys.setswitchinterval()`. - It currently defaults to 5 milliseconds. - - Additional details about the implementation can be read from a `python-dev - mailing-list message - <https://mail.python.org/pipermail/python-dev/2009-October/093321.html>`_ - (however, "priority requests" as exposed in this message have not been kept - for inclusion). - - (Contributed by Antoine Pitrou.) - -* Regular and recursive locks now accept an optional *timeout* argument to their - :meth:`~threading.Lock.acquire` method. (Contributed by Antoine Pitrou; - :issue:`7316`.) - -* Similarly, :meth:`threading.Semaphore.acquire` also gained a *timeout* - argument. (Contributed by Torsten Landschoff; :issue:`850728`.) - -* Regular and recursive lock acquisitions can now be interrupted by signals on - platforms using Pthreads. This means that Python programs that deadlock while - acquiring locks can be successfully killed by repeatedly sending SIGINT to the - process (by pressing :kbd:`Ctrl+C` in most shells). - (Contributed by Reid Kleckner; :issue:`8844`.) - - -Optimizations -============= - -A number of small performance enhancements have been added: - -* Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as - being a test for membership in a set of constants. The optimizer recasts the - :class:`set` as a :class:`frozenset` and stores the pre-built constant. - - Now that the speed penalty is gone, it is practical to start writing - membership tests using set-notation. This style is both semantically clear - and operationally fast:: - - extension = name.rpartition('.')[2] - if extension in {'xml', 'html', 'xhtml', 'css'}: - handle(name) - - (Patch and additional tests contributed by Dave Malcolm; :issue:`6690`). - -* Serializing and unserializing data using the :mod:`pickle` module is now - several times faster. - - (Contributed by Alexandre Vassalotti, Antoine Pitrou - and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.) - -* The `Timsort algorithm <https://en.wikipedia.org/wiki/Timsort>`_ used in - :meth:`list.sort` and :func:`sorted` now runs faster and uses less memory - when called with a :term:`key function`. Previously, every element of - a list was wrapped with a temporary object that remembered the key value - associated with each element. Now, two arrays of keys and values are - sorted in parallel. This saves the memory consumed by the sort wrappers, - and it saves time lost to delegating comparisons. - - (Patch by Daniel Stutzbach in :issue:`9915`.) - -* JSON decoding performance is improved and memory consumption is reduced - whenever the same string is repeated for multiple keys. Also, JSON encoding - now uses the C speedups when the ``sort_keys`` argument is true. - - (Contributed by Antoine Pitrou in :issue:`7451` and by Raymond Hettinger and - Antoine Pitrou in :issue:`10314`.) - -* Recursive locks (created with the :func:`threading.RLock` API) now benefit - from a C implementation which makes them as fast as regular locks, and between - 10x and 15x faster than their previous pure Python implementation. - - (Contributed by Antoine Pitrou; :issue:`3001`.) - -* The fast-search algorithm in stringlib is now used by the :meth:`split`, - :meth:`rsplit`, :meth:`splitlines` and :meth:`replace` methods on - :class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the - algorithm is also used by :meth:`rfind`, :meth:`rindex`, :meth:`rsplit` and - :meth:`rpartition`. - - (Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.) - - -* Integer to string conversions now work two "digits" at a time, reducing the - number of division and modulo operations. - - (:issue:`6713` by Gawain Bolton, Mark Dickinson, and Victor Stinner.) - -There were several other minor optimizations. Set differencing now runs faster -when one operand is much larger than the other (patch by Andress Bennetts in -:issue:`8685`). The :meth:`array.repeat` method has a faster implementation -(:issue:`1569291` by Alexander Belopolsky). The :class:`BaseHTTPRequestHandler` -has more efficient buffering (:issue:`3709` by Andrew Schaaf). The -:func:`operator.attrgetter` function has been sped-up (:issue:`10160` by -Christos Georgiou). And :class:`ConfigParser` loads multi-line arguments a bit -faster (:issue:`7113` by Łukasz Langa). - - -Unicode -======= - -Python has been updated to `Unicode 6.0.0 -<http://unicode.org/versions/Unicode6.0.0/>`_. The update to the standard adds -over 2,000 new characters including `emoji <https://en.wikipedia.org/wiki/Emoji>`_ -symbols which are important for mobile phones. - -In addition, the updated standard has altered the character properties for two -Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character -(U+19DA), making the former eligible for use in identifiers while disqualifying -the latter. For more information, see `Unicode Character Database Changes -<http://www.unicode.org/versions/Unicode6.0.0/#Database_Changes>`_. - - -Codecs -====== - -Support was added for *cp720* Arabic DOS encoding (:issue:`1616979`). - -MBCS encoding no longer ignores the error handler argument. In the default -strict mode, it raises an :exc:`UnicodeDecodeError` when it encounters an -undecodable byte sequence and an :exc:`UnicodeEncodeError` for an unencodable -character. - -The MBCS codec supports ``'strict'`` and ``'ignore'`` error handlers for -decoding, and ``'strict'`` and ``'replace'`` for encoding. - -To emulate Python3.1 MBCS encoding, select the ``'ignore'`` handler for decoding -and the ``'replace'`` handler for encoding. - -On Mac OS X, Python decodes command line arguments with ``'utf-8'`` rather than -the locale encoding. - -By default, :mod:`tarfile` uses ``'utf-8'`` encoding on Windows (instead of -``'mbcs'``) and the ``'surrogateescape'`` error handler on all operating -systems. - - -Documentation -============= - -The documentation continues to be improved. - -* A table of quick links has been added to the top of lengthy sections such as - :ref:`built-in-funcs`. In the case of :mod:`itertools`, the links are - accompanied by tables of cheatsheet-style summaries to provide an overview and - memory jog without having to read all of the docs. - -* In some cases, the pure Python source code can be a helpful adjunct to the - documentation, so now many modules now feature quick links to the latest - version of the source code. For example, the :mod:`functools` module - documentation has a quick link at the top labeled: - - **Source code** :source:`Lib/functools.py`. - - (Contributed by Raymond Hettinger; see - `rationale <https://rhettinger.wordpress.com/2011/01/28/open-your-source-more/>`_.) - -* The docs now contain more examples and recipes. In particular, :mod:`re` - module has an extensive section, :ref:`re-examples`. Likewise, the - :mod:`itertools` module continues to be updated with new - :ref:`itertools-recipes`. - -* The :mod:`datetime` module now has an auxiliary implementation in pure Python. - No functionality was changed. This just provides an easier-to-read alternate - implementation. - - (Contributed by Alexander Belopolsky in :issue:`9528`.) - -* The unmaintained :file:`Demo` directory has been removed. Some demos were - integrated into the documentation, some were moved to the :file:`Tools/demo` - directory, and others were removed altogether. - - (Contributed by Georg Brandl in :issue:`7962`.) - - -IDLE -==== - -* The format menu now has an option to clean source files by stripping - trailing whitespace. - - (Contributed by Raymond Hettinger; :issue:`5150`.) - -* IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa AquaTk. - - (Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren; :issue:`6075`.) - -Code Repository -=============== - -In addition to the existing Subversion code repository at http://svn.python.org -there is now a `Mercurial <https://www.mercurial-scm.org/>`_ repository at -https://hg.python.org/\ . - -After the 3.2 release, there are plans to switch to Mercurial as the primary -repository. This distributed version control system should make it easier for -members of the community to create and share external changesets. See -:pep:`385` for details. - -To learn to use the new version control system, see the `Quick Start -<https://www.mercurial-scm.org/wiki/QuickStart>`_ or the `Guide to -Mercurial Workflows <https://www.mercurial-scm.org/guide>`_. - - -Build and C API Changes -======================= - -Changes to Python's build process and to the C API include: - -* The *idle*, *pydoc* and *2to3* scripts are now installed with a - version-specific suffix on ``make altinstall`` (:issue:`10679`). - -* The C functions that access the Unicode Database now accept and return - characters from the full Unicode range, even on narrow unicode builds - (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference - in Python is that :func:`unicodedata.numeric` now returns the correct value - for large code points, and :func:`repr` may consider more characters as - printable. - - (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.) - -* Computed gotos are now enabled by default on supported compilers (which are - detected by the configure script). They can still be disabled selectively by - specifying ``--without-computed-gotos``. - - (Contributed by Antoine Pitrou; :issue:`9203`.) - -* The option ``--with-wctype-functions`` was removed. The built-in unicode - database is now used for all functions. - - (Contributed by Amaury Forgeot D'Arc; :issue:`9210`.) - -* Hash values are now values of a new type, :c:type:`Py_hash_t`, which is - defined to be the same size as a pointer. Previously they were of type long, - which on some 64-bit operating systems is still only 32 bits long. As a - result of this fix, :class:`set` and :class:`dict` can now hold more than - ``2**32`` entries on builds with 64-bit pointers (previously, they could grow - to that size but their performance degraded catastrophically). - - (Suggested by Raymond Hettinger and implemented by Benjamin Peterson; - :issue:`9778`.) - -* A new macro :c:macro:`Py_VA_COPY` copies the state of the variable argument - list. It is equivalent to C99 *va_copy* but available on all Python platforms - (:issue:`2443`). - -* A new C API function :c:func:`PySys_SetArgvEx` allows an embedded interpreter - to set :attr:`sys.argv` without also modifying :attr:`sys.path` - (:issue:`5753`). - -* :c:macro:`PyEval_CallObject` is now only available in macro form. The - function declaration, which was kept for backwards compatibility reasons, is - now removed -- the macro was introduced in 1997 (:issue:`8276`). - -* There is a new function :c:func:`PyLong_AsLongLongAndOverflow` which - is analogous to :c:func:`PyLong_AsLongAndOverflow`. They both serve to - convert Python :class:`int` into a native fixed-width type while providing - detection of cases where the conversion won't fit (:issue:`7767`). - -* The :c:func:`PyUnicode_CompareWithASCIIString` function now returns *not - equal* if the Python string is *NUL* terminated. - -* There is a new function :c:func:`PyErr_NewExceptionWithDoc` that is - like :c:func:`PyErr_NewException` but allows a docstring to be specified. - This lets C exceptions have the same self-documenting capabilities as - their pure Python counterparts (:issue:`7033`). - -* When compiled with the ``--with-valgrind`` option, the pymalloc - allocator will be automatically disabled when running under Valgrind. This - gives improved memory leak detection when running under Valgrind, while taking - advantage of pymalloc at other times (:issue:`2422`). - -* Removed the ``O?`` format from the *PyArg_Parse* functions. The format is no - longer used and it had never been documented (:issue:`8837`). - -There were a number of other small changes to the C-API. See the -:source:`Misc/NEWS` file for a complete list. - -Also, there were a number of updates to the Mac OS X build, see -:source:`Mac/BuildScript/README.txt` for details. For users running a 32/64-bit -build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6. -Accordingly, we recommend installing an updated alternative such as -`ActiveState Tcl/Tk 8.5.9 <https://www.activestate.com/activetcl/downloads>`_\. -See https://www.python.org/download/mac/tcltk/ for additional details. - -Porting to Python 3.2 -===================== - -This section lists previously described changes and other bugfixes that may -require changes to your code: - -* The :mod:`configparser` module has a number of clean-ups. The major change is - to replace the old :class:`ConfigParser` class with long-standing preferred - alternative :class:`SafeConfigParser`. In addition there are a number of - smaller incompatibilities: - - * The interpolation syntax is now validated on - :meth:`~configparser.ConfigParser.get` and - :meth:`~configparser.ConfigParser.set` operations. In the default - interpolation scheme, only two tokens with percent signs are valid: ``%(name)s`` - and ``%%``, the latter being an escaped percent sign. - - * The :meth:`~configparser.ConfigParser.set` and - :meth:`~configparser.ConfigParser.add_section` methods now verify that - values are actual strings. Formerly, unsupported types could be introduced - unintentionally. - - * Duplicate sections or options from a single source now raise either - :exc:`~configparser.DuplicateSectionError` or - :exc:`~configparser.DuplicateOptionError`. Formerly, duplicates would - silently overwrite a previous entry. - - * Inline comments are now disabled by default so now the **;** character - can be safely used in values. - - * Comments now can be indented. Consequently, for **;** or **#** to appear at - the start of a line in multiline values, it has to be interpolated. This - keeps comment prefix characters in values from being mistaken as comments. - - * ``""`` is now a valid value and is no longer automatically converted to an - empty string. For empty strings, use ``"option ="`` in a line. - -* The :mod:`nntplib` module was reworked extensively, meaning that its APIs - are often incompatible with the 3.1 APIs. - -* :class:`bytearray` objects can no longer be used as filenames; instead, - they should be converted to :class:`bytes`. - -* The :meth:`array.tostring` and :meth:`array.fromstring` have been renamed to - :meth:`array.tobytes` and :meth:`array.frombytes` for clarity. The old names - have been deprecated. (See :issue:`8990`.) - -* ``PyArg_Parse*()`` functions: - - * "t#" format has been removed: use "s#" or "s*" instead - * "w" and "w#" formats has been removed: use "w*" instead - -* The :c:type:`PyCObject` type, deprecated in 3.1, has been removed. To wrap - opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used - instead; the new type has a well-defined interface for passing typing safety - information and a less complicated signature for calling a destructor. - -* The :func:`sys.setfilesystemencoding` function was removed because - it had a flawed design. - -* The :func:`random.seed` function and method now salt string seeds with an - sha512 hash function. To access the previous version of *seed* in order to - reproduce Python 3.1 sequences, set the *version* argument to *1*, - ``random.seed(s, version=1)``. - -* The previously deprecated :func:`string.maketrans` function has been removed - in favor of the static methods :meth:`bytes.maketrans` and - :meth:`bytearray.maketrans`. This change solves the confusion around which - types were supported by the :mod:`string` module. Now, :class:`str`, - :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and - **translate** methods with intermediate translation tables of the appropriate - type. - - (Contributed by Georg Brandl; :issue:`5675`.) - -* The previously deprecated :func:`contextlib.nested` function has been removed - in favor of a plain :keyword:`with` statement which can accept multiple - context managers. The latter technique is faster (because it is built-in), - and it does a better job finalizing multiple context managers when one of them - raises an exception:: - - with open('mylog.txt') as infile, open('a.out', 'w') as outfile: - for line in infile: - if '<critical>' in line: - outfile.write(line) - - (Contributed by Georg Brandl and Mattias Brändström; - `appspot issue 53094 <https://codereview.appspot.com/53094>`_.) - -* :func:`struct.pack` now only allows bytes for the ``s`` string pack code. - Formerly, it would accept text arguments and implicitly encode them to bytes - using UTF-8. This was problematic because it made assumptions about the - correct encoding and because a variable-length encoding can fail when writing - to fixed length segment of a structure. - - Code such as ``struct.pack('<6sHHBBB', 'GIF87a', x, y)`` should be rewritten - with to use bytes instead of text, ``struct.pack('<6sHHBBB', b'GIF87a', x, y)``. - - (Discovered by David Beazley and fixed by Victor Stinner; :issue:`10783`.) - -* The :class:`xml.etree.ElementTree` class now raises an - :exc:`xml.etree.ElementTree.ParseError` when a parse fails. Previously it - raised an :exc:`xml.parsers.expat.ExpatError`. - -* The new, longer :func:`str` value on floats may break doctests which rely on - the old output format. - -* In :class:`subprocess.Popen`, the default value for *close_fds* is now - ``True`` under Unix; under Windows, it is ``True`` if the three standard - streams are set to ``None``, ``False`` otherwise. Previously, *close_fds* - was always ``False`` by default, which produced difficult to solve bugs - or race conditions when open file descriptors would leak into the child - process. - -* Support for legacy HTTP 0.9 has been removed from :mod:`urllib.request` - and :mod:`http.client`. Such support is still present on the server side - (in :mod:`http.server`). - - (Contributed by Antoine Pitrou, :issue:`10711`.) - -* SSL sockets in timeout mode now raise :exc:`socket.timeout` when a timeout - occurs, rather than a generic :exc:`~ssl.SSLError`. - - (Contributed by Antoine Pitrou, :issue:`10272`.) - -* The misleading functions :c:func:`PyEval_AcquireLock()` and - :c:func:`PyEval_ReleaseLock()` have been officially deprecated. The - thread-state aware APIs (such as :c:func:`PyEval_SaveThread()` - and :c:func:`PyEval_RestoreThread()`) should be used instead. - -* Due to security risks, :func:`asyncore.handle_accept` has been deprecated, and - a new function, :func:`asyncore.handle_accepted`, was added to replace it. - - (Contributed by Giampaolo Rodola in :issue:`6706`.) - -* Due to the new :term:`GIL` implementation, :c:func:`PyEval_InitThreads()` - cannot be called before :c:func:`Py_Initialize()` anymore. diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst deleted file mode 100644 index f1a033c..0000000 --- a/Doc/whatsnew/3.3.rst +++ /dev/null @@ -1,2517 +0,0 @@ -**************************** - What's New In Python 3.3 -**************************** - -.. Rules for maintenance: - - * Anyone can add text to this document. Do not spend very much time - on the wording of your changes, because your text will probably - get rewritten to some degree. - - * The maintainer will go through Misc/NEWS periodically and add - changes; it's therefore more important to add your changes to - Misc/NEWS than to this file. - - * This is not a complete list of every single change; completeness - is the purpose of Misc/NEWS. Some changes I consider too small - or esoteric to include. If such a change is added to the text, - I'll just remove it. (This is another reason you shouldn't spend - too much time on writing your addition.) - - * If you want to draw your new text to the attention of the - maintainer, add 'XXX' to the beginning of the paragraph or - section. - - * It's OK to just add a fragmentary note about a change. For - example: "XXX Describe the transmogrify() function added to the - socket module." The maintainer will research the change and - write the necessary text. - - * You can comment out your additions if you like, but it's not - necessary (especially when a final release is some months away). - - * Credit the author of a patch or bugfix. Just the name is - sufficient; the e-mail address isn't necessary. - - * It's helpful to add the bug/patch number as a comment: - - XXX Describe the transmogrify() function added to the socket - module. - (Contributed by P.Y. Developer in :issue:`12345`.) - - This saves the maintainer the effort of going through the Mercurial log - when researching a change. - -This article explains the new features in Python 3.3, compared to 3.2. -Python 3.3 was released on September 29, 2012. For full details, -see the `changelog <https://docs.python.org/3.3/whatsnew/changelog.html>`_. - -.. seealso:: - - :pep:`398` - Python 3.3 Release Schedule - - -Summary -- Release highlights -============================= - -.. This section singles out the most important changes in Python 3.3. - Brevity is key. - -New syntax features: - -* New ``yield from`` expression for :ref:`generator delegation <pep-380>`. -* The ``u'unicode'`` syntax is accepted again for :class:`str` objects. - -New library modules: - -* :mod:`faulthandler` (helps debugging low-level crashes) -* :mod:`ipaddress` (high-level objects representing IP addresses and masks) -* :mod:`lzma` (compress data using the XZ / LZMA algorithm) -* :mod:`unittest.mock` (replace parts of your system under test with mock objects) -* :mod:`venv` (Python :ref:`virtual environments <pep-405>`, as in the - popular ``virtualenv`` package) - -New built-in features: - -* Reworked :ref:`I/O exception hierarchy <pep-3151>`. - -Implementation improvements: - -* Rewritten :ref:`import machinery <importlib>` based on :mod:`importlib`. -* More compact :ref:`unicode strings <pep-393>`. -* More compact :ref:`attribute dictionaries <pep-412>`. - -Significantly Improved Library Modules: - -* C Accelerator for the :ref:`decimal <new-decimal>` module. -* Better unicode handling in the :ref:`email <new-email>` module - (:term:`provisional <provisional package>`). - -Security improvements: - -* Hash randomization is switched on by default. - -Please read on for a comprehensive list of user-facing changes. - - -.. _pep-405: - -PEP 405: Virtual Environments -============================= - -Virtual environments help create separate Python setups while sharing a -system-wide base install, for ease of maintenance. Virtual environments -have their own set of private site packages (i.e. locally-installed -libraries), and are optionally segregated from the system-wide site -packages. Their concept and implementation are inspired by the popular -``virtualenv`` third-party package, but benefit from tighter integration -with the interpreter core. - -This PEP adds the :mod:`venv` module for programmatic access, and the -``pyvenv`` script for command-line access and -administration. The Python interpreter checks for a ``pyvenv.cfg``, -file whose existence signals the base of a virtual environment's directory -tree. - -.. seealso:: - - :pep:`405` - Python Virtual Environments - PEP written by Carl Meyer; implementation by Carl Meyer and Vinay Sajip - - -PEP 420: Implicit Namespace Packages -==================================== - -Native support for package directories that don't require ``__init__.py`` -marker files and can automatically span multiple path segments (inspired by -various third party approaches to namespace packages, as described in -:pep:`420`) - -.. seealso:: - - :pep:`420` - Implicit Namespace Packages - PEP written by Eric V. Smith; implementation by Eric V. Smith - and Barry Warsaw - - -.. _pep-3118-update: - -PEP 3118: New memoryview implementation and buffer protocol documentation -========================================================================= - -The implementation of :pep:`3118` has been significantly improved. - -The new memoryview implementation comprehensively fixes all ownership and -lifetime issues of dynamically allocated fields in the Py_buffer struct -that led to multiple crash reports. Additionally, several functions that -crashed or returned incorrect results for non-contiguous or multi-dimensional -input have been fixed. - -The memoryview object now has a PEP-3118 compliant getbufferproc() -that checks the consumer's request type. Many new features have been -added, most of them work in full generality for non-contiguous arrays -and arrays with suboffsets. - -The documentation has been updated, clearly spelling out responsibilities -for both exporters and consumers. Buffer request flags are grouped into -basic and compound flags. The memory layout of non-contiguous and -multi-dimensional NumPy-style arrays is explained. - -Features --------- - -* All native single character format specifiers in struct module syntax - (optionally prefixed with '@') are now supported. - -* With some restrictions, the cast() method allows changing of format and - shape of C-contiguous arrays. - -* Multi-dimensional list representations are supported for any array type. - -* Multi-dimensional comparisons are supported for any array type. - -* One-dimensional memoryviews of hashable (read-only) types with formats B, - b or c are now hashable. (Contributed by Antoine Pitrou in :issue:`13411`.) - -* Arbitrary slicing of any 1-D arrays type is supported. For example, it - is now possible to reverse a memoryview in O(1) by using a negative step. - -API changes ------------ - -* The maximum number of dimensions is officially limited to 64. - -* The representation of empty shape, strides and suboffsets is now - an empty tuple instead of ``None``. - -* Accessing a memoryview element with format 'B' (unsigned bytes) - now returns an integer (in accordance with the struct module syntax). - For returning a bytes object the view must be cast to 'c' first. - -* memoryview comparisons now use the logical structure of the operands - and compare all array elements by value. All format strings in struct - module syntax are supported. Views with unrecognised format strings - are still permitted, but will always compare as unequal, regardless - of view contents. - -* For further changes see `Build and C API Changes`_ and `Porting C code`_. - -(Contributed by Stefan Krah in :issue:`10181`.) - -.. seealso:: - - :pep:`3118` - Revising the Buffer Protocol - - -.. _pep-393: - -PEP 393: Flexible String Representation -======================================= - -The Unicode string type is changed to support multiple internal -representations, depending on the character with the largest Unicode ordinal -(1, 2, or 4 bytes) in the represented string. This allows a space-efficient -representation in common cases, but gives access to full UCS-4 on all -systems. For compatibility with existing APIs, several representations may -exist in parallel; over time, this compatibility should be phased out. - -On the Python side, there should be no downside to this change. - -On the C API side, :pep:`393` is fully backward compatible. The legacy API -should remain available at least five years. Applications using the legacy -API will not fully benefit of the memory reduction, or - worse - may use -a bit more memory, because Python may have to maintain two versions of each -string (in the legacy format and in the new efficient storage). - -Functionality -------------- - -Changes introduced by :pep:`393` are the following: - -* Python now always supports the full range of Unicode code points, including - non-BMP ones (i.e. from ``U+0000`` to ``U+10FFFF``). The distinction between - narrow and wide builds no longer exists and Python now behaves like a wide - build, even under Windows. - -* With the death of narrow builds, the problems specific to narrow builds have - also been fixed, for example: - - * :func:`len` now always returns 1 for non-BMP characters, - so ``len('\U0010FFFF') == 1``; - - * surrogate pairs are not recombined in string literals, - so ``'\uDBFF\uDFFF' != '\U0010FFFF'``; - - * indexing or slicing non-BMP characters returns the expected value, - so ``'\U0010FFFF'[0]`` now returns ``'\U0010FFFF'`` and not ``'\uDBFF'``; - - * all other functions in the standard library now correctly handle - non-BMP code points. - -* The value of :data:`sys.maxunicode` is now always ``1114111`` (``0x10FFFF`` - in hexadecimal). The :c:func:`PyUnicode_GetMax` function still returns - either ``0xFFFF`` or ``0x10FFFF`` for backward compatibility, and it should - not be used with the new Unicode API (see :issue:`13054`). - -* The :file:`./configure` flag ``--with-wide-unicode`` has been removed. - -Performance and resource usage ------------------------------- - -The storage of Unicode strings now depends on the highest code point in the string: - -* pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per code point; - -* BMP strings (``U+0000-U+FFFF``) use 2 bytes per code point; - -* non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per code point. - -The net effect is that for most applications, memory usage of string -storage should decrease significantly - especially compared to former -wide unicode builds - as, in many cases, strings will be pure ASCII -even in international contexts (because many strings store non-human -language data, such as XML fragments, HTTP headers, JSON-encoded data, -etc.). We also hope that it will, for the same reasons, increase CPU -cache efficiency on non-trivial applications. The memory usage of -Python 3.3 is two to three times smaller than Python 3.2, and a little -bit better than Python 2.7, on a Django benchmark (see the PEP for -details). - -.. seealso:: - - :pep:`393` - Flexible String Representation - PEP written by Martin von Löwis; implementation by Torsten Becker - and Martin von Löwis. - - -.. _pep-397: - -PEP 397: Python Launcher for Windows -==================================== - -The Python 3.3 Windows installer now includes a ``py`` launcher application -that can be used to launch Python applications in a version independent -fashion. - -This launcher is invoked implicitly when double-clicking ``*.py`` files. -If only a single Python version is installed on the system, that version -will be used to run the file. If multiple versions are installed, the most -recent version is used by default, but this can be overridden by including -a Unix-style "shebang line" in the Python script. - -The launcher can also be used explicitly from the command line as the ``py`` -application. Running ``py`` follows the same version selection rules as -implicitly launching scripts, but a more specific version can be selected -by passing appropriate arguments (such as ``-3`` to request Python 3 when -Python 2 is also installed, or ``-2.6`` to specifically request an earlier -Python version when a more recent version is installed). - -In addition to the launcher, the Windows installer now includes an -option to add the newly installed Python to the system PATH. (Contributed -by Brian Curtin in :issue:`3561`.) - -.. seealso:: - - :pep:`397` - Python Launcher for Windows - PEP written by Mark Hammond and Martin v. Löwis; implementation by - Vinay Sajip. - - Launcher documentation: :ref:`launcher` - - Installer PATH modification: :ref:`windows-path-mod` - - -.. _pep-3151: - -PEP 3151: Reworking the OS and IO exception hierarchy -===================================================== - -The hierarchy of exceptions raised by operating system errors is now both -simplified and finer-grained. - -You don't have to worry anymore about choosing the appropriate exception -type between :exc:`OSError`, :exc:`IOError`, :exc:`EnvironmentError`, -:exc:`WindowsError`, :exc:`mmap.error`, :exc:`socket.error` or -:exc:`select.error`. All these exception types are now only one: -:exc:`OSError`. The other names are kept as aliases for compatibility -reasons. - -Also, it is now easier to catch a specific error condition. Instead of -inspecting the ``errno`` attribute (or ``args[0]``) for a particular -constant from the :mod:`errno` module, you can catch the adequate -:exc:`OSError` subclass. The available subclasses are the following: - -* :exc:`BlockingIOError` -* :exc:`ChildProcessError` -* :exc:`ConnectionError` -* :exc:`FileExistsError` -* :exc:`FileNotFoundError` -* :exc:`InterruptedError` -* :exc:`IsADirectoryError` -* :exc:`NotADirectoryError` -* :exc:`PermissionError` -* :exc:`ProcessLookupError` -* :exc:`TimeoutError` - -And the :exc:`ConnectionError` itself has finer-grained subclasses: - -* :exc:`BrokenPipeError` -* :exc:`ConnectionAbortedError` -* :exc:`ConnectionRefusedError` -* :exc:`ConnectionResetError` - -Thanks to the new exceptions, common usages of the :mod:`errno` can now be -avoided. For example, the following code written for Python 3.2:: - - from errno import ENOENT, EACCES, EPERM - - try: - with open("document.txt") as f: - content = f.read() - except IOError as err: - if err.errno == ENOENT: - print("document.txt file is missing") - elif err.errno in (EACCES, EPERM): - print("You are not allowed to read document.txt") - else: - raise - -can now be written without the :mod:`errno` import and without manual -inspection of exception attributes:: - - try: - with open("document.txt") as f: - content = f.read() - except FileNotFoundError: - print("document.txt file is missing") - except PermissionError: - print("You are not allowed to read document.txt") - -.. seealso:: - - :pep:`3151` - Reworking the OS and IO Exception Hierarchy - PEP written and implemented by Antoine Pitrou - - -.. index:: - single: yield; yield from (in What's New) - -.. _pep-380: - -PEP 380: Syntax for Delegating to a Subgenerator -================================================ - -PEP 380 adds the ``yield from`` expression, allowing a :term:`generator` to -delegate -part of its operations to another generator. This allows a section of code -containing :keyword:`yield` to be factored out and placed in another generator. -Additionally, the subgenerator is allowed to return with a value, and the -value is made available to the delegating generator. - -While designed primarily for use in delegating to a subgenerator, the ``yield -from`` expression actually allows delegation to arbitrary subiterators. - -For simple iterators, ``yield from iterable`` is essentially just a shortened -form of ``for item in iterable: yield item``:: - - >>> def g(x): - ... yield from range(x, 0, -1) - ... yield from range(x) - ... - >>> list(g(5)) - [5, 4, 3, 2, 1, 0, 1, 2, 3, 4] - -However, unlike an ordinary loop, ``yield from`` allows subgenerators to -receive sent and thrown values directly from the calling scope, and -return a final value to the outer generator:: - - >>> def accumulate(): - ... tally = 0 - ... while 1: - ... next = yield - ... if next is None: - ... return tally - ... tally += next - ... - >>> def gather_tallies(tallies): - ... while 1: - ... tally = yield from accumulate() - ... tallies.append(tally) - ... - >>> tallies = [] - >>> acc = gather_tallies(tallies) - >>> next(acc) # Ensure the accumulator is ready to accept values - >>> for i in range(4): - ... acc.send(i) - ... - >>> acc.send(None) # Finish the first tally - >>> for i in range(5): - ... acc.send(i) - ... - >>> acc.send(None) # Finish the second tally - >>> tallies - [6, 10] - -The main principle driving this change is to allow even generators that are -designed to be used with the ``send`` and ``throw`` methods to be split into -multiple subgenerators as easily as a single large function can be split into -multiple subfunctions. - -.. seealso:: - - :pep:`380` - Syntax for Delegating to a Subgenerator - PEP written by Greg Ewing; implementation by Greg Ewing, integrated into - 3.3 by Renaud Blanch, Ryan Kelly and Nick Coghlan; documentation by - Zbigniew Jędrzejewski-Szmek and Nick Coghlan - - -PEP 409: Suppressing exception context -====================================== - -PEP 409 introduces new syntax that allows the display of the chained -exception context to be disabled. This allows cleaner error messages in -applications that convert between exception types:: - - >>> class D: - ... def __init__(self, extra): - ... self._extra_attributes = extra - ... def __getattr__(self, attr): - ... try: - ... return self._extra_attributes[attr] - ... except KeyError: - ... raise AttributeError(attr) from None - ... - >>> D({}).x - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - File "<stdin>", line 8, in __getattr__ - AttributeError: x - -Without the ``from None`` suffix to suppress the cause, the original -exception would be displayed by default:: - - >>> class C: - ... def __init__(self, extra): - ... self._extra_attributes = extra - ... def __getattr__(self, attr): - ... try: - ... return self._extra_attributes[attr] - ... except KeyError: - ... raise AttributeError(attr) - ... - >>> C({}).x - Traceback (most recent call last): - File "<stdin>", line 6, in __getattr__ - KeyError: 'x' - - During handling of the above exception, another exception occurred: - - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - File "<stdin>", line 8, in __getattr__ - AttributeError: x - -No debugging capability is lost, as the original exception context remains -available if needed (for example, if an intervening library has incorrectly -suppressed valuable underlying details):: - - >>> try: - ... D({}).x - ... except AttributeError as exc: - ... print(repr(exc.__context__)) - ... - KeyError('x',) - -.. seealso:: - - :pep:`409` - Suppressing exception context - PEP written by Ethan Furman; implemented by Ethan Furman and Nick - Coghlan. - - -PEP 414: Explicit Unicode literals -====================================== - -To ease the transition from Python 2 for Unicode aware Python applications -that make heavy use of Unicode literals, Python 3.3 once again supports the -"``u``" prefix for string literals. This prefix has no semantic significance -in Python 3, it is provided solely to reduce the number of purely mechanical -changes in migrating to Python 3, making it easier for developers to focus on -the more significant semantic changes (such as the stricter default -separation of binary and text data). - -.. seealso:: - - :pep:`414` - Explicit Unicode literals - PEP written by Armin Ronacher. - - -PEP 3155: Qualified name for classes and functions -================================================== - -Functions and class objects have a new ``__qualname__`` attribute representing -the "path" from the module top-level to their definition. For global functions -and classes, this is the same as ``__name__``. For other functions and classes, -it provides better information about where they were actually defined, and -how they might be accessible from the global scope. - -Example with (non-bound) methods:: - - >>> class C: - ... def meth(self): - ... pass - >>> C.meth.__name__ - 'meth' - >>> C.meth.__qualname__ - 'C.meth' - -Example with nested classes:: - - >>> class C: - ... class D: - ... def meth(self): - ... pass - ... - >>> C.D.__name__ - 'D' - >>> C.D.__qualname__ - 'C.D' - >>> C.D.meth.__name__ - 'meth' - >>> C.D.meth.__qualname__ - 'C.D.meth' - -Example with nested functions:: - - >>> def outer(): - ... def inner(): - ... pass - ... return inner - ... - >>> outer().__name__ - 'inner' - >>> outer().__qualname__ - 'outer.<locals>.inner' - -The string representation of those objects is also changed to include the -new, more precise information:: - - >>> str(C.D) - "<class '__main__.C.D'>" - >>> str(C.D.meth) - '<function C.D.meth at 0x7f46b9fe31e0>' - -.. seealso:: - - :pep:`3155` - Qualified name for classes and functions - PEP written and implemented by Antoine Pitrou. - - -.. _pep-412: - -PEP 412: Key-Sharing Dictionary -=============================== - -Dictionaries used for the storage of objects' attributes are now able to -share part of their internal storage between each other (namely, the part -which stores the keys and their respective hashes). This reduces the memory -consumption of programs creating many instances of non-builtin types. - -.. seealso:: - - :pep:`412` - Key-Sharing Dictionary - PEP written and implemented by Mark Shannon. - - -PEP 362: Function Signature Object -================================== - -A new function :func:`inspect.signature` makes introspection of python -callables easy and straightforward. A broad range of callables is supported: -python functions, decorated or not, classes, and :func:`functools.partial` -objects. New classes :class:`inspect.Signature`, :class:`inspect.Parameter` -and :class:`inspect.BoundArguments` hold information about the call signatures, -such as, annotations, default values, parameters kinds, and bound arguments, -which considerably simplifies writing decorators and any code that validates -or amends calling signatures or arguments. - -.. seealso:: - - :pep:`362`: - Function Signature Object - PEP written by Brett Cannon, Yury Selivanov, Larry Hastings, Jiwon Seo; - implemented by Yury Selivanov. - - -PEP 421: Adding sys.implementation -================================== - -A new attribute on the :mod:`sys` module exposes details specific to the -implementation of the currently running interpreter. The initial set of -attributes on :attr:`sys.implementation` are ``name``, ``version``, -``hexversion``, and ``cache_tag``. - -The intention of ``sys.implementation`` is to consolidate into one namespace -the implementation-specific data used by the standard library. This allows -different Python implementations to share a single standard library code base -much more easily. In its initial state, ``sys.implementation`` holds only a -small portion of the implementation-specific data. Over time that ratio will -shift in order to make the standard library more portable. - -One example of improved standard library portability is ``cache_tag``. As of -Python 3.3, ``sys.implementation.cache_tag`` is used by :mod:`importlib` to -support :pep:`3147` compliance. Any Python implementation that uses -``importlib`` for its built-in import system may use ``cache_tag`` to control -the caching behavior for modules. - -SimpleNamespace ---------------- - -The implementation of ``sys.implementation`` also introduces a new type to -Python: :class:`types.SimpleNamespace`. In contrast to a mapping-based -namespace, like :class:`dict`, ``SimpleNamespace`` is attribute-based, like -:class:`object`. However, unlike ``object``, ``SimpleNamespace`` instances -are writable. This means that you can add, remove, and modify the namespace -through normal attribute access. - -.. seealso:: - - :pep:`421` - Adding sys.implementation - PEP written and implemented by Eric Snow. - - -.. _importlib: - -Using importlib as the Implementation of Import -=============================================== -:issue:`2377` - Replace __import__ w/ importlib.__import__ -:issue:`13959` - Re-implement parts of :mod:`imp` in pure Python -:issue:`14605` - Make import machinery explicit -:issue:`14646` - Require loaders set __loader__ and __package__ - -The :func:`__import__` function is now powered by :func:`importlib.__import__`. -This work leads to the completion of "phase 2" of :pep:`302`. There are -multiple benefits to this change. First, it has allowed for more of the -machinery powering import to be exposed instead of being implicit and hidden -within the C code. It also provides a single implementation for all Python VMs -supporting Python 3.3 to use, helping to end any VM-specific deviations in -import semantics. And finally it eases the maintenance of import, allowing for -future growth to occur. - -For the common user, there should be no visible change in semantics. For -those whose code currently manipulates import or calls import -programmatically, the code changes that might possibly be required are covered -in the `Porting Python code`_ section of this document. - -New APIs --------- -One of the large benefits of this work is the exposure of what goes into -making the import statement work. That means the various importers that were -once implicit are now fully exposed as part of the :mod:`importlib` package. - -The abstract base classes defined in :mod:`importlib.abc` have been expanded -to properly delineate between :term:`meta path finders <meta path finder>` -and :term:`path entry finders <path entry finder>` by introducing -:class:`importlib.abc.MetaPathFinder` and -:class:`importlib.abc.PathEntryFinder`, respectively. The old ABC of -:class:`importlib.abc.Finder` is now only provided for backwards-compatibility -and does not enforce any method requirements. - -In terms of finders, :class:`importlib.machinery.FileFinder` exposes the -mechanism used to search for source and bytecode files of a module. Previously -this class was an implicit member of :attr:`sys.path_hooks`. - -For loaders, the new abstract base class :class:`importlib.abc.FileLoader` helps -write a loader that uses the file system as the storage mechanism for a module's -code. The loader for source files -(:class:`importlib.machinery.SourceFileLoader`), sourceless bytecode files -(:class:`importlib.machinery.SourcelessFileLoader`), and extension modules -(:class:`importlib.machinery.ExtensionFileLoader`) are now available for -direct use. - -:exc:`ImportError` now has ``name`` and ``path`` attributes which are set when -there is relevant data to provide. The message for failed imports will also -provide the full name of the module now instead of just the tail end of the -module's name. - -The :func:`importlib.invalidate_caches` function will now call the method with -the same name on all finders cached in :attr:`sys.path_importer_cache` to help -clean up any stored state as necessary. - -Visible Changes ---------------- - -For potential required changes to code, see the `Porting Python code`_ -section. - -Beyond the expanse of what :mod:`importlib` now exposes, there are other -visible changes to import. The biggest is that :attr:`sys.meta_path` and -:attr:`sys.path_hooks` now store all of the meta path finders and path entry -hooks used by import. Previously the finders were implicit and hidden within -the C code of import instead of being directly exposed. This means that one can -now easily remove or change the order of the various finders to fit one's needs. - -Another change is that all modules have a ``__loader__`` attribute, storing the -loader used to create the module. :pep:`302` has been updated to make this -attribute mandatory for loaders to implement, so in the future once 3rd-party -loaders have been updated people will be able to rely on the existence of the -attribute. Until such time, though, import is setting the module post-load. - -Loaders are also now expected to set the ``__package__`` attribute from -:pep:`366`. Once again, import itself is already setting this on all loaders -from :mod:`importlib` and import itself is setting the attribute post-load. - -``None`` is now inserted into :attr:`sys.path_importer_cache` when no finder -can be found on :attr:`sys.path_hooks`. Since :class:`imp.NullImporter` is not -directly exposed on :attr:`sys.path_hooks` it could no longer be relied upon to -always be available to use as a value representing no finder found. - -All other changes relate to semantic changes which should be taken into -consideration when updating code for Python 3.3, and thus should be read about -in the `Porting Python code`_ section of this document. - -(Implementation by Brett Cannon) - - -Other Language Changes -====================== - -Some smaller changes made to the core Python language are: - -* Added support for Unicode name aliases and named sequences. - Both :func:`unicodedata.lookup()` and ``'\N{...}'`` now resolve name aliases, - and :func:`unicodedata.lookup()` resolves named sequences too. - - (Contributed by Ezio Melotti in :issue:`12753`.) - -* Unicode database updated to UCD version 6.1.0 - -* Equality comparisons on :func:`range` objects now return a result reflecting - the equality of the underlying sequences generated by those range objects. - (:issue:`13201`) - -* The ``count()``, ``find()``, ``rfind()``, ``index()`` and ``rindex()`` - methods of :class:`bytes` and :class:`bytearray` objects now accept an - integer between 0 and 255 as their first argument. - - (Contributed by Petri Lehtinen in :issue:`12170`.) - -* The ``rjust()``, ``ljust()``, and ``center()`` methods of :class:`bytes` - and :class:`bytearray` now accept a :class:`bytearray` for the ``fill`` - argument. (Contributed by Petri Lehtinen in :issue:`12380`.) - -* New methods have been added to :class:`list` and :class:`bytearray`: - ``copy()`` and ``clear()`` (:issue:`10516`). Consequently, - :class:`~collections.abc.MutableSequence` now also defines a - :meth:`~collections.abc.MutableSequence.clear` method (:issue:`11388`). - -* Raw bytes literals can now be written ``rb"..."`` as well as ``br"..."``. - - (Contributed by Antoine Pitrou in :issue:`13748`.) - -* :meth:`dict.setdefault` now does only one lookup for the given key, making - it atomic when used with built-in types. - - (Contributed by Filip Gruszczyński in :issue:`13521`.) - -* The error messages produced when a function call does not match the function - signature have been significantly improved. - - (Contributed by Benjamin Peterson.) - - -A Finer-Grained Import Lock -=========================== - -Previous versions of CPython have always relied on a global import lock. -This led to unexpected annoyances, such as deadlocks when importing a module -would trigger code execution in a different thread as a side-effect. -Clumsy workarounds were sometimes employed, such as the -:c:func:`PyImport_ImportModuleNoBlock` C API function. - -In Python 3.3, importing a module takes a per-module lock. This correctly -serializes importation of a given module from multiple threads (preventing -the exposure of incompletely initialized modules), while eliminating the -aforementioned annoyances. - -(Contributed by Antoine Pitrou in :issue:`9260`.) - - -Builtin functions and types -=========================== - -* :func:`open` gets a new *opener* parameter: the underlying file descriptor - for the file object is then obtained by calling *opener* with (*file*, - *flags*). It can be used to use custom flags like :data:`os.O_CLOEXEC` for - example. The ``'x'`` mode was added: open for exclusive creation, failing if - the file already exists. -* :func:`print`: added the *flush* keyword argument. If the *flush* keyword - argument is true, the stream is forcibly flushed. -* :func:`hash`: hash randomization is enabled by default, see - :meth:`object.__hash__` and :envvar:`PYTHONHASHSEED`. -* The :class:`str` type gets a new :meth:`~str.casefold` method: return a - casefolded copy of the string, casefolded strings may be used for caseless - matching. For example, ``'ß'.casefold()`` returns ``'ss'``. -* The sequence documentation has been substantially rewritten to better - explain the binary/text sequence distinction and to provide specific - documentation sections for the individual builtin sequence types - (:issue:`4966`). - - -New Modules -=========== - -faulthandler ------------- - -This new debug module :mod:`faulthandler` contains functions to dump Python tracebacks explicitly, -on a fault (a crash like a segmentation fault), after a timeout, or on a user -signal. Call :func:`faulthandler.enable` to install fault handlers for the -:const:`SIGSEGV`, :const:`SIGFPE`, :const:`SIGABRT`, :const:`SIGBUS`, and -:const:`SIGILL` signals. You can also enable them at startup by setting the -:envvar:`PYTHONFAULTHANDLER` environment variable or by using :option:`-X` -``faulthandler`` command line option. - -Example of a segmentation fault on Linux: - -.. code-block:: shell-session - - $ python -q -X faulthandler - >>> import ctypes - >>> ctypes.string_at(0) - Fatal Python error: Segmentation fault - - Current thread 0x00007fb899f39700: - File "/home/python/cpython/Lib/ctypes/__init__.py", line 486 in string_at - File "<stdin>", line 1 in <module> - Segmentation fault - - -ipaddress ---------- - -The new :mod:`ipaddress` module provides tools for creating and manipulating -objects representing IPv4 and IPv6 addresses, networks and interfaces (i.e. -an IP address associated with a specific IP subnet). - -(Contributed by Google and Peter Moody in :pep:`3144`.) - -lzma ----- - -The newly-added :mod:`lzma` module provides data compression and decompression -using the LZMA algorithm, including support for the ``.xz`` and ``.lzma`` -file formats. - -(Contributed by Nadeem Vawda and Per Øyvind Karlsen in :issue:`6715`.) - - -Improved Modules -================ - -abc ---- - -Improved support for abstract base classes containing descriptors composed with -abstract methods. The recommended approach to declaring abstract descriptors is -now to provide :attr:`__isabstractmethod__` as a dynamically updated -property. The built-in descriptors have been updated accordingly. - - * :class:`abc.abstractproperty` has been deprecated, use :class:`property` - with :func:`abc.abstractmethod` instead. - * :class:`abc.abstractclassmethod` has been deprecated, use - :class:`classmethod` with :func:`abc.abstractmethod` instead. - * :class:`abc.abstractstaticmethod` has been deprecated, use - :class:`staticmethod` with :func:`abc.abstractmethod` instead. - -(Contributed by Darren Dale in :issue:`11610`.) - -:meth:`abc.ABCMeta.register` now returns the registered subclass, which means -it can now be used as a class decorator (:issue:`10868`). - - -array ------ - -The :mod:`array` module supports the :c:type:`long long` type using ``q`` and -``Q`` type codes. - -(Contributed by Oren Tirosh and Hirokazu Yamamoto in :issue:`1172711`.) - - -base64 ------- - -ASCII-only Unicode strings are now accepted by the decoding functions of the -:mod:`base64` modern interface. For example, ``base64.b64decode('YWJj')`` -returns ``b'abc'``. (Contributed by Catalin Iacob in :issue:`13641`.) - - -binascii --------- - -In addition to the binary objects they normally accept, the ``a2b_`` functions -now all also accept ASCII-only strings as input. (Contributed by Antoine -Pitrou in :issue:`13637`.) - - -bz2 ---- - -The :mod:`bz2` module has been rewritten from scratch. In the process, several -new features have been added: - -* New :func:`bz2.open` function: open a bzip2-compressed file in binary or - text mode. - -* :class:`bz2.BZ2File` can now read from and write to arbitrary file-like - objects, by means of its constructor's *fileobj* argument. - - (Contributed by Nadeem Vawda in :issue:`5863`.) - -* :class:`bz2.BZ2File` and :func:`bz2.decompress` can now decompress - multi-stream inputs (such as those produced by the :program:`pbzip2` tool). - :class:`bz2.BZ2File` can now also be used to create this type of file, using - the ``'a'`` (append) mode. - - (Contributed by Nir Aides in :issue:`1625`.) - -* :class:`bz2.BZ2File` now implements all of the :class:`io.BufferedIOBase` API, - except for the :meth:`detach` and :meth:`truncate` methods. - - -codecs ------- - -The :mod:`~encodings.mbcs` codec has been rewritten to handle correctly -``replace`` and ``ignore`` error handlers on all Windows versions. The -:mod:`~encodings.mbcs` codec now supports all error handlers, instead of only -``replace`` to encode and ``ignore`` to decode. - -A new Windows-only codec has been added: ``cp65001`` (:issue:`13216`). It is the -Windows code page 65001 (Windows UTF-8, ``CP_UTF8``). For example, it is used -by ``sys.stdout`` if the console output code page is set to cp65001 (e.g., using -``chcp 65001`` command). - -Multibyte CJK decoders now resynchronize faster. They only ignore the first -byte of an invalid byte sequence. For example, ``b'\xff\n'.decode('gb2312', -'replace')`` now returns a ``\n`` after the replacement character. - -(:issue:`12016`) - -Incremental CJK codec encoders are no longer reset at each call to their -encode() methods. For example:: - - >>> import codecs - >>> encoder = codecs.getincrementalencoder('hz')('strict') - >>> b''.join(encoder.encode(x) for x in '\u52ff\u65bd\u65bc\u4eba\u3002 Bye.') - b'~{NpJ)l6HK!#~} Bye.' - -This example gives ``b'~{Np~}~{J)~}~{l6~}~{HK~}~{!#~} Bye.'`` with older Python -versions. - -(:issue:`12100`) - -The ``unicode_internal`` codec has been deprecated. - - -collections ------------ - -Addition of a new :class:`~collections.ChainMap` class to allow treating a -number of mappings as a single unit. (Written by Raymond Hettinger for -:issue:`11089`, made public in :issue:`11297`.) - -The abstract base classes have been moved in a new :mod:`collections.abc` -module, to better differentiate between the abstract and the concrete -collections classes. Aliases for ABCs are still present in the -:mod:`collections` module to preserve existing imports. (:issue:`11085`) - -.. XXX addition of __slots__ to ABCs not recorded here: internal detail - -The :class:`~collections.Counter` class now supports the unary ``+`` and ``-`` -operators, as well as the in-place operators ``+=``, ``-=``, ``|=``, and -``&=``. (Contributed by Raymond Hettinger in :issue:`13121`.) - - -contextlib ----------- - -:class:`~contextlib.ExitStack` now provides a solid foundation for -programmatic manipulation of context managers and similar cleanup -functionality. Unlike the previous ``contextlib.nested`` API (which was -deprecated and removed), the new API is designed to work correctly -regardless of whether context managers acquire their resources in -their ``__init__`` method (for example, file objects) or in their -``__enter__`` method (for example, synchronisation objects from the -:mod:`threading` module). - -(:issue:`13585`) - - -crypt ------ - -Addition of salt and modular crypt format (hashing method) and the :func:`~crypt.mksalt` -function to the :mod:`crypt` module. - -(:issue:`10924`) - -curses ------- - - * If the :mod:`curses` module is linked to the ncursesw library, use Unicode - functions when Unicode strings or characters are passed (e.g. - :c:func:`waddwstr`), and bytes functions otherwise (e.g. :c:func:`waddstr`). - * Use the locale encoding instead of ``utf-8`` to encode Unicode strings. - * :class:`curses.window` has a new :attr:`curses.window.encoding` attribute. - * The :class:`curses.window` class has a new :meth:`~curses.window.get_wch` - method to get a wide character - * The :mod:`curses` module has a new :meth:`~curses.unget_wch` function to - push a wide character so the next :meth:`~curses.window.get_wch` will return - it - -(Contributed by Iñigo Serna in :issue:`6755`.) - -datetime --------- - - * Equality comparisons between naive and aware :class:`~datetime.datetime` - instances now return :const:`False` instead of raising :exc:`TypeError` - (:issue:`15006`). - * New :meth:`datetime.datetime.timestamp` method: Return POSIX timestamp - corresponding to the :class:`~datetime.datetime` instance. - * The :meth:`datetime.datetime.strftime` method supports formatting years - older than 1000. - * The :meth:`datetime.datetime.astimezone` method can now be - called without arguments to convert datetime instance to the system - timezone. - - -.. _new-decimal: - -decimal -------- - -:issue:`7652` - integrate fast native decimal arithmetic. - C-module and libmpdec written by Stefan Krah. - -The new C version of the decimal module integrates the high speed libmpdec -library for arbitrary precision correctly-rounded decimal floating point -arithmetic. libmpdec conforms to IBM's General Decimal Arithmetic Specification. - -Performance gains range from 10x for database applications to 100x for -numerically intensive applications. These numbers are expected gains -for standard precisions used in decimal floating point arithmetic. Since -the precision is user configurable, the exact figures may vary. For example, -in integer bignum arithmetic the differences can be significantly higher. - -The following table is meant as an illustration. Benchmarks are available -at http://www.bytereef.org/mpdecimal/quickstart.html. - - +---------+-------------+--------------+-------------+ - | | decimal.py | _decimal | speedup | - +=========+=============+==============+=============+ - | pi | 42.02s | 0.345s | 120x | - +---------+-------------+--------------+-------------+ - | telco | 172.19s | 5.68s | 30x | - +---------+-------------+--------------+-------------+ - | psycopg | 3.57s | 0.29s | 12x | - +---------+-------------+--------------+-------------+ - -Features -~~~~~~~~ - -* The :exc:`~decimal.FloatOperation` signal optionally enables stricter - semantics for mixing floats and Decimals. - -* If Python is compiled without threads, the C version automatically - disables the expensive thread local context machinery. In this case, - the variable :data:`~decimal.HAVE_THREADS` is set to ``False``. - -API changes -~~~~~~~~~~~ - -* The C module has the following context limits, depending on the machine - architecture: - - +-------------------+---------------------+------------------------------+ - | | 32-bit | 64-bit | - +===================+=====================+==============================+ - | :const:`MAX_PREC` | :const:`425000000` | :const:`999999999999999999` | - +-------------------+---------------------+------------------------------+ - | :const:`MAX_EMAX` | :const:`425000000` | :const:`999999999999999999` | - +-------------------+---------------------+------------------------------+ - | :const:`MIN_EMIN` | :const:`-425000000` | :const:`-999999999999999999` | - +-------------------+---------------------+------------------------------+ - -* In the context templates (:class:`~decimal.DefaultContext`, - :class:`~decimal.BasicContext` and :class:`~decimal.ExtendedContext`) - the magnitude of :attr:`~decimal.Context.Emax` and - :attr:`~decimal.Context.Emin` has changed to :const:`999999`. - -* The :class:`~decimal.Decimal` constructor in decimal.py does not observe - the context limits and converts values with arbitrary exponents or precision - exactly. Since the C version has internal limits, the following scheme is - used: If possible, values are converted exactly, otherwise - :exc:`~decimal.InvalidOperation` is raised and the result is NaN. In the - latter case it is always possible to use :meth:`~decimal.Context.create_decimal` - in order to obtain a rounded or inexact value. - - -* The power function in decimal.py is always correctly-rounded. In the - C version, it is defined in terms of the correctly-rounded - :meth:`~decimal.Decimal.exp` and :meth:`~decimal.Decimal.ln` functions, - but the final result is only "almost always correctly rounded". - - -* In the C version, the context dictionary containing the signals is a - :class:`~collections.abc.MutableMapping`. For speed reasons, - :attr:`~decimal.Context.flags` and :attr:`~decimal.Context.traps` always - refer to the same :class:`~collections.abc.MutableMapping` that the context - was initialized with. If a new signal dictionary is assigned, - :attr:`~decimal.Context.flags` and :attr:`~decimal.Context.traps` - are updated with the new values, but they do not reference the RHS - dictionary. - - -* Pickling a :class:`~decimal.Context` produces a different output in order - to have a common interchange format for the Python and C versions. - - -* The order of arguments in the :class:`~decimal.Context` constructor has been - changed to match the order displayed by :func:`repr`. - - -* The ``watchexp`` parameter in the :meth:`~decimal.Decimal.quantize` method - is deprecated. - - -.. _new-email: - -email ------ - -Policy Framework -~~~~~~~~~~~~~~~~ - -The email package now has a :mod:`~email.policy` framework. A -:class:`~email.policy.Policy` is an object with several methods and properties -that control how the email package behaves. The primary policy for Python 3.3 -is the :class:`~email.policy.Compat32` policy, which provides backward -compatibility with the email package in Python 3.2. A ``policy`` can be -specified when an email message is parsed by a :mod:`~email.parser`, or when a -:class:`~email.message.Message` object is created, or when an email is -serialized using a :mod:`~email.generator`. Unless overridden, a policy passed -to a ``parser`` is inherited by all the ``Message`` object and sub-objects -created by the ``parser``. By default a ``generator`` will use the policy of -the ``Message`` object it is serializing. The default policy is -:data:`~email.policy.compat32`. - -The minimum set of controls implemented by all ``policy`` objects are: - - .. tabularcolumns:: |l|L| - - =============== ======================================================= - max_line_length The maximum length, excluding the linesep character(s), - individual lines may have when a ``Message`` is - serialized. Defaults to 78. - - linesep The character used to separate individual lines when a - ``Message`` is serialized. Defaults to ``\n``. - - cte_type ``7bit`` or ``8bit``. ``8bit`` applies only to a - ``Bytes`` ``generator``, and means that non-ASCII may - be used where allowed by the protocol (or where it - exists in the original input). - - raise_on_defect Causes a ``parser`` to raise error when defects are - encountered instead of adding them to the ``Message`` - object's ``defects`` list. - =============== ======================================================= - -A new policy instance, with new settings, is created using the -:meth:`~email.policy.Policy.clone` method of policy objects. ``clone`` takes -any of the above controls as keyword arguments. Any control not specified in -the call retains its default value. Thus you can create a policy that uses -``\r\n`` linesep characters like this:: - - mypolicy = compat32.clone(linesep='\r\n') - -Policies can be used to make the generation of messages in the format needed by -your application simpler. Instead of having to remember to specify -``linesep='\r\n'`` in all the places you call a ``generator``, you can specify -it once, when you set the policy used by the ``parser`` or the ``Message``, -whichever your program uses to create ``Message`` objects. On the other hand, -if you need to generate messages in multiple forms, you can still specify the -parameters in the appropriate ``generator`` call. Or you can have custom -policy instances for your different cases, and pass those in when you create -the ``generator``. - - -Provisional Policy with New Header API -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -While the policy framework is worthwhile all by itself, the main motivation for -introducing it is to allow the creation of new policies that implement new -features for the email package in a way that maintains backward compatibility -for those who do not use the new policies. Because the new policies introduce a -new API, we are releasing them in Python 3.3 as a :term:`provisional policy -<provisional package>`. Backwards incompatible changes (up to and including -removal of the code) may occur if deemed necessary by the core developers. - -The new policies are instances of :class:`~email.policy.EmailPolicy`, -and add the following additional controls: - - .. tabularcolumns:: |l|L| - - =============== ======================================================= - refold_source Controls whether or not headers parsed by a - :mod:`~email.parser` are refolded by the - :mod:`~email.generator`. It can be ``none``, ``long``, - or ``all``. The default is ``long``, which means that - source headers with a line longer than - ``max_line_length`` get refolded. ``none`` means no - line get refolded, and ``all`` means that all lines - get refolded. - - header_factory A callable that take a ``name`` and ``value`` and - produces a custom header object. - =============== ======================================================= - -The ``header_factory`` is the key to the new features provided by the new -policies. When one of the new policies is used, any header retrieved from -a ``Message`` object is an object produced by the ``header_factory``, and any -time you set a header on a ``Message`` it becomes an object produced by -``header_factory``. All such header objects have a ``name`` attribute equal -to the header name. Address and Date headers have additional attributes -that give you access to the parsed data of the header. This means you can now -do things like this:: - - >>> m = Message(policy=SMTP) - >>> m['To'] = 'Éric <foo@example.com>' - >>> m['to'] - 'Éric <foo@example.com>' - >>> m['to'].addresses - (Address(display_name='Éric', username='foo', domain='example.com'),) - >>> m['to'].addresses[0].username - 'foo' - >>> m['to'].addresses[0].display_name - 'Éric' - >>> m['Date'] = email.utils.localtime() - >>> m['Date'].datetime - datetime.datetime(2012, 5, 25, 21, 39, 24, 465484, tzinfo=datetime.timezone(datetime.timedelta(-1, 72000), 'EDT')) - >>> m['Date'] - 'Fri, 25 May 2012 21:44:27 -0400' - >>> print(m) - To: =?utf-8?q?=C3=89ric?= <foo@example.com> - Date: Fri, 25 May 2012 21:44:27 -0400 - -You will note that the unicode display name is automatically encoded as -``utf-8`` when the message is serialized, but that when the header is accessed -directly, you get the unicode version. This eliminates any need to deal with -the :mod:`email.header` :meth:`~email.header.decode_header` or -:meth:`~email.header.make_header` functions. - -You can also create addresses from parts:: - - >>> m['cc'] = [Group('pals', [Address('Bob', 'bob', 'example.com'), - ... Address('Sally', 'sally', 'example.com')]), - ... Address('Bonzo', addr_spec='bonz@laugh.com')] - >>> print(m) - To: =?utf-8?q?=C3=89ric?= <foo@example.com> - Date: Fri, 25 May 2012 21:44:27 -0400 - cc: pals: Bob <bob@example.com>, Sally <sally@example.com>;, Bonzo <bonz@laugh.com> - -Decoding to unicode is done automatically:: - - >>> m2 = message_from_string(str(m)) - >>> m2['to'] - 'Éric <foo@example.com>' - -When you parse a message, you can use the ``addresses`` and ``groups`` -attributes of the header objects to access the groups and individual -addresses:: - - >>> m2['cc'].addresses - (Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com'), Address(display_name='Bonzo', username='bonz', domain='laugh.com')) - >>> m2['cc'].groups - (Group(display_name='pals', addresses=(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com')), Group(display_name=None, addresses=(Address(display_name='Bonzo', username='bonz', domain='laugh.com'),)) - -In summary, if you use one of the new policies, header manipulation works the -way it ought to: your application works with unicode strings, and the email -package transparently encodes and decodes the unicode to and from the RFC -standard Content Transfer Encodings. - -Other API Changes -~~~~~~~~~~~~~~~~~ - -New :class:`~email.parser.BytesHeaderParser`, added to the :mod:`~email.parser` -module to complement :class:`~email.parser.HeaderParser` and complete the Bytes -API. - -New utility functions: - - * :func:`~email.utils.format_datetime`: given a :class:`~datetime.datetime`, - produce a string formatted for use in an email header. - - * :func:`~email.utils.parsedate_to_datetime`: given a date string from - an email header, convert it into an aware :class:`~datetime.datetime`, - or a naive :class:`~datetime.datetime` if the offset is ``-0000``. - - * :func:`~email.utils.localtime`: With no argument, returns the - current local time as an aware :class:`~datetime.datetime` using the local - :class:`~datetime.timezone`. Given an aware :class:`~datetime.datetime`, - converts it into an aware :class:`~datetime.datetime` using the - local :class:`~datetime.timezone`. - - -ftplib ------- - -* :class:`ftplib.FTP` now accepts a ``source_address`` keyword argument to - specify the ``(host, port)`` to use as the source address in the bind call - when creating the outgoing socket. (Contributed by Giampaolo Rodolà - in :issue:`8594`.) - -* The :class:`~ftplib.FTP_TLS` class now provides a new - :func:`~ftplib.FTP_TLS.ccc` function to revert control channel back to - plaintext. This can be useful to take advantage of firewalls that know how - to handle NAT with non-secure FTP without opening fixed ports. (Contributed - by Giampaolo Rodolà in :issue:`12139`.) - -* Added :meth:`ftplib.FTP.mlsd` method which provides a parsable directory - listing format and deprecates :meth:`ftplib.FTP.nlst` and - :meth:`ftplib.FTP.dir`. (Contributed by Giampaolo Rodolà in :issue:`11072`.) - - -functools ---------- - -The :func:`functools.lru_cache` decorator now accepts a ``typed`` keyword -argument (that defaults to ``False`` to ensure that it caches values of -different types that compare equal in separate cache slots. (Contributed -by Raymond Hettinger in :issue:`13227`.) - - -gc --- - -It is now possible to register callbacks invoked by the garbage collector -before and after collection using the new :data:`~gc.callbacks` list. - - -hmac ----- - -A new :func:`~hmac.compare_digest` function has been added to prevent side -channel attacks on digests through timing analysis. (Contributed by Nick -Coghlan and Christian Heimes in :issue:`15061`.) - - -http ----- - -:class:`http.server.BaseHTTPRequestHandler` now buffers the headers and writes -them all at once when :meth:`~http.server.BaseHTTPRequestHandler.end_headers` is -called. A new method :meth:`~http.server.BaseHTTPRequestHandler.flush_headers` -can be used to directly manage when the accumulated headers are sent. -(Contributed by Andrew Schaaf in :issue:`3709`.) - -:class:`http.server` now produces valid ``HTML 4.01 strict`` output. -(Contributed by Ezio Melotti in :issue:`13295`.) - -:class:`http.client.HTTPResponse` now has a -:meth:`~http.client.HTTPResponse.readinto` method, which means it can be used -as an :class:`io.RawIOBase` class. (Contributed by John Kuhn in -:issue:`13464`.) - - -html ----- - -:class:`html.parser.HTMLParser` is now able to parse broken markup without -raising errors, therefore the *strict* argument of the constructor and the -:exc:`~html.parser.HTMLParseError` exception are now deprecated. -The ability to parse broken markup is the result of a number of bug fixes that -are also available on the latest bug fix releases of Python 2.7/3.2. -(Contributed by Ezio Melotti in :issue:`15114`, and :issue:`14538`, -:issue:`13993`, :issue:`13960`, :issue:`13358`, :issue:`1745761`, -:issue:`755670`, :issue:`13357`, :issue:`12629`, :issue:`1200313`, -:issue:`670664`, :issue:`13273`, :issue:`12888`, :issue:`7311`.) - -A new :data:`~html.entities.html5` dictionary that maps HTML5 named character -references to the equivalent Unicode character(s) (e.g. ``html5['gt;'] == -'>'``) has been added to the :mod:`html.entities` module. The dictionary is -now also used by :class:`~html.parser.HTMLParser`. (Contributed by Ezio -Melotti in :issue:`11113` and :issue:`15156`.) - - -imaplib -------- - -The :class:`~imaplib.IMAP4_SSL` constructor now accepts an SSLContext -parameter to control parameters of the secure channel. - -(Contributed by Sijin Joseph in :issue:`8808`.) - - -inspect -------- - -A new :func:`~inspect.getclosurevars` function has been added. This function -reports the current binding of all names referenced from the function body and -where those names were resolved, making it easier to verify correct internal -state when testing code that relies on stateful closures. - -(Contributed by Meador Inge and Nick Coghlan in :issue:`13062`.) - -A new :func:`~inspect.getgeneratorlocals` function has been added. This -function reports the current binding of local variables in the generator's -stack frame, making it easier to verify correct internal state when testing -generators. - -(Contributed by Meador Inge in :issue:`15153`.) - -io --- - -The :func:`~io.open` function has a new ``'x'`` mode that can be used to -exclusively create a new file, and raise a :exc:`FileExistsError` if the file -already exists. It is based on the C11 'x' mode to fopen(). - -(Contributed by David Townshend in :issue:`12760`.) - -The constructor of the :class:`~io.TextIOWrapper` class has a new -*write_through* optional argument. If *write_through* is ``True``, calls to -:meth:`~io.TextIOWrapper.write` are guaranteed not to be buffered: any data -written on the :class:`~io.TextIOWrapper` object is immediately handled to its -underlying binary buffer. - - -itertools ---------- - -:func:`~itertools.accumulate` now takes an optional ``func`` argument for -providing a user-supplied binary function. - - -logging -------- - -The :func:`~logging.basicConfig` function now supports an optional ``handlers`` -argument taking an iterable of handlers to be added to the root logger. - -A class level attribute :attr:`~logging.handlers.SysLogHandler.append_nul` has -been added to :class:`~logging.handlers.SysLogHandler` to allow control of the -appending of the ``NUL`` (``\000``) byte to syslog records, since for some -daemons it is required while for others it is passed through to the log. - - - -math ----- - -The :mod:`math` module has a new function, :func:`~math.log2`, which returns -the base-2 logarithm of *x*. - -(Written by Mark Dickinson in :issue:`11888`.) - - -mmap ----- - -The :meth:`~mmap.mmap.read` method is now more compatible with other file-like -objects: if the argument is omitted or specified as ``None``, it returns the -bytes from the current file position to the end of the mapping. (Contributed -by Petri Lehtinen in :issue:`12021`.) - - -multiprocessing ---------------- - -The new :func:`multiprocessing.connection.wait` function allows polling -multiple objects (such as connections, sockets and pipes) with a timeout. -(Contributed by Richard Oudkerk in :issue:`12328`.) - -:class:`multiprocessing.Connection` objects can now be transferred over -multiprocessing connections. -(Contributed by Richard Oudkerk in :issue:`4892`.) - -:class:`multiprocessing.Process` now accepts a ``daemon`` keyword argument -to override the default behavior of inheriting the ``daemon`` flag from -the parent process (:issue:`6064`). - -New attribute :data:`multiprocessing.Process.sentinel` allows a -program to wait on multiple :class:`~multiprocessing.Process` objects at one -time using the appropriate OS primitives (for example, :mod:`select` on -posix systems). - -New methods :meth:`multiprocessing.pool.Pool.starmap` and -:meth:`~multiprocessing.pool.Pool.starmap_async` provide -:func:`itertools.starmap` equivalents to the existing -:meth:`multiprocessing.pool.Pool.map` and -:meth:`~multiprocessing.pool.Pool.map_async` functions. (Contributed by Hynek -Schlawack in :issue:`12708`.) - - -nntplib -------- - -The :class:`nntplib.NNTP` class now supports the context management protocol to -unconditionally consume :exc:`socket.error` exceptions and to close the NNTP -connection when done:: - - >>> from nntplib import NNTP - >>> with NNTP('news.gmane.org') as n: - ... n.group('gmane.comp.python.committers') - ... - ('211 1755 1 1755 gmane.comp.python.committers', 1755, 1, 1755, 'gmane.comp.python.committers') - >>> - -(Contributed by Giampaolo Rodolà in :issue:`9795`.) - - -os --- - -* The :mod:`os` module has a new :func:`~os.pipe2` function that makes it - possible to create a pipe with :data:`~os.O_CLOEXEC` or - :data:`~os.O_NONBLOCK` flags set atomically. This is especially useful to - avoid race conditions in multi-threaded programs. - -* The :mod:`os` module has a new :func:`~os.sendfile` function which provides - an efficient "zero-copy" way for copying data from one file (or socket) - descriptor to another. The phrase "zero-copy" refers to the fact that all of - the copying of data between the two descriptors is done entirely by the - kernel, with no copying of data into userspace buffers. :func:`~os.sendfile` - can be used to efficiently copy data from a file on disk to a network socket, - e.g. for downloading a file. - - (Patch submitted by Ross Lagerwall and Giampaolo Rodolà in :issue:`10882`.) - -* To avoid race conditions like symlink attacks and issues with temporary - files and directories, it is more reliable (and also faster) to manipulate - file descriptors instead of file names. Python 3.3 enhances existing functions - and introduces new functions to work on file descriptors (:issue:`4761`, - :issue:`10755` and :issue:`14626`). - - - The :mod:`os` module has a new :func:`~os.fwalk` function similar to - :func:`~os.walk` except that it also yields file descriptors referring to the - directories visited. This is especially useful to avoid symlink races. - - - The following functions get new optional *dir_fd* (:ref:`paths relative to - directory descriptors <dir_fd>`) and/or *follow_symlinks* (:ref:`not - following symlinks <follow_symlinks>`): - :func:`~os.access`, :func:`~os.chflags`, :func:`~os.chmod`, :func:`~os.chown`, - :func:`~os.link`, :func:`~os.lstat`, :func:`~os.mkdir`, :func:`~os.mkfifo`, - :func:`~os.mknod`, :func:`~os.open`, :func:`~os.readlink`, :func:`~os.remove`, - :func:`~os.rename`, :func:`~os.replace`, :func:`~os.rmdir`, :func:`~os.stat`, - :func:`~os.symlink`, :func:`~os.unlink`, :func:`~os.utime`. Platform - support for using these parameters can be checked via the sets - :data:`os.supports_dir_fd` and :data:`os.supports_follows_symlinks`. - - - The following functions now support a file descriptor for their path argument: - :func:`~os.chdir`, :func:`~os.chmod`, :func:`~os.chown`, - :func:`~os.execve`, :func:`~os.listdir`, :func:`~os.pathconf`, :func:`~os.path.exists`, - :func:`~os.stat`, :func:`~os.statvfs`, :func:`~os.utime`. Platform support - for this can be checked via the :data:`os.supports_fd` set. - -* :func:`~os.access` accepts an ``effective_ids`` keyword argument to turn on - using the effective uid/gid rather than the real uid/gid in the access check. - Platform support for this can be checked via the - :data:`~os.supports_effective_ids` set. - -* The :mod:`os` module has two new functions: :func:`~os.getpriority` and - :func:`~os.setpriority`. They can be used to get or set process - niceness/priority in a fashion similar to :func:`os.nice` but extended to all - processes instead of just the current one. - - (Patch submitted by Giampaolo Rodolà in :issue:`10784`.) - -* The new :func:`os.replace` function allows cross-platform renaming of a - file with overwriting the destination. With :func:`os.rename`, an existing - destination file is overwritten under POSIX, but raises an error under - Windows. - (Contributed by Antoine Pitrou in :issue:`8828`.) - -* The stat family of functions (:func:`~os.stat`, :func:`~os.fstat`, - and :func:`~os.lstat`) now support reading a file's timestamps - with nanosecond precision. Symmetrically, :func:`~os.utime` - can now write file timestamps with nanosecond precision. (Contributed by - Larry Hastings in :issue:`14127`.) - -* The new :func:`os.get_terminal_size` function queries the size of the - terminal attached to a file descriptor. See also - :func:`shutil.get_terminal_size`. - (Contributed by Zbigniew Jędrzejewski-Szmek in :issue:`13609`.) - -.. XXX sort out this mess after beta1 - -* New functions to support Linux extended attributes (:issue:`12720`): - :func:`~os.getxattr`, :func:`~os.listxattr`, :func:`~os.removexattr`, - :func:`~os.setxattr`. - -* New interface to the scheduler. These functions - control how a process is allocated CPU time by the operating system. New - functions: - :func:`~os.sched_get_priority_max`, :func:`~os.sched_get_priority_min`, - :func:`~os.sched_getaffinity`, :func:`~os.sched_getparam`, - :func:`~os.sched_getscheduler`, :func:`~os.sched_rr_get_interval`, - :func:`~os.sched_setaffinity`, :func:`~os.sched_setparam`, - :func:`~os.sched_setscheduler`, :func:`~os.sched_yield`, - -* New functions to control the file system: - - * :func:`~os.posix_fadvise`: Announces an intention to access data in a - specific pattern thus allowing the kernel to make optimizations. - * :func:`~os.posix_fallocate`: Ensures that enough disk space is allocated - for a file. - * :func:`~os.sync`: Force write of everything to disk. - -* Additional new posix functions: - - * :func:`~os.lockf`: Apply, test or remove a POSIX lock on an open file descriptor. - * :func:`~os.pread`: Read from a file descriptor at an offset, the file - offset remains unchanged. - * :func:`~os.pwrite`: Write to a file descriptor from an offset, leaving - the file offset unchanged. - * :func:`~os.readv`: Read from a file descriptor into a number of writable buffers. - * :func:`~os.truncate`: Truncate the file corresponding to *path*, so that - it is at most *length* bytes in size. - * :func:`~os.waitid`: Wait for the completion of one or more child processes. - * :func:`~os.writev`: Write the contents of *buffers* to a file descriptor, - where *buffers* is an arbitrary sequence of buffers. - * :func:`~os.getgrouplist` (:issue:`9344`): Return list of group ids that - specified user belongs to. - -* :func:`~os.times` and :func:`~os.uname`: Return type changed from a tuple to - a tuple-like object with named attributes. - -* Some platforms now support additional constants for the :func:`~os.lseek` - function, such as ``os.SEEK_HOLE`` and ``os.SEEK_DATA``. - -* New constants :data:`~os.RTLD_LAZY`, :data:`~os.RTLD_NOW`, - :data:`~os.RTLD_GLOBAL`, :data:`~os.RTLD_LOCAL`, :data:`~os.RTLD_NODELETE`, - :data:`~os.RTLD_NOLOAD`, and :data:`~os.RTLD_DEEPBIND` are available on - platforms that support them. These are for use with the - :func:`sys.setdlopenflags` function, and supersede the similar constants - defined in :mod:`ctypes` and :mod:`DLFCN`. (Contributed by Victor Stinner - in :issue:`13226`.) - -* :func:`os.symlink` now accepts (and ignores) the ``target_is_directory`` - keyword argument on non-Windows platforms, to ease cross-platform support. - - -pdb ---- - -Tab-completion is now available not only for command names, but also their -arguments. For example, for the ``break`` command, function and file names -are completed. - -(Contributed by Georg Brandl in :issue:`14210`) - - -pickle ------- - -:class:`pickle.Pickler` objects now have an optional -:attr:`~pickle.Pickler.dispatch_table` attribute allowing per-pickler -reduction functions to be set. - -(Contributed by Richard Oudkerk in :issue:`14166`.) - - -pydoc ------ - -The Tk GUI and the :func:`~pydoc.serve` function have been removed from the -:mod:`pydoc` module: ``pydoc -g`` and :func:`~pydoc.serve` have been deprecated -in Python 3.2. - - -re --- - -:class:`str` regular expressions now support ``\u`` and ``\U`` escapes. - -(Contributed by Serhiy Storchaka in :issue:`3665`.) - - -sched ------ - -* :meth:`~sched.scheduler.run` now accepts a *blocking* parameter which when - set to false makes the method execute the scheduled events due to expire - soonest (if any) and then return immediately. - This is useful in case you want to use the :class:`~sched.scheduler` in - non-blocking applications. (Contributed by Giampaolo Rodolà in :issue:`13449`.) - -* :class:`~sched.scheduler` class can now be safely used in multi-threaded - environments. (Contributed by Josiah Carlson and Giampaolo Rodolà in - :issue:`8684`.) - -* *timefunc* and *delayfunct* parameters of :class:`~sched.scheduler` class - constructor are now optional and defaults to :func:`time.time` and - :func:`time.sleep` respectively. (Contributed by Chris Clark in - :issue:`13245`.) - -* :meth:`~sched.scheduler.enter` and :meth:`~sched.scheduler.enterabs` - *argument* parameter is now optional. (Contributed by Chris Clark in - :issue:`13245`.) - -* :meth:`~sched.scheduler.enter` and :meth:`~sched.scheduler.enterabs` - now accept a *kwargs* parameter. (Contributed by Chris Clark in - :issue:`13245`.) - - -select ------- - -Solaris and derivative platforms have a new class :class:`select.devpoll` -for high performance asynchronous sockets via :file:`/dev/poll`. -(Contributed by Jesús Cea Avión in :issue:`6397`.) - - -shlex ------ - -The previously undocumented helper function ``quote`` from the -:mod:`pipes` modules has been moved to the :mod:`shlex` module and -documented. :func:`~shlex.quote` properly escapes all characters in a string -that might be otherwise given special meaning by the shell. - - -shutil ------- - -* New functions: - - * :func:`~shutil.disk_usage`: provides total, used and free disk space - statistics. (Contributed by Giampaolo Rodolà in :issue:`12442`.) - * :func:`~shutil.chown`: allows one to change user and/or group of the given - path also specifying the user/group names and not only their numeric - ids. (Contributed by Sandro Tosi in :issue:`12191`.) - * :func:`shutil.get_terminal_size`: returns the size of the terminal window - to which the interpreter is attached. (Contributed by Zbigniew - Jędrzejewski-Szmek in :issue:`13609`.) - -* :func:`~shutil.copy2` and :func:`~shutil.copystat` now preserve file - timestamps with nanosecond precision on platforms that support it. - They also preserve file "extended attributes" on Linux. (Contributed - by Larry Hastings in :issue:`14127` and :issue:`15238`.) - -* Several functions now take an optional ``symlinks`` argument: when that - parameter is true, symlinks aren't dereferenced and the operation instead - acts on the symlink itself (or creates one, if relevant). - (Contributed by Hynek Schlawack in :issue:`12715`.) - -* When copying files to a different file system, :func:`~shutil.move` now - handles symlinks the way the posix ``mv`` command does, recreating the - symlink rather than copying the target file contents. (Contributed by - Jonathan Niehof in :issue:`9993`.) :func:`~shutil.move` now also returns - the ``dst`` argument as its result. - -* :func:`~shutil.rmtree` is now resistant to symlink attacks on platforms - which support the new ``dir_fd`` parameter in :func:`os.open` and - :func:`os.unlink`. (Contributed by Martin von Löwis and Hynek Schlawack - in :issue:`4489`.) - - -signal ------- - -* The :mod:`signal` module has new functions: - - * :func:`~signal.pthread_sigmask`: fetch and/or change the signal mask of the - calling thread (Contributed by Jean-Paul Calderone in :issue:`8407`); - * :func:`~signal.pthread_kill`: send a signal to a thread; - * :func:`~signal.sigpending`: examine pending functions; - * :func:`~signal.sigwait`: wait a signal; - * :func:`~signal.sigwaitinfo`: wait for a signal, returning detailed - information about it; - * :func:`~signal.sigtimedwait`: like :func:`~signal.sigwaitinfo` but with a - timeout. - -* The signal handler writes the signal number as a single byte instead of - a nul byte into the wakeup file descriptor. So it is possible to wait more - than one signal and know which signals were raised. - -* :func:`signal.signal` and :func:`signal.siginterrupt` raise an OSError, - instead of a RuntimeError: OSError has an errno attribute. - - -smtpd ------ - -The :mod:`smtpd` module now supports :rfc:`5321` (extended SMTP) and :rfc:`1870` -(size extension). Per the standard, these extensions are enabled if and only -if the client initiates the session with an ``EHLO`` command. - -(Initial ``ELHO`` support by Alberto Trevino. Size extension by Juhana -Jauhiainen. Substantial additional work on the patch contributed by Michele -Orrù and Dan Boswell. :issue:`8739`) - - -smtplib -------- - -The :class:`~smtplib.SMTP`, :class:`~smtplib.SMTP_SSL`, and -:class:`~smtplib.LMTP` classes now accept a ``source_address`` keyword argument -to specify the ``(host, port)`` to use as the source address in the bind call -when creating the outgoing socket. (Contributed by Paulo Scardine in -:issue:`11281`.) - -:class:`~smtplib.SMTP` now supports the context management protocol, allowing an -``SMTP`` instance to be used in a ``with`` statement. (Contributed -by Giampaolo Rodolà in :issue:`11289`.) - -The :class:`~smtplib.SMTP_SSL` constructor and the :meth:`~smtplib.SMTP.starttls` -method now accept an SSLContext parameter to control parameters of the secure -channel. (Contributed by Kasun Herath in :issue:`8809`.) - - -socket ------- - -* The :class:`~socket.socket` class now exposes additional methods to process - ancillary data when supported by the underlying platform: - - * :func:`~socket.socket.sendmsg` - * :func:`~socket.socket.recvmsg` - * :func:`~socket.socket.recvmsg_into` - - (Contributed by David Watson in :issue:`6560`, based on an earlier patch by - Heiko Wundram) - -* The :class:`~socket.socket` class now supports the PF_CAN protocol family - (https://en.wikipedia.org/wiki/Socketcan), on Linux - (https://lwn.net/Articles/253425). - - (Contributed by Matthias Fuchs, updated by Tiago Gonçalves in :issue:`10141`.) - -* The :class:`~socket.socket` class now supports the PF_RDS protocol family - (https://en.wikipedia.org/wiki/Reliable_Datagram_Sockets and - https://oss.oracle.com/projects/rds/). - -* The :class:`~socket.socket` class now supports the ``PF_SYSTEM`` protocol - family on OS X. (Contributed by Michael Goderbauer in :issue:`13777`.) - -* New function :func:`~socket.sethostname` allows the hostname to be set - on unix systems if the calling process has sufficient privileges. - (Contributed by Ross Lagerwall in :issue:`10866`.) - - -socketserver ------------- - -:class:`~socketserver.BaseServer` now has an overridable method -:meth:`~socketserver.BaseServer.service_actions` that is called by the -:meth:`~socketserver.BaseServer.serve_forever` method in the service loop. -:class:`~socketserver.ForkingMixIn` now uses this to clean up zombie -child processes. (Contributed by Justin Warkentin in :issue:`11109`.) - - -sqlite3 -------- - -New :class:`sqlite3.Connection` method -:meth:`~sqlite3.Connection.set_trace_callback` can be used to capture a trace of -all sql commands processed by sqlite. (Contributed by Torsten Landschoff -in :issue:`11688`.) - - -ssl ---- - -* The :mod:`ssl` module has two new random generation functions: - - * :func:`~ssl.RAND_bytes`: generate cryptographically strong - pseudo-random bytes. - * :func:`~ssl.RAND_pseudo_bytes`: generate pseudo-random bytes. - - (Contributed by Victor Stinner in :issue:`12049`.) - -* The :mod:`ssl` module now exposes a finer-grained exception hierarchy - in order to make it easier to inspect the various kinds of errors. - (Contributed by Antoine Pitrou in :issue:`11183`.) - -* :meth:`~ssl.SSLContext.load_cert_chain` now accepts a *password* argument - to be used if the private key is encrypted. - (Contributed by Adam Simpkins in :issue:`12803`.) - -* Diffie-Hellman key exchange, both regular and Elliptic Curve-based, is - now supported through the :meth:`~ssl.SSLContext.load_dh_params` and - :meth:`~ssl.SSLContext.set_ecdh_curve` methods. - (Contributed by Antoine Pitrou in :issue:`13626` and :issue:`13627`.) - -* SSL sockets have a new :meth:`~ssl.SSLSocket.get_channel_binding` method - allowing the implementation of certain authentication mechanisms such as - SCRAM-SHA-1-PLUS. (Contributed by Jacek Konieczny in :issue:`12551`.) - -* You can query the SSL compression algorithm used by an SSL socket, thanks - to its new :meth:`~ssl.SSLSocket.compression` method. The new attribute - :attr:`~ssl.OP_NO_COMPRESSION` can be used to disable compression. - (Contributed by Antoine Pitrou in :issue:`13634`.) - -* Support has been added for the Next Protocol Negotiation extension using - the :meth:`ssl.SSLContext.set_npn_protocols` method. - (Contributed by Colin Marc in :issue:`14204`.) - -* SSL errors can now be introspected more easily thanks to - :attr:`~ssl.SSLError.library` and :attr:`~ssl.SSLError.reason` attributes. - (Contributed by Antoine Pitrou in :issue:`14837`.) - -* The :func:`~ssl.get_server_certificate` function now supports IPv6. - (Contributed by Charles-François Natali in :issue:`11811`.) - -* New attribute :attr:`~ssl.OP_CIPHER_SERVER_PREFERENCE` allows setting - SSLv3 server sockets to use the server's cipher ordering preference rather - than the client's (:issue:`13635`). - - -stat ----- - -The undocumented tarfile.filemode function has been moved to -:func:`stat.filemode`. It can be used to convert a file's mode to a string of -the form '-rwxrwxrwx'. - -(Contributed by Giampaolo Rodolà in :issue:`14807`.) - - -struct ------- - -The :mod:`struct` module now supports ``ssize_t`` and ``size_t`` via the -new codes ``n`` and ``N``, respectively. (Contributed by Antoine Pitrou -in :issue:`3163`.) - - -subprocess ----------- - -Command strings can now be bytes objects on posix platforms. (Contributed by -Victor Stinner in :issue:`8513`.) - -A new constant :data:`~subprocess.DEVNULL` allows suppressing output in a -platform-independent fashion. (Contributed by Ross Lagerwall in -:issue:`5870`.) - - -sys ---- - -The :mod:`sys` module has a new :data:`~sys.thread_info` :term:`named -tuple` holding information about the thread implementation -(:issue:`11223`). - - -tarfile -------- - -:mod:`tarfile` now supports ``lzma`` encoding via the :mod:`lzma` module. -(Contributed by Lars Gustäbel in :issue:`5689`.) - - -tempfile --------- - -:class:`tempfile.SpooledTemporaryFile`\'s -:meth:`~tempfile.SpooledTemporaryFile.truncate` method now accepts -a ``size`` parameter. (Contributed by Ryan Kelly in :issue:`9957`.) - - -textwrap --------- - -The :mod:`textwrap` module has a new :func:`~textwrap.indent` that makes -it straightforward to add a common prefix to selected lines in a block -of text (:issue:`13857`). - - -threading ---------- - -:class:`threading.Condition`, :class:`threading.Semaphore`, -:class:`threading.BoundedSemaphore`, :class:`threading.Event`, and -:class:`threading.Timer`, all of which used to be factory functions returning a -class instance, are now classes and may be subclassed. (Contributed by Éric -Araujo in :issue:`10968`.) - -The :class:`threading.Thread` constructor now accepts a ``daemon`` keyword -argument to override the default behavior of inheriting the ``daemon`` flag -value from the parent thread (:issue:`6064`). - -The formerly private function ``_thread.get_ident`` is now available as the -public function :func:`threading.get_ident`. This eliminates several cases of -direct access to the ``_thread`` module in the stdlib. Third party code that -used ``_thread.get_ident`` should likewise be changed to use the new public -interface. - - -time ----- - -The :pep:`418` added new functions to the :mod:`time` module: - -* :func:`~time.get_clock_info`: Get information on a clock. -* :func:`~time.monotonic`: Monotonic clock (cannot go backward), not affected - by system clock updates. -* :func:`~time.perf_counter`: Performance counter with the highest available - resolution to measure a short duration. -* :func:`~time.process_time`: Sum of the system and user CPU time of the - current process. - -Other new functions: - -* :func:`~time.clock_getres`, :func:`~time.clock_gettime` and - :func:`~time.clock_settime` functions with ``CLOCK_xxx`` constants. - (Contributed by Victor Stinner in :issue:`10278`.) - -To improve cross platform consistency, :func:`~time.sleep` now raises a -:exc:`ValueError` when passed a negative sleep value. Previously this was an -error on posix, but produced an infinite sleep on Windows. - - -types ------ - -Add a new :class:`types.MappingProxyType` class: Read-only proxy of a mapping. -(:issue:`14386`) - - -The new functions :func:`types.new_class` and :func:`types.prepare_class` provide support -for :pep:`3115` compliant dynamic type creation. (:issue:`14588`) - - -unittest --------- - -:meth:`.assertRaises`, :meth:`.assertRaisesRegex`, :meth:`.assertWarns`, and -:meth:`.assertWarnsRegex` now accept a keyword argument *msg* when used as -context managers. (Contributed by Ezio Melotti and Winston Ewert in -:issue:`10775`.) - -:meth:`unittest.TestCase.run` now returns the :class:`~unittest.TestResult` -object. - - -urllib ------- - -The :class:`~urllib.request.Request` class, now accepts a *method* argument -used by :meth:`~urllib.request.Request.get_method` to determine what HTTP method -should be used. For example, this will send a ``'HEAD'`` request:: - - >>> urlopen(Request('https://www.python.org', method='HEAD')) - -(:issue:`1673007`) - - -webbrowser ----------- - -The :mod:`webbrowser` module supports more "browsers": Google Chrome (named -:program:`chrome`, :program:`chromium`, :program:`chrome-browser` or -:program:`chromium-browser` depending on the version and operating system), -and the generic launchers :program:`xdg-open`, from the FreeDesktop.org -project, and :program:`gvfs-open`, which is the default URI handler for GNOME -3. (The former contributed by Arnaud Calmettes in :issue:`13620`, the latter -by Matthias Klose in :issue:`14493`.) - - -xml.etree.ElementTree ---------------------- - -The :mod:`xml.etree.ElementTree` module now imports its C accelerator by -default; there is no longer a need to explicitly import -:mod:`xml.etree.cElementTree` (this module stays for backwards compatibility, -but is now deprecated). In addition, the ``iter`` family of methods of -:class:`~xml.etree.ElementTree.Element` has been optimized (rewritten in C). -The module's documentation has also been greatly improved with added examples -and a more detailed reference. - - -zlib ----- - -New attribute :attr:`zlib.Decompress.eof` makes it possible to distinguish -between a properly-formed compressed stream and an incomplete or truncated one. -(Contributed by Nadeem Vawda in :issue:`12646`.) - -New attribute :attr:`zlib.ZLIB_RUNTIME_VERSION` reports the version string of -the underlying ``zlib`` library that is loaded at runtime. (Contributed by -Torsten Landschoff in :issue:`12306`.) - - -Optimizations -============= - -Major performance enhancements have been added: - -* Thanks to :pep:`393`, some operations on Unicode strings have been optimized: - - * the memory footprint is divided by 2 to 4 depending on the text - * encode an ASCII string to UTF-8 doesn't need to encode characters anymore, - the UTF-8 representation is shared with the ASCII representation - * the UTF-8 encoder has been optimized - * repeating a single ASCII letter and getting a substring of an ASCII string - is 4 times faster - -* UTF-8 is now 2x to 4x faster. UTF-16 encoding is now up to 10x faster. - - (Contributed by Serhiy Storchaka, :issue:`14624`, :issue:`14738` and - :issue:`15026`.) - - -Build and C API Changes -======================= - -Changes to Python's build process and to the C API include: - -* New :pep:`3118` related function: - - * :c:func:`PyMemoryView_FromMemory` - -* :pep:`393` added new Unicode types, macros and functions: - - * High-level API: - - * :c:func:`PyUnicode_CopyCharacters` - * :c:func:`PyUnicode_FindChar` - * :c:func:`PyUnicode_GetLength`, :c:macro:`PyUnicode_GET_LENGTH` - * :c:func:`PyUnicode_New` - * :c:func:`PyUnicode_Substring` - * :c:func:`PyUnicode_ReadChar`, :c:func:`PyUnicode_WriteChar` - - * Low-level API: - - * :c:type:`Py_UCS1`, :c:type:`Py_UCS2`, :c:type:`Py_UCS4` types - * :c:type:`PyASCIIObject` and :c:type:`PyCompactUnicodeObject` structures - * :c:macro:`PyUnicode_READY` - * :c:func:`PyUnicode_FromKindAndData` - * :c:func:`PyUnicode_AsUCS4`, :c:func:`PyUnicode_AsUCS4Copy` - * :c:macro:`PyUnicode_DATA`, :c:macro:`PyUnicode_1BYTE_DATA`, - :c:macro:`PyUnicode_2BYTE_DATA`, :c:macro:`PyUnicode_4BYTE_DATA` - * :c:macro:`PyUnicode_KIND` with :c:type:`PyUnicode_Kind` enum: - :c:data:`PyUnicode_WCHAR_KIND`, :c:data:`PyUnicode_1BYTE_KIND`, - :c:data:`PyUnicode_2BYTE_KIND`, :c:data:`PyUnicode_4BYTE_KIND` - * :c:macro:`PyUnicode_READ`, :c:macro:`PyUnicode_READ_CHAR`, :c:macro:`PyUnicode_WRITE` - * :c:macro:`PyUnicode_MAX_CHAR_VALUE` - -* :c:macro:`PyArg_ParseTuple` now accepts a :class:`bytearray` for the ``c`` - format (:issue:`12380`). - - - -Deprecated -========== - -Unsupported Operating Systems ------------------------------ - -OS/2 and VMS are no longer supported due to the lack of a maintainer. - -Windows 2000 and Windows platforms which set ``COMSPEC`` to ``command.com`` -are no longer supported due to maintenance burden. - -OSF support, which was deprecated in 3.2, has been completely removed. - - -Deprecated Python modules, functions and methods ------------------------------------------------- - -* Passing a non-empty string to ``object.__format__()`` is deprecated, and - will produce a :exc:`TypeError` in Python 3.4 (:issue:`9856`). -* The ``unicode_internal`` codec has been deprecated because of the - :pep:`393`, use UTF-8, UTF-16 (``utf-16-le`` or ``utf-16-be``), or UTF-32 - (``utf-32-le`` or ``utf-32-be``) -* :meth:`ftplib.FTP.nlst` and :meth:`ftplib.FTP.dir`: use - :meth:`ftplib.FTP.mlsd` -* :func:`platform.popen`: use the :mod:`subprocess` module. Check especially - the :ref:`subprocess-replacements` section (:issue:`11377`). -* :issue:`13374`: The Windows bytes API has been deprecated in the :mod:`os` - module. Use Unicode filenames, instead of bytes filenames, to not depend on - the ANSI code page anymore and to support any filename. -* :issue:`13988`: The :mod:`xml.etree.cElementTree` module is deprecated. The - accelerator is used automatically whenever available. -* The behaviour of :func:`time.clock` depends on the platform: use the new - :func:`time.perf_counter` or :func:`time.process_time` function instead, - depending on your requirements, to have a well defined behaviour. -* The :func:`os.stat_float_times` function is deprecated. -* :mod:`abc` module: - - * :class:`abc.abstractproperty` has been deprecated, use :class:`property` - with :func:`abc.abstractmethod` instead. - * :class:`abc.abstractclassmethod` has been deprecated, use - :class:`classmethod` with :func:`abc.abstractmethod` instead. - * :class:`abc.abstractstaticmethod` has been deprecated, use - :class:`staticmethod` with :func:`abc.abstractmethod` instead. - -* :mod:`importlib` package: - - * :meth:`importlib.abc.SourceLoader.path_mtime` is now deprecated in favour of - :meth:`importlib.abc.SourceLoader.path_stats` as bytecode files now store - both the modification time and size of the source file the bytecode file was - compiled from. - - - - - -Deprecated functions and types of the C API -------------------------------------------- - -The :c:type:`Py_UNICODE` has been deprecated by :pep:`393` and will be -removed in Python 4. All functions using this type are deprecated: - -Unicode functions and methods using :c:type:`Py_UNICODE` and -:c:type:`Py_UNICODE*` types: - -* :c:macro:`PyUnicode_FromUnicode`: use :c:func:`PyUnicode_FromWideChar` or - :c:func:`PyUnicode_FromKindAndData` -* :c:macro:`PyUnicode_AS_UNICODE`, :c:func:`PyUnicode_AsUnicode`, - :c:func:`PyUnicode_AsUnicodeAndSize`: use :c:func:`PyUnicode_AsWideCharString` -* :c:macro:`PyUnicode_AS_DATA`: use :c:macro:`PyUnicode_DATA` with - :c:macro:`PyUnicode_READ` and :c:macro:`PyUnicode_WRITE` -* :c:macro:`PyUnicode_GET_SIZE`, :c:func:`PyUnicode_GetSize`: use - :c:macro:`PyUnicode_GET_LENGTH` or :c:func:`PyUnicode_GetLength` -* :c:macro:`PyUnicode_GET_DATA_SIZE`: use - ``PyUnicode_GET_LENGTH(str) * PyUnicode_KIND(str)`` (only work on ready - strings) -* :c:func:`PyUnicode_AsUnicodeCopy`: use :c:func:`PyUnicode_AsUCS4Copy` or - :c:func:`PyUnicode_AsWideCharString` -* :c:func:`PyUnicode_GetMax` - - -Functions and macros manipulating Py_UNICODE* strings: - -* :c:macro:`Py_UNICODE_strlen`: use :c:func:`PyUnicode_GetLength` or - :c:macro:`PyUnicode_GET_LENGTH` -* :c:macro:`Py_UNICODE_strcat`: use :c:func:`PyUnicode_CopyCharacters` or - :c:func:`PyUnicode_FromFormat` -* :c:macro:`Py_UNICODE_strcpy`, :c:macro:`Py_UNICODE_strncpy`, - :c:macro:`Py_UNICODE_COPY`: use :c:func:`PyUnicode_CopyCharacters` or - :c:func:`PyUnicode_Substring` -* :c:macro:`Py_UNICODE_strcmp`: use :c:func:`PyUnicode_Compare` -* :c:macro:`Py_UNICODE_strncmp`: use :c:func:`PyUnicode_Tailmatch` -* :c:macro:`Py_UNICODE_strchr`, :c:macro:`Py_UNICODE_strrchr`: use - :c:func:`PyUnicode_FindChar` -* :c:macro:`Py_UNICODE_FILL`: use :c:func:`PyUnicode_Fill` -* :c:macro:`Py_UNICODE_MATCH` - -Encoders: - -* :c:func:`PyUnicode_Encode`: use :c:func:`PyUnicode_AsEncodedObject` -* :c:func:`PyUnicode_EncodeUTF7` -* :c:func:`PyUnicode_EncodeUTF8`: use :c:func:`PyUnicode_AsUTF8` or - :c:func:`PyUnicode_AsUTF8String` -* :c:func:`PyUnicode_EncodeUTF32` -* :c:func:`PyUnicode_EncodeUTF16` -* :c:func:`PyUnicode_EncodeUnicodeEscape:` use - :c:func:`PyUnicode_AsUnicodeEscapeString` -* :c:func:`PyUnicode_EncodeRawUnicodeEscape:` use - :c:func:`PyUnicode_AsRawUnicodeEscapeString` -* :c:func:`PyUnicode_EncodeLatin1`: use :c:func:`PyUnicode_AsLatin1String` -* :c:func:`PyUnicode_EncodeASCII`: use :c:func:`PyUnicode_AsASCIIString` -* :c:func:`PyUnicode_EncodeCharmap` -* :c:func:`PyUnicode_TranslateCharmap` -* :c:func:`PyUnicode_EncodeMBCS`: use :c:func:`PyUnicode_AsMBCSString` or - :c:func:`PyUnicode_EncodeCodePage` (with ``CP_ACP`` code_page) -* :c:func:`PyUnicode_EncodeDecimal`, - :c:func:`PyUnicode_TransformDecimalToASCII` - - -Deprecated features -------------------- - -The :mod:`array` module's ``'u'`` format code is now deprecated and will be -removed in Python 4 together with the rest of the (:c:type:`Py_UNICODE`) API. - - -Porting to Python 3.3 -===================== - -This section lists previously described changes and other bugfixes -that may require changes to your code. - -.. _portingpythoncode: - -Porting Python code -------------------- - -* Hash randomization is enabled by default. Set the :envvar:`PYTHONHASHSEED` - environment variable to ``0`` to disable hash randomization. See also the - :meth:`object.__hash__` method. - -* :issue:`12326`: On Linux, sys.platform doesn't contain the major version - anymore. It is now always 'linux', instead of 'linux2' or 'linux3' depending - on the Linux version used to build Python. Replace sys.platform == 'linux2' - with sys.platform.startswith('linux'), or directly sys.platform == 'linux' if - you don't need to support older Python versions. - -* :issue:`13847`, :issue:`14180`: :mod:`time` and :mod:`datetime`: - :exc:`OverflowError` is now raised instead of :exc:`ValueError` if a - timestamp is out of range. :exc:`OSError` is now raised if C functions - :c:func:`gmtime` or :c:func:`localtime` failed. - -* The default finders used by import now utilize a cache of what is contained - within a specific directory. If you create a Python source file or sourceless - bytecode file, make sure to call :func:`importlib.invalidate_caches` to clear - out the cache for the finders to notice the new file. - -* :exc:`ImportError` now uses the full name of the module that was attempted to - be imported. Doctests that check ImportErrors' message will need to be - updated to use the full name of the module instead of just the tail of the - name. - -* The *index* argument to :func:`__import__` now defaults to 0 instead of -1 - and no longer support negative values. It was an oversight when :pep:`328` was - implemented that the default value remained -1. If you need to continue to - perform a relative import followed by an absolute import, then perform the - relative import using an index of 1, followed by another import using an - index of 0. It is preferred, though, that you use - :func:`importlib.import_module` rather than call :func:`__import__` directly. - -* :func:`__import__` no longer allows one to use an index value other than 0 - for top-level modules. E.g. ``__import__('sys', level=1)`` is now an error. - -* Because :attr:`sys.meta_path` and :attr:`sys.path_hooks` now have finders on - them by default, you will most likely want to use :meth:`list.insert` instead - of :meth:`list.append` to add to those lists. - -* Because ``None`` is now inserted into :attr:`sys.path_importer_cache`, if you - are clearing out entries in the dictionary of paths that do not have a - finder, you will need to remove keys paired with values of ``None`` **and** - :class:`imp.NullImporter` to be backwards-compatible. This will lead to extra - overhead on older versions of Python that re-insert ``None`` into - :attr:`sys.path_importer_cache` where it represents the use of implicit - finders, but semantically it should not change anything. - -* :class:`importlib.abc.Finder` no longer specifies a `find_module()` abstract - method that must be implemented. If you were relying on subclasses to - implement that method, make sure to check for the method's existence first. - You will probably want to check for `find_loader()` first, though, in the - case of working with :term:`path entry finders <path entry finder>`. - -* :mod:`pkgutil` has been converted to use :mod:`importlib` internally. This - eliminates many edge cases where the old behaviour of the :pep:`302` import - emulation failed to match the behaviour of the real import system. The - import emulation itself is still present, but is now deprecated. The - :func:`pkgutil.iter_importers` and :func:`pkgutil.walk_packages` functions - special case the standard import hooks so they are still supported even - though they do not provide the non-standard ``iter_modules()`` method. - -* A longstanding RFC-compliance bug (:issue:`1079`) in the parsing done by - :func:`email.header.decode_header` has been fixed. Code that uses the - standard idiom to convert encoded headers into unicode - (``str(make_header(decode_header(h))``) will see no change, but code that - looks at the individual tuples returned by decode_header will see that - whitespace that precedes or follows ``ASCII`` sections is now included in the - ``ASCII`` section. Code that builds headers using ``make_header`` should - also continue to work without change, since ``make_header`` continues to add - whitespace between ``ASCII`` and non-``ASCII`` sections if it is not already - present in the input strings. - -* :func:`email.utils.formataddr` now does the correct content transfer - encoding when passed non-``ASCII`` display names. Any code that depended on - the previous buggy behavior that preserved the non-``ASCII`` unicode in the - formatted output string will need to be changed (:issue:`1690608`). - -* :meth:`poplib.POP3.quit` may now raise protocol errors like all other - ``poplib`` methods. Code that assumes ``quit`` does not raise - :exc:`poplib.error_proto` errors may need to be changed if errors on ``quit`` - are encountered by a particular application (:issue:`11291`). - -* The ``strict`` argument to :class:`email.parser.Parser`, deprecated since - Python 2.4, has finally been removed. - -* The deprecated method ``unittest.TestCase.assertSameElements`` has been - removed. - -* The deprecated variable ``time.accept2dyear`` has been removed. - -* The deprecated ``Context._clamp`` attribute has been removed from the - :mod:`decimal` module. It was previously replaced by the public attribute - :attr:`~decimal.Context.clamp`. (See :issue:`8540`.) - -* The undocumented internal helper class ``SSLFakeFile`` has been removed - from :mod:`smtplib`, since its functionality has long been provided directly - by :meth:`socket.socket.makefile`. - -* Passing a negative value to :func:`time.sleep` on Windows now raises an - error instead of sleeping forever. It has always raised an error on posix. - -* The ``ast.__version__`` constant has been removed. If you need to - make decisions affected by the AST version, use :attr:`sys.version_info` - to make the decision. - -* Code that used to work around the fact that the :mod:`threading` module used - factory functions by subclassing the private classes will need to change to - subclass the now-public classes. - -* The undocumented debugging machinery in the threading module has been - removed, simplifying the code. This should have no effect on production - code, but is mentioned here in case any application debug frameworks were - interacting with it (:issue:`13550`). - - -Porting C code --------------- - -* In the course of changes to the buffer API the undocumented - :c:member:`~Py_buffer.smalltable` member of the - :c:type:`Py_buffer` structure has been removed and the - layout of the :c:type:`PyMemoryViewObject` has changed. - - All extensions relying on the relevant parts in ``memoryobject.h`` - or ``object.h`` must be rebuilt. - -* Due to :ref:`PEP 393 <pep-393>`, the :c:type:`Py_UNICODE` type and all - functions using this type are deprecated (but will stay available for - at least five years). If you were using low-level Unicode APIs to - construct and access unicode objects and you want to benefit of the - memory footprint reduction provided by :pep:`393`, you have to convert - your code to the new :doc:`Unicode API <../c-api/unicode>`. - - However, if you only have been using high-level functions such as - :c:func:`PyUnicode_Concat()`, :c:func:`PyUnicode_Join` or - :c:func:`PyUnicode_FromFormat()`, your code will automatically take - advantage of the new unicode representations. - -* :c:func:`PyImport_GetMagicNumber` now returns ``-1`` upon failure. - -* As a negative value for the *level* argument to :func:`__import__` is no - longer valid, the same now holds for :c:func:`PyImport_ImportModuleLevel`. - This also means that the value of *level* used by - :c:func:`PyImport_ImportModuleEx` is now ``0`` instead of ``-1``. - - -Building C extensions ---------------------- - -* The range of possible file names for C extensions has been narrowed. - Very rarely used spellings have been suppressed: under POSIX, files - named ``xxxmodule.so``, ``xxxmodule.abi3.so`` and - ``xxxmodule.cpython-*.so`` are no longer recognized as implementing - the ``xxx`` module. If you had been generating such files, you have - to switch to the other spellings (i.e., remove the ``module`` string - from the file names). - - (implemented in :issue:`14040`.) - - -Command Line Switch Changes ---------------------------- - -* The -Q command-line flag and related artifacts have been removed. Code - checking sys.flags.division_warning will need updating. - - (:issue:`10998`, contributed by Éric Araujo.) - -* When :program:`python` is started with :option:`-S`, ``import site`` - will no longer add site-specific paths to the module search paths. In - previous versions, it did. - - (:issue:`11591`, contributed by Carl Meyer with editions by Éric Araujo.) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst deleted file mode 100644 index 99d0408..0000000 --- a/Doc/whatsnew/3.4.rst +++ /dev/null @@ -1,2545 +0,0 @@ -**************************** - What's New In Python 3.4 -**************************** - -:Author: R. David Murray <rdmurray@bitdance.com> (Editor) - -.. Rules for maintenance: - - * Anyone can add text to this document, but the maintainer reserves the - right to rewrite any additions. In particular, for obscure or esoteric - features, the maintainer may reduce any addition to a simple reference to - the new documentation rather than explaining the feature inline. - - * While the maintainer will periodically go through Misc/NEWS - and add changes, it's best not to rely on this. We know from experience - that any changes that aren't in the What's New documentation around the - time of the original release will remain largely unknown to the community - for years, even if they're added later. We also know from experience that - other priorities can arise, and the maintainer will run out of time to do - updates -- in such cases, end users will be much better served by partial - notifications that at least give a hint about new features to - investigate. - - * This is not a complete list of every single change; completeness - is the purpose of Misc/NEWS. The What's New should focus on changes that - are visible to Python *users* and that *require* a feature release (i.e. - most bug fixes should only be recorded in Misc/NEWS) - - * PEPs should not be marked Final until they have an entry in What's New. - A placeholder entry that is just a section header and a link to the PEP - (e.g ":pep:`397` has been implemented") is acceptable. If a PEP has been - implemented and noted in What's New, don't forget to mark it as Final! - - * If you want to draw your new text to the attention of the - maintainer, add 'XXX' to the beginning of the paragraph or - section. - - * It's OK to add just a very brief note about a change. For - example: "The :ref:`~socket.transmogrify()` function was added to the - :mod:`socket` module." The maintainer will research the change and - write the necessary text (if appropriate). The advantage of doing this - is that even if no more descriptive text is ever added, readers will at - least have a notification that the new feature exists and a link to the - relevant documentation. - - * You can comment out your additions if you like, but it's not - necessary (especially when a final release is some months away). - - * Credit the author of a patch or bugfix. Just the name is - sufficient; the e-mail address isn't necessary. - - * It's helpful to add the bug/patch number as a comment: - - The :ref:`~socket.transmogrify()` function was added to the - :mod:`socket` module. (Contributed by P.Y. Developer in :issue:`12345`.) - - This saves the maintainer the effort of going through the Mercurial log - when researching a change. - - * Cross referencing tip: :ref:`mod.attr` will display as ``mod.attr``, - while :ref:`~mod.attr` will display as ``attr``. - -This article explains the new features in Python 3.4, compared to 3.3. -Python 3.4 was released on March 16, 2014. For full details, see the -`changelog <https://docs.python.org/3.4/whatsnew/changelog.html>`_. - - -.. seealso:: - - :pep:`429` -- Python 3.4 Release Schedule - - - -Summary -- Release Highlights -============================= - -.. This section singles out the most important changes in Python 3.4. - Brevity is key. - -New syntax features: - -* No new syntax features were added in Python 3.4. - -Other new features: - -* :ref:`pip should always be available <whatsnew-pep-453>` (:pep:`453`). -* :ref:`Newly created file descriptors are non-inheritable <whatsnew-pep-446>` - (:pep:`446`). -* command line option for :ref:`isolated mode <whatsnew-isolated-mode>` - (:issue:`16499`). -* :ref:`improvements in the handling of codecs <codec-handling-improvements>` - that are not text encodings (multiple issues). -* :ref:`A ModuleSpec Type <whatsnew-pep-451>` for the Import System - (:pep:`451`). (Affects importer authors.) -* The :mod:`marshal` format has been made :ref:`more compact and efficient - <whatsnew-marshal-3>` (:issue:`16475`). - -New library modules: - -* :mod:`asyncio`: :ref:`New provisional API for asynchronous IO - <whatsnew-asyncio>` (:pep:`3156`). -* :mod:`ensurepip`: :ref:`Bootstrapping the pip installer <whatsnew-ensurepip>` - (:pep:`453`). -* :mod:`enum`: :ref:`Support for enumeration types <whatsnew-enum>` - (:pep:`435`). -* :mod:`pathlib`: :ref:`Object-oriented filesystem paths <whatsnew-pathlib>` - (:pep:`428`). -* :mod:`selectors`: :ref:`High-level and efficient I/O multiplexing - <whatsnew-selectors>`, built upon the :mod:`select` module primitives (part - of :pep:`3156`). -* :mod:`statistics`: A basic :ref:`numerically stable statistics library - <whatsnew-statistics>` (:pep:`450`). -* :mod:`tracemalloc`: :ref:`Trace Python memory allocations - <whatsnew-tracemalloc>` (:pep:`454`). - -Significantly improved library modules: - -* :ref:`Single-dispatch generic functions <whatsnew-singledispatch>` in - :mod:`functools` (:pep:`443`). -* New :mod:`pickle` :ref:`protocol 4 <whatsnew-protocol-4>` (:pep:`3154`). -* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork - on Unix <whatsnew-multiprocessing-no-fork>` (:issue:`8713`). -* :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and - a new :mod:`~email.message.Message` subclass - (:class:`~email.contentmanager.EmailMessage`) that :ref:`simplify MIME - handling <whatsnew_email_contentmanager>` (:issue:`18891`). -* The :mod:`inspect` and :mod:`pydoc` modules are now capable of - correct introspection of a much wider variety of callable objects, - which improves the output of the Python :func:`help` system. -* The :mod:`ipaddress` module API has been declared stable - -Security improvements: - -* :ref:`Secure and interchangeable hash algorithm <whatsnew-pep-456>` - (:pep:`456`). -* :ref:`Make newly created file descriptors non-inheritable <whatsnew-pep-446>` - (:pep:`446`) to avoid leaking file descriptors to child processes. -* New command line option for :ref:`isolated mode <whatsnew-isolated-mode>`, - (:issue:`16499`). -* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork - on Unix <whatsnew-multiprocessing-no-fork>`. *spawn* and *forkserver* are - more secure because they avoid sharing data with child processes. -* :mod:`multiprocessing` child processes on Windows no longer inherit - all of the parent's inheritable handles, only the necessary ones. -* A new :func:`hashlib.pbkdf2_hmac` function provides - the `PKCS#5 password-based key derivation function 2 - <https://en.wikipedia.org/wiki/PBKDF2>`_. -* :ref:`TLSv1.1 and TLSv1.2 support <whatsnew-tls-11-12>` for :mod:`ssl`. -* :ref:`Retrieving certificates from the Windows system cert store support - <whatsnew34-win-cert-store>` for :mod:`ssl`. -* :ref:`Server-side SNI (Server Name Indication) support - <whatsnew34-sni>` for :mod:`ssl`. -* The :class:`ssl.SSLContext` class has a :ref:`lot of improvements - <whatsnew34-sslcontext>`. -* All modules in the standard library that support SSL now support server - certificate verification, including hostname matching - (:func:`ssl.match_hostname`) and CRLs (Certificate Revocation lists, see - :func:`ssl.SSLContext.load_verify_locations`). - -CPython implementation improvements: - -* :ref:`Safe object finalization <whatsnew-pep-442>` (:pep:`442`). -* Leveraging :pep:`442`, in most cases :ref:`module globals are no longer set - to None during finalization <whatsnew-pep-442>` (:issue:`18214`). -* :ref:`Configurable memory allocators <whatsnew-pep-445>` (:pep:`445`). -* :ref:`Argument Clinic <whatsnew-pep-436>` (:pep:`436`). - -Please read on for a comprehensive list of user-facing changes, including many -other smaller improvements, CPython optimizations, deprecations, and potential -porting issues. - - - -New Features -============ - -.. _whatsnew-pep-453: - -PEP 453: Explicit Bootstrapping of PIP in Python Installations --------------------------------------------------------------- - -Bootstrapping pip By Default -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard -cross-platform mechanism to bootstrap the pip installer into Python -installations and virtual environments. The version of ``pip`` included -with Python 3.4.0 is ``pip`` 1.5.4, and future 3.4.x maintenance releases -will update the bundled version to the latest version of ``pip`` that is -available at the time of creating the release candidate. - -By default, the commands ``pipX`` and ``pipX.Y`` will be installed on all -platforms (where X.Y stands for the version of the Python installation), -along with the ``pip`` Python package and its dependencies. On Windows and -in virtual environments on all platforms, the unversioned ``pip`` command -will also be installed. On other platforms, the system wide unversioned -``pip`` command typically refers to the separately installed Python 2 -version. - -The ``pyvenv`` command line utility and the :mod:`venv` -module make use of the :mod:`ensurepip` module to make ``pip`` readily -available in virtual environments. When using the command line utility, -``pip`` is installed by default, while when using the :mod:`venv` module -:ref:`venv-api` installation of ``pip`` must be requested explicitly. - -For CPython :ref:`source builds on POSIX systems <building-python-on-unix>`, -the ``make install`` and ``make altinstall`` commands bootstrap ``pip`` by -default. This behaviour can be controlled through configure options, and -overridden through Makefile options. - -On Windows and Mac OS X, the CPython installers now default to installing -``pip`` along with CPython itself (users may opt out of installing it -during the installation process). Window users will need to opt in to the -automatic ``PATH`` modifications to have ``pip`` available from the command -line by default, otherwise it can still be accessed through the Python -launcher for Windows as ``py -m pip``. - -As `discussed in the PEP`__, platform packagers may choose not to install -these commands by default, as long as, when invoked, they provide clear and -simple directions on how to install them on that platform (usually using -the system package manager). - -__ https://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distributors - -.. note:: - - To avoid conflicts between parallel Python 2 and Python 3 installations, - only the versioned ``pip3`` and ``pip3.4`` commands are bootstrapped by - default when ``ensurepip`` is invoked directly - the ``--default-pip`` - option is needed to also request the unversioned ``pip`` command. - ``pyvenv`` and the Windows installer ensure that the unqualified ``pip`` - command is made available in those environments, and ``pip`` can always be - invoked via the ``-m`` switch rather than directly to avoid ambiguity on - systems with multiple Python installations. - - -Documentation Changes -~~~~~~~~~~~~~~~~~~~~~ - -As part of this change, the :ref:`installing-index` and -:ref:`distributing-index` sections of the documentation have been -completely redesigned as short getting started and FAQ documents. Most -packaging documentation has now been moved out to the Python Packaging -Authority maintained `Python Packaging User Guide -<https://packaging.python.org>`__ and the documentation of the individual -projects. - -However, as this migration is currently still incomplete, the legacy -versions of those guides remaining available as :ref:`install-index` -and :ref:`distutils-index`. - -.. seealso:: - - :pep:`453` -- Explicit bootstrapping of pip in Python installations - PEP written by Donald Stufft and Nick Coghlan, implemented by - Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily. - - -.. _whatsnew-pep-446: - -PEP 446: Newly Created File Descriptors Are Non-Inheritable ------------------------------------------------------------ - -:pep:`446` makes newly created file descriptors :ref:`non-inheritable -<fd_inheritance>`. In general, this is the behavior an application will -want: when launching a new process, having currently open files also -open in the new process can lead to all sorts of hard to find bugs, -and potentially to security issues. - -However, there are occasions when inheritance is desired. To support -these cases, the following new functions and methods are available: - -* :func:`os.get_inheritable`, :func:`os.set_inheritable` -* :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable` -* :meth:`socket.socket.get_inheritable`, :meth:`socket.socket.set_inheritable` - -.. seealso:: - - :pep:`446` -- Make newly created file descriptors non-inheritable - PEP written and implemented by Victor Stinner. - - -.. _codec-handling-improvements: - -Improvements to Codec Handling ------------------------------- - -Since it was first introduced, the :mod:`codecs` module has always been -intended to operate as a type-neutral dynamic encoding and decoding -system. However, its close coupling with the Python text model, especially -the type restricted convenience methods on the builtin :class:`str`, -:class:`bytes` and :class:`bytearray` types, has historically obscured that -fact. - -As a key step in clarifying the situation, the :meth:`codecs.encode` and -:meth:`codecs.decode` convenience functions are now properly documented in -Python 2.7, 3.3 and 3.4. These functions have existed in the :mod:`codecs` -module (and have been covered by the regression test suite) since Python 2.4, -but were previously only discoverable through runtime introspection. - -Unlike the convenience methods on :class:`str`, :class:`bytes` and -:class:`bytearray`, the :mod:`codecs` convenience functions support arbitrary -codecs in both Python 2 and Python 3, rather than being limited to Unicode text -encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions (in -Python 2). - -In Python 3.4, the interpreter is able to identify the known non-text -encodings provided in the standard library and direct users towards these -general purpose convenience functions when appropriate:: - - >>> b"abcdef".decode("hex") - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - LookupError: 'hex' is not a text encoding; use codecs.decode() to handle arbitrary codecs - - >>> "hello".encode("rot13") - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - LookupError: 'rot13' is not a text encoding; use codecs.encode() to handle arbitrary codecs - - >>> open("foo.txt", encoding="hex") - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - LookupError: 'hex' is not a text encoding; use codecs.open() to handle arbitrary codecs - -In a related change, whenever it is feasible without breaking backwards -compatibility, exceptions raised during encoding and decoding operations -are wrapped in a chained exception of the same type that mentions the -name of the codec responsible for producing the error:: - - >>> import codecs - - >>> codecs.decode(b"abcdefgh", "hex") - Traceback (most recent call last): - File "/usr/lib/python3.4/encodings/hex_codec.py", line 20, in hex_decode - return (binascii.a2b_hex(input), len(input)) - binascii.Error: Non-hexadecimal digit found - - The above exception was the direct cause of the following exception: - - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - binascii.Error: decoding with 'hex' codec failed (Error: Non-hexadecimal digit found) - - >>> codecs.encode("hello", "bz2") - Traceback (most recent call last): - File "/usr/lib/python3.4/encodings/bz2_codec.py", line 17, in bz2_encode - return (bz2.compress(input), len(input)) - File "/usr/lib/python3.4/bz2.py", line 498, in compress - return comp.compress(data) + comp.flush() - TypeError: 'str' does not support the buffer interface - - The above exception was the direct cause of the following exception: - - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - TypeError: encoding with 'bz2' codec failed (TypeError: 'str' does not support the buffer interface) - -Finally, as the examples above show, these improvements have permitted -the restoration of the convenience aliases for the non-Unicode codecs that -were themselves restored in Python 3.2. This means that encoding binary data -to and from its hexadecimal representation (for example) can now be written -as:: - - >>> from codecs import encode, decode - >>> encode(b"hello", "hex") - b'68656c6c6f' - >>> decode(b"68656c6c6f", "hex") - b'hello' - -The binary and text transforms provided in the standard library are detailed -in :ref:`binary-transforms` and :ref:`text-transforms`. - -(Contributed by Nick Coghlan in :issue:`7475`, :issue:`17827`, -:issue:`17828` and :issue:`19619`.) - - -.. _whatsnew-pep-451: - -PEP 451: A ModuleSpec Type for the Import System ------------------------------------------------- - -:pep:`451` provides an encapsulation of the information about a module that the -import machinery will use to load it (that is, a module specification). This -helps simplify both the import implementation and several import-related APIs. -The change is also a stepping stone for `several future import-related -improvements`__. - -__ https://mail.python.org/pipermail/python-dev/2013-November/130111.html - -The public-facing changes from the PEP are entirely backward-compatible. -Furthermore, they should be transparent to everyone but importer authors. Key -finder and loader methods have been deprecated, but they will continue working. -New importers should use the new methods described in the PEP. Existing -importers should be updated to implement the new methods. See the -:ref:`deprecated-3.4` section for a list of methods that should be replaced and -their replacements. - - -Other Language Changes ----------------------- - -Some smaller changes made to the core Python language are: - -* Unicode database updated to UCD version 6.3. - -* :func:`min` and :func:`max` now accept a *default* keyword-only argument that - can be used to specify the value they return if the iterable they are - evaluating has no elements. (Contributed by Julian Berman in - :issue:`18111`.) - -* Module objects are now :mod:`weakref`'able. - -* Module ``__file__`` attributes (and related values) should now always - contain absolute paths by default, with the sole exception of - ``__main__.__file__`` when a script has been executed directly using - a relative path. (Contributed by Brett Cannon in :issue:`18416`.) - -* All the UTF-\* codecs (except UTF-7) now reject surrogates during both - encoding and decoding unless the ``surrogatepass`` error handler is used, - with the exception of the UTF-16 decoder (which accepts valid surrogate pairs) - and the UTF-16 encoder (which produces them while encoding non-BMP characters). - (Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in - :issue:`12892`.) - -* New German EBCDIC :ref:`codec <standard-encodings>` ``cp273``. (Contributed - by Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.) - -* New Ukrainian :ref:`codec <standard-encodings>` ``cp1125``. (Contributed by - Serhiy Storchaka in :issue:`19668`.) - -* :class:`bytes`.join() and :class:`bytearray`.join() now accept arbitrary - buffer objects as arguments. (Contributed by Antoine Pitrou in - :issue:`15958`.) - -* The :class:`int` constructor now accepts any object that has an ``__index__`` - method for its *base* argument. (Contributed by Mark Dickinson in - :issue:`16772`.) - -* Frame objects now have a :func:`~frame.clear` method that clears all - references to local variables from the frame. (Contributed by Antoine Pitrou - in :issue:`17934`.) - -* :class:`memoryview` is now registered as a :class:`Sequence <collections.abc>`, - and supports the :func:`reversed` builtin. (Contributed by Nick Coghlan - and Claudiu Popa in :issue:`18690` and :issue:`19078`.) - -* Signatures reported by :func:`help` have been modified and improved in - several cases as a result of the introduction of Argument Clinic and other - changes to the :mod:`inspect` and :mod:`pydoc` modules. - -* :meth:`~object.__length_hint__` is now part of the formal language - specification (see :pep:`424`). (Contributed by Armin Ronacher in - :issue:`16148`.) - - -New Modules -=========== - - -.. _whatsnew-asyncio: - -asyncio -------- - -The new :mod:`asyncio` module (defined in :pep:`3156`) provides a standard -pluggable event loop model for Python, providing solid asynchronous IO -support in the standard library, and making it easier for other event loop -implementations to interoperate with the standard library and each other. - -For Python 3.4, this module is considered a :term:`provisional API`. - -.. seealso:: - - :pep:`3156` -- Asynchronous IO Support Rebooted: the "asyncio" Module - PEP written and implementation led by Guido van Rossum. - - -.. _whatsnew-ensurepip: - -ensurepip ---------- - -The new :mod:`ensurepip` module is the primary infrastructure for the -:pep:`453` implementation. In the normal course of events end users will not -need to interact with this module, but it can be used to manually bootstrap -``pip`` if the automated bootstrapping into an installation or virtual -environment was declined. - -:mod:`ensurepip` includes a bundled copy of ``pip``, up-to-date as of the first -release candidate of the release of CPython with which it ships (this applies -to both maintenance releases and feature releases). ``ensurepip`` does not -access the internet. If the installation has Internet access, after -``ensurepip`` is run the bundled ``pip`` can be used to upgrade ``pip`` to a -more recent release than the bundled one. (Note that such an upgraded version -of ``pip`` is considered to be a separately installed package and will not be -removed if Python is uninstalled.) - -The module is named *ensure*\ pip because if called when ``pip`` is already -installed, it does nothing. It also has an ``--upgrade`` option that will -cause it to install the bundled copy of ``pip`` if the existing installed -version of ``pip`` is older than the bundled copy. - - -.. _whatsnew-enum: - -enum ----- - -The new :mod:`enum` module (defined in :pep:`435`) provides a standard -implementation of enumeration types, allowing other modules (such as -:mod:`socket`) to provide more informative error messages and better -debugging support by replacing opaque integer constants with backwards -compatible enumeration values. - -.. seealso:: - - :pep:`435` -- Adding an Enum type to the Python standard library - PEP written by Barry Warsaw, Eli Bendersky and Ethan Furman, - implemented by Ethan Furman. - - -.. _whatsnew-pathlib: - -pathlib -------- - -The new :mod:`pathlib` module offers classes representing filesystem paths -with semantics appropriate for different operating systems. Path classes are -divided between *pure paths*, which provide purely computational operations -without I/O, and *concrete paths*, which inherit from pure paths but also -provide I/O operations. - -For Python 3.4, this module is considered a :term:`provisional API`. - -.. seealso:: - - :pep:`428` -- The pathlib module -- object-oriented filesystem paths - PEP written and implemented by Antoine Pitrou. - - -.. _whatsnew-selectors: - -selectors ---------- - -The new :mod:`selectors` module (created as part of implementing :pep:`3156`) -allows high-level and efficient I/O multiplexing, built upon the -:mod:`select` module primitives. - - -.. _whatsnew-statistics: - -statistics ----------- - -The new :mod:`statistics` module (defined in :pep:`450`) offers some core -statistics functionality directly in the standard library. This module -supports calculation of the mean, median, mode, variance and standard -deviation of a data series. - -.. seealso:: - - :pep:`450` -- Adding A Statistics Module To The Standard Library - PEP written and implemented by Steven D'Aprano - -.. _whatsnew-tracemalloc: - - -tracemalloc ------------ - -The new :mod:`tracemalloc` module (defined in :pep:`454`) is a debug tool to -trace memory blocks allocated by Python. It provides the following information: - -* Trace where an object was allocated -* Statistics on allocated memory blocks per filename and per line number: - total size, number and average size of allocated memory blocks -* Compute the differences between two snapshots to detect memory leaks - -.. seealso:: - - :pep:`454` -- Add a new tracemalloc module to trace Python memory allocations - PEP written and implemented by Victor Stinner - - - -Improved Modules -================ - - -abc ---- - -New function :func:`abc.get_cache_token` can be used to know when to invalidate -caches that are affected by changes in the object graph. (Contributed -by Łukasz Langa in :issue:`16832`.) - -New class :class:`~abc.ABC` has :class:`~abc.ABCMeta` as its meta class. -Using ``ABC`` as a base class has essentially the same effect as specifying -``metaclass=abc.ABCMeta``, but is simpler to type and easier to read. -(Contributed by Bruno Dupuis in :issue:`16049`.) - - -aifc ----- - -The :meth:`~aifc.aifc.getparams` method now returns a namedtuple rather than a -plain tuple. (Contributed by Claudiu Popa in :issue:`17818`.) - -:func:`aifc.open` now supports the context management protocol: when used in a -:keyword:`with` block, the :meth:`~aifc.aifc.close` method of the returned -object will be called automatically at the end of the block. (Contributed by -Serhiy Storchacha in :issue:`16486`.) - -The :meth:`~aifc.aifc.writeframesraw` and :meth:`~aifc.aifc.writeframes` -methods now accept any :term:`bytes-like object`. (Contributed by Serhiy -Storchaka in :issue:`8311`.) - - -argparse --------- - -The :class:`~argparse.FileType` class now accepts *encoding* and -*errors* arguments, which are passed through to :func:`open`. (Contributed -by Lucas Maystre in :issue:`11175`.) - - -audioop -------- - -:mod:`audioop` now supports 24-bit samples. (Contributed by Serhiy Storchaka -in :issue:`12866`.) - -New :func:`~audioop.byteswap` function converts big-endian samples to -little-endian and vice versa. (Contributed by Serhiy Storchaka in -:issue:`19641`.) - -All :mod:`audioop` functions now accept any :term:`bytes-like object`. Strings -are not accepted: they didn't work before, now they raise an error right away. -(Contributed by Serhiy Storchaka in :issue:`16685`.) - - -base64 ------- - -The encoding and decoding functions in :mod:`base64` now accept any -:term:`bytes-like object` in cases where it previously required a -:class:`bytes` or :class:`bytearray` instance. (Contributed by Nick Coghlan in -:issue:`17839`.) - -New functions :func:`~base64.a85encode`, :func:`~base64.a85decode`, -:func:`~base64.b85encode`, and :func:`~base64.b85decode` provide the ability to -encode and decode binary data from and to ``Ascii85`` and the git/mercurial -``Base85`` formats, respectively. The ``a85`` functions have options that can -be used to make them compatible with the variants of the ``Ascii85`` encoding, -including the Adobe variant. (Contributed by Martin Morrison, the Mercurial -project, Serhiy Storchaka, and Antoine Pitrou in :issue:`17618`.) - - -collections ------------ - -The :meth:`.ChainMap.new_child` method now accepts an *m* argument specifying -the child map to add to the chain. This allows an existing mapping and/or a -custom mapping type to be used for the child. (Contributed by Vinay Sajip in -:issue:`16613`.) - - -colorsys --------- - -The number of digits in the coefficients for the RGB --- YIQ conversions have -been expanded so that they match the FCC NTSC versions. The change in -results should be less than 1% and may better match results found elsewhere. -(Contributed by Brian Landers and Serhiy Storchaka in :issue:`14323`.) - - -contextlib ----------- - -The new :class:`contextlib.suppress` context manager helps to clarify the -intent of code that deliberately suppresses exceptions from a single -statement. (Contributed by Raymond Hettinger in :issue:`15806` and -Zero Piraeus in :issue:`19266`.) - -The new :func:`contextlib.redirect_stdout` context manager makes it easier -for utility scripts to handle inflexible APIs that write their output to -:data:`sys.stdout` and don't provide any options to redirect it. Using the -context manager, the :data:`sys.stdout` output can be redirected to any -other stream or, in conjunction with :class:`io.StringIO`, to a string. -The latter can be especially useful, for example, to capture output -from a function that was written to implement a command line interface. -It is recommended only for utility scripts because it affects the -global state of :data:`sys.stdout`. (Contributed by Raymond Hettinger -in :issue:`15805`.) - -The :mod:`contextlib` documentation has also been updated to include a -:ref:`discussion <single-use-reusable-and-reentrant-cms>` of the -differences between single use, reusable and reentrant context managers. - - -dbm ---- - -:func:`dbm.open` objects now support the context management protocol. When -used in a :keyword:`with` statement, the ``close`` method of the database -object will be called automatically at the end of the block. (Contributed by -Claudiu Popa and Nick Coghlan in :issue:`19282`.) - - -dis ---- - -Functions :func:`~dis.show_code`, :func:`~dis.dis`, :func:`~dis.distb`, and -:func:`~dis.disassemble` now accept a keyword-only *file* argument that -controls where they write their output. - -The :mod:`dis` module is now built around an :class:`~dis.Instruction` class -that provides object oriented access to the details of each individual bytecode -operation. - -A new method, :func:`~dis.get_instructions`, provides an iterator that emits -the Instruction stream for a given piece of Python code. Thus it is now -possible to write a program that inspects and manipulates a bytecode -object in ways different from those provided by the :mod:`~dis` module -itself. For example:: - - >>> import dis - >>> for instr in dis.get_instructions(lambda x: x + 1): - ... print(instr.opname) - LOAD_FAST - LOAD_CONST - BINARY_ADD - RETURN_VALUE - -The various display tools in the :mod:`dis` module have been rewritten to use -these new components. - -In addition, a new application-friendly class :class:`~dis.Bytecode` provides -an object-oriented API for inspecting bytecode in both in human-readable form -and for iterating over instructions. The :class:`~dis.Bytecode` constructor -takes the same arguments that :func:`~dis.get_instruction` does (plus an -optional *current_offset*), and the resulting object can be iterated to produce -:class:`~dis.Instruction` objects. But it also has a :mod:`~dis.Bytecode.dis` -method, equivalent to calling :mod:`~dis.dis` on the constructor argument, but -returned as a multi-line string:: - - >>> bytecode = dis.Bytecode(lambda x: x + 1, current_offset=3) - >>> for instr in bytecode: - ... print('{} ({})'.format(instr.opname, instr.opcode)) - LOAD_FAST (124) - LOAD_CONST (100) - BINARY_ADD (23) - RETURN_VALUE (83) - >>> bytecode.dis().splitlines() # doctest: +NORMALIZE_WHITESPACE - [' 1 0 LOAD_FAST 0 (x)', - ' --> 3 LOAD_CONST 1 (1)', - ' 6 BINARY_ADD', - ' 7 RETURN_VALUE'] - -:class:`~dis.Bytecode` also has a class method, -:meth:`~dis.Bytecode.from_traceback`, that provides the ability to manipulate a -traceback (that is, ``print(Bytecode.from_traceback(tb).dis())`` is equivalent -to ``distb(tb)``). - -(Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in :issue:`11816` -and Claudiu Popa in :issue:`17916`.) - -New function :func:`~dis.stack_effect` computes the effect on the Python stack -of a given opcode and argument, information that is not otherwise available. -(Contributed by Larry Hastings in :issue:`19722`.) - - -doctest -------- - -A new :ref:`option flag <doctest-options>`, :data:`~doctest.FAIL_FAST`, halts -test running as soon as the first failure is detected. (Contributed by R. -David Murray and Daniel Urban in :issue:`16522`.) - -The :mod:`doctest` command line interface now uses :mod:`argparse`, and has two -new options, ``-o`` and ``-f``. ``-o`` allows :ref:`doctest options -<doctest-options>` to be specified on the command line, and ``-f`` is a -shorthand for ``-o FAIL_FAST`` (to parallel the similar option supported by the -:mod:`unittest` CLI). (Contributed by R. David Murray in :issue:`11390`.) - -:mod:`doctest` will now find doctests in extension module ``__doc__`` strings. -(Contributed by Zachary Ware in :issue:`3158`.) - - -email ------ - -:meth:`~email.message.Message.as_string` now accepts a *policy* argument to -override the default policy of the message when generating a string -representation of it. This means that ``as_string`` can now be used in more -circumstances, instead of having to create and use a :mod:`~email.generator` in -order to pass formatting parameters to its ``flatten`` method. (Contributed by -R. David Murray in :issue:`18600`.) - -New method :meth:`~email.message.Message.as_bytes` added to produce a bytes -representation of the message in a fashion similar to how ``as_string`` -produces a string representation. It does not accept the *maxheaderlen* -argument, but does accept the *unixfrom* and *policy* arguments. The -:class:`~email.message.Message` :meth:`~email.message.Message.__bytes__` method -calls it, meaning that ``bytes(mymsg)`` will now produce the intuitive -result: a bytes object containing the fully formatted message. (Contributed -by R. David Murray in :issue:`18600`.) - -The :meth:`.Message.set_param` message now accepts a *replace* keyword argument. -When specified, the associated header will be updated without changing -its location in the list of headers. For backward compatibility, the default -is ``False``. (Contributed by R. David Murray in :issue:`18891`.) - - -.. _whatsnew_email_contentmanager: - -A pair of new subclasses of :class:`~email.message.Message` have been added -(:class:`.EmailMessage` and :class:`.MIMEPart`), along with a new sub-module, -:mod:`~email.contentmanager` and a new :mod:`~email.policy` attribute -:attr:`~email.policy.EmailPolicy.content_manager`. All documentation is -currently in the new module, which is being added as part of email's new -:term:`provisional API`. These classes provide a number of new methods that -make extracting content from and inserting content into email messages much -easier. For details, see the :mod:`~email.contentmanager` documentation and -the :ref:`email-examples`. These API additions complete the -bulk of the work that was planned as part of the email6 project. The currently -provisional API is scheduled to become final in Python 3.5 (possibly with a few -minor additions in the area of error handling). (Contributed by R. David -Murray in :issue:`18891`.) - - -filecmp -------- - -A new :func:`~filecmp.clear_cache` function provides the ability to clear the -:mod:`filecmp` comparison cache, which uses :func:`os.stat` information to -determine if the file has changed since the last compare. This can be used, -for example, if the file might have been changed and re-checked in less time -than the resolution of a particular filesystem's file modification time field. -(Contributed by Mark Levitt in :issue:`18149`.) - -New module attribute :data:`~filecmp.DEFAULT_IGNORES` provides the list of -directories that are used as the default value for the *ignore* parameter of -the :func:`~filecmp.dircmp` function. (Contributed by Eli Bendersky in -:issue:`15442`.) - - -functools ---------- - -The new :func:`~functools.partialmethod` descriptor brings partial argument -application to descriptors, just as :func:`~functools.partial` provides -for normal callables. The new descriptor also makes it easier to get -arbitrary callables (including :func:`~functools.partial` instances) -to behave like normal instance methods when included in a class definition. -(Contributed by Alon Horev and Nick Coghlan in :issue:`4331`.) - -.. _whatsnew-singledispatch: - -The new :func:`~functools.singledispatch` decorator brings support for -single-dispatch generic functions to the Python standard library. Where -object oriented programming focuses on grouping multiple operations on a -common set of data into a class, a generic function focuses on grouping -multiple implementations of an operation that allows it to work with -*different* kinds of data. - -.. seealso:: - - :pep:`443` -- Single-dispatch generic functions - PEP written and implemented by Łukasz Langa. - -:func:`~functools.total_ordering` now supports a return value of -:const:`NotImplemented` from the underlying comparison function. (Contributed -by Katie Miller in :issue:`10042`.) - -A pure-python version of the :func:`~functools.partial` function is now in the -stdlib; in CPython it is overridden by the C accelerated version, but it is -available for other implementations to use. (Contributed by Brian Thorne in -:issue:`12428`.) - - -gc --- - -New function :func:`~gc.get_stats` returns a list of three per-generation -dictionaries containing the collections statistics since interpreter startup. -(Contributed by Antoine Pitrou in :issue:`16351`.) - - -glob ----- - -A new function :func:`~glob.escape` provides a way to escape special characters -in a filename so that they do not become part of the globbing expansion but are -instead matched literally. (Contributed by Serhiy Storchaka in :issue:`8402`.) - - -hashlib -------- - -A new :func:`hashlib.pbkdf2_hmac` function provides -the `PKCS#5 password-based key derivation function 2 -<https://en.wikipedia.org/wiki/PBKDF2>`_. (Contributed by Christian -Heimes in :issue:`18582`.) - -The :attr:`~hashlib.hash.name` attribute of :mod:`hashlib` hash objects is now -a formally supported interface. It has always existed in CPython's -:mod:`hashlib` (although it did not return lower case names for all supported -hashes), but it was not a public interface and so some other Python -implementations have not previously supported it. (Contributed by Jason R. -Coombs in :issue:`18532`.) - - -hmac ----- - -:mod:`hmac` now accepts ``bytearray`` as well as ``bytes`` for the *key* -argument to the :func:`~hmac.new` function, and the *msg* parameter to both the -:func:`~hmac.new` function and the :meth:`~hmac.HMAC.update` method now -accepts any type supported by the :mod:`hashlib` module. (Contributed -by Jonas Borgström in :issue:`18240`.) - -The *digestmod* argument to the :func:`hmac.new` function may now be any hash -digest name recognized by :mod:`hashlib`. In addition, the current behavior in -which the value of *digestmod* defaults to ``MD5`` is deprecated: in a -future version of Python there will be no default value. (Contributed by -Christian Heimes in :issue:`17276`.) - -With the addition of :attr:`~hmac.HMAC.block_size` and :attr:`~hmac.HMAC.name` -attributes (and the formal documentation of the :attr:`~hmac.HMAC.digest_size` -attribute), the :mod:`hmac` module now conforms fully to the :pep:`247` API. -(Contributed by Christian Heimes in :issue:`18775`.) - - -html ----- - -New function :func:`~html.unescape` function converts HTML5 character references to -the corresponding Unicode characters. (Contributed by Ezio Melotti in -:issue:`2927`.) - -:class:`~html.parser.HTMLParser` accepts a new keyword argument -*convert_charrefs* that, when ``True``, automatically converts all character -references. For backward-compatibility, its value defaults to ``False``, but -it will change to ``True`` in a future version of Python, so you are invited to -set it explicitly and update your code to use this new feature. (Contributed -by Ezio Melotti in :issue:`13633`.) - -The *strict* argument of :class:`~html.parser.HTMLParser` is now deprecated. -(Contributed by Ezio Melotti in :issue:`15114`.) - - -http ----- - -:meth:`~http.server.BaseHTTPRequestHandler.send_error` now accepts an -optional additional *explain* parameter which can be used to provide an -extended error description, overriding the hardcoded default if there is one. -This extended error description will be formatted using the -:attr:`~http.server.HTTP.error_message_format` attribute and sent as the body -of the error response. (Contributed by Karl Cow in :issue:`12921`.) - -The :mod:`http.server` :ref:`command line interface <http-server-cli>` now has -a ``-b/--bind`` option that causes the server to listen on a specific address. -(Contributed by Malte Swart in :issue:`17764`.) - - -idlelib and IDLE ----------------- - -Since idlelib implements the IDLE shell and editor and is not intended for -import by other programs, it gets improvements with every release. See -:file:`Lib/idlelib/NEWS.txt` for a cumulative list of changes since 3.3.0, -as well as changes made in future 3.4.x releases. This file is also available -from the IDLE :menuselection:`Help --> About IDLE` dialog. - - -importlib ---------- - -The :class:`~importlib.abc.InspectLoader` ABC defines a new method, -:meth:`~importlib.abc.InspectLoader.source_to_code` that accepts source -data and a path and returns a code object. The default implementation -is equivalent to ``compile(data, path, 'exec', dont_inherit=True)``. -(Contributed by Eric Snow and Brett Cannon in :issue:`15627`.) - -:class:`~importlib.abc.InspectLoader` also now has a default implementation -for the :meth:`~importlib.abc.InspectLoader.get_code` method. However, -it will normally be desirable to override the default implementation -for performance reasons. (Contributed by Brett Cannon in :issue:`18072`.) - -The :func:`~importlib.reload` function has been moved from :mod:`imp` to -:mod:`importlib` as part of the :mod:`imp` module deprecation. (Contributed by -Berker Peksag in :issue:`18193`.) - -:mod:`importlib.util` now has a :data:`~importlib.util.MAGIC_NUMBER` attribute -providing access to the bytecode version number. This replaces the -:func:`~imp.get_magic` function in the deprecated :mod:`imp` module. -(Contributed by Brett Cannon in :issue:`18192`.) - -New :mod:`importlib.util` functions :func:`~importlib.util.cache_from_source` -and :func:`~importlib.util.source_from_cache` replace the same-named functions -in the deprecated :mod:`imp` module. (Contributed by Brett Cannon in -:issue:`18194`.) - -The :mod:`importlib` bootstrap :class:`.NamespaceLoader` now conforms to -the :class:`.InspectLoader` ABC, which means that ``runpy`` and -``python -m`` can now be used with namespace packages. (Contributed -by Brett Cannon in :issue:`18058`.) - -:mod:`importlib.util` has a new function :func:`~importlib.util.decode_source` -that decodes source from bytes using universal newline processing. This is -useful for implementing :meth:`.InspectLoader.get_source` methods. - -:class:`importlib.machinery.ExtensionFileLoader` now has a -:meth:`~importlib.machinery.ExtensionFileLoader.get_filename` method. This was -inadvertently omitted in the original implementation. (Contributed by Eric -Snow in :issue:`19152`.) - - -inspect -------- - -The :mod:`inspect` module now offers a basic :ref:`command line interface -<inspect-module-cli>` to quickly display source code and other -information for modules, classes and functions. (Contributed by Claudiu Popa -and Nick Coghlan in :issue:`18626`.) - -:func:`~inspect.unwrap` makes it easy to unravel wrapper function chains -created by :func:`functools.wraps` (and any other API that sets the -``__wrapped__`` attribute on a wrapper function). (Contributed by -Daniel Urban, Aaron Iles and Nick Coghlan in :issue:`13266`.) - -As part of the implementation of the new :mod:`enum` module, the -:mod:`inspect` module now has substantially better support for custom -``__dir__`` methods and dynamic class attributes provided through -metaclasses. (Contributed by Ethan Furman in :issue:`18929` and -:issue:`19030`.) - -:func:`~inspect.getfullargspec` and :func:`~inspect.getargspec` -now use the :func:`~inspect.signature` API. This allows them to -support a much broader range of callables, including those with -``__signature__`` attributes, those with metadata provided by argument -clinic, :func:`functools.partial` objects and more. Note that, unlike -:func:`~inspect.signature`, these functions still ignore ``__wrapped__`` -attributes, and report the already bound first argument for bound methods, -so it is still necessary to update your code to use -:func:`~inspect.signature` directly if those features are desired. -(Contributed by Yury Selivanov in :issue:`17481`.) - -:func:`~inspect.signature` now supports duck types of CPython functions, -which adds support for functions compiled with Cython. (Contributed -by Stefan Behnel and Yury Selivanov in :issue:`17159`.) - - -ipaddress ---------- - -:mod:`ipaddress` was added to the standard library in Python 3.3 as a -:term:`provisional API`. With the release of Python 3.4, this qualification -has been removed: :mod:`ipaddress` is now considered a stable API, covered -by the normal standard library requirements to maintain backwards -compatibility. - -A new :attr:`~ipaddress.IPv4Address.is_global` property is ``True`` if -an address is globally routeable. (Contributed by Peter Moody in -:issue:`17400`.) - - -logging -------- - -The :class:`~logging.handlers.TimedRotatingFileHandler` has a new *atTime* -parameter that can be used to specify the time of day when rollover should -happen. (Contributed by Ronald Oussoren in :issue:`9556`.) - -:class:`~logging.handlers.SocketHandler` and -:class:`~logging.handlers.DatagramHandler` now support Unix domain sockets (by -setting *port* to ``None``). (Contributed by Vinay Sajip in commit -ce46195b56a9.) - -:func:`~logging.config.fileConfig` now accepts a -:class:`configparser.RawConfigParser` subclass instance for the *fname* -parameter. This facilitates using a configuration file when logging -configuration is just a part of the overall application configuration, or where -the application modifies the configuration before passing it to -:func:`~logging.config.fileConfig`. (Contributed by Vinay Sajip in -:issue:`16110`.) - -Logging configuration data received from a socket via the -:func:`logging.config.listen` function can now be validated before being -processed by supplying a verification function as the argument to the new -*verify* keyword argument. (Contributed by Vinay Sajip in :issue:`15452`.) - - -.. _whatsnew-marshal-3: - -marshal -------- - -The default :mod:`marshal` version has been bumped to 3. The code implementing -the new version restores the Python2 behavior of recording only one copy of -interned strings and preserving the interning on deserialization, and extends -this "one copy" ability to any object type (including handling recursive -references). This reduces both the size of ``.pyc`` files and the amount of -memory a module occupies in memory when it is loaded from a ``.pyc`` (or -``.pyo``) file. (Contributed by Kristján Valur Jónsson in :issue:`16475`, -with additional speedups by Antoine Pitrou in :issue:`19219`.) - - -mmap ----- - -mmap objects can now be :mod:`weakref`\ ed. (Contributed by Valerie Lambert in -:issue:`4885`.) - - -multiprocessing ---------------- - -.. _whatsnew-multiprocessing-no-fork: - -On Unix two new :ref:`start methods <multiprocessing-start-methods>`, -``spawn`` and ``forkserver``, have been added for starting processes using -:mod:`multiprocessing`. These make the mixing of processes with threads more -robust, and the ``spawn`` method matches the semantics that multiprocessing has -always used on Windows. New function -:func:`~multiprocessing.get_all_start_methods` reports all start methods -available on the platform, :func:`~multiprocessing.get_start_method` reports -the current start method, and :func:`~multiprocessing.set_start_method` sets -the start method. (Contributed by Richard Oudkerk in :issue:`8713`.) - -:mod:`multiprocessing` also now has the concept of a ``context``, which -determines how child processes are created. New function -:func:`~multiprocessing.get_context` returns a context that uses a specified -start method. It has the same API as the :mod:`multiprocessing` module itself, -so you can use it to create :class:`~multiprocessing.pool.Pool`\ s and other -objects that will operate within that context. This allows a framework and an -application or different parts of the same application to use multiprocessing -without interfering with each other. (Contributed by Richard Oudkerk in -:issue:`18999`.) - -Except when using the old *fork* start method, child processes no longer -inherit unneeded handles/file descriptors from their parents (part of -:issue:`8713`). - -:mod:`multiprocessing` now relies on :mod:`runpy` (which implements the -``-m`` switch) to initialise ``__main__`` appropriately in child processes -when using the ``spawn`` or ``forkserver`` start methods. This resolves some -edge cases where combining multiprocessing, the ``-m`` command line switch, -and explicit relative imports could cause obscure failures in child -processes. (Contributed by Nick Coghlan in :issue:`19946`.) - - -operator --------- - -New function :func:`~operator.length_hint` provides an implementation of the -specification for how the :meth:`~object.__length_hint__` special method should -be used, as part of the :pep:`424` formal specification of this language -feature. (Contributed by Armin Ronacher in :issue:`16148`.) - -There is now a pure-python version of the :mod:`operator` module available for -reference and for use by alternate implementations of Python. (Contributed by -Zachary Ware in :issue:`16694`.) - - -os --- - -There are new functions to get and set the :ref:`inheritable flag -<fd_inheritance>` of a file descriptor (:func:`os.get_inheritable`, -:func:`os.set_inheritable`) or a Windows handle -(:func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable`). - -New function :func:`~os.cpu_count` reports the number of CPUs available on the -platform on which Python is running (or ``None`` if the count can't be -determined). The :func:`multiprocessing.cpu_count` function is now implemented -in terms of this function). (Contributed by Trent Nelson, Yogesh Chaudhari, -Victor Stinner, and Charles-François Natali in :issue:`17914`.) - -:func:`os.path.samestat` is now available on the Windows platform (and the -:func:`os.path.samefile` implementation is now shared between Unix and -Windows). (Contributed by Brian Curtin in :issue:`11939`.) - -:func:`os.path.ismount` now recognizes volumes mounted below a drive -root on Windows. (Contributed by Tim Golden in :issue:`9035`.) - -:func:`os.open` supports two new flags on platforms that provide them, -:data:`~os.O_PATH` (un-opened file descriptor), and :data:`~os.O_TMPFILE` -(unnamed temporary file; as of 3.4.0 release available only on Linux systems -with a kernel version of 3.11 or newer that have uapi headers). (Contributed -by Christian Heimes in :issue:`18673` and Benjamin Peterson, respectively.) - - -pdb ---- - -:mod:`pdb` has been enhanced to handle generators, :keyword:`yield`, and -``yield from`` in a more useful fashion. This is especially helpful when -debugging :mod:`asyncio` based programs. (Contributed by Andrew Svetlov and -Xavier de Gaye in :issue:`16596`.) - -The ``print`` command has been removed from :mod:`pdb`, restoring access to the -Python :func:`print` function from the pdb command line. Python2's ``pdb`` did -not have a ``print`` command; instead, entering ``print`` executed the -``print`` statement. In Python3 ``print`` was mistakenly made an alias for the -pdb :pdbcmd:`p` command. ``p``, however, prints the ``repr`` of its argument, -not the ``str`` like the Python2 ``print`` command did. Worse, the Python3 -``pdb print`` command shadowed the Python3 ``print`` function, making it -inaccessible at the ``pdb`` prompt. (Contributed by Connor Osborn in -:issue:`18764`.) - - -.. _whatsnew-protocol-4: - -pickle ------- - -:mod:`pickle` now supports (but does not use by default) a new pickle protocol, -protocol 4. This new protocol addresses a number of issues that were present -in previous protocols, such as the serialization of nested classes, very large -strings and containers, and classes whose :meth:`__new__` method takes -keyword-only arguments. It also provides some efficiency improvements. - -.. seealso:: - - :pep:`3154` -- Pickle protocol 4 - PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti. - - -plistlib --------- - -:mod:`plistlib` now has an API that is similar to the standard pattern for -stdlib serialization protocols, with new :func:`~plistlib.load`, -:func:`~plistlib.dump`, :func:`~plistlib.loads`, and :func:`~plistlib.dumps` -functions. (The older API is now deprecated.) In addition to the already -supported XML plist format (:data:`~plistlib.FMT_XML`), it also now supports -the binary plist format (:data:`~plistlib.FMT_BINARY`). (Contributed by Ronald -Oussoren and others in :issue:`14455`.) - - -poplib ------- - -Two new methods have been added to :mod:`poplib`: :meth:`~poplib.POP3.capa`, -which returns the list of capabilities advertised by the POP server, and -:meth:`~poplib.POP3.stls`, which switches a clear-text POP3 session into an -encrypted POP3 session if the POP server supports it. (Contributed by Lorenzo -Catucci in :issue:`4473`.) - - -pprint ------- - -The :mod:`pprint` module's :class:`~pprint.PrettyPrinter` class and its -:func:`~pprint.pformat`, and :func:`~pprint.pprint` functions have a new -option, *compact*, that controls how the output is formatted. Currently -setting *compact* to ``True`` means that sequences will be printed with as many -sequence elements as will fit within *width* on each (indented) line. -(Contributed by Serhiy Storchaka in :issue:`19132`.) - -Long strings are now wrapped using Python's normal line continuation -syntax. (Contributed by Antoine Pitrou in :issue:`17150`.) - - -pty ---- - -:func:`pty.spawn` now returns the status value from :func:`os.waitpid` on -the child process, instead of ``None``. (Contributed by Gregory P. Smith.) - - -pydoc ------ - -The :mod:`pydoc` module is now based directly on the :func:`inspect.signature` -introspection API, allowing it to provide signature information for a wider -variety of callable objects. This change also means that ``__wrapped__`` -attributes are now taken into account when displaying help information. -(Contributed by Larry Hastings in :issue:`19674`.) - -The :mod:`pydoc` module no longer displays the ``self`` parameter for -already bound methods. Instead, it aims to always display the exact current -signature of the supplied callable. (Contributed by Larry Hastings in -:issue:`20710`.) - -In addition to the changes that have been made to :mod:`pydoc` directly, -its handling of custom ``__dir__`` methods and various descriptor -behaviours has also been improved substantially by the underlying changes in -the :mod:`inspect` module. - -As the :func:`help` builtin is based on :mod:`pydoc`, the above changes also -affect the behaviour of :func:`help`. - - -re --- - -New :func:`~re.fullmatch` function and :meth:`.regex.fullmatch` method anchor -the pattern at both ends of the string to match. This provides a way to be -explicit about the goal of the match, which avoids a class of subtle bugs where -``$`` characters get lost during code changes or the addition of alternatives -to an existing regular expression. (Contributed by Matthew Barnett in -:issue:`16203`.) - -The repr of :ref:`regex objects <re-objects>` now includes the pattern -and the flags; the repr of :ref:`match objects <match-objects>` now -includes the start, end, and the part of the string that matched. (Contributed -by Hugo Lopes Tavares and Serhiy Storchaka in :issue:`13592` and -:issue:`17087`.) - - -resource --------- - -New :func:`~resource.prlimit` function, available on Linux platforms with a -kernel version of 2.6.36 or later and glibc of 2.13 or later, provides the -ability to query or set the resource limits for processes other than the one -making the call. (Contributed by Christian Heimes in :issue:`16595`.) - -On Linux kernel version 2.6.36 or later, there are also some new -Linux specific constants: :attr:`~resource.RLIMIT_MSGQUEUE`, -:attr:`~resource.RLIMIT_NICE`, :attr:`~resource.RLIMIT_RTPRIO`, -:attr:`~resource.RLIMIT_RTTIME`, and :attr:`~resource.RLIMIT_SIGPENDING`. -(Contributed by Christian Heimes in :issue:`19324`.) - -On FreeBSD version 9 and later, there some new FreeBSD specific constants: -:attr:`~resource.RLIMIT_SBSIZE`, :attr:`~resource.RLIMIT_SWAP`, and -:attr:`~resource.RLIMIT_NPTS`. (Contributed by Claudiu Popa in -:issue:`19343`.) - - -select ------- - -:class:`~select.epoll` objects now support the context management protocol. -When used in a :keyword:`with` statement, the :meth:`~select.epoll.close` -method will be called automatically at the end of the block. (Contributed -by Serhiy Storchaka in :issue:`16488`.) - -:class:`~select.devpoll` objects now have :meth:`~select.devpoll.fileno` and -:meth:`~select.devpoll.close` methods, as well as a new attribute -:attr:`~select.devpoll.closed`. (Contributed by Victor Stinner in -:issue:`18794`.) - - -shelve ------- - -:class:`~shelve.Shelf` instances may now be used in :keyword:`with` statements, -and will be automatically closed at the end of the :keyword:`!with` block. -(Contributed by Filip Gruszczyński in :issue:`13896`.) - - -shutil ------- - -:func:`~shutil.copyfile` now raises a specific :exc:`~shutil.Error` subclass, -:exc:`~shutil.SameFileError`, when the source and destination are the same -file, which allows an application to take appropriate action on this specific -error. (Contributed by Atsuo Ishimoto and Hynek Schlawack in -:issue:`1492704`.) - - -smtpd ------ - -The :class:`~smtpd.SMTPServer` and :class:`~smtpd.SMTPChannel` classes now -accept a *map* keyword argument which, if specified, is passed in to -:class:`asynchat.async_chat` as its *map* argument. This allows an application -to avoid affecting the global socket map. (Contributed by Vinay Sajip in -:issue:`11959`.) - - -smtplib -------- - -:exc:`~smtplib.SMTPException` is now a subclass of :exc:`OSError`, which allows -both socket level errors and SMTP protocol level errors to be caught in one -try/except statement by code that only cares whether or not an error occurred. -(Contributed by Ned Jackson Lovely in :issue:`2118`.) - - -socket ------- - -The socket module now supports the :data:`~socket.CAN_BCM` protocol on -platforms that support it. (Contributed by Brian Thorne in :issue:`15359`.) - -Socket objects have new methods to get or set their :ref:`inheritable flag -<fd_inheritance>`, :meth:`~socket.socket.get_inheritable` and -:meth:`~socket.socket.set_inheritable`. - -The ``socket.AF_*`` and ``socket.SOCK_*`` constants are now enumeration values -using the new :mod:`enum` module. This allows meaningful names to be printed -during debugging, instead of integer "magic numbers". - -The :data:`~socket.AF_LINK` constant is now available on BSD and OSX. - -:func:`~socket.inet_pton` and :func:`~socket.inet_ntop` are now supported -on Windows. (Contributed by Atsuo Ishimoto in :issue:`7171`.) - - -sqlite3 -------- - -A new boolean parameter to the :func:`~sqlite3.connect` function, *uri*, can be -used to indicate that the *database* parameter is a ``uri`` (see the `SQLite -URI documentation <https://www.sqlite.org/uri.html>`_). (Contributed by poq in -:issue:`13773`.) - - -ssl ---- - -.. _whatsnew-tls-11-12: - -:data:`~ssl.PROTOCOL_TLSv1_1` and :data:`~ssl.PROTOCOL_TLSv1_2` (TLSv1.1 and -TLSv1.2 support) have been added; support for these protocols is only available if -Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orrù and -Antoine Pitrou in :issue:`16692`.) - -.. _whatsnew34-sslcontext: - -New function :func:`~ssl.create_default_context` provides a standard way to -obtain an :class:`~ssl.SSLContext` whose settings are intended to be a -reasonable balance between compatibility and security. These settings are -more stringent than the defaults provided by the :class:`~ssl.SSLContext` -constructor, and may be adjusted in the future, without prior deprecation, if -best-practice security requirements change. The new recommended best -practice for using stdlib libraries that support SSL is to use -:func:`~ssl.create_default_context` to obtain an :class:`~ssl.SSLContext` -object, modify it if needed, and then pass it as the *context* argument -of the appropriate stdlib API. (Contributed by Christian Heimes -in :issue:`19689`.) - -:class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_verify_locations` -accepts a new optional argument *cadata*, which can be used to provide PEM or -DER encoded certificates directly via strings or bytes, respectively. -(Contributed by Christian Heimes in :issue:`18138`.) - -New function :func:`~ssl.get_default_verify_paths` returns -a named tuple of the paths and environment variables that the -:meth:`~ssl.SSLContext.set_default_verify_paths` method uses to set -OpenSSL's default ``cafile`` and ``capath``. This can be an aid in -debugging default verification issues. (Contributed by Christian Heimes -in :issue:`18143`.) - -:class:`~ssl.SSLContext` has a new method, -:meth:`~ssl.SSLContext.cert_store_stats`, that reports the number of loaded -``X.509`` certs, ``X.509 CA`` certs, and certificate revocation lists -(``crl``\ s), as well as a :meth:`~ssl.SSLContext.get_ca_certs` method that -returns a list of the loaded ``CA`` certificates. (Contributed by Christian -Heimes in :issue:`18147`.) - -If OpenSSL 0.9.8 or later is available, :class:`~ssl.SSLContext` has a new -attribute :attr:`~ssl.SSLContext.verify_flags` that can be used to control the -certificate verification process by setting it to some combination of the new -constants :data:`~ssl.VERIFY_DEFAULT`, :data:`~ssl.VERIFY_CRL_CHECK_LEAF`, -:data:`~ssl.VERIFY_CRL_CHECK_CHAIN`, or :data:`~ssl.VERIFY_X509_STRICT`. -OpenSSL does not do any CRL verification by default. (Contributed by -Christien Heimes in :issue:`8813`.) - -New :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_default_certs` -loads a set of default "certificate authority" (CA) certificates from default -locations, which vary according to the platform. It can be used to load both -TLS web server authentication certificates -(``purpose=``:data:`~ssl.Purpose.SERVER_AUTH`) for a client to use to verify a -server, and certificates for a server to use in verifying client certificates -(``purpose=``:data:`~ssl.Purpose.CLIENT_AUTH`). (Contributed by Christian -Heimes in :issue:`19292`.) - -.. _whatsnew34-win-cert-store: - -Two new windows-only functions, :func:`~ssl.enum_certificates` and -:func:`~ssl.enum_crls` provide the ability to retrieve certificates, -certificate information, and CRLs from the Windows cert store. (Contributed -by Christian Heimes in :issue:`17134`.) - -.. _whatsnew34-sni: - -Support for server-side SNI (Server Name Indication) using the new -:meth:`ssl.SSLContext.set_servername_callback` method. -(Contributed by Daniel Black in :issue:`8109`.) - -The dictionary returned by :meth:`.SSLSocket.getpeercert` contains additional -``X509v3`` extension items: ``crlDistributionPoints``, ``calIssuers``, and -``OCSP`` URIs. (Contributed by Christian Heimes in :issue:`18379`.) - - -stat ----- - -The :mod:`stat` module is now backed by a C implementation in :mod:`_stat`. A C -implementation is required as most of the values aren't standardized and -are platform-dependent. (Contributed by Christian Heimes in :issue:`11016`.) - -The module supports new :mod:`~stat.ST_MODE` flags, :mod:`~stat.S_IFDOOR`, -:attr:`~stat.S_IFPORT`, and :attr:`~stat.S_IFWHT`. (Contributed by -Christian Hiemes in :issue:`11016`.) - - -struct ------- - -New function :mod:`~struct.iter_unpack` and a new -:meth:`struct.Struct.iter_unpack` method on compiled formats provide streamed -unpacking of a buffer containing repeated instances of a given format of data. -(Contributed by Antoine Pitrou in :issue:`17804`.) - - -subprocess ----------- - -:func:`~subprocess.check_output` now accepts an *input* argument that can -be used to provide the contents of ``stdin`` for the command that is run. -(Contributed by Zack Weinberg in :issue:`16624`.) - -:func:`~subprocess.getstatus` and :func:`~subprocess.getstatusoutput` now -work on Windows. This change was actually inadvertently made in 3.3.4. -(Contributed by Tim Golden in :issue:`10197`.) - - -sunau ------ - -The :meth:`~sunau.getparams` method now returns a namedtuple rather than a -plain tuple. (Contributed by Claudiu Popa in :issue:`18901`.) - -:meth:`sunau.open` now supports the context management protocol: when used in a -:keyword:`with` block, the ``close`` method of the returned object will be -called automatically at the end of the block. (Contributed by Serhiy Storchaka -in :issue:`18878`.) - -:meth:`.AU_write.setsampwidth` now supports 24 bit samples, thus adding -support for writing 24 sample using the module. (Contributed by -Serhiy Storchaka in :issue:`19261`.) - -The :meth:`~sunau.AU_write.writeframesraw` and -:meth:`~sunau.AU_write.writeframes` methods now accept any :term:`bytes-like -object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) - - -sys ---- - -New function :func:`sys.getallocatedblocks` returns the current number of -blocks allocated by the interpreter. (In CPython with the default -``--with-pymalloc`` setting, this is allocations made through the -:c:func:`PyObject_Malloc` API.) This can be useful for tracking memory leaks, -especially if automated via a test suite. (Contributed by Antoine Pitrou -in :issue:`13390`.) - -When the Python interpreter starts in :ref:`interactive mode -<tut-interactive>`, it checks for an :data:`~sys.__interactivehook__` attribute -on the :mod:`sys` module. If the attribute exists, its value is called with no -arguments just before interactive mode is started. The check is made after the -:envvar:`PYTHONSTARTUP` file is read, so it can be set there. The :mod:`site` -module :ref:`sets it <rlcompleter-config>` to a function that enables tab -completion and history saving (in :file:`~/.python-history`) if the platform -supports :mod:`readline`. If you do not want this (new) behavior, you can -override it in :envvar:`PYTHONSTARTUP`, :mod:`sitecustomize`, or -:mod:`usercustomize` by deleting this attribute from :mod:`sys` (or setting it -to some other callable). (Contributed by Éric Araujo and Antoine Pitrou in -:issue:`5845`.) - - -tarfile -------- - -The :mod:`tarfile` module now supports a simple :ref:`tarfile-commandline` when -called as a script directly or via ``-m``. This can be used to create and -extract tarfile archives. (Contributed by Berker Peksag in :issue:`13477`.) - - -textwrap --------- - -The :class:`~textwrap.TextWrapper` class has two new attributes/constructor -arguments: :attr:`~textwrap.TextWrapper.max_lines`, which limits the number of -lines in the output, and :attr:`~textwrap.TextWrapper.placeholder`, which is a -string that will appear at the end of the output if it has been truncated -because of *max_lines*. Building on these capabilities, a new convenience -function :func:`~textwrap.shorten` collapses all of the whitespace in the input -to single spaces and produces a single line of a given *width* that ends with -the *placeholder* (by default, ``[...]``). (Contributed by Antoine Pitrou and -Serhiy Storchaka in :issue:`18585` and :issue:`18725`.) - - -threading ---------- - -The :class:`~threading.Thread` object representing the main thread can be -obtained from the new :func:`~threading.main_thread` function. In normal -conditions this will be the thread from which the Python interpreter was -started. (Contributed by Andrew Svetlov in :issue:`18882`.) - - -traceback ---------- - -A new :func:`traceback.clear_frames` function takes a traceback object -and clears the local variables in all of the frames it references, -reducing the amount of memory consumed. (Contributed by Andrew Kuchling in -:issue:`1565525`.) - - -types ------ - -A new :func:`~types.DynamicClassAttribute` descriptor provides a way to define -an attribute that acts normally when looked up through an instance object, but -which is routed to the *class* ``__getattr__`` when looked up through the -class. This allows one to have properties active on a class, and have virtual -attributes on the class with the same name (see :mod:`Enum` for an example). -(Contributed by Ethan Furman in :issue:`19030`.) - - -urllib ------- - -:mod:`urllib.request` now supports ``data:`` URLs via the -:class:`~urllib.request.DataHandler` class. (Contributed by Mathias Panzenböck -in :issue:`16423`.) - -The http method that will be used by a :class:`~urllib.request.Request` class -can now be specified by setting a :class:`~urllib.request.Request.method` -class attribute on the subclass. (Contributed by Jason R Coombs in -:issue:`18978`.) - -:class:`~urllib.request.Request` objects are now reusable: if the -:attr:`~urllib.request.Request.full_url` or :attr:`~urllib.request.Request.data` -attributes are modified, all relevant internal properties are updated. This -means, for example, that it is now possible to use the same -:class:`~urllib.request.Request` object in more than one -:meth:`.OpenerDirector.open` call with different *data* arguments, or to -modify a :class:`~urllib.request.Request`\ 's ``url`` rather than recomputing it -from scratch. There is also a new -:meth:`~urllib.request.Request.remove_header` method that can be used to remove -headers from a :class:`~urllib.request.Request`. (Contributed by Alexey -Kachayev in :issue:`16464`, Daniel Wozniak in :issue:`17485`, and Damien Brecht -and Senthil Kumaran in :issue:`17272`.) - -:class:`~urllib.error.HTTPError` objects now have a -:attr:`~urllib.error.HTTPError.headers` attribute that provides access to the -HTTP response headers associated with the error. (Contributed by -Berker Peksag in :issue:`15701`.) - - -unittest --------- - -The :class:`~unittest.TestCase` class has a new method, -:meth:`~unittest.TestCase.subTest`, that produces a context manager whose -:keyword:`with` block becomes a "sub-test". This context manager allows a test -method to dynamically generate subtests by, say, calling the ``subTest`` -context manager inside a loop. A single test method can thereby produce an -indefinite number of separately-identified and separately-counted tests, all of -which will run even if one or more of them fail. For example:: - - class NumbersTest(unittest.TestCase): - def test_even(self): - for i in range(6): - with self.subTest(i=i): - self.assertEqual(i % 2, 0) - -will result in six subtests, each identified in the unittest verbose output -with a label consisting of the variable name ``i`` and a particular value for -that variable (``i=0``, ``i=1``, etc). See :ref:`subtests` for the full -version of this example. (Contributed by Antoine Pitrou in :issue:`16997`.) - -:func:`unittest.main` now accepts an iterable of test names for -*defaultTest*, where previously it only accepted a single test name as a -string. (Contributed by Jyrki Pulliainen in :issue:`15132`.) - -If :class:`~unittest.SkipTest` is raised during test discovery (that is, at the -module level in the test file), it is now reported as a skip instead of an -error. (Contributed by Zach Ware in :issue:`16935`.) - -:meth:`~unittest.TestLoader.discover` now sorts the discovered files to provide -consistent test ordering. (Contributed by Martin Melin and Jeff Ramnani in -:issue:`16709`.) - -:class:`~unittest.TestSuite` now drops references to tests as soon as the test -has been run, if the test is successful. On Python interpreters that do -garbage collection, this allows the tests to be garbage collected if nothing -else is holding a reference to the test. It is possible to override this -behavior by creating a :class:`~unittest.TestSuite` subclass that defines a -custom ``_removeTestAtIndex`` method. (Contributed by Tom Wardill, Matt -McClure, and Andrew Svetlov in :issue:`11798`.) - -A new test assertion context-manager, :meth:`~unittest.TestCase.assertLogs`, -will ensure that a given block of code emits a log message using the -:mod:`logging` module. By default the message can come from any logger and -have a priority of ``INFO`` or higher, but both the logger name and an -alternative minimum logging level may be specified. The object returned by the -context manager can be queried for the :class:`~logging.LogRecord`\ s and/or -formatted messages that were logged. (Contributed by Antoine Pitrou in -:issue:`18937`.) - -Test discovery now works with namespace packages (Contributed by Claudiu Popa -in :issue:`17457`.) - -:mod:`unittest.mock` objects now inspect their specification signatures when -matching calls, which means an argument can now be matched by either position -or name, instead of only by position. (Contributed by Antoine Pitrou in -:issue:`17015`.) - -:func:`~mock.mock_open` objects now have ``readline`` and ``readlines`` -methods. (Contributed by Toshio Kuratomi in :issue:`17467`.) - - -venv ----- - -:mod:`venv` now includes activation scripts for the ``csh`` and ``fish`` -shells. (Contributed by Andrew Svetlov in :issue:`15417`.) - -:class:`~venv.EnvBuilder` and the :func:`~venv.create` convenience function -take a new keyword argument *with_pip*, which defaults to ``False``, that -controls whether or not :class:`~venv.EnvBuilder` ensures that ``pip`` is -installed in the virtual environment. (Contributed by Nick Coghlan in -:issue:`19552` as part of the :pep:`453` implementation.) - - -wave ----- - -The :meth:`~wave.getparams` method now returns a namedtuple rather than a -plain tuple. (Contributed by Claudiu Popa in :issue:`17487`.) - -:meth:`wave.open` now supports the context management protocol. (Contributed -by Claudiu Popa in :issue:`17616`.) - -:mod:`wave` can now :ref:`write output to unseekable files -<wave-write-objects>`. (Contributed by David Jones, Guilherme Polo, and Serhiy -Storchaka in :issue:`5202`.) - -The :meth:`~wave.Wave_write.writeframesraw` and -:meth:`~wave.Wave_write.writeframes` methods now accept any :term:`bytes-like -object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) - - -weakref -------- - -New :class:`~weakref.WeakMethod` class simulates weak references to bound -methods. (Contributed by Antoine Pitrou in :issue:`14631`.) - -New :class:`~weakref.finalize` class makes it possible to register a callback -to be invoked when an object is garbage collected, without needing to -carefully manage the lifecycle of the weak reference itself. (Contributed by -Richard Oudkerk in :issue:`15528`.) - -The callback, if any, associated with a :class:`~weakref.ref` is now -exposed via the :attr:`~weakref.ref.__callback__` attribute. (Contributed -by Mark Dickinson in :issue:`17643`.) - - -xml.etree ---------- - -A new parser, :class:`~xml.etree.ElementTree.XMLPullParser`, allows a -non-blocking applications to parse XML documents. An example can be -seen at :ref:`elementtree-pull-parsing`. (Contributed by Antoine -Pitrou in :issue:`17741`.) - -The :mod:`xml.etree.ElementTree` :func:`~xml.etree.ElementTree.tostring` and -:func:`~xml.etree.ElementTree.tostringlist` functions, and the -:class:`~xml.etree.ElementTree.ElementTree` -:meth:`~xml.etree.ElementTree.ElementTree.write` method, now have a -*short_empty_elements* :ref:`keyword-only parameter <keyword-only_parameter>` -providing control over whether elements with no content are written in -abbreviated (``<tag />``) or expanded (``<tag></tag>``) form. (Contributed by -Ariel Poliak and Serhiy Storchaka in :issue:`14377`.) - - -zipfile -------- - -The :meth:`~zipfile.PyZipFile.writepy` method of the -:class:`~zipfile.PyZipFile` class has a new *filterfunc* option that can be -used to control which directories and files are added to the archive. For -example, this could be used to exclude test files from the archive. -(Contributed by Christian Tismer in :issue:`19274`.) - -The *allowZip64* parameter to :class:`~zipfile.ZipFile` and -:class:`~zipfile.PyZipfile` is now ``True`` by default. (Contributed by -William Mallard in :issue:`17201`.) - - - -CPython Implementation Changes -============================== - - -.. _whatsnew-pep-445: - -PEP 445: Customization of CPython Memory Allocators ---------------------------------------------------- - -:pep:`445` adds new C level interfaces to customize memory allocation in -the CPython interpreter. - -.. seealso:: - - :pep:`445` -- Add new APIs to customize Python memory allocators - PEP written and implemented by Victor Stinner. - - -.. _whatsnew-pep-442: - -PEP 442: Safe Object Finalization ---------------------------------- - -:pep:`442` removes the current limitations and quirks of object finalization -in CPython. With it, objects with :meth:`__del__` methods, as well as -generators with :keyword:`finally` clauses, can be finalized when they are -part of a reference cycle. - -As part of this change, module globals are no longer forcibly set to -:const:`None` during interpreter shutdown in most cases, instead relying -on the normal operation of the cyclic garbage collector. This avoids a -whole class of interpreter-shutdown-time errors, usually involving -``__del__`` methods, that have plagued Python since the cyclic GC -was first introduced. - -.. seealso:: - - :pep:`442` -- Safe object finalization - PEP written and implemented by Antoine Pitrou. - - -.. _whatsnew-pep-456: - -PEP 456: Secure and Interchangeable Hash Algorithm --------------------------------------------------- - -:pep:`456` follows up on earlier security fix work done on Python's hash -algorithm to address certain DOS attacks to which public facing APIs backed by -dictionary lookups may be subject. (See :issue:`14621` for the start of the -current round of improvements.) The PEP unifies CPython's hash code to make it -easier for a packager to substitute a different hash algorithm, and switches -Python's default implementation to a SipHash implementation on platforms that -have a 64 bit data type. Any performance differences in comparison with the -older FNV algorithm are trivial. - -The PEP adds additional fields to the :attr:`sys.hash_info` named tuple to -describe the hash algorithm in use by the currently executing binary. Otherwise, -the PEP does not alter any existing CPython APIs. - - -.. _whatsnew-pep-436: - -PEP 436: Argument Clinic ------------------------- - -"Argument Clinic" (:pep:`436`) is now part of the CPython build process -and can be used to simplify the process of defining and maintaining -accurate signatures for builtins and standard library extension modules -implemented in C. - -Some standard library extension modules have been converted to use Argument -Clinic in Python 3.4, and :mod:`pydoc` and :mod:`inspect` have been updated -accordingly. - -It is expected that signature metadata for programmatic introspection will -be added to additional callables implemented in C as part of Python 3.4 -maintenance releases. - -.. note:: - The Argument Clinic PEP is not fully up to date with the state of the - implementation. This has been deemed acceptable by the release manager - and core development team in this case, as Argument Clinic will not - be made available as a public API for third party use in Python 3.4. - -.. seealso:: - - :pep:`436` -- The Argument Clinic DSL - PEP written and implemented by Larry Hastings. - - -Other Build and C API Changes ------------------------------ - -* The new :c:func:`PyType_GetSlot` function has been added to the stable ABI, - allowing retrieval of function pointers from named type slots when using - the limited API. (Contributed by Martin von Löwis in :issue:`17162`.) - -* The new :c:func:`Py_SetStandardStreamEncoding` pre-initialization API - allows applications embedding the CPython interpreter to reliably force - a particular encoding and error handler for the standard streams. - (Contributed by Bastien Montagne and Nick Coghlan in :issue:`16129`.) - -* Most Python C APIs that don't mutate string arguments are now correctly - marked as accepting ``const char *`` rather than ``char *``. (Contributed - by Serhiy Storchaka in :issue:`1772673`.) - -* A new shell version of ``python-config`` can be used even when a python - interpreter is not available (for example, in cross compilation scenarios). - -* :c:func:`PyUnicode_FromFormat` now supports width and precision - specifications for ``%s``, ``%A``, ``%U``, ``%V``, ``%S``, and ``%R``. - (Contributed by Ysj Ray and Victor Stinner in :issue:`7330`.) - -* New function :c:func:`PyStructSequence_InitType2` supplements the - existing :c:func:`PyStructSequence_InitType` function. The difference - is that it returns ``0`` on success and ``-1`` on failure. - -* The CPython source can now be compiled using the address sanity checking - features of recent versions of GCC and clang: the false alarms in the small - object allocator have been silenced. (Contributed by Dhiru Kholia in - :issue:`18596`.) - -* The Windows build now uses `Address Space Layout Randomization - <https://en.wikipedia.org/wiki/Address_space_layout_randomization>`_ and `Data Execution Prevention - <https://en.wikipedia.org/wiki/Data_Execution_Prevention>`_. (Contributed by - Christian Heimes in :issue:`16632`.) - -* New function :c:func:`PyObject_LengthHint` is the C API equivalent - of :func:`operator.length_hint`. (Contributed by Armin Ronacher in - :issue:`16148`.) - - -.. _other-improvements-3.4: - -Other Improvements ------------------- - -.. _whatsnew-isolated-mode: - -* The :ref:`python <using-on-cmdline>` command has a new :ref:`option - <using-on-misc-options>`, ``-I``, which causes it to run in "isolated mode", - which means that :data:`sys.path` contains neither the script's directory nor - the user's ``site-packages`` directory, and all :envvar:`PYTHON*` environment - variables are ignored (it implies both ``-s`` and ``-E``). Other - restrictions may also be applied in the future, with the goal being to - isolate the execution of a script from the user's environment. This is - appropriate, for example, when Python is used to run a system script. On - most POSIX systems it can and should be used in the ``#!`` line of system - scripts. (Contributed by Christian Heimes in :issue:`16499`.) - -* Tab-completion is now enabled by default in the interactive interpreter - on systems that support :mod:`readline`. History is also enabled by default, - and is written to (and read from) the file :file:`~/.python-history`. - (Contributed by Antoine Pitrou and Éric Araujo in :issue:`5845`.) - -* Invoking the Python interpreter with ``--version`` now outputs the version to - standard output instead of standard error (:issue:`18338`). Similar changes - were made to :mod:`argparse` (:issue:`18920`) and other modules that have - script-like invocation capabilities (:issue:`18922`). - -* The CPython Windows installer now adds ``.py`` to the :envvar:`PATHEXT` - variable when extensions are registered, allowing users to run a python - script at the windows command prompt by just typing its name without the - ``.py`` extension. (Contributed by Paul Moore in :issue:`18569`.) - -* A new ``make`` target `coverage-report - <https://devguide.python.org/coverage/#measuring-coverage-of-c-code-with-gcov-and-lcov>`_ - will build python, run the test suite, and generate an HTML coverage report - for the C codebase using ``gcov`` and `lcov - <http://ltp.sourceforge.net/coverage/lcov.php>`_. - -* The ``-R`` option to the :ref:`python regression test suite <regrtest>` now - also checks for memory allocation leaks, using - :func:`sys.getallocatedblocks()`. (Contributed by Antoine Pitrou in - :issue:`13390`.) - -* ``python -m`` now works with namespace packages. - -* The :mod:`stat` module is now implemented in C, which means it gets the - values for its constants from the C header files, instead of having the - values hard-coded in the python module as was previously the case. - -* Loading multiple python modules from a single OS module (``.so``, ``.dll``) - now works correctly (previously it silently returned the first python - module in the file). (Contributed by Václav Šmilauer in :issue:`16421`.) - -* A new opcode, :opcode:`LOAD_CLASSDEREF`, has been added to fix a bug in the - loading of free variables in class bodies that could be triggered by certain - uses of :ref:`__prepare__ <prepare>`. (Contributed by Benjamin Peterson in - :issue:`17853`.) - -* A number of MemoryError-related crashes were identified and fixed by Victor - Stinner using his :pep:`445`-based ``pyfailmalloc`` tool (:issue:`18408`, - :issue:`18520`). - -* The ``pyvenv`` command now accepts a ``--copies`` option - to use copies rather than symlinks even on systems where symlinks are the - default. (Contributed by Vinay Sajip in :issue:`18807`.) - -* The ``pyvenv`` command also accepts a ``--without-pip`` - option to suppress the otherwise-automatic bootstrapping of pip into - the virtual environment. (Contributed by Nick Coghlan in :issue:`19552` - as part of the :pep:`453` implementation.) - -* The encoding name is now optional in the value set for the - :envvar:`PYTHONIOENCODING` environment variable. This makes it possible to - set just the error handler, without changing the default encoding. - (Contributed by Serhiy Storchaka in :issue:`18818`.) - -* The :mod:`bz2`, :mod:`lzma`, and :mod:`gzip` module ``open`` functions now - support ``x`` (exclusive creation) mode. (Contributed by Tim Heaney and - Vajrasky Kok in :issue:`19201`, :issue:`19222`, and :issue:`19223`.) - - -Significant Optimizations -------------------------- - -* The UTF-32 decoder is now 3x to 4x faster. (Contributed by Serhiy Storchaka - in :issue:`14625`.) - -* The cost of hash collisions for sets is now reduced. Each hash table - probe now checks a series of consecutive, adjacent key/hash pairs before - continuing to make random probes through the hash table. This exploits - cache locality to make collision resolution less expensive. - The collision resolution scheme can be described as a hybrid of linear - probing and open addressing. The number of additional linear probes - defaults to nine. This can be changed at compile-time by defining - LINEAR_PROBES to be any value. Set LINEAR_PROBES=0 to turn-off - linear probing entirely. (Contributed by Raymond Hettinger in - :issue:`18771`.) - -* The interpreter starts about 30% faster. A couple of measures lead to the - speedup. The interpreter loads fewer modules on startup, e.g. the :mod:`re`, - :mod:`collections` and :mod:`locale` modules and their dependencies are no - longer imported by default. The marshal module has been improved to load - compiled Python code faster. (Contributed by Antoine Pitrou, Christian - Heimes and Victor Stinner in :issue:`19219`, :issue:`19218`, :issue:`19209`, - :issue:`19205` and :issue:`9548`.) - -* :class:`bz2.BZ2File` is now as fast or faster than the Python2 version for - most cases. :class:`lzma.LZMAFile` has also been optimized. (Contributed by - Serhiy Storchaka and Nadeem Vawda in :issue:`16034`.) - -* :func:`random.getrandbits` is 20%-40% faster for small integers (the most - common use case). (Contributed by Serhiy Storchaka in :issue:`16674`.) - -* By taking advantage of the new storage format for strings, pickling of - strings is now significantly faster. (Contributed by Victor Stinner and - Antoine Pitrou in :issue:`15596`.) - -* A performance issue in :meth:`io.FileIO.readall` has been solved. This - particularly affects Windows, and significantly speeds up the case of piping - significant amounts of data through :mod:`subprocess`. (Contributed - by Richard Oudkerk in :issue:`15758`.) - -* :func:`html.escape` is now 10x faster. (Contributed by Matt Bryant in - :issue:`18020`.) - -* On Windows, the native ``VirtualAlloc`` is now used instead of the CRT - ``malloc`` in ``obmalloc``. Artificial benchmarks show about a 3% memory - savings. - -* :func:`os.urandom` now uses a lazily-opened persistent file descriptor - so as to avoid using many file descriptors when run in parallel from - multiple threads. (Contributed by Antoine Pitrou in :issue:`18756`.) - - -.. _deprecated-3.4: - -Deprecated -========== - -This section covers various APIs and other features that have been deprecated -in Python 3.4, and will be removed in Python 3.5 or later. In most (but not -all) cases, using the deprecated APIs will produce a :exc:`DeprecationWarning` -when the interpreter is run with deprecation warnings enabled (for example, by -using ``-Wd``). - - -Deprecations in the Python API ------------------------------- - -* As mentioned in :ref:`whatsnew-pep-451`, a number of :mod:`importlib` - methods and functions are deprecated: :meth:`importlib.find_loader` is - replaced by :func:`importlib.util.find_spec`; - :meth:`importlib.machinery.PathFinder.find_module` is replaced by - :meth:`importlib.machinery.PathFinder.find_spec`; - :meth:`importlib.abc.MetaPathFinder.find_module` is replaced by - :meth:`importlib.abc.MetaPathFinder.find_spec`; - :meth:`importlib.abc.PathEntryFinder.find_loader` and - :meth:`~importlib.abc.PathEntryFinder.find_module` are replaced by - :meth:`importlib.abc.PathEntryFinder.find_spec`; all of the ``xxxLoader`` ABC - ``load_module`` methods (:meth:`importlib.abc.Loader.load_module`, - :meth:`importlib.abc.InspectLoader.load_module`, - :meth:`importlib.abc.FileLoader.load_module`, - :meth:`importlib.abc.SourceLoader.load_module`) should no longer be - implemented, instead loaders should implement an - ``exec_module`` method - (:meth:`importlib.abc.Loader.exec_module`, - :meth:`importlib.abc.InspectLoader.exec_module` - :meth:`importlib.abc.SourceLoader.exec_module`) and let the import system - take care of the rest; and - :meth:`importlib.abc.Loader.module_repr`, - :meth:`importlib.util.module_for_loader`, :meth:`importlib.util.set_loader`, - and :meth:`importlib.util.set_package` are no longer needed because their - functions are now handled automatically by the import system. - -* The :mod:`imp` module is pending deprecation. To keep compatibility with - Python 2/3 code bases, the module's removal is currently not scheduled. - -* The :mod:`formatter` module is pending deprecation and is slated for removal - in Python 3.6. - -* ``MD5`` as the default *digestmod* for the :func:`hmac.new` function is - deprecated. Python 3.6 will require an explicit digest name or constructor as - *digestmod* argument. - -* The internal ``Netrc`` class in the :mod:`ftplib` module has been documented - as deprecated in its docstring for quite some time. It now emits a - :exc:`DeprecationWarning` and will be removed completely in Python 3.5. - -* The undocumented *endtime* argument to :meth:`subprocess.Popen.wait` should - not have been exposed and is hopefully not in use; it is deprecated and - will mostly likely be removed in Python 3.5. - -* The *strict* argument of :class:`~html.parser.HTMLParser` is deprecated. - -* The :mod:`plistlib` :func:`~plistlib.readPlist`, - :func:`~plistlib.writePlist`, :func:`~plistlib.readPlistFromBytes`, and - :func:`~plistlib.writePlistToBytes` functions are deprecated in favor of the - corresponding new functions :func:`~plistlib.load`, :func:`~plistlib.dump`, - :func:`~plistlib.loads`, and :func:`~plistlib.dumps`. :func:`~plistlib.Data` - is deprecated in favor of just using the :class:`bytes` constructor. - -* The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by - ``EXT_SUFFIX``. - -* The ``U`` mode accepted by various ``open`` functions is deprecated. - In Python3 it does not do anything useful, and should be replaced by - appropriate uses of :class:`io.TextIOWrapper` (if needed) and its *newline* - argument. - -* The *parser* argument of :func:`xml.etree.ElementTree.iterparse` has - been deprecated, as has the *html* argument of - :func:`~xml.etree.ElementTree.XMLParser`. To prepare for the removal of the - latter, all arguments to ``XMLParser`` should be passed by keyword. - - -Deprecated Features -------------------- - -* Running :ref:`idle` with the ``-n`` flag (no subprocess) is deprecated. - However, the feature will not be removed until :issue:`18823` is resolved. - -* The site module adding a "site-python" directory to sys.path, if it - exists, is deprecated (:issue:`19375`). - - - -Removed -======= - - -Operating Systems No Longer Supported -------------------------------------- - -Support for the following operating systems has been removed from the source -and build tools: - -* OS/2 (:issue:`16135`). -* Windows 2000 (changeset e52df05b496a). -* Windows systems where ``COMSPEC`` points to ``command.com`` (:issue:`14470`). -* VMS (:issue:`16136`). - - -API and Feature Removals ------------------------- - -The following obsolete and previously deprecated APIs and features have been -removed: - -* The unmaintained ``Misc/TextMate`` and ``Misc/vim`` directories have been - removed (see the `devguide <https://devguide.python.org>`_ - for suggestions on what to use instead). - -* The ``SO`` makefile macro is removed (it was replaced by the - ``SHLIB_SUFFIX`` and ``EXT_SUFFIX`` macros) (:issue:`16754`). - -* The ``PyThreadState.tick_counter`` field has been removed; its value has - been meaningless since Python 3.2, when the "new GIL" was introduced - (:issue:`19199`). - -* ``PyLoader`` and ``PyPycLoader`` have been removed from :mod:`importlib`. - (Contributed by Taras Lyapun in :issue:`15641`.) - -* The *strict* argument to :class:`~http.client.HTTPConnection` and - :class:`~http.client.HTTPSConnection` has been removed. HTTP 0.9-style - "Simple Responses" are no longer supported. - -* The deprecated :mod:`urllib.request.Request` getter and setter methods - ``add_data``, ``has_data``, ``get_data``, ``get_type``, ``get_host``, - ``get_selector``, ``set_proxy``, ``get_origin_req_host``, and - ``is_unverifiable`` have been removed (use direct attribute access instead). - -* Support for loading the deprecated ``TYPE_INT64`` has been removed from - :mod:`marshal`. (Contributed by Dan Riti in :issue:`15480`.) - -* :class:`inspect.Signature`: positional-only parameters are now required - to have a valid name. - -* :meth:`object.__format__` no longer accepts non-empty format strings, it now - raises a :exc:`TypeError` instead. Using a non-empty string has been - deprecated since Python 3.2. This change has been made to prevent a - situation where previously working (but incorrect) code would start failing - if an object gained a __format__ method, which means that your code may now - raise a :exc:`TypeError` if you are using an ``'s'`` format code with objects - that do not have a __format__ method that handles it. See :issue:`7994` for - background. - -* :meth:`difflib.SequenceMatcher.isbjunk` and - :meth:`difflib.SequenceMatcher.isbpopular` were deprecated in 3.2, and have - now been removed: use ``x in sm.bjunk`` and - ``x in sm.bpopular``, where *sm* is a :class:`~difflib.SequenceMatcher` object - (:issue:`13248`). - - -Code Cleanups -------------- - -* The unused and undocumented internal ``Scanner`` class has been removed from - the :mod:`pydoc` module. - -* The private and effectively unused ``_gestalt`` module has been removed, - along with the private :mod:`platform` functions ``_mac_ver_lookup``, - ``_mac_ver_gstalt``, and ``_bcd2str``, which would only have ever been called - on badly broken OSX systems (see :issue:`18393`). - -* The hardcoded copies of certain :mod:`stat` constants that were included in - the :mod:`tarfile` module namespace have been removed. - - - -Porting to Python 3.4 -===================== - -This section lists previously described changes and other bugfixes -that may require changes to your code. - - -Changes in 'python' Command Behavior ------------------------------------- - -* In a posix shell, setting the :envvar:`PATH` environment variable to - an empty value is equivalent to not setting it at all. However, setting - :envvar:`PYTHONPATH` to an empty value was *not* equivalent to not setting it - at all: setting :envvar:`PYTHONPATH` to an empty value was equivalent to - setting it to ``.``, which leads to confusion when reasoning by analogy to - how :envvar:`PATH` works. The behavior now conforms to the posix convention - for :envvar:`PATH`. - -* The [X refs, Y blocks] output of a debug (``--with-pydebug``) build of the - CPython interpreter is now off by default. It can be re-enabled using the - ``-X showrefcount`` option. (Contributed by Ezio Melotti in :issue:`17323`.) - -* The python command and most stdlib scripts (as well as :mod:`argparse`) now - output ``--version`` information to ``stdout`` instead of ``stderr`` (for - issue list see :ref:`other-improvements-3.4` above). - - -Changes in the Python API -------------------------- - -* The ABCs defined in :mod:`importlib.abc` now either raise the appropriate - exception or return a default value instead of raising - :exc:`NotImplementedError` blindly. This will only affect code calling - :func:`super` and falling through all the way to the ABCs. For compatibility, - catch both :exc:`NotImplementedError` or the appropriate exception as needed. - -* The module type now initializes the :attr:`__package__` and :attr:`__loader__` - attributes to ``None`` by default. To determine if these attributes were set - in a backwards-compatible fashion, use e.g. - ``getattr(module, '__loader__', None) is not None``. (:issue:`17115`.) - -* :meth:`importlib.util.module_for_loader` now sets ``__loader__`` and - ``__package__`` unconditionally to properly support reloading. If this is not - desired then you will need to set these attributes manually. You can use - :func:`importlib.util.module_to_load` for module management. - -* Import now resets relevant attributes (e.g. ``__name__``, ``__loader__``, - ``__package__``, ``__file__``, ``__cached__``) unconditionally when reloading. - Note that this restores a pre-3.3 behavior in that it means a module is - re-found when re-loaded (:issue:`19413`). - -* Frozen packages no longer set ``__path__`` to a list containing the package - name, they now set it to an empty list. The previous behavior could cause - the import system to do the wrong thing on submodule imports if there was - also a directory with the same name as the frozen package. The correct way - to determine if a module is a package or not is to use ``hasattr(module, - '__path__')`` (:issue:`18065`). - -* Frozen modules no longer define a ``__file__`` attribute. It's semantically - incorrect for frozen modules to set the attribute as they are not loaded from - any explicit location. If you must know that a module comes from frozen code - then you can see if the module's ``__spec__.location`` is set to ``'frozen'``, - check if the loader is a subclass of - :class:`importlib.machinery.FrozenImporter`, - or if Python 2 compatibility is necessary you can use :func:`imp.is_frozen`. - -* :func:`py_compile.compile` now raises :exc:`FileExistsError` if the file path - it would write to is a symlink or a non-regular file. This is to act as a - warning that import will overwrite those files with a regular file regardless - of what type of file path they were originally. - -* :meth:`importlib.abc.SourceLoader.get_source` no longer raises - :exc:`ImportError` when the source code being loaded triggers a - :exc:`SyntaxError` or :exc:`UnicodeDecodeError`. As :exc:`ImportError` is - meant to be raised only when source code cannot be found but it should, it was - felt to be over-reaching/overloading of that meaning when the source code is - found but improperly structured. If you were catching ImportError before and - wish to continue to ignore syntax or decoding issues, catch all three - exceptions now. - -* :func:`functools.update_wrapper` and :func:`functools.wraps` now correctly - set the ``__wrapped__`` attribute to the function being wrapped, even if - that function also had its ``__wrapped__`` attribute set. This means - ``__wrapped__`` attributes now correctly link a stack of decorated - functions rather than every ``__wrapped__`` attribute in the chain - referring to the innermost function. Introspection libraries that - assumed the previous behaviour was intentional can use - :func:`inspect.unwrap` to access the first function in the chain that has - no ``__wrapped__`` attribute. - -* :func:`inspect.getfullargspec` has been reimplemented on top of - :func:`inspect.signature` and hence handles a much wider variety of callable - objects than it did in the past. It is expected that additional builtin and - extension module callables will gain signature metadata over the course of - the Python 3.4 series. Code that assumes that - :func:`inspect.getfullargspec` will fail on non-Python callables may need - to be adjusted accordingly. - -* :class:`importlib.machinery.PathFinder` now passes on the current working - directory to objects in :data:`sys.path_hooks` for the empty string. This - results in :data:`sys.path_importer_cache` never containing ``''``, thus - iterating through :data:`sys.path_importer_cache` based on :data:`sys.path` - will not find all keys. A module's ``__file__`` when imported in the current - working directory will also now have an absolute path, including when using - ``-m`` with the interpreter (except for ``__main__.__file__`` when a script - has been executed directly using a relative path) (Contributed by Brett - Cannon in :issue:`18416`). is specified on the command-line) - (:issue:`18416`). - -* The removal of the *strict* argument to :class:`~http.client.HTTPConnection` - and :class:`~http.client.HTTPSConnection` changes the meaning of the - remaining arguments if you are specifying them positionally rather than by - keyword. If you've been paying attention to deprecation warnings your code - should already be specifying any additional arguments via keywords. - -* Strings between ``from __future__ import ...`` statements now *always* raise - a :exc:`SyntaxError`. Previously if there was no leading docstring, an - interstitial string would sometimes be ignored. This brings CPython into - compliance with the language spec; Jython and PyPy already were. - (:issue:`17434`). - -* :meth:`ssl.SSLSocket.getpeercert` and :meth:`ssl.SSLSocket.do_handshake` - now raise an :exc:`OSError` with ``ENOTCONN`` when the ``SSLSocket`` is not - connected, instead of the previous behavior of raising an - :exc:`AttributeError`. In addition, :meth:`~ssl.SSLSocket.getpeercert` - will raise a :exc:`ValueError` if the handshake has not yet been done. - -* :func:`base64.b32decode` now raises a :exc:`binascii.Error` when the - input string contains non-b32-alphabet characters, instead of a - :exc:`TypeError`. This particular :exc:`TypeError` was missed when the other - :exc:`TypeError`\ s were converted. (Contributed by Serhiy Storchaka in - :issue:`18011`.) Note: this change was also inadvertently applied in Python - 3.3.3. - -* The :attr:`~cgi.FieldStorage.file` attribute is now automatically closed when - the creating :class:`cgi.FieldStorage` instance is garbage collected. If you - were pulling the file object out separately from the :class:`cgi.FieldStorage` - instance and not keeping the instance alive, then you should either store the - entire :class:`cgi.FieldStorage` instance or read the contents of the file - before the :class:`cgi.FieldStorage` instance is garbage collected. - -* Calling ``read`` or ``write`` on a closed SSL socket now raises an - informative :exc:`ValueError` rather than the previous more mysterious - :exc:`AttributeError` (:issue:`9177`). - -* :meth:`slice.indices` no longer produces an :exc:`OverflowError` for huge - values. As a consequence of this fix, :meth:`slice.indices` now raises a - :exc:`ValueError` if given a negative length; previously it returned nonsense - values (:issue:`14794`). - -* The :class:`complex` constructor, unlike the :mod:`cmath` functions, was - incorrectly accepting :class:`float` values if an object's ``__complex__`` - special method returned one. This now raises a :exc:`TypeError`. - (:issue:`16290`.) - -* The :class:`int` constructor in 3.2 and 3.3 erroneously accepts :class:`float` - values for the *base* parameter. It is unlikely anyone was doing this, but - if so, it will now raise a :exc:`TypeError` (:issue:`16772`). - -* Defaults for keyword-only arguments are now evaluated *after* defaults for - regular keyword arguments, instead of before. Hopefully no one wrote any - code that depends on the previous buggy behavior (:issue:`16967`). - -* Stale thread states are now cleared after :func:`~os.fork`. This may cause - some system resources to be released that previously were incorrectly kept - perpetually alive (for example, database connections kept in thread-local - storage). (:issue:`17094`.) - -* Parameter names in ``__annotations__`` dicts are now mangled properly, - similarly to ``__kwdefaults__``. (Contributed by Yury Selivanov in - :issue:`20625`.) - -* :attr:`hashlib.hash.name` now always returns the identifier in lower case. - Previously some builtin hashes had uppercase names, but now that it is a - formal public interface the naming has been made consistent (:issue:`18532`). - -* Because :mod:`unittest.TestSuite` now drops references to tests after they - are run, test harnesses that re-use a :class:`~unittest.TestSuite` to re-run - a set of tests may fail. Test suites should not be re-used in this fashion - since it means state is retained between test runs, breaking the test - isolation that :mod:`unittest` is designed to provide. However, if the lack - of isolation is considered acceptable, the old behavior can be restored by - creating a :mod:`~unittest.TestSuite` subclass that defines a - ``_removeTestAtIndex`` method that does nothing (see - :meth:`.TestSuite.__iter__`) (:issue:`11798`). - -* :mod:`unittest` now uses :mod:`argparse` for command line parsing. There are - certain invalid command forms that used to work that are no longer allowed; - in theory this should not cause backward compatibility issues since the - disallowed command forms didn't make any sense and are unlikely to be in use. - -* The :func:`re.split`, :func:`re.findall`, and :func:`re.sub` functions, and - the :meth:`~re.match.group` and :meth:`~re.match.groups` methods of - ``match`` objects now always return a *bytes* object when the string - to be matched is a :term:`bytes-like object`. Previously the return type - matched the input type, so if your code was depending on the return value - being, say, a ``bytearray``, you will need to change your code. - -* :mod:`audioop` functions now raise an error immediately if passed string - input, instead of failing randomly later on (:issue:`16685`). - -* The new *convert_charrefs* argument to :class:`~html.parser.HTMLParser` - currently defaults to ``False`` for backward compatibility, but will - eventually be changed to default to ``True``. It is recommended that you add - this keyword, with the appropriate value, to any - :class:`~html.parser.HTMLParser` calls in your code (:issue:`13633`). - -* Since the *digestmod* argument to the :func:`hmac.new` function will in the - future have no default, all calls to :func:`hmac.new` should be changed to - explicitly specify a *digestmod* (:issue:`17276`). - -* Calling :func:`sysconfig.get_config_var` with the ``SO`` key, or looking - ``SO`` up in the results of a call to :func:`sysconfig.get_config_vars` - is deprecated. This key should be replaced by ``EXT_SUFFIX`` or - ``SHLIB_SUFFIX``, depending on the context (:issue:`19555`). - -* Any calls to ``open`` functions that specify ``U`` should be modified. - ``U`` is ineffective in Python3 and will eventually raise an error if used. - Depending on the function, the equivalent of its old Python2 behavior can be - achieved using either a *newline* argument, or if necessary by wrapping the - stream in :mod:`~io.TextIOWrapper` to use its *newline* argument - (:issue:`15204`). - -* If you use ``pyvenv`` in a script and desire that pip - *not* be installed, you must add ``--without-pip`` to your command - invocation. - -* The default behavior of :func:`json.dump` and :func:`json.dumps` when - an indent is specified has changed: it no longer produces trailing - spaces after the item separating commas at the ends of lines. This - will matter only if you have tests that are doing white-space-sensitive - comparisons of such output (:issue:`16333`). - -* :mod:`doctest` now looks for doctests in extension module ``__doc__`` - strings, so if your doctest test discovery includes extension modules that - have things that look like doctests in them you may see test failures you've - never seen before when running your tests (:issue:`3158`). - -* The :mod:`collections.abc` module has been slightly refactored as - part of the Python startup improvements. As a consequence of this, it is no - longer the case that importing :mod:`collections` automatically imports - :mod:`collections.abc`. If your program depended on the (undocumented) - implicit import, you will need to add an explicit ``import collections.abc`` - (:issue:`20784`). - - -Changes in the C API --------------------- - -* :c:func:`PyEval_EvalFrameEx`, :c:func:`PyObject_Repr`, and - :c:func:`PyObject_Str`, along with some other internal C APIs, now include - a debugging assertion that ensures they are not used in situations where - they may silently discard a currently active exception. In cases where - discarding the active exception is expected and desired (for example, - because it has already been saved locally with :c:func:`PyErr_Fetch` or - is being deliberately replaced with a different exception), an explicit - :c:func:`PyErr_Clear` call will be needed to avoid triggering the - assertion when invoking these operations (directly or indirectly) and - running against a version of Python that is compiled with assertions - enabled. - -* :c:func:`PyErr_SetImportError` now sets :exc:`TypeError` when its **msg** - argument is not set. Previously only ``NULL`` was returned with no exception - set. - -* The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must - now be a string allocated by :c:func:`PyMem_RawMalloc` or - :c:func:`PyMem_RawRealloc`, or ``NULL`` if an error occurred, instead of a - string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc` - (:issue:`16742`) - -* :c:func:`PyThread_set_key_value` now always set the value. In Python - 3.3, the function did nothing if the key already exists (if the current - value is a non-``NULL`` pointer). - -* The ``f_tstate`` (thread state) field of the :c:type:`PyFrameObject` - structure has been removed to fix a bug: see :issue:`14432` for the - rationale. - -Changed in 3.4.3 -================ - -.. _pep-476: - -PEP 476: Enabling certificate verification by default for stdlib http clients ------------------------------------------------------------------------------ - -:mod:`http.client` and modules which use it, such as :mod:`urllib.request` and -:mod:`xmlrpc.client`, will now verify that the server presents a certificate -which is signed by a CA in the platform trust store and whose hostname matches -the hostname being requested by default, significantly improving security for -many applications. - -For applications which require the old previous behavior, they can pass an -alternate context:: - - import urllib.request - import ssl - - # This disables all verification - context = ssl._create_unverified_context() - - # This allows using a specific certificate for the host, which doesn't need - # to be in the trust store - context = ssl.create_default_context(cafile="/path/to/file.crt") - - urllib.request.urlopen("https://invalid-cert", context=context) diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst deleted file mode 100644 index b4540ac..0000000 --- a/Doc/whatsnew/3.5.rst +++ /dev/null @@ -1,2574 +0,0 @@ -**************************** - What's New In Python 3.5 -**************************** - -:Editors: Elvis Pranskevichus <elvis@magic.io>, Yury Selivanov <yury@magic.io> - -.. Rules for maintenance: - - * Anyone can add text to this document. Do not spend very much time - on the wording of your changes, because your text will probably - get rewritten to some degree. - - * The maintainer will go through Misc/NEWS periodically and add - changes; it's therefore more important to add your changes to - Misc/NEWS than to this file. - - * This is not a complete list of every single change; completeness - is the purpose of Misc/NEWS. Some changes I consider too small - or esoteric to include. If such a change is added to the text, - I'll just remove it. (This is another reason you shouldn't spend - too much time on writing your addition.) - - * If you want to draw your new text to the attention of the - maintainer, add 'XXX' to the beginning of the paragraph or - section. - - * It's OK to just add a fragmentary note about a change. For - example: "XXX Describe the transmogrify() function added to the - socket module." The maintainer will research the change and - write the necessary text. - - * You can comment out your additions if you like, but it's not - necessary (especially when a final release is some months away). - - * Credit the author of a patch or bugfix. Just the name is - sufficient; the e-mail address isn't necessary. - - * It's helpful to add the bug/patch number as a comment: - - XXX Describe the transmogrify() function added to the socket - module. - (Contributed by P.Y. Developer in :issue:`12345`.) - - This saves the maintainer the effort of going through the Mercurial log - when researching a change. - -This article explains the new features in Python 3.5, compared to 3.4. -Python 3.5 was released on September 13, 2015. See the -`changelog <https://docs.python.org/3.5/whatsnew/changelog.html>`_ for a full -list of changes. - -.. seealso:: - - :pep:`478` - Python 3.5 Release Schedule - - -Summary -- Release highlights -============================= - -New syntax features: - -* :ref:`PEP 492 <whatsnew-pep-492>`, coroutines with async and await syntax. -* :ref:`PEP 465 <whatsnew-pep-465>`, a new matrix multiplication operator: ``a @ b``. -* :ref:`PEP 448 <whatsnew-pep-448>`, additional unpacking generalizations. - - -New library modules: - -* :mod:`typing`: :ref:`PEP 484 -- Type Hints <whatsnew-pep-484>`. -* :mod:`zipapp`: :ref:`PEP 441 Improving Python ZIP Application Support - <whatsnew-zipapp>`. - - -New built-in features: - -* ``bytes % args``, ``bytearray % args``: :ref:`PEP 461 <whatsnew-pep-461>` -- - Adding ``%`` formatting to bytes and bytearray. - -* New :meth:`bytes.hex`, :meth:`bytearray.hex` and :meth:`memoryview.hex` - methods. (Contributed by Arnon Yaari in :issue:`9951`.) - -* :class:`memoryview` now supports tuple indexing (including multi-dimensional). - (Contributed by Antoine Pitrou in :issue:`23632`.) - -* Generators have a new ``gi_yieldfrom`` attribute, which returns the - object being iterated by ``yield from`` expressions. (Contributed - by Benno Leslie and Yury Selivanov in :issue:`24450`.) - -* A new :exc:`RecursionError` exception is now raised when maximum - recursion depth is reached. (Contributed by Georg Brandl - in :issue:`19235`.) - - -CPython implementation improvements: - -* When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), - :py:data:`sys.stdin` and :py:data:`sys.stdout` now use the - ``surrogateescape`` error handler, instead of the ``strict`` error handler. - (Contributed by Victor Stinner in :issue:`19977`.) - -* ``.pyo`` files are no longer used and have been replaced by a more flexible - scheme that includes the optimization level explicitly in ``.pyc`` name. - (See :ref:`PEP 488 overview <whatsnew-pep-488>`.) - -* Builtin and extension modules are now initialized in a multi-phase process, - which is similar to how Python modules are loaded. - (See :ref:`PEP 489 overview <whatsnew-pep-489>`.) - - -Significant improvements in the standard library: - -* :class:`collections.OrderedDict` is now - :ref:`implemented in C <whatsnew-ordereddict>`, which makes it - 4 to 100 times faster. - -* The :mod:`ssl` module gained - :ref:`support for Memory BIO <whatsnew-sslmemorybio>`, which decouples SSL - protocol handling from network IO. - -* The new :func:`os.scandir` function provides a - :ref:`better and significantly faster way <whatsnew-pep-471>` - of directory traversal. - -* :func:`functools.lru_cache` has been mostly - :ref:`reimplemented in C <whatsnew-lrucache>`, yielding much better - performance. - -* The new :func:`subprocess.run` function provides a - :ref:`streamlined way to run subprocesses <whatsnew-subprocess>`. - -* The :mod:`traceback` module has been significantly - :ref:`enhanced <whatsnew-traceback>` for improved - performance and developer convenience. - - -Security improvements: - -* SSLv3 is now disabled throughout the standard library. - It can still be enabled by instantiating a :class:`ssl.SSLContext` - manually. (See :issue:`22638` for more details; this change was - backported to CPython 3.4 and 2.7.) - -* HTTP cookie parsing is now stricter, in order to protect - against potential injection attacks. (Contributed by Antoine Pitrou - in :issue:`22796`.) - - -Windows improvements: - -* A new installer for Windows has replaced the old MSI. - See :ref:`using-on-windows` for more information. - -* Windows builds now use Microsoft Visual C++ 14.0, and extension modules - should use the same. - - -Please read on for a comprehensive list of user-facing changes, including many -other smaller improvements, CPython optimizations, deprecations, and potential -porting issues. - - -New Features -============ - -.. _whatsnew-pep-492: - -PEP 492 - Coroutines with async and await syntax ------------------------------------------------- - -:pep:`492` greatly improves support for asynchronous programming in Python -by adding :term:`awaitable objects <awaitable>`, -:term:`coroutine functions <coroutine function>`, -:term:`asynchronous iteration <asynchronous iterable>`, -and :term:`asynchronous context managers <asynchronous context manager>`. - -Coroutine functions are declared using the new :keyword:`async def` syntax:: - - >>> async def coro(): - ... return 'spam' - -Inside a coroutine function, the new :keyword:`await` expression can be used -to suspend coroutine execution until the result is available. Any object -can be *awaited*, as long as it implements the :term:`awaitable` protocol by -defining the :meth:`__await__` method. - -PEP 492 also adds :keyword:`async for` statement for convenient iteration -over asynchronous iterables. - -An example of a rudimentary HTTP client written using the new syntax:: - - import asyncio - - async def http_get(domain): - reader, writer = await asyncio.open_connection(domain, 80) - - writer.write(b'\r\n'.join([ - b'GET / HTTP/1.1', - b'Host: %b' % domain.encode('latin-1'), - b'Connection: close', - b'', b'' - ])) - - async for line in reader: - print('>>>', line) - - writer.close() - - loop = asyncio.get_event_loop() - try: - loop.run_until_complete(http_get('example.com')) - finally: - loop.close() - - -Similarly to asynchronous iteration, there is a new syntax for asynchronous -context managers. The following script:: - - import asyncio - - async def coro(name, lock): - print('coro {}: waiting for lock'.format(name)) - async with lock: - print('coro {}: holding the lock'.format(name)) - await asyncio.sleep(1) - print('coro {}: releasing the lock'.format(name)) - - loop = asyncio.get_event_loop() - lock = asyncio.Lock() - coros = asyncio.gather(coro(1, lock), coro(2, lock)) - try: - loop.run_until_complete(coros) - finally: - loop.close() - -will output:: - - coro 2: waiting for lock - coro 2: holding the lock - coro 1: waiting for lock - coro 2: releasing the lock - coro 1: holding the lock - coro 1: releasing the lock - -Note that both :keyword:`async for` and :keyword:`async with` can only -be used inside a coroutine function declared with :keyword:`async def`. - -Coroutine functions are intended to be run inside a compatible event loop, -such as the :ref:`asyncio loop <asyncio-event-loop>`. - - -.. note:: - - .. versionchanged:: 3.5.2 - Starting with CPython 3.5.2, ``__aiter__`` can directly return - :term:`asynchronous iterators <asynchronous iterator>`. Returning - an :term:`awaitable` object will result in a - :exc:`PendingDeprecationWarning`. - - See more details in the :ref:`async-iterators` documentation - section. - - -.. seealso:: - - :pep:`492` -- Coroutines with async and await syntax - PEP written and implemented by Yury Selivanov. - - -.. _whatsnew-pep-465: - -PEP 465 - A dedicated infix operator for matrix multiplication --------------------------------------------------------------- - -:pep:`465` adds the ``@`` infix operator for matrix multiplication. -Currently, no builtin Python types implement the new operator, however, it -can be implemented by defining :meth:`__matmul__`, :meth:`__rmatmul__`, -and :meth:`__imatmul__` for regular, reflected, and in-place matrix -multiplication. The semantics of these methods is similar to that of -methods defining other infix arithmetic operators. - -Matrix multiplication is a notably common operation in many fields of -mathematics, science, engineering, and the addition of ``@`` allows writing -cleaner code:: - - S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r) - -instead of:: - - S = dot((dot(H, beta) - r).T, - dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r)) - -NumPy 1.10 has support for the new operator:: - - >>> import numpy - - >>> x = numpy.ones(3) - >>> x - array([ 1., 1., 1.]) - - >>> m = numpy.eye(3) - >>> m - array([[ 1., 0., 0.], - [ 0., 1., 0.], - [ 0., 0., 1.]]) - - >>> x @ m - array([ 1., 1., 1.]) - - -.. seealso:: - - :pep:`465` -- A dedicated infix operator for matrix multiplication - PEP written by Nathaniel J. Smith; implemented by Benjamin Peterson. - - -.. _whatsnew-pep-448: - -PEP 448 - Additional Unpacking Generalizations ----------------------------------------------- - -:pep:`448` extends the allowed uses of the ``*`` iterable unpacking -operator and ``**`` dictionary unpacking operator. It is now possible -to use an arbitrary number of unpackings in :ref:`function calls <calls>`:: - - >>> print(*[1], *[2], 3, *[4, 5]) - 1 2 3 4 5 - - >>> def fn(a, b, c, d): - ... print(a, b, c, d) - ... - - >>> fn(**{'a': 1, 'c': 3}, **{'b': 2, 'd': 4}) - 1 2 3 4 - -Similarly, tuple, list, set, and dictionary displays allow multiple -unpackings (see :ref:`exprlists` and :ref:`dict`):: - - >>> *range(4), 4 - (0, 1, 2, 3, 4) - - >>> [*range(4), 4] - [0, 1, 2, 3, 4] - - >>> {*range(4), 4, *(5, 6, 7)} - {0, 1, 2, 3, 4, 5, 6, 7} - - >>> {'x': 1, **{'y': 2}} - {'x': 1, 'y': 2} - -.. seealso:: - - :pep:`448` -- Additional Unpacking Generalizations - PEP written by Joshua Landau; implemented by Neil Girdhar, - Thomas Wouters, and Joshua Landau. - - -.. _whatsnew-pep-461: - -PEP 461 - percent formatting support for bytes and bytearray ------------------------------------------------------------- - -:pep:`461` adds support for the ``%`` -:ref:`interpolation operator <bytes-formatting>` to :class:`bytes` -and :class:`bytearray`. - -While interpolation is usually thought of as a string operation, there are -cases where interpolation on ``bytes`` or ``bytearrays`` makes sense, and the -work needed to make up for this missing functionality detracts from the -overall readability of the code. This issue is particularly important when -dealing with wire format protocols, which are often a mixture of binary -and ASCII compatible text. - -Examples:: - - >>> b'Hello %b!' % b'World' - b'Hello World!' - - >>> b'x=%i y=%f' % (1, 2.5) - b'x=1 y=2.500000' - -Unicode is not allowed for ``%b``, but it is accepted by ``%a`` (equivalent of -``repr(obj).encode('ascii', 'backslashreplace')``):: - - >>> b'Hello %b!' % 'World' - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - TypeError: %b requires bytes, or an object that implements __bytes__, not 'str' - - >>> b'price: %a' % '10€' - b"price: '10\\u20ac'" - -Note that ``%s`` and ``%r`` conversion types, although supported, should -only be used in codebases that need compatibility with Python 2. - -.. seealso:: - - :pep:`461` -- Adding % formatting to bytes and bytearray - PEP written by Ethan Furman; implemented by Neil Schemenauer and - Ethan Furman. - - -.. _whatsnew-pep-484: - -PEP 484 - Type Hints --------------------- - -Function annotation syntax has been a Python feature since version 3.0 -(:pep:`3107`), however the semantics of annotations has been left undefined. - -Experience has shown that the majority of function annotation -uses were to provide type hints to function parameters and return values. It -became evident that it would be beneficial for Python users, if the -standard library included the base definitions and tools for type annotations. - -:pep:`484` introduces a :term:`provisional module <provisional api>` to -provide these standard definitions and tools, along with some conventions -for situations where annotations are not available. - -For example, here is a simple function whose argument and return type -are declared in the annotations:: - - def greeting(name: str) -> str: - return 'Hello ' + name - -While these annotations are available at runtime through the usual -:attr:`__annotations__` attribute, *no automatic type checking happens at -runtime*. Instead, it is assumed that a separate off-line type checker -(e.g. `mypy <http://mypy-lang.org>`_) will be used for on-demand -source code analysis. - -The type system supports unions, generic types, and a special type -named :class:`~typing.Any` which is consistent with (i.e. assignable to -and from) all types. - -.. seealso:: - - * :mod:`typing` module documentation - * :pep:`484` -- Type Hints - PEP written by Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa; - implemented by Guido van Rossum. - * :pep:`483` -- The Theory of Type Hints - PEP written by Guido van Rossum - - -.. _whatsnew-pep-471: - -PEP 471 - os.scandir() function -- a better and faster directory iterator -------------------------------------------------------------------------- - -:pep:`471` adds a new directory iteration function, :func:`os.scandir`, -to the standard library. Additionally, :func:`os.walk` is now -implemented using ``scandir``, which makes it 3 to 5 times faster -on POSIX systems and 7 to 20 times faster on Windows systems. This is -largely achieved by greatly reducing the number of calls to :func:`os.stat` -required to walk a directory tree. - -Additionally, ``scandir`` returns an iterator, as opposed to returning -a list of file names, which improves memory efficiency when iterating -over very large directories. - -The following example shows a simple use of :func:`os.scandir` to display all -the files (excluding directories) in the given *path* that don't start with -``'.'``. The :meth:`entry.is_file() <os.DirEntry.is_file>` call will generally -not make an additional system call:: - - for entry in os.scandir(path): - if not entry.name.startswith('.') and entry.is_file(): - print(entry.name) - -.. seealso:: - - :pep:`471` -- os.scandir() function -- a better and faster directory iterator - PEP written and implemented by Ben Hoyt with the help of Victor Stinner. - - -.. _whatsnew-pep-475: - -PEP 475: Retry system calls failing with EINTR ----------------------------------------------- - -An :py:data:`errno.EINTR` error code is returned whenever a system call, that -is waiting for I/O, is interrupted by a signal. Previously, Python would -raise :exc:`InterruptedError` in such cases. This meant that, when writing a -Python application, the developer had two choices: - -#. Ignore the ``InterruptedError``. -#. Handle the ``InterruptedError`` and attempt to restart the interrupted - system call at every call site. - -The first option makes an application fail intermittently. -The second option adds a large amount of boilerplate that makes the -code nearly unreadable. Compare:: - - print("Hello World") - -and:: - - while True: - try: - print("Hello World") - break - except InterruptedError: - continue - -:pep:`475` implements automatic retry of system calls on -``EINTR``. This removes the burden of dealing with ``EINTR`` -or :exc:`InterruptedError` in user code in most situations and makes -Python programs, including the standard library, more robust. Note that -the system call is only retried if the signal handler does not raise an -exception. - -Below is a list of functions which are now retried when interrupted -by a signal: - -* :func:`open` and :func:`io.open`; - -* functions of the :mod:`faulthandler` module; - -* :mod:`os` functions: :func:`~os.fchdir`, :func:`~os.fchmod`, - :func:`~os.fchown`, :func:`~os.fdatasync`, :func:`~os.fstat`, - :func:`~os.fstatvfs`, :func:`~os.fsync`, :func:`~os.ftruncate`, - :func:`~os.mkfifo`, :func:`~os.mknod`, :func:`~os.open`, - :func:`~os.posix_fadvise`, :func:`~os.posix_fallocate`, :func:`~os.pread`, - :func:`~os.pwrite`, :func:`~os.read`, :func:`~os.readv`, :func:`~os.sendfile`, - :func:`~os.wait3`, :func:`~os.wait4`, :func:`~os.wait`, - :func:`~os.waitid`, :func:`~os.waitpid`, :func:`~os.write`, - :func:`~os.writev`; - -* special cases: :func:`os.close` and :func:`os.dup2` now ignore - :py:data:`~errno.EINTR` errors; the syscall is not retried (see the PEP - for the rationale); - -* :mod:`select` functions: :func:`devpoll.poll() <select.devpoll.poll>`, - :func:`epoll.poll() <select.epoll.poll>`, - :func:`kqueue.control() <select.kqueue.control>`, - :func:`poll.poll() <select.poll.poll>`, :func:`~select.select`; - -* methods of the :class:`~socket.socket` class: :meth:`~socket.socket.accept`, - :meth:`~socket.socket.connect` (except for non-blocking sockets), - :meth:`~socket.socket.recv`, :meth:`~socket.socket.recvfrom`, - :meth:`~socket.socket.recvmsg`, :meth:`~socket.socket.send`, - :meth:`~socket.socket.sendall`, :meth:`~socket.socket.sendmsg`, - :meth:`~socket.socket.sendto`; - -* :func:`signal.sigtimedwait` and :func:`signal.sigwaitinfo`; - -* :func:`time.sleep`. - -.. seealso:: - - :pep:`475` -- Retry system calls failing with EINTR - PEP and implementation written by Charles-François Natali and - Victor Stinner, with the help of Antoine Pitrou (the French connection). - - -.. _whatsnew-pep-479: - -PEP 479: Change StopIteration handling inside generators --------------------------------------------------------- - -The interaction of generators and :exc:`StopIteration` in Python 3.4 and -earlier was sometimes surprising, and could conceal obscure bugs. Previously, -``StopIteration`` raised accidentally inside a generator function was -interpreted as the end of the iteration by the loop construct driving the -generator. - -:pep:`479` changes the behavior of generators: when a ``StopIteration`` -exception is raised inside a generator, it is replaced with a -:exc:`RuntimeError` before it exits the generator frame. The main goal of -this change is to ease debugging in the situation where an unguarded -:func:`next` call raises ``StopIteration`` and causes the iteration controlled -by the generator to terminate silently. This is particularly pernicious in -combination with the ``yield from`` construct. - -This is a backwards incompatible change, so to enable the new behavior, -a :term:`__future__` import is necessary:: - - >>> from __future__ import generator_stop - - >>> def gen(): - ... next(iter([])) - ... yield - ... - >>> next(gen()) - Traceback (most recent call last): - File "<stdin>", line 2, in gen - StopIteration - - The above exception was the direct cause of the following exception: - - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - RuntimeError: generator raised StopIteration - -Without a ``__future__`` import, a :exc:`PendingDeprecationWarning` will be -raised whenever a :exc:`StopIteration` exception is raised inside a generator. - -.. seealso:: - - :pep:`479` -- Change StopIteration handling inside generators - PEP written by Chris Angelico and Guido van Rossum. Implemented by - Chris Angelico, Yury Selivanov and Nick Coghlan. - - -.. _whatsnew-pep-485: - -PEP 485: A function for testing approximate equality ----------------------------------------------------- - -:pep:`485` adds the :func:`math.isclose` and :func:`cmath.isclose` -functions which tell whether two values are approximately equal or -"close" to each other. Whether or not two values are considered -close is determined according to given absolute and relative tolerances. -Relative tolerance is the maximum allowed difference between ``isclose`` -arguments, relative to the larger absolute value:: - - >>> import math - >>> a = 5.0 - >>> b = 4.99998 - >>> math.isclose(a, b, rel_tol=1e-5) - True - >>> math.isclose(a, b, rel_tol=1e-6) - False - -It is also possible to compare two values using absolute tolerance, which -must be a non-negative value:: - - >>> import math - >>> a = 5.0 - >>> b = 4.99998 - >>> math.isclose(a, b, abs_tol=0.00003) - True - >>> math.isclose(a, b, abs_tol=0.00001) - False - -.. seealso:: - - :pep:`485` -- A function for testing approximate equality - PEP written by Christopher Barker; implemented by Chris Barker and - Tal Einat. - - -.. _whatsnew-pep-486: - -PEP 486: Make the Python Launcher aware of virtual environments ---------------------------------------------------------------- - -:pep:`486` makes the Windows launcher (see :pep:`397`) aware of an active -virtual environment. When the default interpreter would be used and the -``VIRTUAL_ENV`` environment variable is set, the interpreter in the virtual -environment will be used. - -.. seealso:: - - :pep:`486` -- Make the Python Launcher aware of virtual environments - PEP written and implemented by Paul Moore. - - -.. _whatsnew-pep-488: - -PEP 488: Elimination of PYO files ---------------------------------- - -:pep:`488` does away with the concept of ``.pyo`` files. This means that -``.pyc`` files represent both unoptimized and optimized bytecode. To prevent the -need to constantly regenerate bytecode files, ``.pyc`` files now have an -optional ``opt-`` tag in their name when the bytecode is optimized. This has the -side-effect of no more bytecode file name clashes when running under either -:option:`-O` or :option:`-OO`. Consequently, bytecode files generated from -:option:`-O`, and :option:`-OO` may now exist simultaneously. -:func:`importlib.util.cache_from_source` has an updated API to help with -this change. - -.. seealso:: - - :pep:`488` -- Elimination of PYO files - PEP written and implemented by Brett Cannon. - - -.. _whatsnew-pep-489: - -PEP 489: Multi-phase extension module initialization ----------------------------------------------------- - -:pep:`489` updates extension module initialization to take advantage of the -two step module loading mechanism introduced by :pep:`451` in Python 3.4. - -This change brings the import semantics of extension modules that opt-in to -using the new mechanism much closer to those of Python source and bytecode -modules, including the ability to use any valid identifier as a module name, -rather than being restricted to ASCII. - -.. seealso:: - - :pep:`489` -- Multi-phase extension module initialization - PEP written by Petr Viktorin, Stefan Behnel, and Nick Coghlan; - implemented by Petr Viktorin. - - -Other Language Changes -====================== - -Some smaller changes made to the core Python language are: - -* Added the ``"namereplace"`` error handlers. The ``"backslashreplace"`` - error handlers now work with decoding and translating. - (Contributed by Serhiy Storchaka in :issue:`19676` and :issue:`22286`.) - -* The :option:`-b` option now affects comparisons of :class:`bytes` with - :class:`int`. (Contributed by Serhiy Storchaka in :issue:`23681`.) - -* New Kazakh ``kz1048`` and Tajik ``koi8_t`` :ref:`codecs <standard-encodings>`. - (Contributed by Serhiy Storchaka in :issue:`22682` and :issue:`22681`.) - -* Property docstrings are now writable. This is especially useful for - :func:`collections.namedtuple` docstrings. - (Contributed by Berker Peksag in :issue:`24064`.) - -* Circular imports involving relative imports are now supported. - (Contributed by Brett Cannon and Antoine Pitrou in :issue:`17636`.) - - -New Modules -=========== - -typing ------- - -The new :mod:`typing` :term:`provisional <provisional api>` module -provides standard definitions and tools for function type annotations. -See :ref:`Type Hints <whatsnew-pep-484>` for more information. - -.. _whatsnew-zipapp: - -zipapp ------- - -The new :mod:`zipapp` module (specified in :pep:`441`) provides an API and -command line tool for creating executable Python Zip Applications, which -were introduced in Python 2.6 in :issue:`1739468`, but which were not well -publicized, either at the time or since. - -With the new module, bundling your application is as simple as putting all -the files, including a ``__main__.py`` file, into a directory ``myapp`` -and running: - -.. code-block:: shell-session - - $ python -m zipapp myapp - $ python myapp.pyz - -The module implementation has been contributed by Paul Moore in -:issue:`23491`. - -.. seealso:: - - :pep:`441` -- Improving Python ZIP Application Support - - -Improved Modules -================ - -argparse --------- - -The :class:`~argparse.ArgumentParser` class now allows disabling -:ref:`abbreviated usage <prefix-matching>` of long options by setting -:ref:`allow_abbrev` to ``False``. (Contributed by Jonathan Paugh, -Steven Bethard, paul j3 and Daniel Eriksson in :issue:`14910`.) - - -asyncio -------- - -Since the :mod:`asyncio` module is :term:`provisional <provisional api>`, -all changes introduced in Python 3.5 have also been backported to Python 3.4.x. - -Notable changes in the :mod:`asyncio` module since Python 3.4.0: - -* New debugging APIs: :meth:`loop.set_debug() <asyncio.loop.set_debug>` - and :meth:`loop.get_debug() <asyncio.loop.get_debug>` methods. - (Contributed by Victor Stinner.) - -* The proactor event loop now supports SSL. - (Contributed by Antoine Pitrou and Victor Stinner in :issue:`22560`.) - -* A new :meth:`loop.is_closed() <asyncio.loop.is_closed>` method to - check if the event loop is closed. - (Contributed by Victor Stinner in :issue:`21326`.) - -* A new :meth:`loop.create_task() <asyncio.loop.create_task>` - to conveniently create and schedule a new :class:`~asyncio.Task` - for a coroutine. The ``create_task`` method is also used by all - asyncio functions that wrap coroutines into tasks, such as - :func:`asyncio.wait`, :func:`asyncio.gather`, etc. - (Contributed by Victor Stinner.) - -* A new :meth:`transport.get_write_buffer_limits() <asyncio.WriteTransport.get_write_buffer_limits>` - method to inquire for *high-* and *low-* water limits of the flow - control. - (Contributed by Victor Stinner.) - -* The :func:`~asyncio.async` function is deprecated in favor of - :func:`~asyncio.ensure_future`. - (Contributed by Yury Selivanov.) - -* New :meth:`loop.set_task_factory() - <asyncio.loop.set_task_factory>` and - :meth:`loop.get_task_factory() <asyncio.loop.get_task_factory>` - methods to customize the task factory that :meth:`loop.create_task() - <asyncio.loop.create_task>` method uses. (Contributed by Yury - Selivanov.) - -* New :meth:`Queue.join() <asyncio.Queue.join>` and - :meth:`Queue.task_done() <asyncio.Queue.task_done>` queue methods. - (Contributed by Victor Stinner.) - -* The ``JoinableQueue`` class was removed, in favor of the - :class:`asyncio.Queue` class. - (Contributed by Victor Stinner.) - -Updates in 3.5.1: - -* The :func:`~asyncio.ensure_future` function and all functions that - use it, such as :meth:`loop.run_until_complete() <asyncio.loop.run_until_complete>`, - now accept all kinds of :term:`awaitable objects <awaitable>`. - (Contributed by Yury Selivanov.) - -* New :func:`~asyncio.run_coroutine_threadsafe` function to submit - coroutines to event loops from other threads. - (Contributed by Vincent Michel.) - -* New :meth:`Transport.is_closing() <asyncio.BaseTransport.is_closing>` - method to check if the transport is closing or closed. - (Contributed by Yury Selivanov.) - -* The :meth:`loop.create_server() <asyncio.loop.create_server>` - method can now accept a list of hosts. - (Contributed by Yann Sionneau.) - -Updates in 3.5.2: - -* New :meth:`loop.create_future() <asyncio.loop.create_future>` - method to create Future objects. This allows alternative event - loop implementations, such as - `uvloop <https://github.com/MagicStack/uvloop>`_, to provide a faster - :class:`asyncio.Future` implementation. - (Contributed by Yury Selivanov.) - -* New :meth:`loop.get_exception_handler() <asyncio.loop.get_exception_handler>` - method to get the current exception handler. - (Contributed by Yury Selivanov.) - -* New :meth:`StreamReader.readuntil() <asyncio.StreamReader.readuntil>` - method to read data from the stream until a separator bytes - sequence appears. - (Contributed by Mark Korenberg.) - -* The :meth:`loop.create_connection() <asyncio.loop.create_connection>` - and :meth:`loop.create_server() <asyncio.loop.create_server>` - methods are optimized to avoid calling the system ``getaddrinfo`` - function if the address is already resolved. - (Contributed by A. Jesse Jiryu Davis.) - -* The :meth:`loop.sock_connect(sock, address) <asyncio.loop.sock_connect>` - no longer requires the *address* to be resolved prior to the call. - (Contributed by A. Jesse Jiryu Davis.) - - -bz2 ---- - -The :meth:`BZ2Decompressor.decompress <bz2.BZ2Decompressor.decompress>` -method now accepts an optional *max_length* argument to limit the maximum -size of decompressed data. (Contributed by Nikolaus Rath in :issue:`15955`.) - - -cgi ---- - -The :class:`~cgi.FieldStorage` class now supports the :term:`context manager` -protocol. (Contributed by Berker Peksag in :issue:`20289`.) - - -cmath ------ - -A new function :func:`~cmath.isclose` provides a way to test for approximate -equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) - - -code ----- - -The :func:`InteractiveInterpreter.showtraceback() <code.InteractiveInterpreter.showtraceback>` -method now prints the full chained traceback, just like the interactive -interpreter. (Contributed by Claudiu Popa in :issue:`17442`.) - - -collections ------------ - -.. _whatsnew-ordereddict: - -The :class:`~collections.OrderedDict` class is now implemented in C, which -makes it 4 to 100 times faster. (Contributed by Eric Snow in :issue:`16991`.) - -:meth:`OrderedDict.items() <collections.OrderedDict.items>`, -:meth:`OrderedDict.keys() <collections.OrderedDict.keys>`, -:meth:`OrderedDict.values() <collections.OrderedDict.values>` views now support -:func:`reversed` iteration. -(Contributed by Serhiy Storchaka in :issue:`19505`.) - -The :class:`~collections.deque` class now defines -:meth:`~collections.deque.index`, :meth:`~collections.deque.insert`, and -:meth:`~collections.deque.copy`, and supports the ``+`` and ``*`` operators. -This allows deques to be recognized as a :class:`~collections.abc.MutableSequence` -and improves their substitutability for lists. -(Contributed by Raymond Hettinger in :issue:`23704`.) - -Docstrings produced by :func:`~collections.namedtuple` can now be updated:: - - Point = namedtuple('Point', ['x', 'y']) - Point.__doc__ += ': Cartesian coodinate' - Point.x.__doc__ = 'abscissa' - Point.y.__doc__ = 'ordinate' - -(Contributed by Berker Peksag in :issue:`24064`.) - -The :class:`~collections.UserString` class now implements the -:meth:`__getnewargs__`, :meth:`__rmod__`, :meth:`~str.casefold`, -:meth:`~str.format_map`, :meth:`~str.isprintable`, and :meth:`~str.maketrans` -methods to match the corresponding methods of :class:`str`. -(Contributed by Joe Jevnik in :issue:`22189`.) - - -collections.abc ---------------- - -The :meth:`Sequence.index() <collections.abc.Sequence.index>` method now -accepts *start* and *stop* arguments to match the corresponding methods -of :class:`tuple`, :class:`list`, etc. -(Contributed by Devin Jeanpierre in :issue:`23086`.) - -A new :class:`~collections.abc.Generator` abstract base class. (Contributed -by Stefan Behnel in :issue:`24018`.) - -New :class:`~collections.abc.Awaitable`, :class:`~collections.abc.Coroutine`, -:class:`~collections.abc.AsyncIterator`, and -:class:`~collections.abc.AsyncIterable` abstract base classes. -(Contributed by Yury Selivanov in :issue:`24184`.) - -For earlier Python versions, a backport of the new ABCs is available in an -external `PyPI package <https://pypi.org/project/backports_abc>`_. - - -compileall ----------- - -A new :mod:`compileall` option, :samp:`-j {N}`, allows running *N* workers -simultaneously to perform parallel bytecode compilation. -The :func:`~compileall.compile_dir` function has a corresponding ``workers`` -parameter. (Contributed by Claudiu Popa in :issue:`16104`.) - -Another new option, ``-r``, allows controlling the maximum recursion -level for subdirectories. (Contributed by Claudiu Popa in :issue:`19628`.) - -The ``-q`` command line option can now be specified more than once, in -which case all output, including errors, will be suppressed. The corresponding -``quiet`` parameter in :func:`~compileall.compile_dir`, -:func:`~compileall.compile_file`, and :func:`~compileall.compile_path` can now -accept an integer value indicating the level of output suppression. -(Contributed by Thomas Kluyver in :issue:`21338`.) - - -concurrent.futures ------------------- - -The :meth:`Executor.map() <concurrent.futures.Executor.map>` method now accepts a -*chunksize* argument to allow batching of tasks to improve performance when -:meth:`~concurrent.futures.ProcessPoolExecutor` is used. -(Contributed by Dan O'Reilly in :issue:`11271`.) - -The number of workers in the :class:`~concurrent.futures.ThreadPoolExecutor` -constructor is optional now. The default value is 5 times the number of CPUs. -(Contributed by Claudiu Popa in :issue:`21527`.) - - -configparser ------------- - -:mod:`configparser` now provides a way to customize the conversion -of values by specifying a dictionary of converters in the -:class:`~configparser.ConfigParser` constructor, or by defining them -as methods in ``ConfigParser`` subclasses. Converters defined in -a parser instance are inherited by its section proxies. - -Example:: - - >>> import configparser - >>> conv = {} - >>> conv['list'] = lambda v: [e.strip() for e in v.split() if e.strip()] - >>> cfg = configparser.ConfigParser(converters=conv) - >>> cfg.read_string(""" - ... [s] - ... list = a b c d e f g - ... """) - >>> cfg.get('s', 'list') - 'a b c d e f g' - >>> cfg.getlist('s', 'list') - ['a', 'b', 'c', 'd', 'e', 'f', 'g'] - >>> section = cfg['s'] - >>> section.getlist('list') - ['a', 'b', 'c', 'd', 'e', 'f', 'g'] - -(Contributed by Łukasz Langa in :issue:`18159`.) - - -contextlib ----------- - -The new :func:`~contextlib.redirect_stderr` :term:`context manager` (similar to -:func:`~contextlib.redirect_stdout`) makes it easier for utility scripts to -handle inflexible APIs that write their output to :data:`sys.stderr` and -don't provide any options to redirect it:: - - >>> import contextlib, io, logging - >>> f = io.StringIO() - >>> with contextlib.redirect_stderr(f): - ... logging.warning('warning') - ... - >>> f.getvalue() - 'WARNING:root:warning\n' - -(Contributed by Berker Peksag in :issue:`22389`.) - - -csv ---- - -The :meth:`~csv.csvwriter.writerow` method now supports arbitrary iterables, -not just sequences. (Contributed by Serhiy Storchaka in :issue:`23171`.) - - -curses ------- - -The new :func:`~curses.update_lines_cols` function updates the :envvar:`LINES` -and :envvar:`COLS` environment variables. This is useful for detecting -manual screen resizing. (Contributed by Arnon Yaari in :issue:`4254`.) - - -dbm ---- - -:func:`dumb.open <dbm.dumb.open>` always creates a new database when the flag -has the value ``"n"``. (Contributed by Claudiu Popa in :issue:`18039`.) - - -difflib -------- - -The charset of HTML documents generated by -:meth:`HtmlDiff.make_file() <difflib.HtmlDiff.make_file>` -can now be customized by using a new *charset* keyword-only argument. -The default charset of HTML document changed from ``"ISO-8859-1"`` -to ``"utf-8"``. -(Contributed by Berker Peksag in :issue:`2052`.) - -The :func:`~difflib.diff_bytes` function can now compare lists of byte -strings. This fixes a regression from Python 2. -(Contributed by Terry J. Reedy and Greg Ward in :issue:`17445`.) - - -distutils ---------- - -Both the ``build`` and ``build_ext`` commands now accept a ``-j`` option to -enable parallel building of extension modules. -(Contributed by Antoine Pitrou in :issue:`5309`.) - -The :mod:`distutils` module now supports ``xz`` compression, and can be -enabled by passing ``xztar`` as an argument to ``bdist --format``. -(Contributed by Serhiy Storchaka in :issue:`16314`.) - - -doctest -------- - -The :func:`~doctest.DocTestSuite` function returns an empty -:class:`unittest.TestSuite` if *module* contains no docstrings, instead of -raising :exc:`ValueError`. (Contributed by Glenn Jones in :issue:`15916`.) - - -email ------ - -A new policy option :attr:`Policy.mangle_from_ <email.policy.Policy.mangle_from_>` -controls whether or not lines that start with ``"From "`` in email bodies are -prefixed with a ``">"`` character by generators. The default is ``True`` for -:attr:`~email.policy.compat32` and ``False`` for all other policies. -(Contributed by Milan Oberkirch in :issue:`20098`.) - -A new -:meth:`Message.get_content_disposition() <email.message.Message.get_content_disposition>` -method provides easy access to a canonical value for the -:mailheader:`Content-Disposition` header. -(Contributed by Abhilash Raj in :issue:`21083`.) - -A new policy option :attr:`EmailPolicy.utf8 <email.policy.EmailPolicy.utf8>` -can be set to ``True`` to encode email headers using the UTF-8 charset instead -of using encoded words. This allows ``Messages`` to be formatted according to -:rfc:`6532` and used with an SMTP server that supports the :rfc:`6531` -``SMTPUTF8`` extension. (Contributed by R. David Murray in -:issue:`24211`.) - -The :class:`mime.text.MIMEText <email.mime.text.MIMEText>` constructor now -accepts a :class:`charset.Charset <email.charset.Charset>` instance. -(Contributed by Claude Paroz and Berker Peksag in :issue:`16324`.) - - -enum ----- - -The :class:`~enum.Enum` callable has a new parameter *start* to -specify the initial number of enum values if only *names* are provided:: - - >>> Animal = enum.Enum('Animal', 'cat dog', start=10) - >>> Animal.cat - <Animal.cat: 10> - >>> Animal.dog - <Animal.dog: 11> - -(Contributed by Ethan Furman in :issue:`21706`.) - - -faulthandler ------------- - -The :func:`~faulthandler.enable`, :func:`~faulthandler.register`, -:func:`~faulthandler.dump_traceback` and -:func:`~faulthandler.dump_traceback_later` functions now accept file -descriptors in addition to file-like objects. -(Contributed by Wei Wu in :issue:`23566`.) - - -functools ---------- - -.. _whatsnew-lrucache: - -Most of the :func:`~functools.lru_cache` machinery is now implemented in C, making -it significantly faster. (Contributed by Matt Joiner, Alexey Kachayev, and -Serhiy Storchaka in :issue:`14373`.) - - -glob ----- - -The :func:`~glob.iglob` and :func:`~glob.glob` functions now support recursive -search in subdirectories, using the ``"**"`` pattern. -(Contributed by Serhiy Storchaka in :issue:`13968`.) - - -gzip ----- - -The *mode* argument of the :class:`~gzip.GzipFile` constructor now -accepts ``"x"`` to request exclusive creation. -(Contributed by Tim Heaney in :issue:`19222`.) - - -heapq ------ - -Element comparison in :func:`~heapq.merge` can now be customized by -passing a :term:`key function` in a new optional *key* keyword argument, -and a new optional *reverse* keyword argument can be used to reverse element -comparison:: - - >>> import heapq - >>> a = ['9', '777', '55555'] - >>> b = ['88', '6666'] - >>> list(heapq.merge(a, b, key=len)) - ['9', '88', '777', '6666', '55555'] - >>> list(heapq.merge(reversed(a), reversed(b), key=len, reverse=True)) - ['55555', '6666', '777', '88', '9'] - -(Contributed by Raymond Hettinger in :issue:`13742`.) - - -http ----- - -A new :class:`HTTPStatus <http.HTTPStatus>` enum that defines a set of -HTTP status codes, reason phrases and long descriptions written in English. -(Contributed by Demian Brecht in :issue:`21793`.) - - -http.client ------------ - -:meth:`HTTPConnection.getresponse() <http.client.HTTPConnection.getresponse>` -now raises a :exc:`~http.client.RemoteDisconnected` exception when a -remote server connection is closed unexpectedly. Additionally, if a -:exc:`ConnectionError` (of which ``RemoteDisconnected`` -is a subclass) is raised, the client socket is now closed automatically, -and will reconnect on the next request:: - - import http.client - conn = http.client.HTTPConnection('www.python.org') - for retries in range(3): - try: - conn.request('GET', '/') - resp = conn.getresponse() - except http.client.RemoteDisconnected: - pass - -(Contributed by Martin Panter in :issue:`3566`.) - - -idlelib and IDLE ----------------- - -Since idlelib implements the IDLE shell and editor and is not intended for -import by other programs, it gets improvements with every release. See -:file:`Lib/idlelib/NEWS.txt` for a cumulative list of changes since 3.4.0, -as well as changes made in future 3.5.x releases. This file is also available -from the IDLE :menuselection:`Help --> About IDLE` dialog. - - -imaplib -------- - -The :class:`~imaplib.IMAP4` class now supports the :term:`context manager` protocol. -When used in a :keyword:`with` statement, the IMAP4 ``LOGOUT`` -command will be called automatically at the end of the block. -(Contributed by Tarek Ziadé and Serhiy Storchaka in :issue:`4972`.) - -The :mod:`imaplib` module now supports :rfc:`5161` (ENABLE Extension) -and :rfc:`6855` (UTF-8 Support) via the :meth:`IMAP4.enable() <imaplib.IMAP4.enable>` -method. A new :attr:`IMAP4.utf8_enabled <imaplib.IMAP4.utf8_enabled>` -attribute tracks whether or not :rfc:`6855` support is enabled. -(Contributed by Milan Oberkirch, R. David Murray, and Maciej Szulik in -:issue:`21800`.) - -The :mod:`imaplib` module now automatically encodes non-ASCII string usernames -and passwords using UTF-8, as recommended by the RFCs. (Contributed by Milan -Oberkirch in :issue:`21800`.) - - -imghdr ------- - -The :func:`~imghdr.what` function now recognizes the -`OpenEXR <http://www.openexr.com>`_ format -(contributed by Martin Vignali and Claudiu Popa in :issue:`20295`), -and the `WebP <https://en.wikipedia.org/wiki/WebP>`_ format -(contributed by Fabrice Aneche and Claudiu Popa in :issue:`20197`.) - - -importlib ---------- - -The :class:`util.LazyLoader <importlib.util.LazyLoader>` class allows for -lazy loading of modules in applications where startup time is important. -(Contributed by Brett Cannon in :issue:`17621`.) - -The :func:`abc.InspectLoader.source_to_code() <importlib.abc.InspectLoader.source_to_code>` -method is now a static method. This makes it easier to initialize a module -object with code compiled from a string by running -``exec(code, module.__dict__)``. -(Contributed by Brett Cannon in :issue:`21156`.) - -The new :func:`util.module_from_spec() <importlib.util.module_from_spec>` -function is now the preferred way to create a new module. As opposed to -creating a :class:`types.ModuleType` instance directly, this new function -will set the various import-controlled attributes based on the passed-in -spec object. (Contributed by Brett Cannon in :issue:`20383`.) - - -inspect -------- - -Both the :class:`~inspect.Signature` and :class:`~inspect.Parameter` classes are -now picklable and hashable. (Contributed by Yury Selivanov in :issue:`20726` -and :issue:`20334`.) - -A new -:meth:`BoundArguments.apply_defaults() <inspect.BoundArguments.apply_defaults>` -method provides a way to set default values for missing arguments:: - - >>> def foo(a, b='ham', *args): pass - >>> ba = inspect.signature(foo).bind('spam') - >>> ba.apply_defaults() - >>> ba.arguments - OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())]) - -(Contributed by Yury Selivanov in :issue:`24190`.) - -A new class method -:meth:`Signature.from_callable() <inspect.Signature.from_callable>` makes -subclassing of :class:`~inspect.Signature` easier. (Contributed -by Yury Selivanov and Eric Snow in :issue:`17373`.) - -The :func:`~inspect.signature` function now accepts a *follow_wrapped* -optional keyword argument, which, when set to ``False``, disables automatic -following of ``__wrapped__`` links. -(Contributed by Yury Selivanov in :issue:`20691`.) - -A set of new functions to inspect -:term:`coroutine functions <coroutine function>` and -:term:`coroutine objects <coroutine>` has been added: -:func:`~inspect.iscoroutine`, :func:`~inspect.iscoroutinefunction`, -:func:`~inspect.isawaitable`, :func:`~inspect.getcoroutinelocals`, -and :func:`~inspect.getcoroutinestate`. -(Contributed by Yury Selivanov in :issue:`24017` and :issue:`24400`.) - -The :func:`~inspect.stack`, :func:`~inspect.trace`, -:func:`~inspect.getouterframes`, and :func:`~inspect.getinnerframes` -functions now return a list of named tuples. -(Contributed by Daniel Shahaf in :issue:`16808`.) - - -io --- - -A new :meth:`BufferedIOBase.readinto1() <io.BufferedIOBase.readinto1>` -method, that uses at most one call to the underlying raw stream's -:meth:`RawIOBase.read() <io.RawIOBase.read>` or -:meth:`RawIOBase.readinto() <io.RawIOBase.readinto>` methods. -(Contributed by Nikolaus Rath in :issue:`20578`.) - - -ipaddress ---------- - -Both the :class:`~ipaddress.IPv4Network` and :class:`~ipaddress.IPv6Network` classes -now accept an ``(address, netmask)`` tuple argument, so as to easily construct -network objects from existing addresses:: - - >>> import ipaddress - >>> ipaddress.IPv4Network(('127.0.0.0', 8)) - IPv4Network('127.0.0.0/8') - >>> ipaddress.IPv4Network(('127.0.0.0', '255.0.0.0')) - IPv4Network('127.0.0.0/8') - -(Contributed by Peter Moody and Antoine Pitrou in :issue:`16531`.) - -A new :attr:`~ipaddress.IPv4Network.reverse_pointer` attribute for the -:class:`~ipaddress.IPv4Network` and :class:`~ipaddress.IPv6Network` classes -returns the name of the reverse DNS PTR record:: - - >>> import ipaddress - >>> addr = ipaddress.IPv4Address('127.0.0.1') - >>> addr.reverse_pointer - '1.0.0.127.in-addr.arpa' - >>> addr6 = ipaddress.IPv6Address('::1') - >>> addr6.reverse_pointer - '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa' - -(Contributed by Leon Weber in :issue:`20480`.) - - -json ----- - -The :mod:`json.tool` command line interface now preserves the order of keys in -JSON objects passed in input. The new ``--sort-keys`` option can be used -to sort the keys alphabetically. (Contributed by Berker Peksag -in :issue:`21650`.) - -JSON decoder now raises :exc:`~json.JSONDecodeError` instead of -:exc:`ValueError` to provide better context information about the error. -(Contributed by Serhiy Storchaka in :issue:`19361`.) - - -linecache ---------- - -A new :func:`~linecache.lazycache` function can be used to capture information -about a non-file-based module to permit getting its lines later via -:func:`~linecache.getline`. This avoids doing I/O until a line is actually -needed, without having to carry the module globals around indefinitely. -(Contributed by Robert Collins in :issue:`17911`.) - - -locale ------- - -A new :func:`~locale.delocalize` function can be used to convert a string into -a normalized number string, taking the ``LC_NUMERIC`` settings into account:: - - >>> import locale - >>> locale.setlocale(locale.LC_NUMERIC, 'de_DE.UTF-8') - 'de_DE.UTF-8' - >>> locale.delocalize('1.234,56') - '1234.56' - >>> locale.setlocale(locale.LC_NUMERIC, 'en_US.UTF-8') - 'en_US.UTF-8' - >>> locale.delocalize('1,234.56') - '1234.56' - -(Contributed by Cédric Krier in :issue:`13918`.) - - -logging -------- - -All logging methods (:class:`~logging.Logger` :meth:`~logging.Logger.log`, -:meth:`~logging.Logger.exception`, :meth:`~logging.Logger.critical`, -:meth:`~logging.Logger.debug`, etc.), now accept exception instances -as an *exc_info* argument, in addition to boolean values and exception -tuples:: - - >>> import logging - >>> try: - ... 1/0 - ... except ZeroDivisionError as ex: - ... logging.error('exception', exc_info=ex) - ERROR:root:exception - -(Contributed by Yury Selivanov in :issue:`20537`.) - -The :class:`handlers.HTTPHandler <logging.handlers.HTTPHandler>` class now -accepts an optional :class:`ssl.SSLContext` instance to configure SSL -settings used in an HTTP connection. -(Contributed by Alex Gaynor in :issue:`22788`.) - -The :class:`handlers.QueueListener <logging.handlers.QueueListener>` class now -takes a *respect_handler_level* keyword argument which, if set to ``True``, -will pass messages to handlers taking handler levels into account. -(Contributed by Vinay Sajip.) - - -lzma ----- - -The :meth:`LZMADecompressor.decompress() <lzma.LZMADecompressor.decompress>` -method now accepts an optional *max_length* argument to limit the maximum -size of decompressed data. -(Contributed by Martin Panter in :issue:`15955`.) - - -math ----- - -Two new constants have been added to the :mod:`math` module: :data:`~math.inf` -and :data:`~math.nan`. (Contributed by Mark Dickinson in :issue:`23185`.) - -A new function :func:`~math.isclose` provides a way to test for approximate -equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) - -A new :func:`~math.gcd` function has been added. The :func:`fractions.gcd` -function is now deprecated. (Contributed by Mark Dickinson and Serhiy -Storchaka in :issue:`22486`.) - - -multiprocessing ---------------- - -:func:`sharedctypes.synchronized() <multiprocessing.sharedctypes.synchronized>` -objects now support the :term:`context manager` protocol. -(Contributed by Charles-François Natali in :issue:`21565`.) - - -operator --------- - -:func:`~operator.attrgetter`, :func:`~operator.itemgetter`, -and :func:`~operator.methodcaller` objects now support pickling. -(Contributed by Josh Rosenberg and Serhiy Storchaka in :issue:`22955`.) - -New :func:`~operator.matmul` and :func:`~operator.imatmul` functions -to perform matrix multiplication. -(Contributed by Benjamin Peterson in :issue:`21176`.) - - -os --- - -The new :func:`~os.scandir` function returning an iterator of -:class:`~os.DirEntry` objects has been added. If possible, :func:`~os.scandir` -extracts file attributes while scanning a directory, removing the need to -perform subsequent system calls to determine file type or attributes, which may -significantly improve performance. (Contributed by Ben Hoyt with the help -of Victor Stinner in :issue:`22524`.) - -On Windows, a new -:attr:`stat_result.st_file_attributes <os.stat_result.st_file_attributes>` -attribute is now available. It corresponds to the ``dwFileAttributes`` member -of the ``BY_HANDLE_FILE_INFORMATION`` structure returned by -``GetFileInformationByHandle()``. (Contributed by Ben Hoyt in :issue:`21719`.) - -The :func:`~os.urandom` function now uses the ``getrandom()`` syscall on Linux 3.17 -or newer, and ``getentropy()`` on OpenBSD 5.6 and newer, removing the need to -use ``/dev/urandom`` and avoiding failures due to potential file descriptor -exhaustion. (Contributed by Victor Stinner in :issue:`22181`.) - -New :func:`~os.get_blocking` and :func:`~os.set_blocking` functions allow -getting and setting a file descriptor's blocking mode (:data:`~os.O_NONBLOCK`.) -(Contributed by Victor Stinner in :issue:`22054`.) - -The :func:`~os.truncate` and :func:`~os.ftruncate` functions are now supported -on Windows. (Contributed by Steve Dower in :issue:`23668`.) - -There is a new :func:`os.path.commonpath` function returning the longest -common sub-path of each passed pathname. Unlike the -:func:`os.path.commonprefix` function, it always returns a valid -path:: - - >>> os.path.commonprefix(['/usr/lib', '/usr/local/lib']) - '/usr/l' - - >>> os.path.commonpath(['/usr/lib', '/usr/local/lib']) - '/usr' - -(Contributed by Rafik Draoui and Serhiy Storchaka in :issue:`10395`.) - - -pathlib -------- - -The new :meth:`Path.samefile() <pathlib.Path.samefile>` method can be used -to check whether the path points to the same file as another path, which can -be either another :class:`~pathlib.Path` object, or a string:: - - >>> import pathlib - >>> p1 = pathlib.Path('/etc/hosts') - >>> p2 = pathlib.Path('/etc/../etc/hosts') - >>> p1.samefile(p2) - True - -(Contributed by Vajrasky Kok and Antoine Pitrou in :issue:`19775`.) - -The :meth:`Path.mkdir() <pathlib.Path.mkdir>` method now accepts a new optional -*exist_ok* argument to match ``mkdir -p`` and :func:`os.makedirs` -functionality. (Contributed by Berker Peksag in :issue:`21539`.) - -There is a new :meth:`Path.expanduser() <pathlib.Path.expanduser>` method to -expand ``~`` and ``~user`` prefixes. (Contributed by Serhiy Storchaka and -Claudiu Popa in :issue:`19776`.) - -A new :meth:`Path.home() <pathlib.Path.home>` class method can be used to get -a :class:`~pathlib.Path` instance representing the user’s home -directory. -(Contributed by Victor Salgado and Mayank Tripathi in :issue:`19777`.) - -New :meth:`Path.write_text() <pathlib.Path.write_text>`, -:meth:`Path.read_text() <pathlib.Path.read_text>`, -:meth:`Path.write_bytes() <pathlib.Path.write_bytes>`, -:meth:`Path.read_bytes() <pathlib.Path.read_bytes>` methods to simplify -read/write operations on files. - -The following code snippet will create or rewrite existing file -``~/spam42``:: - - >>> import pathlib - >>> p = pathlib.Path('~/spam42') - >>> p.expanduser().write_text('ham') - 3 - -(Contributed by Christopher Welborn in :issue:`20218`.) - - -pickle ------- - -Nested objects, such as unbound methods or nested classes, can now be pickled -using :ref:`pickle protocols <pickle-protocols>` older than protocol version 4. -Protocol version 4 already supports these cases. (Contributed by Serhiy -Storchaka in :issue:`23611`.) - - -poplib ------- - -A new :meth:`POP3.utf8() <poplib.POP3.utf8>` command enables :rfc:`6856` -(Internationalized Email) support, if a POP server supports it. -(Contributed by Milan OberKirch in :issue:`21804`.) - - -re --- - -References and conditional references to groups with fixed length are now -allowed in lookbehind assertions:: - - >>> import re - >>> pat = re.compile(r'(a|b).(?<=\1)c') - >>> pat.match('aac') - <_sre.SRE_Match object; span=(0, 3), match='aac'> - >>> pat.match('bbc') - <_sre.SRE_Match object; span=(0, 3), match='bbc'> - -(Contributed by Serhiy Storchaka in :issue:`9179`.) - -The number of capturing groups in regular expressions is no longer limited to -100. (Contributed by Serhiy Storchaka in :issue:`22437`.) - -The :func:`~re.sub` and :func:`~re.subn` functions now replace unmatched -groups with empty strings instead of raising an exception. -(Contributed by Serhiy Storchaka in :issue:`1519638`.) - -The :class:`re.error` exceptions have new attributes, -:attr:`~re.error.msg`, :attr:`~re.error.pattern`, -:attr:`~re.error.pos`, :attr:`~re.error.lineno`, -and :attr:`~re.error.colno`, that provide better context -information about the error:: - - >>> re.compile(""" - ... (?x) - ... .++ - ... """) - Traceback (most recent call last): - ... - sre_constants.error: multiple repeat at position 16 (line 3, column 7) - -(Contributed by Serhiy Storchaka in :issue:`22578`.) - - -readline --------- - -A new :func:`~readline.append_history_file` function can be used to append -the specified number of trailing elements in history to the given file. -(Contributed by Bruno Cauet in :issue:`22940`.) - - -selectors ---------- - -The new :class:`~selectors.DevpollSelector` supports efficient -``/dev/poll`` polling on Solaris. -(Contributed by Giampaolo Rodola' in :issue:`18931`.) - - -shutil ------- - -The :func:`~shutil.move` function now accepts a *copy_function* argument, -allowing, for example, the :func:`~shutil.copy` function to be used instead of -the default :func:`~shutil.copy2` if there is a need to ignore file metadata -when moving. -(Contributed by Claudiu Popa in :issue:`19840`.) - -The :func:`~shutil.make_archive` function now supports the *xztar* format. -(Contributed by Serhiy Storchaka in :issue:`5411`.) - - -signal ------- - -On Windows, the :func:`~signal.set_wakeup_fd` function now also supports -socket handles. (Contributed by Victor Stinner in :issue:`22018`.) - -Various ``SIG*`` constants in the :mod:`signal` module have been converted into -:mod:`Enums <enum>`. This allows meaningful names to be printed -during debugging, instead of integer "magic numbers". -(Contributed by Giampaolo Rodola' in :issue:`21076`.) - - -smtpd ------ - -Both the :class:`~smtpd.SMTPServer` and :class:`~smtpd.SMTPChannel` classes now -accept a *decode_data* keyword argument to determine if the ``DATA`` portion of -the SMTP transaction is decoded using the ``"utf-8"`` codec or is instead -provided to the -:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` -method as a byte string. The default is ``True`` for backward compatibility -reasons, but will change to ``False`` in Python 3.6. If *decode_data* is set -to ``False``, the ``process_message`` method must be prepared to accept keyword -arguments. -(Contributed by Maciej Szulik in :issue:`19662`.) - -The :class:`~smtpd.SMTPServer` class now advertises the ``8BITMIME`` extension -(:rfc:`6152`) if *decode_data* has been set ``True``. If the client -specifies ``BODY=8BITMIME`` on the ``MAIL`` command, it is passed to -:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` -via the *mail_options* keyword. -(Contributed by Milan Oberkirch and R. David Murray in :issue:`21795`.) - -The :class:`~smtpd.SMTPServer` class now also supports the ``SMTPUTF8`` -extension (:rfc:`6531`: Internationalized Email). If the client specified -``SMTPUTF8 BODY=8BITMIME`` on the ``MAIL`` command, they are passed to -:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` -via the *mail_options* keyword. It is the responsibility of the -``process_message`` method to correctly handle the ``SMTPUTF8`` data. -(Contributed by Milan Oberkirch in :issue:`21725`.) - -It is now possible to provide, directly or via name resolution, IPv6 -addresses in the :class:`~smtpd.SMTPServer` constructor, and have it -successfully connect. (Contributed by Milan Oberkirch in :issue:`14758`.) - - -smtplib -------- - -A new :meth:`SMTP.auth() <smtplib.SMTP.auth>` method provides a convenient way to -implement custom authentication mechanisms. (Contributed by Milan -Oberkirch in :issue:`15014`.) - -The :meth:`SMTP.set_debuglevel() <smtplib.SMTP.set_debuglevel>` method now -accepts an additional debuglevel (2), which enables timestamps in debug -messages. (Contributed by Gavin Chappell and Maciej Szulik in :issue:`16914`.) - -Both the :meth:`SMTP.sendmail() <smtplib.SMTP.sendmail>` and -:meth:`SMTP.send_message() <smtplib.SMTP.send_message>` methods now -support :rfc:`6531` (SMTPUTF8). -(Contributed by Milan Oberkirch and R. David Murray in :issue:`22027`.) - - -sndhdr ------- - -The :func:`~sndhdr.what` and :func:`~sndhdr.whathdr` functions now return -a :func:`~collections.namedtuple`. (Contributed by Claudiu Popa in -:issue:`18615`.) - - -socket ------- - -Functions with timeouts now use a monotonic clock, instead of a system clock. -(Contributed by Victor Stinner in :issue:`22043`.) - -A new :meth:`socket.sendfile() <socket.socket.sendfile>` method allows -sending a file over a socket by using the high-performance :func:`os.sendfile` -function on UNIX, resulting in uploads being from 2 to 3 times faster than when -using plain :meth:`socket.send() <socket.socket.send>`. -(Contributed by Giampaolo Rodola' in :issue:`17552`.) - -The :meth:`socket.sendall() <socket.socket.sendall>` method no longer resets the -socket timeout every time bytes are received or sent. The socket timeout is -now the maximum total duration to send all data. -(Contributed by Victor Stinner in :issue:`23853`.) - -The *backlog* argument of the :meth:`socket.listen() <socket.socket.listen>` -method is now optional. By default it is set to -:data:`SOMAXCONN <socket.SOMAXCONN>` or to ``128``, whichever is less. -(Contributed by Charles-François Natali in :issue:`21455`.) - - -ssl ---- - -.. _whatsnew-sslmemorybio: - -Memory BIO Support -~~~~~~~~~~~~~~~~~~ - -(Contributed by Geert Jansen in :issue:`21965`.) - -The new :class:`~ssl.SSLObject` class has been added to provide SSL protocol -support for cases when the network I/O capabilities of :class:`~ssl.SSLSocket` -are not necessary or are suboptimal. ``SSLObject`` represents -an SSL protocol instance, but does not implement any network I/O methods, and -instead provides a memory buffer interface. The new :class:`~ssl.MemoryBIO` -class can be used to pass data between Python and an SSL protocol instance. - -The memory BIO SSL support is primarily intended to be used in frameworks -implementing asynchronous I/O for which :class:`~ssl.SSLSocket`'s readiness -model ("select/poll") is inefficient. - -A new :meth:`SSLContext.wrap_bio() <ssl.SSLContext.wrap_bio>` method can be used -to create a new ``SSLObject`` instance. - - -Application-Layer Protocol Negotiation Support -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -(Contributed by Benjamin Peterson in :issue:`20188`.) - -Where OpenSSL support is present, the :mod:`ssl` module now implements -the *Application-Layer Protocol Negotiation* TLS extension as described -in :rfc:`7301`. - -The new :meth:`SSLContext.set_alpn_protocols() <ssl.SSLContext.set_alpn_protocols>` -can be used to specify which protocols a socket should advertise during -the TLS handshake. - -The new -:meth:`SSLSocket.selected_alpn_protocol() <ssl.SSLSocket.selected_alpn_protocol>` -returns the protocol that was selected during the TLS handshake. -The :data:`~ssl.HAS_ALPN` flag indicates whether ALPN support is present. - - -Other Changes -~~~~~~~~~~~~~ - -There is a new :meth:`SSLSocket.version() <ssl.SSLSocket.version>` method to -query the actual protocol version in use. -(Contributed by Antoine Pitrou in :issue:`20421`.) - -The :class:`~ssl.SSLSocket` class now implements -a :meth:`SSLSocket.sendfile() <ssl.SSLSocket.sendfile>` method. -(Contributed by Giampaolo Rodola' in :issue:`17552`.) - -The :meth:`SSLSocket.send() <ssl.SSLSocket.send>` method now raises either -the :exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError` exception on a -non-blocking socket if the operation would block. Previously, it would return -``0``. (Contributed by Nikolaus Rath in :issue:`20951`.) - -The :func:`~ssl.cert_time_to_seconds` function now interprets the input time -as UTC and not as local time, per :rfc:`5280`. Additionally, the return -value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.) - -New :meth:`SSLObject.shared_ciphers() <ssl.SSLObject.shared_ciphers>` and -:meth:`SSLSocket.shared_ciphers() <ssl.SSLSocket.shared_ciphers>` methods return -the list of ciphers sent by the client during the handshake. -(Contributed by Benjamin Peterson in :issue:`23186`.) - -The :meth:`SSLSocket.do_handshake() <ssl.SSLSocket.do_handshake>`, -:meth:`SSLSocket.read() <ssl.SSLSocket.read>`, -:meth:`SSLSocket.shutdown() <ssl.SSLSocket.shutdown>`, and -:meth:`SSLSocket.write() <ssl.SSLSocket.write>` methods of the :class:`~ssl.SSLSocket` -class no longer reset the socket timeout every time bytes are received or sent. -The socket timeout is now the maximum total duration of the method. -(Contributed by Victor Stinner in :issue:`23853`.) - -The :func:`~ssl.match_hostname` function now supports matching of IP addresses. -(Contributed by Antoine Pitrou in :issue:`23239`.) - - -sqlite3 -------- - -The :class:`~sqlite3.Row` class now fully supports the sequence protocol, -in particular :func:`reversed` iteration and slice indexing. -(Contributed by Claudiu Popa in :issue:`10203`; by Lucas Sinclair, -Jessica McKellar, and Serhiy Storchaka in :issue:`13583`.) - - -.. _whatsnew-subprocess: - -subprocess ----------- - -The new :func:`~subprocess.run` function has been added. -It runs the specified command and returns a -:class:`~subprocess.CompletedProcess` object, which describes a finished -process. The new API is more consistent and is the recommended approach -to invoking subprocesses in Python code that does not need to maintain -compatibility with earlier Python versions. -(Contributed by Thomas Kluyver in :issue:`23342`.) - -Examples:: - - >>> subprocess.run(["ls", "-l"]) # doesn't capture output - CompletedProcess(args=['ls', '-l'], returncode=0) - - >>> subprocess.run("exit 1", shell=True, check=True) - Traceback (most recent call last): - ... - subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 - - >>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE) - CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0, - stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n') - - -sys ---- - -A new :func:`~sys.set_coroutine_wrapper` function allows setting a global -hook that will be called whenever a :term:`coroutine object <coroutine>` -is created by an :keyword:`async def` function. A corresponding -:func:`~sys.get_coroutine_wrapper` can be used to obtain a currently set -wrapper. Both functions are :term:`provisional <provisional api>`, -and are intended for debugging purposes only. (Contributed by Yury Selivanov -in :issue:`24017`.) - -A new :func:`~sys.is_finalizing` function can be used to check if the Python -interpreter is :term:`shutting down <interpreter shutdown>`. -(Contributed by Antoine Pitrou in :issue:`22696`.) - - -sysconfig ---------- - -The name of the user scripts directory on Windows now includes the first -two components of the Python version. (Contributed by Paul Moore -in :issue:`23437`.) - - -tarfile -------- - -The *mode* argument of the :func:`~tarfile.open` function now accepts ``"x"`` -to request exclusive creation. (Contributed by Berker Peksag in :issue:`21717`.) - -The :meth:`TarFile.extractall() <tarfile.TarFile.extractall>` and -:meth:`TarFile.extract() <tarfile.TarFile.extract>` methods now take a keyword -argument *numeric_owner*. If set to ``True``, the extracted files and -directories will be owned by the numeric ``uid`` and ``gid`` from the tarfile. -If set to ``False`` (the default, and the behavior in versions prior to 3.5), -they will be owned by the named user and group in the tarfile. -(Contributed by Michael Vogt and Eric Smith in :issue:`23193`.) - -The :meth:`TarFile.list() <tarfile.TarFile.list>` now accepts an optional -*members* keyword argument that can be set to a subset of the list returned -by :meth:`TarFile.getmembers() <tarfile.TarFile.getmembers>`. -(Contributed by Serhiy Storchaka in :issue:`21549`.) - - -threading ---------- - -Both the :meth:`Lock.acquire() <threading.Lock.acquire>` and -:meth:`RLock.acquire() <threading.RLock.acquire>` methods -now use a monotonic clock for timeout management. -(Contributed by Victor Stinner in :issue:`22043`.) - - -time ----- - -The :func:`~time.monotonic` function is now always available. -(Contributed by Victor Stinner in :issue:`22043`.) - - -timeit ------- - -A new command line option ``-u`` or :samp:`--unit={U}` can be used to specify the time -unit for the timer output. Supported options are ``usec``, ``msec``, -or ``sec``. (Contributed by Julian Gindi in :issue:`18983`.) - -The :func:`~timeit.timeit` function has a new *globals* parameter for -specifying the namespace in which the code will be running. -(Contributed by Ben Roberts in :issue:`2527`.) - - -tkinter -------- - -The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment -on Windows has been replaced by a private function in the :mod:`_tkinter` -module which makes no permanent changes to environment variables. -(Contributed by Zachary Ware in :issue:`20035`.) - - -.. _whatsnew-traceback: - -traceback ---------- - -New :func:`~traceback.walk_stack` and :func:`~traceback.walk_tb` -functions to conveniently traverse frame and traceback objects. -(Contributed by Robert Collins in :issue:`17911`.) - -New lightweight classes: :class:`~traceback.TracebackException`, -:class:`~traceback.StackSummary`, and :class:`~traceback.FrameSummary`. -(Contributed by Robert Collins in :issue:`17911`.) - -Both the :func:`~traceback.print_tb` and :func:`~traceback.print_stack` functions -now support negative values for the *limit* argument. -(Contributed by Dmitry Kazakov in :issue:`22619`.) - - -types ------ - -A new :func:`~types.coroutine` function to transform -:term:`generator <generator iterator>` and -:class:`generator-like <collections.abc.Generator>` objects into -:term:`awaitables <awaitable>`. -(Contributed by Yury Selivanov in :issue:`24017`.) - -A new type called :class:`~types.CoroutineType`, which is used for -:term:`coroutine` objects created by :keyword:`async def` functions. -(Contributed by Yury Selivanov in :issue:`24400`.) - - -unicodedata ------------ - -The :mod:`unicodedata` module now uses data from `Unicode 8.0.0 -<http://unicode.org/versions/Unicode8.0.0/>`_. - - -unittest --------- - -The :meth:`TestLoader.loadTestsFromModule() <unittest.TestLoader.loadTestsFromModule>` -method now accepts a keyword-only argument *pattern* which is passed to -``load_tests`` as the third argument. Found packages are now checked for -``load_tests`` regardless of whether their path matches *pattern*, because it -is impossible for a package name to match the default pattern. -(Contributed by Robert Collins and Barry A. Warsaw in :issue:`16662`.) - -Unittest discovery errors now are exposed in the -:data:`TestLoader.errors <unittest.TestLoader.errors>` attribute of the -:class:`~unittest.TestLoader` instance. -(Contributed by Robert Collins in :issue:`19746`.) - -A new command line option ``--locals`` to show local variables in -tracebacks. (Contributed by Robert Collins in :issue:`22936`.) - - -unittest.mock -------------- - -The :class:`~unittest.mock.Mock` class has the following improvements: - -* The class constructor has a new *unsafe* parameter, which causes mock - objects to raise :exc:`AttributeError` on attribute names starting - with ``"assert"``. - (Contributed by Kushal Das in :issue:`21238`.) - -* A new :meth:`Mock.assert_not_called() <unittest.mock.Mock.assert_not_called>` - method to check if the mock object was called. - (Contributed by Kushal Das in :issue:`21262`.) - -The :class:`~unittest.mock.MagicMock` class now supports :meth:`__truediv__`, -:meth:`__divmod__` and :meth:`__matmul__` operators. -(Contributed by Johannes Baiter in :issue:`20968`, and Håkan Lövdahl -in :issue:`23581` and :issue:`23568`.) - -It is no longer necessary to explicitly pass ``create=True`` to the -:func:`~unittest.mock.patch` function when patching builtin names. -(Contributed by Kushal Das in :issue:`17660`.) - - -urllib ------- - -A new -:class:`request.HTTPPasswordMgrWithPriorAuth <urllib.request.HTTPPasswordMgrWithPriorAuth>` -class allows HTTP Basic Authentication credentials to be managed so as to -eliminate unnecessary ``401`` response handling, or to unconditionally send -credentials on the first request in order to communicate with servers that -return a ``404`` response instead of a ``401`` if the ``Authorization`` header -is not sent. (Contributed by Matej Cepl in :issue:`19494` and Akshit Khurana in -:issue:`7159`.) - -A new *quote_via* argument for the -:func:`parse.urlencode() <urllib.parse.urlencode>` -function provides a way to control the encoding of query parts if needed. -(Contributed by Samwyse and Arnon Yaari in :issue:`13866`.) - -The :func:`request.urlopen() <urllib.request.urlopen>` function accepts an -:class:`ssl.SSLContext` object as a *context* argument, which will be used for -the HTTPS connection. (Contributed by Alex Gaynor in :issue:`22366`.) - -The :func:`parse.urljoin() <urllib.parse.urljoin>` was updated to use the -:rfc:`3986` semantics for the resolution of relative URLs, rather than -:rfc:`1808` and :rfc:`2396`. -(Contributed by Demian Brecht and Senthil Kumaran in :issue:`22118`.) - - -wsgiref -------- - -The *headers* argument of the :class:`headers.Headers <wsgiref.headers.Headers>` -class constructor is now optional. -(Contributed by Pablo Torres Navarrete and SilentGhost in :issue:`5800`.) - - -xmlrpc ------- - -The :class:`client.ServerProxy <xmlrpc.client.ServerProxy>` class now supports -the :term:`context manager` protocol. -(Contributed by Claudiu Popa in :issue:`20627`.) - -The :class:`client.ServerProxy <xmlrpc.client.ServerProxy>` constructor now accepts -an optional :class:`ssl.SSLContext` instance. -(Contributed by Alex Gaynor in :issue:`22960`.) - - -xml.sax -------- - -SAX parsers now support a character stream of the -:class:`xmlreader.InputSource <xml.sax.xmlreader.InputSource>` object. -(Contributed by Serhiy Storchaka in :issue:`2175`.) - -:func:`~xml.sax.parseString` now accepts a :class:`str` instance. -(Contributed by Serhiy Storchaka in :issue:`10590`.) - - -zipfile -------- - -ZIP output can now be written to unseekable streams. -(Contributed by Serhiy Storchaka in :issue:`23252`.) - -The *mode* argument of :meth:`ZipFile.open() <zipfile.ZipFile.open>` method now -accepts ``"x"`` to request exclusive creation. -(Contributed by Serhiy Storchaka in :issue:`21717`.) - - -Other module-level changes -========================== - -Many functions in the :mod:`mmap`, :mod:`ossaudiodev`, :mod:`socket`, -:mod:`ssl`, and :mod:`codecs` modules now accept writable -:term:`bytes-like objects <bytes-like object>`. -(Contributed by Serhiy Storchaka in :issue:`23001`.) - - -Optimizations -============= - -The :func:`os.walk` function has been sped up by 3 to 5 times on POSIX systems, -and by 7 to 20 times on Windows. This was done using the new :func:`os.scandir` -function, which exposes file information from the underlying ``readdir`` or -``FindFirstFile``/``FindNextFile`` system calls. (Contributed by -Ben Hoyt with help from Victor Stinner in :issue:`23605`.) - -Construction of ``bytes(int)`` (filled by zero bytes) is faster and uses less -memory for large objects. ``calloc()`` is used instead of ``malloc()`` to -allocate memory for these objects. -(Contributed by Victor Stinner in :issue:`21233`.) - -Some operations on :mod:`ipaddress` :class:`~ipaddress.IPv4Network` and -:class:`~ipaddress.IPv6Network` have been massively sped up, such as -:meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`, -:func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`. -The speed up can range from 3 to 15 times. -(Contributed by Antoine Pitrou, Michel Albert, and Markus in -:issue:`21486`, :issue:`21487`, :issue:`20826`, :issue:`23266`.) - -Pickling of :mod:`ipaddress` objects was optimized to produce significantly -smaller output. (Contributed by Serhiy Storchaka in :issue:`23133`.) - -Many operations on :class:`io.BytesIO` are now 50% to 100% faster. -(Contributed by Serhiy Storchaka in :issue:`15381` and David Wilson in -:issue:`22003`.) - -The :func:`marshal.dumps` function is now faster: 65--85% with versions 3 -and 4, 20--25% with versions 0 to 2 on typical data, and up to 5 times in -best cases. -(Contributed by Serhiy Storchaka in :issue:`20416` and :issue:`23344`.) - -The UTF-32 encoder is now 3 to 7 times faster. -(Contributed by Serhiy Storchaka in :issue:`15027`.) - -Regular expressions are now parsed up to 10% faster. -(Contributed by Serhiy Storchaka in :issue:`19380`.) - -The :func:`json.dumps` function was optimized to run with -``ensure_ascii=False`` as fast as with ``ensure_ascii=True``. -(Contributed by Naoki Inada in :issue:`23206`.) - -The :c:func:`PyObject_IsInstance` and :c:func:`PyObject_IsSubclass` -functions have been sped up in the common case that the second argument -has :class:`type` as its metaclass. -(Contributed Georg Brandl by in :issue:`22540`.) - -Method caching was slightly improved, yielding up to 5% performance -improvement in some benchmarks. -(Contributed by Antoine Pitrou in :issue:`22847`.) - -Objects from the :mod:`random` module now use 50% less memory on 64-bit -builds. (Contributed by Serhiy Storchaka in :issue:`23488`.) - -The :func:`property` getter calls are up to 25% faster. -(Contributed by Joe Jevnik in :issue:`23910`.) - -Instantiation of :class:`fractions.Fraction` is now up to 30% faster. -(Contributed by Stefan Behnel in :issue:`22464`.) - -String methods :meth:`~str.find`, :meth:`~str.rfind`, :meth:`~str.split`, -:meth:`~str.partition` and the :keyword:`in` string operator are now significantly -faster for searching 1-character substrings. -(Contributed by Serhiy Storchaka in :issue:`23573`.) - - -Build and C API Changes -======================= - -New ``calloc`` functions were added: - -* :c:func:`PyMem_RawCalloc`, -* :c:func:`PyMem_Calloc`, -* :c:func:`PyObject_Calloc`. - -(Contributed by Victor Stinner in :issue:`21233`.) - -New encoding/decoding helper functions: - -* :c:func:`Py_DecodeLocale` (replaced ``_Py_char2wchar()``), -* :c:func:`Py_EncodeLocale` (replaced ``_Py_wchar2char()``). - -(Contributed by Victor Stinner in :issue:`18395`.) - -A new :c:func:`PyCodec_NameReplaceErrors` function to replace the unicode -encode error with ``\N{...}`` escapes. -(Contributed by Serhiy Storchaka in :issue:`19676`.) - -A new :c:func:`PyErr_FormatV` function similar to :c:func:`PyErr_Format`, -but accepts a ``va_list`` argument. -(Contributed by Antoine Pitrou in :issue:`18711`.) - -A new :c:data:`PyExc_RecursionError` exception. -(Contributed by Georg Brandl in :issue:`19235`.) - -New :c:func:`PyModule_FromDefAndSpec`, :c:func:`PyModule_FromDefAndSpec2`, -and :c:func:`PyModule_ExecDef` functions introduced by :pep:`489` -- -multi-phase extension module initialization. -(Contributed by Petr Viktorin in :issue:`24268`.) - -New :c:func:`PyNumber_MatrixMultiply` and -:c:func:`PyNumber_InPlaceMatrixMultiply` functions to perform matrix -multiplication. -(Contributed by Benjamin Peterson in :issue:`21176`. See also :pep:`465` -for details.) - -The :c:member:`PyTypeObject.tp_finalize` slot is now part of the stable ABI. - -Windows builds now require Microsoft Visual C++ 14.0, which -is available as part of `Visual Studio 2015 <https://www.visualstudio.com/>`_. - -Extension modules now include a platform information tag in their filename on -some platforms (the tag is optional, and CPython will import extensions without -it, although if the tag is present and mismatched, the extension won't be -loaded): - -* On Linux, extension module filenames end with - ``.cpython-<major><minor>m-<architecture>-<os>.pyd``: - - * ``<major>`` is the major number of the Python version; - for Python 3.5 this is ``3``. - - * ``<minor>`` is the minor number of the Python version; - for Python 3.5 this is ``5``. - - * ``<architecture>`` is the hardware architecture the extension module - was built to run on. It's most commonly either ``i386`` for 32-bit Intel - platforms or ``x86_64`` for 64-bit Intel (and AMD) platforms. - - * ``<os>`` is always ``linux-gnu``, except for extensions built to - talk to the 32-bit ABI on 64-bit platforms, in which case it is - ``linux-gnu32`` (and ``<architecture>`` will be ``x86_64``). - -* On Windows, extension module filenames end with - ``<debug>.cp<major><minor>-<platform>.pyd``: - - * ``<major>`` is the major number of the Python version; - for Python 3.5 this is ``3``. - - * ``<minor>`` is the minor number of the Python version; - for Python 3.5 this is ``5``. - - * ``<platform>`` is the platform the extension module was built for, - either ``win32`` for Win32, ``win_amd64`` for Win64, ``win_ia64`` for - Windows Itanium 64, and ``win_arm`` for Windows on ARM. - - * If built in debug mode, ``<debug>`` will be ``_d``, - otherwise it will be blank. - -* On OS X platforms, extension module filenames now end with ``-darwin.so``. - -* On all other platforms, extension module filenames are the same as they were - with Python 3.4. - - -Deprecated -========== - -New Keywords ------------- - -``async`` and ``await`` are not recommended to be used as variable, class, -function or module names. Introduced by :pep:`492` in Python 3.5, they will -become proper keywords in Python 3.7. - - -Deprecated Python Behavior --------------------------- - -Raising the :exc:`StopIteration` exception inside a generator will now generate a silent -:exc:`PendingDeprecationWarning`, which will become a non-silent deprecation -warning in Python 3.6 and will trigger a :exc:`RuntimeError` in Python 3.7. -See :ref:`PEP 479: Change StopIteration handling inside generators <whatsnew-pep-479>` -for details. - - -Unsupported Operating Systems ------------------------------ - -Windows XP is no longer supported by Microsoft, thus, per :PEP:`11`, CPython -3.5 is no longer officially supported on this OS. - - -Deprecated Python modules, functions and methods ------------------------------------------------- - -The :mod:`formatter` module has now graduated to full deprecation and is still -slated for removal in Python 3.6. - -The :func:`asyncio.async` function is deprecated in favor of -:func:`~asyncio.ensure_future`. - -The :mod:`smtpd` module has in the past always decoded the DATA portion of -email messages using the ``utf-8`` codec. This can now be controlled by the -new *decode_data* keyword to :class:`~smtpd.SMTPServer`. The default value is -``True``, but this default is deprecated. Specify the *decode_data* keyword -with an appropriate value to avoid the deprecation warning. - -Directly assigning values to the :attr:`~http.cookies.Morsel.key`, -:attr:`~http.cookies.Morsel.value` and -:attr:`~http.cookies.Morsel.coded_value` of :class:`http.cookies.Morsel` -objects is deprecated. Use the :meth:`~http.cookies.Morsel.set` method -instead. In addition, the undocumented *LegalChars* parameter of -:meth:`~http.cookies.Morsel.set` is deprecated, and is now ignored. - -Passing a format string as keyword argument *format_string* to the -:meth:`~string.Formatter.format` method of the :class:`string.Formatter` -class has been deprecated. -(Contributed by Serhiy Storchaka in :issue:`23671`.) - -The :func:`platform.dist` and :func:`platform.linux_distribution` functions -are now deprecated. Linux distributions use too many different ways of -describing themselves, so the functionality is left to a package. -(Contributed by Vajrasky Kok and Berker Peksag in :issue:`1322`.) - -The previously undocumented ``from_function`` and ``from_builtin`` methods of -:class:`inspect.Signature` are deprecated. Use the new -:meth:`Signature.from_callable() <inspect.Signature.from_callable>` -method instead. (Contributed by Yury Selivanov in :issue:`24248`.) - -The :func:`inspect.getargspec` function is deprecated and scheduled to be -removed in Python 3.6. (See :issue:`20438` for details.) - -The :mod:`inspect` :func:`~inspect.getfullargspec`, -:func:`~inspect.getcallargs`, and :func:`~inspect.formatargspec` functions are -deprecated in favor of the :func:`inspect.signature` API. (Contributed by Yury -Selivanov in :issue:`20438`.) - -:func:`~inspect.getargvalues` and :func:`~inspect.formatargvalues` functions -were inadvertently marked as deprecated with the release of Python 3.5.0. - -Use of :const:`re.LOCALE` flag with str patterns or :const:`re.ASCII` is now -deprecated. (Contributed by Serhiy Storchaka in :issue:`22407`.) - -Use of unrecognized special sequences consisting of ``'\'`` and an ASCII letter -in regular expression patterns and replacement patterns now raises a -deprecation warning and will be forbidden in Python 3.6. -(Contributed by Serhiy Storchaka in :issue:`23622`.) - -The undocumented and unofficial *use_load_tests* default argument of the -:meth:`unittest.TestLoader.loadTestsFromModule` method now is -deprecated and ignored. -(Contributed by Robert Collins and Barry A. Warsaw in :issue:`16662`.) - - -Removed -======= - -API and Feature Removals ------------------------- - -The following obsolete and previously deprecated APIs and features have been -removed: - -* The ``__version__`` attribute has been dropped from the email package. The - email code hasn't been shipped separately from the stdlib for a long time, - and the ``__version__`` string was not updated in the last few releases. - -* The internal ``Netrc`` class in the :mod:`ftplib` module was deprecated in - 3.4, and has now been removed. - (Contributed by Matt Chaput in :issue:`6623`.) - -* The concept of ``.pyo`` files has been removed. - -* The JoinableQueue class in the provisional :mod:`asyncio` module was - deprecated in 3.4.4 and is now removed. - (Contributed by A. Jesse Jiryu Davis in :issue:`23464`.) - - -Porting to Python 3.5 -===================== - -This section lists previously described changes and other bugfixes -that may require changes to your code. - - -Changes in Python behavior --------------------------- - -* Due to an oversight, earlier Python versions erroneously accepted the - following syntax:: - - f(1 for x in [1], *args) - f(1 for x in [1], **kwargs) - - Python 3.5 now correctly raises a :exc:`SyntaxError`, as generator - expressions must be put in parentheses if not a sole argument to a function. - - -Changes in the Python API -------------------------- - -* :pep:`475`: System calls are now retried when interrupted by a signal instead - of raising :exc:`InterruptedError` if the Python signal handler does not - raise an exception. - -* Before Python 3.5, a :class:`datetime.time` object was considered to be false - if it represented midnight in UTC. This behavior was considered obscure and - error-prone and has been removed in Python 3.5. See :issue:`13936` for full - details. - -* The :meth:`ssl.SSLSocket.send()` method now raises either - :exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError` - on a non-blocking socket if the operation would block. Previously, - it would return ``0``. (Contributed by Nikolaus Rath in :issue:`20951`.) - -* The ``__name__`` attribute of generators is now set from the function name, - instead of being set from the code name. Use ``gen.gi_code.co_name`` to - retrieve the code name. Generators also have a new ``__qualname__`` - attribute, the qualified name, which is now used for the representation - of a generator (``repr(gen)``). - (Contributed by Victor Stinner in :issue:`21205`.) - -* The deprecated "strict" mode and argument of :class:`~html.parser.HTMLParser`, - :meth:`HTMLParser.error`, and the :exc:`HTMLParserError` exception have been - removed. (Contributed by Ezio Melotti in :issue:`15114`.) - The *convert_charrefs* argument of :class:`~html.parser.HTMLParser` is - now ``True`` by default. (Contributed by Berker Peksag in :issue:`21047`.) - -* Although it is not formally part of the API, it is worth noting for porting - purposes (ie: fixing tests) that error messages that were previously of the - form "'sometype' does not support the buffer protocol" are now of the form "a - :term:`bytes-like object` is required, not 'sometype'". - (Contributed by Ezio Melotti in :issue:`16518`.) - -* If the current directory is set to a directory that no longer exists then - :exc:`FileNotFoundError` will no longer be raised and instead - :meth:`~importlib.machinery.FileFinder.find_spec` will return ``None`` - **without** caching ``None`` in :data:`sys.path_importer_cache`, which is - different than the typical case (:issue:`22834`). - -* HTTP status code and messages from :mod:`http.client` and :mod:`http.server` - were refactored into a common :class:`~http.HTTPStatus` enum. The values in - :mod:`http.client` and :mod:`http.server` remain available for backwards - compatibility. (Contributed by Demian Brecht in :issue:`21793`.) - -* When an import loader defines :meth:`importlib.machinery.Loader.exec_module` - it is now expected to also define - :meth:`~importlib.machinery.Loader.create_module` (raises a - :exc:`DeprecationWarning` now, will be an error in Python 3.6). If the loader - inherits from :class:`importlib.abc.Loader` then there is nothing to do, else - simply define :meth:`~importlib.machinery.Loader.create_module` to return - ``None``. (Contributed by Brett Cannon in :issue:`23014`.) - -* The :func:`re.split` function always ignored empty pattern matches, so the - ``"x*"`` pattern worked the same as ``"x+"``, and the ``"\b"`` pattern never - worked. Now :func:`re.split` raises a warning if the pattern could match - an empty string. For compatibility, use patterns that never match an empty - string (e.g. ``"x+"`` instead of ``"x*"``). Patterns that could only match - an empty string (such as ``"\b"``) now raise an error. - (Contributed by Serhiy Storchaka in :issue:`22818`.) - -* The :class:`http.cookies.Morsel` dict-like interface has been made self - consistent: morsel comparison now takes the :attr:`~http.cookies.Morsel.key` - and :attr:`~http.cookies.Morsel.value` into account, - :meth:`~http.cookies.Morsel.copy` now results in a - :class:`~http.cookies.Morsel` instance rather than a :class:`dict`, and - :meth:`~http.cookies.Morsel.update` will now raise an exception if any of the - keys in the update dictionary are invalid. In addition, the undocumented - *LegalChars* parameter of :func:`~http.cookies.Morsel.set` is deprecated and - is now ignored. (Contributed by Demian Brecht in :issue:`2211`.) - -* :pep:`488` has removed ``.pyo`` files from Python and introduced the optional - ``opt-`` tag in ``.pyc`` file names. The - :func:`importlib.util.cache_from_source` has gained an *optimization* - parameter to help control the ``opt-`` tag. Because of this, the - *debug_override* parameter of the function is now deprecated. `.pyo` files - are also no longer supported as a file argument to the Python interpreter and - thus serve no purpose when distributed on their own (i.e. sourceless code - distribution). Due to the fact that the magic number for bytecode has changed - in Python 3.5, all old `.pyo` files from previous versions of Python are - invalid regardless of this PEP. - -* The :mod:`socket` module now exports the :data:`~socket.CAN_RAW_FD_FRAMES` - constant on linux 3.6 and greater. - -* The :func:`ssl.cert_time_to_seconds` function now interprets the input time - as UTC and not as local time, per :rfc:`5280`. Additionally, the return - value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.) - -* The ``pygettext.py`` Tool now uses the standard +NNNN format for timezones in - the POT-Creation-Date header. - -* The :mod:`smtplib` module now uses :data:`sys.stderr` instead of the previous - module-level :data:`stderr` variable for debug output. If your (test) - program depends on patching the module-level variable to capture the debug - output, you will need to update it to capture sys.stderr instead. - -* The :meth:`str.startswith` and :meth:`str.endswith` methods no longer return - ``True`` when finding the empty string and the indexes are completely out of - range. (Contributed by Serhiy Storchaka in :issue:`24284`.) - -* The :func:`inspect.getdoc` function now returns documentation strings - inherited from base classes. Documentation strings no longer need to be - duplicated if the inherited documentation is appropriate. To suppress an - inherited string, an empty string must be specified (or the documentation - may be filled in). This change affects the output of the :mod:`pydoc` - module and the :func:`help` function. - (Contributed by Serhiy Storchaka in :issue:`15582`.) - -* Nested :func:`functools.partial` calls are now flattened. If you were - relying on the previous behavior, you can now either add an attribute to a - :func:`functools.partial` object or you can create a subclass of - :func:`functools.partial`. - (Contributed by Alexander Belopolsky in :issue:`7830`.) - -Changes in the C API --------------------- - -* The undocumented :c:member:`~PyMemoryViewObject.format` member of the - (non-public) :c:type:`PyMemoryViewObject` structure has been removed. - All extensions relying on the relevant parts in ``memoryobject.h`` - must be rebuilt. - -* The :c:type:`PyMemAllocator` structure was renamed to - :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. - -* Removed non-documented macro :c:macro:`PyObject_REPR` which leaked references. - Use format character ``%R`` in :c:func:`PyUnicode_FromFormat`-like functions - to format the :func:`repr` of the object. - (Contributed by Serhiy Storchaka in :issue:`22453`.) - -* Because the lack of the :attr:`__module__` attribute breaks pickling and - introspection, a deprecation warning is now raised for builtin types without - the :attr:`__module__` attribute. This would be an AttributeError in - the future. - (Contributed by Serhiy Storchaka in :issue:`20204`.) - -* As part of the :pep:`492` implementation, the ``tp_reserved`` slot of - :c:type:`PyTypeObject` was replaced with a - :c:member:`tp_as_async` slot. Refer to :ref:`coro-objects` for - new types, structures and functions. - - -Notable changes in Python 3.5.4 -=============================== - -New ``make regen-all`` build target ------------------------------------ - -To simplify cross-compilation, and to ensure that CPython can reliably be -compiled without requiring an existing version of Python to already be -available, the autotools-based build system no longer attempts to implicitly -recompile generated files based on file modification times. - -Instead, a new ``make regen-all`` command has been added to force regeneration -of these files when desired (e.g. after an initial version of Python has -already been built based on the pregenerated versions). - -More selective regeneration targets are also defined - see -:source:`Makefile.pre.in` for details. - -(Contributed by Victor Stinner in :issue:`23404`.) - -.. versionadded:: 3.5.4 - - -Removal of ``make touch`` build target --------------------------------------- - -The ``make touch`` build target previously used to request implicit regeneration -of generated files by updating their modification times has been removed. - -It has been replaced by the new ``make regen-all`` target. - -(Contributed by Victor Stinner in :issue:`23404`.) - -.. versionchanged:: 3.5.4 - diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst deleted file mode 100644 index 3f5f520..0000000 --- a/Doc/whatsnew/3.6.rst +++ /dev/null @@ -1,2435 +0,0 @@ -**************************** - What's New In Python 3.6 -**************************** - -:Editors: Elvis Pranskevichus <elvis@magic.io>, Yury Selivanov <yury@magic.io> - -.. Rules for maintenance: - - * Anyone can add text to this document. Do not spend very much time - on the wording of your changes, because your text will probably - get rewritten to some degree. - - * The maintainer will go through Misc/NEWS periodically and add - changes; it's therefore more important to add your changes to - Misc/NEWS than to this file. - - * This is not a complete list of every single change; completeness - is the purpose of Misc/NEWS. Some changes I consider too small - or esoteric to include. If such a change is added to the text, - I'll just remove it. (This is another reason you shouldn't spend - too much time on writing your addition.) - - * If you want to draw your new text to the attention of the - maintainer, add 'XXX' to the beginning of the paragraph or - section. - - * It's OK to just add a fragmentary note about a change. For - example: "XXX Describe the transmogrify() function added to the - socket module." The maintainer will research the change and - write the necessary text. - - * You can comment out your additions if you like, but it's not - necessary (especially when a final release is some months away). - - * Credit the author of a patch or bugfix. Just the name is - sufficient; the e-mail address isn't necessary. - - * It's helpful to add the bug/patch number as a comment: - - XXX Describe the transmogrify() function added to the socket - module. - (Contributed by P.Y. Developer in :issue:`12345`.) - - This saves the maintainer the effort of going through the Mercurial log - when researching a change. - -This article explains the new features in Python 3.6, compared to 3.5. -Python 3.6 was released on December 23, 2016. See the -`changelog <https://docs.python.org/3.6/whatsnew/changelog.html>`_ for a full -list of changes. - -.. seealso:: - - :pep:`494` - Python 3.6 Release Schedule - - -Summary -- Release highlights -============================= - -New syntax features: - -* :ref:`PEP 498 <whatsnew36-pep498>`, formatted string literals. - -* :ref:`PEP 515 <whatsnew36-pep515>`, underscores in numeric literals. - -* :ref:`PEP 526 <whatsnew36-pep526>`, syntax for variable annotations. - -* :ref:`PEP 525 <whatsnew36-pep525>`, asynchronous generators. - -* :ref:`PEP 530 <whatsnew36-pep530>`: asynchronous comprehensions. - - -New library modules: - -* :mod:`secrets`: :ref:`PEP 506 -- Adding A Secrets Module To The Standard Library <whatsnew36-pep506>`. - - -CPython implementation improvements: - -* The :ref:`dict <typesmapping>` type has been reimplemented to use - a :ref:`more compact representation <whatsnew36-compactdict>` - based on `a proposal by Raymond Hettinger - <https://mail.python.org/pipermail/python-dev/2012-December/123028.html>`_ - and similar to the `PyPy dict implementation`_. This resulted in dictionaries - using 20% to 25% less memory when compared to Python 3.5. - -* Customization of class creation has been simplified with the - :ref:`new protocol <whatsnew36-pep487>`. - -* The class attribute definition order is - :ref:`now preserved <whatsnew36-pep520>`. - -* The order of elements in ``**kwargs`` now - :ref:`corresponds to the order <whatsnew36-pep468>` in which keyword - arguments were passed to the function. - -* DTrace and SystemTap :ref:`probing support <whatsnew36-tracing>` has - been added. - -* The new :ref:`PYTHONMALLOC <whatsnew36-pythonmalloc>` environment variable - can now be used to debug the interpreter memory allocation and access - errors. - - -Significant improvements in the standard library: - -* The :mod:`asyncio` module has received new features, significant - usability and performance improvements, and a fair amount of bug fixes. - Starting with Python 3.6 the ``asyncio`` module is no longer provisional - and its API is considered stable. - -* A new :ref:`file system path protocol <whatsnew36-pep519>` has been - implemented to support :term:`path-like objects <path-like object>`. - All standard library functions operating on paths have been updated to - work with the new protocol. - -* The :mod:`datetime` module has gained support for - :ref:`Local Time Disambiguation <whatsnew36-pep495>`. - -* The :mod:`typing` module received a number of - :ref:`improvements <whatsnew36-typing>`. - -* The :mod:`tracemalloc` module has been significantly reworked - and is now used to provide better output for :exc:`ResourceWarning` - as well as provide better diagnostics for memory allocation errors. - See the :ref:`PYTHONMALLOC section <whatsnew36-pythonmalloc>` for more - information. - - -Security improvements: - -* The new :mod:`secrets` module has been added to simplify the generation of - cryptographically strong pseudo-random numbers suitable for - managing secrets such as account authentication, tokens, and similar. - -* On Linux, :func:`os.urandom` now blocks until the system urandom entropy - pool is initialized to increase the security. See the :pep:`524` for the - rationale. - -* The :mod:`hashlib` and :mod:`ssl` modules now support OpenSSL 1.1.0. - -* The default settings and feature set of the :mod:`ssl` module have been - improved. - -* The :mod:`hashlib` module received support for the BLAKE2, SHA-3 and SHAKE - hash algorithms and the :func:`~hashlib.scrypt` key derivation function. - - -Windows improvements: - -* :ref:`PEP 528 <whatsnew36-pep528>` and :ref:`PEP 529 <whatsnew36-pep529>`, - Windows filesystem and console encoding changed to UTF-8. - -* The ``py.exe`` launcher, when used interactively, no longer prefers - Python 2 over Python 3 when the user doesn't specify a version (via - command line arguments or a config file). Handling of shebang lines - remains unchanged - "python" refers to Python 2 in that case. - -* ``python.exe`` and ``pythonw.exe`` have been marked as long-path aware, - which means that the 260 character path limit may no longer apply. - See :ref:`removing the MAX_PATH limitation <max-path>` for details. - -* A ``._pth`` file can be added to force isolated mode and fully specify - all search paths to avoid registry and environment lookup. See - :ref:`the documentation <finding_modules>` for more information. - -* A ``python36.zip`` file now works as a landmark to infer - :envvar:`PYTHONHOME`. See :ref:`the documentation <finding_modules>` for - more information. - - -.. _PyPy dict implementation: https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html - - -New Features -============ - -.. _whatsnew36-pep498: - -PEP 498: Formatted string literals ----------------------------------- - -:pep:`498` introduces a new kind of string literals: *f-strings*, or -:ref:`formatted string literals <f-strings>`. - -Formatted string literals are prefixed with ``'f'`` and are similar to -the format strings accepted by :meth:`str.format`. They contain replacement -fields surrounded by curly braces. The replacement fields are expressions, -which are evaluated at run time, and then formatted using the -:func:`format` protocol:: - - >>> name = "Fred" - >>> f"He said his name is {name}." - 'He said his name is Fred.' - >>> width = 10 - >>> precision = 4 - >>> value = decimal.Decimal("12.34567") - >>> f"result: {value:{width}.{precision}}" # nested fields - 'result: 12.35' - -.. seealso:: - - :pep:`498` -- Literal String Interpolation. - PEP written and implemented by Eric V. Smith. - - :ref:`Feature documentation <f-strings>`. - - -.. _whatsnew36-pep526: - -PEP 526: Syntax for variable annotations ----------------------------------------- - -:pep:`484` introduced the standard for type annotations of function -parameters, a.k.a. type hints. This PEP adds syntax to Python for annotating -the types of variables including class variables and instance variables:: - - primes: List[int] = [] - - captain: str # Note: no initial value! - - class Starship: - stats: Dict[str, int] = {} - -Just as for function annotations, the Python interpreter does not attach any -particular meaning to variable annotations and only stores them in the -``__annotations__`` attribute of a class or module. - -In contrast to variable declarations in statically typed languages, -the goal of annotation syntax is to provide an easy way to specify structured -type metadata for third party tools and libraries via the abstract syntax tree -and the ``__annotations__`` attribute. - -.. seealso:: - - :pep:`526` -- Syntax for variable annotations. - PEP written by Ryan Gonzalez, Philip House, Ivan Levkivskyi, Lisa Roach, - and Guido van Rossum. Implemented by Ivan Levkivskyi. - - Tools that use or will use the new syntax: - `mypy <http://www.mypy-lang.org/>`_, - `pytype <https://github.com/google/pytype>`_, PyCharm, etc. - - -.. _whatsnew36-pep515: - -PEP 515: Underscores in Numeric Literals ----------------------------------------- - -:pep:`515` adds the ability to use underscores in numeric literals for -improved readability. For example:: - - >>> 1_000_000_000_000_000 - 1000000000000000 - >>> 0x_FF_FF_FF_FF - 4294967295 - -Single underscores are allowed between digits and after any base -specifier. Leading, trailing, or multiple underscores in a row are not -allowed. - -The :ref:`string formatting <formatspec>` language also now has support -for the ``'_'`` option to signal the use of an underscore for a thousands -separator for floating point presentation types and for integer -presentation type ``'d'``. For integer presentation types ``'b'``, -``'o'``, ``'x'``, and ``'X'``, underscores will be inserted every 4 -digits:: - - >>> '{:_}'.format(1000000) - '1_000_000' - >>> '{:_x}'.format(0xFFFFFFFF) - 'ffff_ffff' - -.. seealso:: - - :pep:`515` -- Underscores in Numeric Literals - PEP written by Georg Brandl and Serhiy Storchaka. - - -.. _whatsnew36-pep525: - -PEP 525: Asynchronous Generators --------------------------------- - -:pep:`492` introduced support for native coroutines and ``async`` / ``await`` -syntax to Python 3.5. A notable limitation of the Python 3.5 implementation -is that it was not possible to use ``await`` and ``yield`` in the same -function body. In Python 3.6 this restriction has been lifted, making it -possible to define *asynchronous generators*:: - - async def ticker(delay, to): - """Yield numbers from 0 to *to* every *delay* seconds.""" - for i in range(to): - yield i - await asyncio.sleep(delay) - -The new syntax allows for faster and more concise code. - -.. seealso:: - - :pep:`525` -- Asynchronous Generators - PEP written and implemented by Yury Selivanov. - - -.. _whatsnew36-pep530: - -PEP 530: Asynchronous Comprehensions ------------------------------------- - -:pep:`530` adds support for using ``async for`` in list, set, dict -comprehensions and generator expressions:: - - result = [i async for i in aiter() if i % 2] - -Additionally, ``await`` expressions are supported in all kinds -of comprehensions:: - - result = [await fun() for fun in funcs if await condition()] - -.. seealso:: - - :pep:`530` -- Asynchronous Comprehensions - PEP written and implemented by Yury Selivanov. - - -.. _whatsnew36-pep487: - -PEP 487: Simpler customization of class creation ------------------------------------------------- - -It is now possible to customize subclass creation without using a metaclass. -The new ``__init_subclass__`` classmethod will be called on the base class -whenever a new subclass is created:: - - class PluginBase: - subclasses = [] - - def __init_subclass__(cls, **kwargs): - super().__init_subclass__(**kwargs) - cls.subclasses.append(cls) - - class Plugin1(PluginBase): - pass - - class Plugin2(PluginBase): - pass - -In order to allow zero-argument :func:`super` calls to work correctly from -:meth:`~object.__init_subclass__` implementations, custom metaclasses must -ensure that the new ``__classcell__`` namespace entry is propagated to -``type.__new__`` (as described in :ref:`class-object-creation`). - -.. seealso:: - - :pep:`487` -- Simpler customization of class creation - PEP written and implemented by Martin Teichmann. - - :ref:`Feature documentation <class-customization>` - - -.. _whatsnew36-pep487-descriptors: - -PEP 487: Descriptor Protocol Enhancements ------------------------------------------ - -:pep:`487` extends the descriptor protocol to include the new optional -:meth:`~object.__set_name__` method. Whenever a new class is defined, the new -method will be called on all descriptors included in the definition, providing -them with a reference to the class being defined and the name given to the -descriptor within the class namespace. In other words, instances of -descriptors can now know the attribute name of the descriptor in the -owner class:: - - class IntField: - def __get__(self, instance, owner): - return instance.__dict__[self.name] - - def __set__(self, instance, value): - if not isinstance(value, int): - raise ValueError(f'expecting integer in {self.name}') - instance.__dict__[self.name] = value - - # this is the new initializer: - def __set_name__(self, owner, name): - self.name = name - - class Model: - int_field = IntField() - - -.. seealso:: - - :pep:`487` -- Simpler customization of class creation - PEP written and implemented by Martin Teichmann. - - :ref:`Feature documentation <descriptors>` - - -.. _whatsnew36-pep519: - -PEP 519: Adding a file system path protocol -------------------------------------------- - -File system paths have historically been represented as :class:`str` -or :class:`bytes` objects. This has led to people who write code which -operate on file system paths to assume that such objects are only one -of those two types (an :class:`int` representing a file descriptor -does not count as that is not a file path). Unfortunately that -assumption prevents alternative object representations of file system -paths like :mod:`pathlib` from working with pre-existing code, -including Python's standard library. - -To fix this situation, a new interface represented by -:class:`os.PathLike` has been defined. By implementing the -:meth:`~os.PathLike.__fspath__` method, an object signals that it -represents a path. An object can then provide a low-level -representation of a file system path as a :class:`str` or -:class:`bytes` object. This means an object is considered -:term:`path-like <path-like object>` if it implements -:class:`os.PathLike` or is a :class:`str` or :class:`bytes` object -which represents a file system path. Code can use :func:`os.fspath`, -:func:`os.fsdecode`, or :func:`os.fsencode` to explicitly get a -:class:`str` and/or :class:`bytes` representation of a path-like -object. - -The built-in :func:`open` function has been updated to accept -:class:`os.PathLike` objects, as have all relevant functions in the -:mod:`os` and :mod:`os.path` modules, and most other functions and -classes in the standard library. The :class:`os.DirEntry` class -and relevant classes in :mod:`pathlib` have also been updated to -implement :class:`os.PathLike`. - -The hope is that updating the fundamental functions for operating -on file system paths will lead to third-party code to implicitly -support all :term:`path-like objects <path-like object>` without any -code changes, or at least very minimal ones (e.g. calling -:func:`os.fspath` at the beginning of code before operating on a -path-like object). - -Here are some examples of how the new interface allows for -:class:`pathlib.Path` to be used more easily and transparently with -pre-existing code:: - - >>> import pathlib - >>> with open(pathlib.Path("README")) as f: - ... contents = f.read() - ... - >>> import os.path - >>> os.path.splitext(pathlib.Path("some_file.txt")) - ('some_file', '.txt') - >>> os.path.join("/a/b", pathlib.Path("c")) - '/a/b/c' - >>> import os - >>> os.fspath(pathlib.Path("some_file.txt")) - 'some_file.txt' - -(Implemented by Brett Cannon, Ethan Furman, Dusty Phillips, and Jelle Zijlstra.) - -.. seealso:: - - :pep:`519` -- Adding a file system path protocol - PEP written by Brett Cannon and Koos Zevenhoven. - - -.. _whatsnew36-pep495: - -PEP 495: Local Time Disambiguation ----------------------------------- - -In most world locations, there have been and will be times when local clocks -are moved back. In those times, intervals are introduced in which local -clocks show the same time twice in the same day. In these situations, the -information displayed on a local clock (or stored in a Python datetime -instance) is insufficient to identify a particular moment in time. - -:pep:`495` adds the new *fold* attribute to instances of -:class:`datetime.datetime` and :class:`datetime.time` classes to differentiate -between two moments in time for which local times are the same:: - - >>> u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc) - >>> for i in range(4): - ... u = u0 + i*HOUR - ... t = u.astimezone(Eastern) - ... print(u.time(), 'UTC =', t.time(), t.tzname(), t.fold) - ... - 04:00:00 UTC = 00:00:00 EDT 0 - 05:00:00 UTC = 01:00:00 EDT 0 - 06:00:00 UTC = 01:00:00 EST 1 - 07:00:00 UTC = 02:00:00 EST 0 - -The values of the :attr:`fold <datetime.datetime.fold>` attribute have the -value ``0`` for all instances except those that represent the second -(chronologically) moment in time in an ambiguous case. - -.. seealso:: - - :pep:`495` -- Local Time Disambiguation - PEP written by Alexander Belopolsky and Tim Peters, implementation - by Alexander Belopolsky. - - -.. _whatsnew36-pep529: - -PEP 529: Change Windows filesystem encoding to UTF-8 ----------------------------------------------------- - -Representing filesystem paths is best performed with str (Unicode) rather than -bytes. However, there are some situations where using bytes is sufficient and -correct. - -Prior to Python 3.6, data loss could result when using bytes paths on Windows. -With this change, using bytes to represent paths is now supported on Windows, -provided those bytes are encoded with the encoding returned by -:func:`sys.getfilesystemencoding()`, which now defaults to ``'utf-8'``. - -Applications that do not use str to represent paths should use -:func:`os.fsencode()` and :func:`os.fsdecode()` to ensure their bytes are -correctly encoded. To revert to the previous behaviour, set -:envvar:`PYTHONLEGACYWINDOWSFSENCODING` or call -:func:`sys._enablelegacywindowsfsencoding`. - -See :pep:`529` for more information and discussion of code modifications that -may be required. - - -.. _whatsnew36-pep528: - -PEP 528: Change Windows console encoding to UTF-8 -------------------------------------------------- - -The default console on Windows will now accept all Unicode characters and -provide correctly read str objects to Python code. ``sys.stdin``, -``sys.stdout`` and ``sys.stderr`` now default to utf-8 encoding. - -This change only applies when using an interactive console, and not when -redirecting files or pipes. To revert to the previous behaviour for interactive -console use, set :envvar:`PYTHONLEGACYWINDOWSSTDIO`. - -.. seealso:: - - :pep:`528` -- Change Windows console encoding to UTF-8 - PEP written and implemented by Steve Dower. - - -.. _whatsnew36-pep520: - -PEP 520: Preserving Class Attribute Definition Order ----------------------------------------------------- - -Attributes in a class definition body have a natural ordering: the same -order in which the names appear in the source. This order is now -preserved in the new class's :attr:`~object.__dict__` attribute. - -Also, the effective default class *execution* namespace (returned from -:ref:`type.__prepare__() <prepare>`) is now an insertion-order-preserving -mapping. - -.. seealso:: - - :pep:`520` -- Preserving Class Attribute Definition Order - PEP written and implemented by Eric Snow. - - -.. _whatsnew36-pep468: - -PEP 468: Preserving Keyword Argument Order ------------------------------------------- - -``**kwargs`` in a function signature is now guaranteed to be an -insertion-order-preserving mapping. - -.. seealso:: - - :pep:`468` -- Preserving Keyword Argument Order - PEP written and implemented by Eric Snow. - - -.. _whatsnew36-compactdict: - -New :ref:`dict <typesmapping>` implementation ---------------------------------------------- - -The :ref:`dict <typesmapping>` type now uses a "compact" representation -based on `a proposal by Raymond Hettinger -<https://mail.python.org/pipermail/python-dev/2012-December/123028.html>`_ -which was `first implemented by PyPy -<https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html>`_. -The memory usage of the new :func:`dict` is between 20% and 25% smaller -compared to Python 3.5. - -The order-preserving aspect of this new implementation is considered an -implementation detail and should not be relied upon (this may change in -the future, but it is desired to have this new dict implementation in -the language for a few releases before changing the language spec to mandate -order-preserving semantics for all current and future Python -implementations; this also helps preserve backwards-compatibility -with older versions of the language where random iteration order is -still in effect, e.g. Python 3.5). - -(Contributed by INADA Naoki in :issue:`27350`. Idea -`originally suggested by Raymond Hettinger -<https://mail.python.org/pipermail/python-dev/2012-December/123028.html>`_.) - - -.. _whatsnew36-pep523: - -PEP 523: Adding a frame evaluation API to CPython -------------------------------------------------- - -While Python provides extensive support to customize how code -executes, one place it has not done so is in the evaluation of frame -objects. If you wanted some way to intercept frame evaluation in -Python there really wasn't any way without directly manipulating -function pointers for defined functions. - -:pep:`523` changes this by providing an API to make frame -evaluation pluggable at the C level. This will allow for tools such -as debuggers and JITs to intercept frame evaluation before the -execution of Python code begins. This enables the use of alternative -evaluation implementations for Python code, tracking frame -evaluation, etc. - -This API is not part of the limited C API and is marked as private to -signal that usage of this API is expected to be limited and only -applicable to very select, low-level use-cases. Semantics of the -API will change with Python as necessary. - -.. seealso:: - - :pep:`523` -- Adding a frame evaluation API to CPython - PEP written by Brett Cannon and Dino Viehland. - - -.. _whatsnew36-pythonmalloc: - -PYTHONMALLOC environment variable ---------------------------------- - -The new :envvar:`PYTHONMALLOC` environment variable allows setting the Python -memory allocators and installing debug hooks. - -It is now possible to install debug hooks on Python memory allocators on Python -compiled in release mode using ``PYTHONMALLOC=debug``. Effects of debug hooks: - -* Newly allocated memory is filled with the byte ``0xCB`` -* Freed memory is filled with the byte ``0xDB`` -* Detect violations of the Python memory allocator API. For example, - :c:func:`PyObject_Free` called on a memory block allocated by - :c:func:`PyMem_Malloc`. -* Detect writes before the start of a buffer (buffer underflows) -* Detect writes after the end of a buffer (buffer overflows) -* Check that the :term:`GIL <global interpreter lock>` is held when allocator - functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex: :c:func:`PyObject_Malloc`) and - :c:data:`PYMEM_DOMAIN_MEM` (ex: :c:func:`PyMem_Malloc`) domains are called. - -Checking if the GIL is held is also a new feature of Python 3.6. - -See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python -memory allocators. - -It is now also possible to force the usage of the :c:func:`malloc` allocator of -the C library for all Python memory allocations using ``PYTHONMALLOC=malloc``. -This is helpful when using external memory debuggers like Valgrind on -a Python compiled in release mode. - -On error, the debug hooks on Python memory allocators now use the -:mod:`tracemalloc` module to get the traceback where a memory block was -allocated. - -Example of fatal error on buffer overflow using -``python3.6 -X tracemalloc=5`` (store 5 frames in traces):: - - Debug memory block at address p=0x7fbcd41666f8: API 'o' - 4 bytes originally requested - The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. - The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb): - at tail+0: 0x02 *** OUCH - at tail+1: 0xfb - at tail+2: 0xfb - at tail+3: 0xfb - at tail+4: 0xfb - at tail+5: 0xfb - at tail+6: 0xfb - at tail+7: 0xfb - The block was made by call #1233329 to debug malloc/realloc. - Data at p: 1a 2b 30 00 - - Memory block allocated at (most recent call first): - File "test/test_bytes.py", line 323 - File "unittest/case.py", line 600 - File "unittest/case.py", line 648 - File "unittest/suite.py", line 122 - File "unittest/suite.py", line 84 - - Fatal Python error: bad trailing pad byte - - Current thread 0x00007fbcdbd32700 (most recent call first): - File "test/test_bytes.py", line 323 in test_hex - File "unittest/case.py", line 600 in run - File "unittest/case.py", line 648 in __call__ - File "unittest/suite.py", line 122 in run - File "unittest/suite.py", line 84 in __call__ - File "unittest/suite.py", line 122 in run - File "unittest/suite.py", line 84 in __call__ - ... - -(Contributed by Victor Stinner in :issue:`26516` and :issue:`26564`.) - - -.. _whatsnew36-tracing: - -DTrace and SystemTap probing support ------------------------------------- - -Python can now be built ``--with-dtrace`` which enables static markers -for the following events in the interpreter: - -* function call/return - -* garbage collection started/finished - -* line of code executed. - -This can be used to instrument running interpreters in production, -without the need to recompile specific debug builds or providing -application-specific profiling/debugging code. - -More details in :ref:`instrumentation`. - -The current implementation is tested on Linux and macOS. Additional -markers may be added in the future. - -(Contributed by Łukasz Langa in :issue:`21590`, based on patches by -Jesús Cea Avión, David Malcolm, and Nikhil Benesch.) - - -Other Language Changes -====================== - -Some smaller changes made to the core Python language are: - -* A ``global`` or ``nonlocal`` statement must now textually appear - before the first use of the affected name in the same scope. - Previously this was a :exc:`SyntaxWarning`. - -* It is now possible to set a :ref:`special method <specialnames>` to - ``None`` to indicate that the corresponding operation is not available. - For example, if a class sets :meth:`__iter__` to ``None``, the class - is not iterable. - (Contributed by Andrew Barnert and Ivan Levkivskyi in :issue:`25958`.) - -* Long sequences of repeated traceback lines are now abbreviated as - ``"[Previous line repeated {count} more times]"`` (see - :ref:`whatsnew36-traceback` for an example). - (Contributed by Emanuel Barry in :issue:`26823`.) - -* Import now raises the new exception :exc:`ModuleNotFoundError` - (subclass of :exc:`ImportError`) when it cannot find a module. Code - that currently checks for ImportError (in try-except) will still work. - (Contributed by Eric Snow in :issue:`15767`.) - -* Class methods relying on zero-argument ``super()`` will now work correctly - when called from metaclass methods during class creation. - (Contributed by Martin Teichmann in :issue:`23722`.) - - -New Modules -=========== - -.. _whatsnew36-pep506: - -secrets -------- - -The main purpose of the new :mod:`secrets` module is to provide an obvious way -to reliably generate cryptographically strong pseudo-random values suitable -for managing secrets, such as account authentication, tokens, and similar. - -.. warning:: - - Note that the pseudo-random generators in the :mod:`random` module - should *NOT* be used for security purposes. Use :mod:`secrets` - on Python 3.6+ and :func:`os.urandom()` on Python 3.5 and earlier. - -.. seealso:: - - :pep:`506` -- Adding A Secrets Module To The Standard Library - PEP written and implemented by Steven D'Aprano. - - -Improved Modules -================ - -array ------ - -Exhausted iterators of :class:`array.array` will now stay exhausted even -if the iterated array is extended. This is consistent with the behavior -of other mutable sequences. - -Contributed by Serhiy Storchaka in :issue:`26492`. - -ast ---- - -The new :class:`ast.Constant` AST node has been added. It can be used -by external AST optimizers for the purposes of constant folding. - -Contributed by Victor Stinner in :issue:`26146`. - - -asyncio -------- - -Starting with Python 3.6 the ``asyncio`` module is no longer provisional and its -API is considered stable. - -Notable changes in the :mod:`asyncio` module since Python 3.5.0 -(all backported to 3.5.x due to the provisional status): - -* The :func:`~asyncio.get_event_loop` function has been changed to - always return the currently running loop when called from coroutines - and callbacks. - (Contributed by Yury Selivanov in :issue:`28613`.) - -* The :func:`~asyncio.ensure_future` function and all functions that - use it, such as :meth:`loop.run_until_complete() <asyncio.loop.run_until_complete>`, - now accept all kinds of :term:`awaitable objects <awaitable>`. - (Contributed by Yury Selivanov.) - -* New :func:`~asyncio.run_coroutine_threadsafe` function to submit - coroutines to event loops from other threads. - (Contributed by Vincent Michel.) - -* New :meth:`Transport.is_closing() <asyncio.BaseTransport.is_closing>` - method to check if the transport is closing or closed. - (Contributed by Yury Selivanov.) - -* The :meth:`loop.create_server() <asyncio.loop.create_server>` - method can now accept a list of hosts. - (Contributed by Yann Sionneau.) - -* New :meth:`loop.create_future() <asyncio.loop.create_future>` - method to create Future objects. This allows alternative event - loop implementations, such as - `uvloop <https://github.com/MagicStack/uvloop>`_, to provide a faster - :class:`asyncio.Future` implementation. - (Contributed by Yury Selivanov in :issue:`27041`.) - -* New :meth:`loop.get_exception_handler() <asyncio.loop.get_exception_handler>` - method to get the current exception handler. - (Contributed by Yury Selivanov in :issue:`27040`.) - -* New :meth:`StreamReader.readuntil() <asyncio.StreamReader.readuntil>` - method to read data from the stream until a separator bytes - sequence appears. - (Contributed by Mark Korenberg.) - -* The performance of :meth:`StreamReader.readexactly() <asyncio.StreamReader.readexactly>` - has been improved. - (Contributed by Mark Korenberg in :issue:`28370`.) - -* The :meth:`loop.getaddrinfo() <asyncio.loop.getaddrinfo>` - method is optimized to avoid calling the system ``getaddrinfo`` - function if the address is already resolved. - (Contributed by A. Jesse Jiryu Davis.) - -* The :meth:`loop.stop() <asyncio.loop.stop>` - method has been changed to stop the loop immediately after - the current iteration. Any new callbacks scheduled as a result - of the last iteration will be discarded. - (Contributed by Guido van Rossum in :issue:`25593`.) - -* :meth:`Future.set_exception <asyncio.futures.Future.set_exception>` - will now raise :exc:`TypeError` when passed an instance of - the :exc:`StopIteration` exception. - (Contributed by Chris Angelico in :issue:`26221`.) - -* New :meth:`loop.connect_accepted_socket() <asyncio.loop.connect_accepted_socket>` - method to be used by servers that accept connections outside of asyncio, - but that use asyncio to handle them. - (Contributed by Jim Fulton in :issue:`27392`.) - -* ``TCP_NODELAY`` flag is now set for all TCP transports by default. - (Contributed by Yury Selivanov in :issue:`27456`.) - -* New :meth:`loop.shutdown_asyncgens() <asyncio.loop.shutdown_asyncgens>` - to properly close pending asynchronous generators before closing the - loop. - (Contributed by Yury Selivanov in :issue:`28003`.) - -* :class:`Future <asyncio.Future>` and :class:`Task <asyncio.Task>` - classes now have an optimized C implementation which makes asyncio - code up to 30% faster. - (Contributed by Yury Selivanov and INADA Naoki in :issue:`26081` - and :issue:`28544`.) - - -binascii --------- - -The :func:`~binascii.b2a_base64` function now accepts an optional *newline* -keyword argument to control whether the newline character is appended to the -return value. -(Contributed by Victor Stinner in :issue:`25357`.) - - -cmath ------ - -The new :const:`cmath.tau` (*τ*) constant has been added. -(Contributed by Lisa Roach in :issue:`12345`, see :pep:`628` for details.) - -New constants: :const:`cmath.inf` and :const:`cmath.nan` to -match :const:`math.inf` and :const:`math.nan`, and also :const:`cmath.infj` -and :const:`cmath.nanj` to match the format used by complex repr. -(Contributed by Mark Dickinson in :issue:`23229`.) - - -collections ------------ - -The new :class:`~collections.abc.Collection` abstract base class has been -added to represent sized iterable container classes. -(Contributed by Ivan Levkivskyi, docs by Neil Girdhar in :issue:`27598`.) - -The new :class:`~collections.abc.Reversible` abstract base class represents -iterable classes that also provide the :meth:`__reversed__` method. -(Contributed by Ivan Levkivskyi in :issue:`25987`.) - -The new :class:`~collections.abc.AsyncGenerator` abstract base class represents -asynchronous generators. -(Contributed by Yury Selivanov in :issue:`28720`.) - -The :func:`~collections.namedtuple` function now accepts an optional -keyword argument *module*, which, when specified, is used for -the ``__module__`` attribute of the returned named tuple class. -(Contributed by Raymond Hettinger in :issue:`17941`.) - -The *verbose* and *rename* arguments for -:func:`~collections.namedtuple` are now keyword-only. -(Contributed by Raymond Hettinger in :issue:`25628`.) - -Recursive :class:`collections.deque` instances can now be pickled. -(Contributed by Serhiy Storchaka in :issue:`26482`.) - - -concurrent.futures ------------------- - -The :class:`ThreadPoolExecutor <concurrent.futures.ThreadPoolExecutor>` -class constructor now accepts an optional *thread_name_prefix* argument -to make it possible to customize the names of the threads created by the -pool. -(Contributed by Gregory P. Smith in :issue:`27664`.) - - -contextlib ----------- - -The :class:`contextlib.AbstractContextManager` class has been added to -provide an abstract base class for context managers. It provides a -sensible default implementation for `__enter__()` which returns -``self`` and leaves `__exit__()` an abstract method. A matching -class has been added to the :mod:`typing` module as -:class:`typing.ContextManager`. -(Contributed by Brett Cannon in :issue:`25609`.) - - -datetime --------- - -The :class:`~datetime.datetime` and :class:`~datetime.time` classes have -the new :attr:`~time.fold` attribute used to disambiguate local time -when necessary. Many functions in the :mod:`datetime` have been -updated to support local time disambiguation. -See :ref:`Local Time Disambiguation <whatsnew36-pep495>` section for more -information. -(Contributed by Alexander Belopolsky in :issue:`24773`.) - -The :meth:`datetime.strftime() <datetime.datetime.strftime>` and -:meth:`date.strftime() <datetime.date.strftime>` methods now support -ISO 8601 date directives ``%G``, ``%u`` and ``%V``. -(Contributed by Ashley Anderson in :issue:`12006`.) - -The :func:`datetime.isoformat() <datetime.datetime.isoformat>` function -now accepts an optional *timespec* argument that specifies the number -of additional components of the time value to include. -(Contributed by Alessandro Cucci and Alexander Belopolsky in :issue:`19475`.) - -The :meth:`datetime.combine() <datetime.datetime.combine>` now -accepts an optional *tzinfo* argument. -(Contributed by Alexander Belopolsky in :issue:`27661`.) - - -decimal -------- - -New :meth:`Decimal.as_integer_ratio() <decimal.Decimal.as_integer_ratio>` -method that returns a pair ``(n, d)`` of integers that represent the given -:class:`~decimal.Decimal` instance as a fraction, in lowest terms and -with a positive denominator:: - - >>> Decimal('-3.14').as_integer_ratio() - (-157, 50) - -(Contributed by Stefan Krah amd Mark Dickinson in :issue:`25928`.) - - - -distutils ---------- - -The ``default_format`` attribute has been removed from -:class:`distutils.command.sdist.sdist` and the ``formats`` -attribute defaults to ``['gztar']``. Although not anticipated, -any code relying on the presence of ``default_format`` may -need to be adapted. See :issue:`27819` for more details. - - -email ------ - -The new email API, enabled via the *policy* keyword to various constructors, is -no longer provisional. The :mod:`email` documentation has been reorganized and -rewritten to focus on the new API, while retaining the old documentation for -the legacy API. (Contributed by R. David Murray in :issue:`24277`.) - -The :mod:`email.mime` classes now all accept an optional *policy* keyword. -(Contributed by Berker Peksag in :issue:`27331`.) - -The :class:`~email.generator.DecodedGenerator` now supports the *policy* -keyword. - -There is a new :mod:`~email.policy` attribute, -:attr:`~email.policy.Policy.message_factory`, that controls what class is used -by default when the parser creates new message objects. For the -:attr:`email.policy.compat32` policy this is :class:`~email.message.Message`, -for the new policies it is :class:`~email.message.EmailMessage`. -(Contributed by R. David Murray in :issue:`20476`.) - - -encodings ---------- - -On Windows, added the ``'oem'`` encoding to use ``CP_OEMCP``, and the ``'ansi'`` -alias for the existing ``'mbcs'`` encoding, which uses the ``CP_ACP`` code page. -(Contributed by Steve Dower in :issue:`27959`.) - - -enum ----- - -Two new enumeration base classes have been added to the :mod:`enum` module: -:class:`~enum.Flag` and :class:`~enum.IntFlags`. Both are used to define -constants that can be combined using the bitwise operators. -(Contributed by Ethan Furman in :issue:`23591`.) - -Many standard library modules have been updated to use the -:class:`~enum.IntFlags` class for their constants. - -The new :class:`enum.auto` value can be used to assign values to enum -members automatically:: - - >>> from enum import Enum, auto - >>> class Color(Enum): - ... red = auto() - ... blue = auto() - ... green = auto() - ... - >>> list(Color) - [<Color.red: 1>, <Color.blue: 2>, <Color.green: 3>] - - -faulthandler ------------- - -On Windows, the :mod:`faulthandler` module now installs a handler for Windows -exceptions: see :func:`faulthandler.enable`. (Contributed by Victor Stinner in -:issue:`23848`.) - - -fileinput ---------- - -:func:`~fileinput.hook_encoded` now supports the *errors* argument. -(Contributed by Joseph Hackman in :issue:`25788`.) - - -hashlib -------- - -:mod:`hashlib` supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. -(Contributed by Christian Heimes in :issue:`26470`.) - -BLAKE2 hash functions were added to the module. :func:`~hashlib.blake2b` -and :func:`~hashlib.blake2s` are always available and support the full -feature set of BLAKE2. -(Contributed by Christian Heimes in :issue:`26798` based on code by -Dmitry Chestnykh and Samuel Neves. Documentation written by Dmitry Chestnykh.) - -The SHA-3 hash functions :func:`~hashlib.sha3_224`, :func:`~hashlib.sha3_256`, -:func:`~hashlib.sha3_384`, :func:`~hashlib.sha3_512`, and SHAKE hash functions -:func:`~hashlib.shake_128` and :func:`~hashlib.shake_256` were added. -(Contributed by Christian Heimes in :issue:`16113`. Keccak Code Package -by Guido Bertoni, Joan Daemen, Michaël Peeters, Gilles Van Assche, and -Ronny Van Keer.) - -The password-based key derivation function :func:`~hashlib.scrypt` is now -available with OpenSSL 1.1.0 and newer. -(Contributed by Christian Heimes in :issue:`27928`.) - -http.client ------------ - -:meth:`HTTPConnection.request() <http.client.HTTPConnection.request>` and -:meth:`~http.client.HTTPConnection.endheaders` both now support -chunked encoding request bodies. -(Contributed by Demian Brecht and Rolf Krahl in :issue:`12319`.) - - -idlelib and IDLE ----------------- - -The idlelib package is being modernized and refactored to make IDLE look and -work better and to make the code easier to understand, test, and improve. Part -of making IDLE look better, especially on Linux and Mac, is using ttk widgets, -mostly in the dialogs. As a result, IDLE no longer runs with tcl/tk 8.4. It -now requires tcl/tk 8.5 or 8.6. We recommend running the latest release of -either. - -'Modernizing' includes renaming and consolidation of idlelib modules. The -renaming of files with partial uppercase names is similar to the renaming of, -for instance, Tkinter and TkFont to tkinter and tkinter.font in 3.0. As a -result, imports of idlelib files that worked in 3.5 will usually not work in -3.6. At least a module name change will be needed (see idlelib/README.txt), -sometimes more. (Name changes contributed by Al Swiegart and Terry Reedy in -:issue:`24225`. Most idlelib patches since have been and will be part of the -process.) - -In compensation, the eventual result with be that some idlelib classes will be -easier to use, with better APIs and docstrings explaining them. Additional -useful information will be added to idlelib when available. - -New in 3.6.2: - -Multiple fixes for autocompletion. (Contributed by Louie Lu in :issue:`15786`.) - -New in 3.6.3: - -Module Browser (on the File menu, formerly called Class Browser), -now displays nested functions and classes in addition to top-level -functions and classes. -(Contributed by Guilherme Polo, Cheryl Sabella, and Terry Jan Reedy -in :issue:`1612262`.) - -The IDLE features formerly implemented as extensions have been reimplemented -as normal features. Their settings have been moved from the Extensions tab -to other dialog tabs. -(Contributed by Charles Wohlganger and Terry Jan Reedy in :issue:`27099`.) - -The Settings dialog (Options, Configure IDLE) has been partly rewritten -to improve both appearance and function. -(Contributed by Cheryl Sabella and Terry Jan Reedy in multiple issues.) - -New in 3.6.4: - -The font sample now includes a selection of non-Latin characters so that -users can better see the effect of selecting a particular font. -(Contributed by Terry Jan Reedy in :issue:`13802`.) -The sample can be edited to include other characters. -(Contributed by Serhiy Storchaka in :issue:`31860`.) - -New in 3.6.6: - -Editor code context option revised. Box displays all context lines up to -maxlines. Clicking on a context line jumps the editor to that line. Context -colors for custom themes is added to Highlights tab of Settings dialog. -(Contributed by Cheryl Sabella and Terry Jan Reedy in :issue:`33642`, -:issue:`33768`, and :issue:`33679`.) - -On Windows, a new API call tells Windows that tk scales for DPI. On Windows -8.1+ or 10, with DPI compatibility properties of the Python binary -unchanged, and a monitor resolution greater than 96 DPI, this should -make text and lines sharper. It should otherwise have no effect. -(Contributed by Terry Jan Reedy in :issue:`33656`.) - -New in 3.6.7: - -Output over N lines (50 by default) is squeezed down to a button. -N can be changed in the PyShell section of the General page of the -Settings dialog. Fewer, but possibly extra long, lines can be squeezed by -right clicking on the output. Squeezed output can be expanded in place -by double-clicking the button or into the clipboard or a separate window -by right-clicking the button. (Contributed by Tal Einat in :issue:`1529353`.) - - -importlib ---------- - -Import now raises the new exception :exc:`ModuleNotFoundError` -(subclass of :exc:`ImportError`) when it cannot find a module. Code -that current checks for ``ImportError`` (in try-except) will still work. -(Contributed by Eric Snow in :issue:`15767`.) - -:class:`importlib.util.LazyLoader` now calls -:meth:`~importlib.abc.Loader.create_module` on the wrapped loader, removing the -restriction that :class:`importlib.machinery.BuiltinImporter` and -:class:`importlib.machinery.ExtensionFileLoader` couldn't be used with -:class:`importlib.util.LazyLoader`. - -:func:`importlib.util.cache_from_source`, -:func:`importlib.util.source_from_cache`, and -:func:`importlib.util.spec_from_file_location` now accept a -:term:`path-like object`. - - -inspect -------- - -The :func:`inspect.signature() <inspect.signature>` function now reports the -implicit ``.0`` parameters generated by the compiler for comprehension and -generator expression scopes as if they were positional-only parameters called -``implicit0``. (Contributed by Jelle Zijlstra in :issue:`19611`.) - -To reduce code churn when upgrading from Python 2.7 and the legacy -:func:`inspect.getargspec` API, the previously documented deprecation of -:func:`inspect.getfullargspec` has been reversed. While this function is -convenient for single/source Python 2/3 code bases, the richer -:func:`inspect.signature` interface remains the recommended approach for new -code. (Contributed by Nick Coghlan in :issue:`27172`) - - -json ----- - -:func:`json.load` and :func:`json.loads` now support binary input. Encoded -JSON should be represented using either UTF-8, UTF-16, or UTF-32. -(Contributed by Serhiy Storchaka in :issue:`17909`.) - - -logging -------- - -The new :meth:`WatchedFileHandler.reopenIfNeeded() <logging.handlers.WatchedFileHandler.reopenIfNeeded>` -method has been added to add the ability to check if the log file needs to -be reopened. -(Contributed by Marian Horban in :issue:`24884`.) - - -math ----- - -The tau (*τ*) constant has been added to the :mod:`math` and :mod:`cmath` -modules. -(Contributed by Lisa Roach in :issue:`12345`, see :pep:`628` for details.) - - -multiprocessing ---------------- - -:ref:`Proxy Objects <multiprocessing-proxy_objects>` returned by -:func:`multiprocessing.Manager` can now be nested. -(Contributed by Davin Potts in :issue:`6766`.) - - -os --- - -See the summary of :ref:`PEP 519 <whatsnew36-pep519>` for details on how the -:mod:`os` and :mod:`os.path` modules now support -:term:`path-like objects <path-like object>`. - -:func:`~os.scandir` now supports :class:`bytes` paths on Windows. - -A new :meth:`~os.scandir.close` method allows explicitly closing a -:func:`~os.scandir` iterator. The :func:`~os.scandir` iterator now -supports the :term:`context manager` protocol. If a :func:`scandir` -iterator is neither exhausted nor explicitly closed a :exc:`ResourceWarning` -will be emitted in its destructor. -(Contributed by Serhiy Storchaka in :issue:`25994`.) - -On Linux, :func:`os.urandom` now blocks until the system urandom entropy pool -is initialized to increase the security. See the :pep:`524` for the rationale. - -The Linux ``getrandom()`` syscall (get random bytes) is now exposed as the new -:func:`os.getrandom` function. -(Contributed by Victor Stinner, part of the :pep:`524`) - - -pathlib -------- - -:mod:`pathlib` now supports :term:`path-like objects <path-like object>`. -(Contributed by Brett Cannon in :issue:`27186`.) - -See the summary of :ref:`PEP 519 <whatsnew36-pep519>` for details. - - -pdb ---- - -The :class:`~pdb.Pdb` class constructor has a new optional *readrc* argument -to control whether ``.pdbrc`` files should be read. - - -pickle ------- - -Objects that need ``__new__`` called with keyword arguments can now be pickled -using :ref:`pickle protocols <pickle-protocols>` older than protocol version 4. -Protocol version 4 already supports this case. (Contributed by Serhiy -Storchaka in :issue:`24164`.) - - -pickletools ------------ - -:func:`pickletools.dis()` now outputs the implicit memo index for the -``MEMOIZE`` opcode. -(Contributed by Serhiy Storchaka in :issue:`25382`.) - - -pydoc ------ - -The :mod:`pydoc` module has learned to respect the ``MANPAGER`` -environment variable. -(Contributed by Matthias Klose in :issue:`8637`.) - -:func:`help` and :mod:`pydoc` can now list named tuple fields in the -order they were defined rather than alphabetically. -(Contributed by Raymond Hettinger in :issue:`24879`.) - - -random -------- - -The new :func:`~random.choices` function returns a list of elements of -specified size from the given population with optional weights. -(Contributed by Raymond Hettinger in :issue:`18844`.) - - -re --- - -Added support of modifier spans in regular expressions. Examples: -``'(?i:p)ython'`` matches ``'python'`` and ``'Python'``, but not ``'PYTHON'``; -``'(?i)g(?-i:v)r'`` matches ``'GvR'`` and ``'gvr'``, but not ``'GVR'``. -(Contributed by Serhiy Storchaka in :issue:`433028`.) - -Match object groups can be accessed by ``__getitem__``, which is -equivalent to ``group()``. So ``mo['name']`` is now equivalent to -``mo.group('name')``. (Contributed by Eric Smith in :issue:`24454`.) - -:class:`~re.Match` objects now support -:meth:`index-like objects <object.__index__>` as group -indices. -(Contributed by Jeroen Demeyer and Xiang Zhang in :issue:`27177`.) - - -readline --------- - -Added :func:`~readline.set_auto_history` to enable or disable -automatic addition of input to the history list. (Contributed by -Tyler Crompton in :issue:`26870`.) - - -rlcompleter ------------ - -Private and special attribute names now are omitted unless the prefix starts -with underscores. A space or a colon is added after some completed keywords. -(Contributed by Serhiy Storchaka in :issue:`25011` and :issue:`25209`.) - - -shlex ------ - -The :class:`~shlex.shlex` has much -:ref:`improved shell compatibility <improved-shell-compatibility>` -through the new *punctuation_chars* argument to control which characters -are treated as punctuation. -(Contributed by Vinay Sajip in :issue:`1521950`.) - - -site ----- - -When specifying paths to add to :attr:`sys.path` in a `.pth` file, -you may now specify file paths on top of directories (e.g. zip files). -(Contributed by Wolfgang Langner in :issue:`26587`). - - -sqlite3 -------- - -:attr:`sqlite3.Cursor.lastrowid` now supports the ``REPLACE`` statement. -(Contributed by Alex LordThorsen in :issue:`16864`.) - - -socket ------- - -The :func:`~socket.socket.ioctl` function now supports the -:data:`~socket.SIO_LOOPBACK_FAST_PATH` control code. -(Contributed by Daniel Stokes in :issue:`26536`.) - -The :meth:`~socket.socket.getsockopt` constants ``SO_DOMAIN``, -``SO_PROTOCOL``, ``SO_PEERSEC``, and ``SO_PASSSEC`` are now supported. -(Contributed by Christian Heimes in :issue:`26907`.) - -The :meth:`~socket.socket.setsockopt` now supports the -``setsockopt(level, optname, None, optlen: int)`` form. -(Contributed by Christian Heimes in :issue:`27744`.) - -The socket module now supports the address family -:data:`~socket.AF_ALG` to interface with Linux Kernel crypto API. ``ALG_*``, -``SOL_ALG`` and :meth:`~socket.socket.sendmsg_afalg` were added. -(Contributed by Christian Heimes in :issue:`27744` with support from -Victor Stinner.) - -New Linux constants ``TCP_USER_TIMEOUT`` and ``TCP_CONGESTION`` were added. -(Contributed by Omar Sandoval, issue:`26273`). - - -socketserver ------------- - -Servers based on the :mod:`socketserver` module, including those -defined in :mod:`http.server`, :mod:`xmlrpc.server` and -:mod:`wsgiref.simple_server`, now support the :term:`context manager` -protocol. -(Contributed by Aviv Palivoda in :issue:`26404`.) - -The :attr:`~socketserver.StreamRequestHandler.wfile` attribute of -:class:`~socketserver.StreamRequestHandler` classes now implements -the :class:`io.BufferedIOBase` writable interface. In particular, -calling :meth:`~io.BufferedIOBase.write` is now guaranteed to send the -data in full. (Contributed by Martin Panter in :issue:`26721`.) - - -ssl ---- - -:mod:`ssl` supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. -(Contributed by Christian Heimes in :issue:`26470`.) - -3DES has been removed from the default cipher suites and ChaCha20 Poly1305 -cipher suites have been added. -(Contributed by Christian Heimes in :issue:`27850` and :issue:`27766`.) - -:class:`~ssl.SSLContext` has better default configuration for options -and ciphers. -(Contributed by Christian Heimes in :issue:`28043`.) - -SSL session can be copied from one client-side connection to another -with the new :class:`~ssl.SSLSession` class. TLS session resumption can -speed up the initial handshake, reduce latency and improve performance -(Contributed by Christian Heimes in :issue:`19500` based on a draft by -Alex Warhawk.) - -The new :meth:`~ssl.SSLContext.get_ciphers` method can be used to -get a list of enabled ciphers in order of cipher priority. - -All constants and flags have been converted to :class:`~enum.IntEnum` and -:class:`~enum.IntFlags`. -(Contributed by Christian Heimes in :issue:`28025`.) - -Server and client-side specific TLS protocols for :class:`~ssl.SSLContext` -were added. -(Contributed by Christian Heimes in :issue:`28085`.) - - -statistics ----------- - -A new :func:`~statistics.harmonic_mean` function has been added. -(Contributed by Steven D'Aprano in :issue:`27181`.) - - -struct ------- - -:mod:`struct` now supports IEEE 754 half-precision floats via the ``'e'`` -format specifier. -(Contributed by Eli Stevens, Mark Dickinson in :issue:`11734`.) - - -subprocess ----------- - -:class:`subprocess.Popen` destructor now emits a :exc:`ResourceWarning` warning -if the child process is still running. Use the context manager protocol (``with -proc: ...``) or explicitly call the :meth:`~subprocess.Popen.wait` method to -read the exit status of the child process. (Contributed by Victor Stinner in -:issue:`26741`.) - -The :class:`subprocess.Popen` constructor and all functions that pass arguments -through to it now accept *encoding* and *errors* arguments. Specifying either -of these will enable text mode for the *stdin*, *stdout* and *stderr* streams. -(Contributed by Steve Dower in :issue:`6135`.) - - -sys ---- - -The new :func:`~sys.getfilesystemencodeerrors` function returns the name of -the error mode used to convert between Unicode filenames and bytes filenames. -(Contributed by Steve Dower in :issue:`27781`.) - -On Windows the return value of the :func:`~sys.getwindowsversion` function -now includes the *platform_version* field which contains the accurate major -version, minor version and build number of the current operating system, -rather than the version that is being emulated for the process -(Contributed by Steve Dower in :issue:`27932`.) - - -telnetlib ---------- - -:class:`~telnetlib.Telnet` is now a context manager (contributed by -Stéphane Wirtel in :issue:`25485`). - - -time ----- - -The :class:`~time.struct_time` attributes :attr:`tm_gmtoff` and -:attr:`tm_zone` are now available on all platforms. - - -timeit ------- - -The new :meth:`Timer.autorange() <timeit.Timer.autorange>` convenience -method has been added to call :meth:`Timer.timeit() <timeit.Timer.timeit>` -repeatedly so that the total run time is greater or equal to 200 milliseconds. -(Contributed by Steven D'Aprano in :issue:`6422`.) - -:mod:`timeit` now warns when there is substantial (4x) variance -between best and worst times. -(Contributed by Serhiy Storchaka in :issue:`23552`.) - - -tkinter -------- - -Added methods :meth:`~tkinter.Variable.trace_add`, -:meth:`~tkinter.Variable.trace_remove` and :meth:`~tkinter.Variable.trace_info` -in the :class:`tkinter.Variable` class. They replace old methods -:meth:`~tkinter.Variable.trace_variable`, :meth:`~tkinter.Variable.trace`, -:meth:`~tkinter.Variable.trace_vdelete` and -:meth:`~tkinter.Variable.trace_vinfo` that use obsolete Tcl commands and might -not work in future versions of Tcl. -(Contributed by Serhiy Storchaka in :issue:`22115`). - - -.. _whatsnew36-traceback: - -traceback ---------- - -Both the traceback module and the interpreter's builtin exception display now -abbreviate long sequences of repeated lines in tracebacks as shown in the -following example:: - - >>> def f(): f() - ... - >>> f() - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - File "<stdin>", line 1, in f - File "<stdin>", line 1, in f - File "<stdin>", line 1, in f - [Previous line repeated 995 more times] - RecursionError: maximum recursion depth exceeded - -(Contributed by Emanuel Barry in :issue:`26823`.) - - -tracemalloc ------------ - -The :mod:`tracemalloc` module now supports tracing memory allocations in -multiple different address spaces. - -The new :class:`~tracemalloc.DomainFilter` filter class has been added -to filter block traces by their address space (domain). - -(Contributed by Victor Stinner in :issue:`26588`.) - - -.. _whatsnew36-typing: - -typing ------- - -Since the :mod:`typing` module is :term:`provisional <provisional api>`, -all changes introduced in Python 3.6 have also been -backported to Python 3.5.x. - -The :mod:`typing` module has a much improved support for generic type -aliases. For example ``Dict[str, Tuple[S, T]]`` is now a valid -type annotation. -(Contributed by Guido van Rossum in `Github #195 -<https://github.com/python/typing/pull/195>`_.) - -The :class:`typing.ContextManager` class has been added for -representing :class:`contextlib.AbstractContextManager`. -(Contributed by Brett Cannon in :issue:`25609`.) - -The :class:`typing.Collection` class has been added for -representing :class:`collections.abc.Collection`. -(Contributed by Ivan Levkivskyi in :issue:`27598`.) - -The :const:`typing.ClassVar` type construct has been added to -mark class variables. As introduced in :pep:`526`, a variable annotation -wrapped in ClassVar indicates that a given attribute is intended to be used as -a class variable and should not be set on instances of that class. -(Contributed by Ivan Levkivskyi in `Github #280 -<https://github.com/python/typing/pull/280>`_.) - -A new :const:`~typing.TYPE_CHECKING` constant that is assumed to be -``True`` by the static type checkers, but is ``False`` at runtime. -(Contributed by Guido van Rossum in `Github #230 -<https://github.com/python/typing/issues/230>`_.) - -A new :func:`~typing.NewType` helper function has been added to create -lightweight distinct types for annotations:: - - from typing import NewType - - UserId = NewType('UserId', int) - some_id = UserId(524313) - -The static type checker will treat the new type as if it were a subclass -of the original type. (Contributed by Ivan Levkivskyi in `Github #189 -<https://github.com/python/typing/issues/189>`_.) - - -unicodedata ------------ - -The :mod:`unicodedata` module now uses data from `Unicode 9.0.0 -<http://unicode.org/versions/Unicode9.0.0/>`_. -(Contributed by Benjamin Peterson.) - - -unittest.mock -------------- - -The :class:`~unittest.mock.Mock` class has the following improvements: - -* Two new methods, :meth:`Mock.assert_called() - <unittest.mock.Mock.assert_called>` and :meth:`Mock.assert_called_once() - <unittest.mock.Mock.assert_called_once>` to check if the mock object - was called. - (Contributed by Amit Saha in :issue:`26323`.) - -* The :meth:`Mock.reset_mock() <unittest.mock.Mock.reset_mock>` method - now has two optional keyword only arguments: *return_value* and - *side_effect*. - (Contributed by Kushal Das in :issue:`21271`.) - - -urllib.request --------------- - -If a HTTP request has a file or iterable body (other than a -bytes object) but no ``Content-Length`` header, rather than -throwing an error, :class:`~urllib.request.AbstractHTTPHandler` now -falls back to use chunked transfer encoding. -(Contributed by Demian Brecht and Rolf Krahl in :issue:`12319`.) - - -urllib.robotparser ------------------- - -:class:`~urllib.robotparser.RobotFileParser` now supports the ``Crawl-delay`` and -``Request-rate`` extensions. -(Contributed by Nikolay Bogoychev in :issue:`16099`.) - - -venv ----- - -:mod:`venv` accepts a new parameter ``--prompt``. This parameter provides an -alternative prefix for the virtual environment. (Proposed by Łukasz Balcerzak -and ported to 3.6 by Stéphane Wirtel in :issue:`22829`.) - - -warnings --------- - -A new optional *source* parameter has been added to the -:func:`warnings.warn_explicit` function: the destroyed object which emitted a -:exc:`ResourceWarning`. A *source* attribute has also been added to -:class:`warnings.WarningMessage` (contributed by Victor Stinner in -:issue:`26568` and :issue:`26567`). - -When a :exc:`ResourceWarning` warning is logged, the :mod:`tracemalloc` module is now -used to try to retrieve the traceback where the destroyed object was allocated. - -Example with the script ``example.py``:: - - import warnings - - def func(): - return open(__file__) - - f = func() - f = None - -Output of the command ``python3.6 -Wd -X tracemalloc=5 example.py``:: - - example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper name='example.py' mode='r' encoding='UTF-8'> - f = None - Object allocated at (most recent call first): - File "example.py", lineno 4 - return open(__file__) - File "example.py", lineno 6 - f = func() - -The "Object allocated at" traceback is new and is only displayed if -:mod:`tracemalloc` is tracing Python memory allocations and if the -:mod:`warnings` module was already imported. - - -winreg ------- - -Added the 64-bit integer type :data:`REG_QWORD <winreg.REG_QWORD>`. -(Contributed by Clement Rouault in :issue:`23026`.) - - -winsound --------- - -Allowed keyword arguments to be passed to :func:`Beep <winsound.Beep>`, -:func:`MessageBeep <winsound.MessageBeep>`, and :func:`PlaySound -<winsound.PlaySound>` (:issue:`27982`). - - -xmlrpc.client -------------- - -The :mod:`xmlrpc.client` module now supports unmarshalling -additional data types used by the Apache XML-RPC implementation -for numerics and ``None``. -(Contributed by Serhiy Storchaka in :issue:`26885`.) - - -zipfile -------- - -A new :meth:`ZipInfo.from_file() <zipfile.ZipInfo.from_file>` class method -allows making a :class:`~zipfile.ZipInfo` instance from a filesystem file. -A new :meth:`ZipInfo.is_dir() <zipfile.ZipInfo.is_dir>` method can be used -to check if the :class:`~zipfile.ZipInfo` instance represents a directory. -(Contributed by Thomas Kluyver in :issue:`26039`.) - -The :meth:`ZipFile.open() <zipfile.ZipFile.open>` method can now be used to -write data into a ZIP file, as well as for extracting data. -(Contributed by Thomas Kluyver in :issue:`26039`.) - - -zlib ----- - -The :func:`~zlib.compress` and :func:`~zlib.decompress` functions now accept -keyword arguments. -(Contributed by Aviv Palivoda in :issue:`26243` and -Xiang Zhang in :issue:`16764` respectively.) - - -Optimizations -============= - -* The Python interpreter now uses a 16-bit wordcode instead of bytecode which - made a number of opcode optimizations possible. - (Contributed by Demur Rumed with input and reviews from - Serhiy Storchaka and Victor Stinner in :issue:`26647` and :issue:`28050`.) - -* The :class:`asyncio.Future` class now has an optimized C implementation. - (Contributed by Yury Selivanov and INADA Naoki in :issue:`26081`.) - -* The :class:`asyncio.Task` class now has an optimized - C implementation. (Contributed by Yury Selivanov in :issue:`28544`.) - -* Various implementation improvements in the :mod:`typing` module - (such as caching of generic types) allow up to 30 times performance - improvements and reduced memory footprint. - -* The ASCII decoder is now up to 60 times as fast for error handlers - ``surrogateescape``, ``ignore`` and ``replace`` (Contributed - by Victor Stinner in :issue:`24870`). - -* The ASCII and the Latin1 encoders are now up to 3 times as fast for the - error handler ``surrogateescape`` - (Contributed by Victor Stinner in :issue:`25227`). - -* The UTF-8 encoder is now up to 75 times as fast for error handlers - ``ignore``, ``replace``, ``surrogateescape``, ``surrogatepass`` (Contributed - by Victor Stinner in :issue:`25267`). - -* The UTF-8 decoder is now up to 15 times as fast for error handlers - ``ignore``, ``replace`` and ``surrogateescape`` (Contributed - by Victor Stinner in :issue:`25301`). - -* ``bytes % args`` is now up to 2 times faster. (Contributed by Victor Stinner - in :issue:`25349`). - -* ``bytearray % args`` is now between 2.5 and 5 times faster. (Contributed by - Victor Stinner in :issue:`25399`). - -* Optimize :meth:`bytes.fromhex` and :meth:`bytearray.fromhex`: they are now - between 2x and 3.5x faster. (Contributed by Victor Stinner in :issue:`25401`). - -* Optimize ``bytes.replace(b'', b'.')`` and ``bytearray.replace(b'', b'.')``: - up to 80% faster. (Contributed by Josh Snider in :issue:`26574`). - -* Allocator functions of the :c:func:`PyMem_Malloc` domain - (:c:data:`PYMEM_DOMAIN_MEM`) now use the :ref:`pymalloc memory allocator - <pymalloc>` instead of :c:func:`malloc` function of the C library. The - pymalloc allocator is optimized for objects smaller or equal to 512 bytes - with a short lifetime, and use :c:func:`malloc` for larger memory blocks. - (Contributed by Victor Stinner in :issue:`26249`). - -* :func:`pickle.load` and :func:`pickle.loads` are now up to 10% faster when - deserializing many small objects (Contributed by Victor Stinner in - :issue:`27056`). - -* Passing :term:`keyword arguments <keyword argument>` to a function has an - overhead in comparison with passing :term:`positional arguments - <positional argument>`. Now in extension functions implemented with using - Argument Clinic this overhead is significantly decreased. - (Contributed by Serhiy Storchaka in :issue:`27574`). - -* Optimized :func:`~glob.glob` and :func:`~glob.iglob` functions in the - :mod:`glob` module; they are now about 3--6 times faster. - (Contributed by Serhiy Storchaka in :issue:`25596`). - -* Optimized globbing in :mod:`pathlib` by using :func:`os.scandir`; - it is now about 1.5--4 times faster. - (Contributed by Serhiy Storchaka in :issue:`26032`). - -* :class:`xml.etree.ElementTree` parsing, iteration and deepcopy performance - has been significantly improved. - (Contributed by Serhiy Storchaka in :issue:`25638`, :issue:`25873`, - and :issue:`25869`.) - -* Creation of :class:`fractions.Fraction` instances from floats and - decimals is now 2 to 3 times faster. - (Contributed by Serhiy Storchaka in :issue:`25971`.) - - -Build and C API Changes -======================= - -* Python now requires some C99 support in the toolchain to build. - Most notably, Python now uses standard integer types and macros in - place of custom macros like ``PY_LONG_LONG``. - For more information, see :pep:`7` and :issue:`17884`. - -* Cross-compiling CPython with the Android NDK and the Android API level set to - 21 (Android 5.0 Lollipop) or greater runs successfully. While Android is not - yet a supported platform, the Python test suite runs on the Android emulator - with only about 16 tests failures. See the Android meta-issue :issue:`26865`. - -* The ``--enable-optimizations`` configure flag has been added. Turning it on - will activate expensive optimizations like PGO. - (Original patch by Alecsandru Patrascu of Intel in :issue:`26359`.) - -* The :term:`GIL <global interpreter lock>` must now be held when allocator - functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex: :c:func:`PyObject_Malloc`) and - :c:data:`PYMEM_DOMAIN_MEM` (ex: :c:func:`PyMem_Malloc`) domains are called. - -* New :c:func:`Py_FinalizeEx` API which indicates if flushing buffered data - failed. - (Contributed by Martin Panter in :issue:`5319`.) - -* :c:func:`PyArg_ParseTupleAndKeywords` now supports :ref:`positional-only - parameters <positional-only_parameter>`. Positional-only parameters are - defined by empty names. - (Contributed by Serhiy Storchaka in :issue:`26282`). - -* ``PyTraceback_Print`` method now abbreviates long sequences of repeated lines - as ``"[Previous line repeated {count} more times]"``. - (Contributed by Emanuel Barry in :issue:`26823`.) - -* The new :c:func:`PyErr_SetImportErrorSubclass` function allows for - specifying a subclass of :exc:`ImportError` to raise. - (Contributed by Eric Snow in :issue:`15767`.) - -* The new :c:func:`PyErr_ResourceWarning` function can be used to generate - a :exc:`ResourceWarning` providing the source of the resource allocation. - (Contributed by Victor Stinner in :issue:`26567`.) - -* The new :c:func:`PyOS_FSPath` function returns the file system - representation of a :term:`path-like object`. - (Contributed by Brett Cannon in :issue:`27186`.) - -* The :c:func:`PyUnicode_FSConverter` and :c:func:`PyUnicode_FSDecoder` - functions will now accept :term:`path-like objects <path-like object>`. - - -Other Improvements -================== - -* When :option:`--version` (short form: :option:`-V`) is supplied twice, - Python prints :data:`sys.version` for detailed information. - - .. code-block:: shell-session - - $ ./python -VV - Python 3.6.0b4+ (3.6:223967b49e49+, Nov 21 2016, 20:55:04) - [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] - - -Deprecated -========== - -New Keywords ------------- - -``async`` and ``await`` are not recommended to be used as variable, class, -function or module names. Introduced by :pep:`492` in Python 3.5, they will -become proper keywords in Python 3.7. Starting in Python 3.6, the use of -``async`` or ``await`` as names will generate a :exc:`DeprecationWarning`. - - -Deprecated Python behavior --------------------------- - -Raising the :exc:`StopIteration` exception inside a generator will now -generate a :exc:`DeprecationWarning`, and will trigger a :exc:`RuntimeError` -in Python 3.7. See :ref:`whatsnew-pep-479` for details. - -The :meth:`__aiter__` method is now expected to return an asynchronous -iterator directly instead of returning an awaitable as previously. -Doing the former will trigger a :exc:`DeprecationWarning`. Backward -compatibility will be removed in Python 3.7. -(Contributed by Yury Selivanov in :issue:`27243`.) - -A backslash-character pair that is not a valid escape sequence now generates -a :exc:`DeprecationWarning`. Although this will eventually become a -:exc:`SyntaxError`, that will not be for several Python releases. -(Contributed by Emanuel Barry in :issue:`27364`.) - -When performing a relative import, falling back on ``__name__`` and -``__path__`` from the calling module when ``__spec__`` or -``__package__`` are not defined now raises an :exc:`ImportWarning`. -(Contributed by Rose Ames in :issue:`25791`.) - - -Deprecated Python modules, functions and methods ------------------------------------------------- - -asynchat -~~~~~~~~ - -The :mod:`asynchat` has been deprecated in favor of :mod:`asyncio`. -(Contributed by Mariatta in :issue:`25002`.) - - -asyncore -~~~~~~~~ - -The :mod:`asyncore` has been deprecated in favor of :mod:`asyncio`. -(Contributed by Mariatta in :issue:`25002`.) - - -dbm -~~~ - -Unlike other :mod:`dbm` implementations, the :mod:`dbm.dumb` module -creates databases with the ``'rw'`` mode and allows modifying the database -opened with the ``'r'`` mode. This behavior is now deprecated and will -be removed in 3.8. -(Contributed by Serhiy Storchaka in :issue:`21708`.) - - -distutils -~~~~~~~~~ - -The undocumented ``extra_path`` argument to the -:class:`~distutils.Distribution` constructor is now considered deprecated -and will raise a warning if set. Support for this parameter will be -removed in a future Python release. See :issue:`27919` for details. - - -grp -~~~ - -The support of non-integer arguments in :func:`~grp.getgrgid` has been -deprecated. -(Contributed by Serhiy Storchaka in :issue:`26129`.) - - -importlib -~~~~~~~~~ - -The :meth:`importlib.machinery.SourceFileLoader.load_module` and -:meth:`importlib.machinery.SourcelessFileLoader.load_module` methods -are now deprecated. They were the only remaining implementations of -:meth:`importlib.abc.Loader.load_module` in :mod:`importlib` that had not -been deprecated in previous versions of Python in favour of -:meth:`importlib.abc.Loader.exec_module`. - -The :class:`importlib.machinery.WindowsRegistryFinder` class is now -deprecated. As of 3.6.0, it is still added to :attr:`sys.meta_path` by -default (on Windows), but this may change in future releases. - -os -~~ - -Undocumented support of general :term:`bytes-like objects <bytes-like object>` -as paths in :mod:`os` functions, :func:`compile` and similar functions is -now deprecated. -(Contributed by Serhiy Storchaka in :issue:`25791` and :issue:`26754`.) - -re -~~ - -Support for inline flags ``(?letters)`` in the middle of the regular -expression has been deprecated and will be removed in a future Python -version. Flags at the start of a regular expression are still allowed. -(Contributed by Serhiy Storchaka in :issue:`22493`.) - -ssl -~~~ - -OpenSSL 0.9.8, 1.0.0 and 1.0.1 are deprecated and no longer supported. -In the future the :mod:`ssl` module will require at least OpenSSL 1.0.2 or -1.1.0. - -SSL-related arguments like ``certfile``, ``keyfile`` and ``check_hostname`` -in :mod:`ftplib`, :mod:`http.client`, :mod:`imaplib`, :mod:`poplib`, -and :mod:`smtplib` have been deprecated in favor of ``context``. -(Contributed by Christian Heimes in :issue:`28022`.) - -A couple of protocols and functions of the :mod:`ssl` module are now -deprecated. Some features will no longer be available in future versions -of OpenSSL. Other features are deprecated in favor of a different API. -(Contributed by Christian Heimes in :issue:`28022` and :issue:`26470`.) - -tkinter -~~~~~~~ - -The :mod:`tkinter.tix` module is now deprecated. :mod:`tkinter` users -should use :mod:`tkinter.ttk` instead. - -venv -~~~~ - -The ``pyvenv`` script has been deprecated in favour of ``python3 -m venv``. -This prevents confusion as to what Python interpreter ``pyvenv`` is -connected to and thus what Python interpreter will be used by the virtual -environment. (Contributed by Brett Cannon in :issue:`25154`.) - - -Deprecated functions and types of the C API -------------------------------------------- - -Undocumented functions :c:func:`PyUnicode_AsEncodedObject`, -:c:func:`PyUnicode_AsDecodedObject`, :c:func:`PyUnicode_AsEncodedUnicode` -and :c:func:`PyUnicode_AsDecodedUnicode` are deprecated now. -Use the :ref:`generic codec based API <codec-registry>` instead. - - -Deprecated Build Options ------------------------- - -The ``--with-system-ffi`` configure flag is now on by default on non-macOS -UNIX platforms. It may be disabled by using ``--without-system-ffi``, but -using the flag is deprecated and will not be accepted in Python 3.7. -macOS is unaffected by this change. Note that many OS distributors already -use the ``--with-system-ffi`` flag when building their system Python. - - -Removed -======= - -API and Feature Removals ------------------------- - -* Unknown escapes consisting of ``'\'`` and an ASCII letter in - regular expressions will now cause an error. In replacement templates for - :func:`re.sub` they are still allowed, but deprecated. - The :const:`re.LOCALE` flag can now only be used with binary patterns. - -* ``inspect.getmoduleinfo()`` was removed (was deprecated since CPython 3.3). - :func:`inspect.getmodulename` should be used for obtaining the module - name for a given path. - (Contributed by Yury Selivanov in :issue:`13248`.) - -* ``traceback.Ignore`` class and ``traceback.usage``, ``traceback.modname``, - ``traceback.fullmodname``, ``traceback.find_lines_from_code``, - ``traceback.find_lines``, ``traceback.find_strings``, - ``traceback.find_executable_lines`` methods were removed from the - :mod:`traceback` module. They were undocumented methods deprecated since - Python 3.2 and equivalent functionality is available from private methods. - -* The ``tk_menuBar()`` and ``tk_bindForTraversal()`` dummy methods in - :mod:`tkinter` widget classes were removed (corresponding Tk commands - were obsolete since Tk 4.0). - -* The :meth:`~zipfile.ZipFile.open` method of the :class:`zipfile.ZipFile` - class no longer supports the ``'U'`` mode (was deprecated since Python 3.4). - Use :class:`io.TextIOWrapper` for reading compressed text files in - :term:`universal newlines` mode. - -* The undocumented ``IN``, ``CDROM``, ``DLFCN``, ``TYPES``, ``CDIO``, and - ``STROPTS`` modules have been removed. They had been available in the - platform specific ``Lib/plat-*/`` directories, but were chronically out of - date, inconsistently available across platforms, and unmaintained. The - script that created these modules is still available in the source - distribution at :source:`Tools/scripts/h2py.py`. - -* The deprecated ``asynchat.fifo`` class has been removed. - - -Porting to Python 3.6 -===================== - -This section lists previously described changes and other bugfixes -that may require changes to your code. - -Changes in 'python' Command Behavior ------------------------------------- - -* The output of a special Python build with defined ``COUNT_ALLOCS``, - ``SHOW_ALLOC_COUNT`` or ``SHOW_TRACK_COUNT`` macros is now off by - default. It can be re-enabled using the ``-X showalloccount`` option. - It now outputs to ``stderr`` instead of ``stdout``. - (Contributed by Serhiy Storchaka in :issue:`23034`.) - - -Changes in the Python API -------------------------- - -* :func:`open() <open>` will no longer allow combining the ``'U'`` mode flag - with ``'+'``. - (Contributed by Jeff Balogh and John O'Connor in :issue:`2091`.) - -* :mod:`sqlite3` no longer implicitly commits an open transaction before DDL - statements. - -* On Linux, :func:`os.urandom` now blocks until the system urandom entropy pool - is initialized to increase the security. - -* When :meth:`importlib.abc.Loader.exec_module` is defined, - :meth:`importlib.abc.Loader.create_module` must also be defined. - -* :c:func:`PyErr_SetImportError` now sets :exc:`TypeError` when its **msg** - argument is not set. Previously only ``NULL`` was returned. - -* The format of the ``co_lnotab`` attribute of code objects changed to support - a negative line number delta. By default, Python does not emit bytecode with - a negative line number delta. Functions using ``frame.f_lineno``, - ``PyFrame_GetLineNumber()`` or ``PyCode_Addr2Line()`` are not affected. - Functions directly decoding ``co_lnotab`` should be updated to use a signed - 8-bit integer type for the line number delta, but this is only required to - support applications using a negative line number delta. See - ``Objects/lnotab_notes.txt`` for the ``co_lnotab`` format and how to decode - it, and see the :pep:`511` for the rationale. - -* The functions in the :mod:`compileall` module now return booleans instead - of ``1`` or ``0`` to represent success or failure, respectively. Thanks to - booleans being a subclass of integers, this should only be an issue if you - were doing identity checks for ``1`` or ``0``. See :issue:`25768`. - -* Reading the :attr:`~urllib.parse.SplitResult.port` attribute of - :func:`urllib.parse.urlsplit` and :func:`~urllib.parse.urlparse` results - now raises :exc:`ValueError` for out-of-range values, rather than - returning :const:`None`. See :issue:`20059`. - -* The :mod:`imp` module now raises a :exc:`DeprecationWarning` instead of - :exc:`PendingDeprecationWarning`. - -* The following modules have had missing APIs added to their :attr:`__all__` - attributes to match the documented APIs: - :mod:`calendar`, :mod:`cgi`, :mod:`csv`, - :mod:`~xml.etree.ElementTree`, :mod:`enum`, - :mod:`fileinput`, :mod:`ftplib`, :mod:`logging`, :mod:`mailbox`, - :mod:`mimetypes`, :mod:`optparse`, :mod:`plistlib`, :mod:`smtpd`, - :mod:`subprocess`, :mod:`tarfile`, :mod:`threading` and - :mod:`wave`. This means they will export new symbols when ``import *`` - is used. - (Contributed by Joel Taddei and Jacek Kołodziej in :issue:`23883`.) - -* When performing a relative import, if ``__package__`` does not compare equal - to ``__spec__.parent`` then :exc:`ImportWarning` is raised. - (Contributed by Brett Cannon in :issue:`25791`.) - -* When a relative import is performed and no parent package is known, then - :exc:`ImportError` will be raised. Previously, :exc:`SystemError` could be - raised. (Contributed by Brett Cannon in :issue:`18018`.) - -* Servers based on the :mod:`socketserver` module, including those - defined in :mod:`http.server`, :mod:`xmlrpc.server` and - :mod:`wsgiref.simple_server`, now only catch exceptions derived - from :exc:`Exception`. Therefore if a request handler raises - an exception like :exc:`SystemExit` or :exc:`KeyboardInterrupt`, - :meth:`~socketserver.BaseServer.handle_error` is no longer called, and - the exception will stop a single-threaded server. (Contributed by - Martin Panter in :issue:`23430`.) - -* :func:`spwd.getspnam` now raises a :exc:`PermissionError` instead of - :exc:`KeyError` if the user doesn't have privileges. - -* The :meth:`socket.socket.close` method now raises an exception if - an error (e.g. ``EBADF``) was reported by the underlying system call. - (Contributed by Martin Panter in :issue:`26685`.) - -* The *decode_data* argument for the :class:`smtpd.SMTPChannel` and - :class:`smtpd.SMTPServer` constructors is now ``False`` by default. - This means that the argument passed to - :meth:`~smtpd.SMTPServer.process_message` is now a bytes object by - default, and ``process_message()`` will be passed keyword arguments. - Code that has already been updated in accordance with the deprecation - warning generated by 3.5 will not be affected. - -* All optional arguments of the :func:`~json.dump`, :func:`~json.dumps`, - :func:`~json.load` and :func:`~json.loads` functions and - :class:`~json.JSONEncoder` and :class:`~json.JSONDecoder` class - constructors in the :mod:`json` module are now :ref:`keyword-only - <keyword-only_parameter>`. - (Contributed by Serhiy Storchaka in :issue:`18726`.) - -* Subclasses of :class:`type` which don't override ``type.__new__`` may no - longer use the one-argument form to get the type of an object. - -* As part of :pep:`487`, the handling of keyword arguments passed to - :class:`type` (other than the metaclass hint, ``metaclass``) is now - consistently delegated to :meth:`object.__init_subclass__`. This means that - :meth:`type.__new__` and :meth:`type.__init__` both now accept arbitrary - keyword arguments, but :meth:`object.__init_subclass__` (which is called from - :meth:`type.__new__`) will reject them by default. Custom metaclasses - accepting additional keyword arguments will need to adjust their calls to - :meth:`type.__new__` (whether direct or via :class:`super`) accordingly. - -* In :class:`distutils.command.sdist.sdist`, the ``default_format`` - attribute has been removed and is no longer honored. Instead, the - gzipped tarfile format is the default on all platforms and no - platform-specific selection is made. - In environments where distributions are - built on Windows and zip distributions are required, configure - the project with a ``setup.cfg`` file containing the following: - - .. code-block:: ini - - [sdist] - formats=zip - - This behavior has also been backported to earlier Python versions - by Setuptools 26.0.0. - -* In the :mod:`urllib.request` module and the - :meth:`http.client.HTTPConnection.request` method, if no Content-Length - header field has been specified and the request body is a file object, - it is now sent with HTTP 1.1 chunked encoding. If a file object has to - be sent to a HTTP 1.0 server, the Content-Length value now has to be - specified by the caller. - (Contributed by Demian Brecht and Rolf Krahl with tweaks from - Martin Panter in :issue:`12319`.) - -* The :class:`~csv.DictReader` now returns rows of type - :class:`~collections.OrderedDict`. - (Contributed by Steve Holden in :issue:`27842`.) - -* The :const:`crypt.METHOD_CRYPT` will no longer be added to ``crypt.methods`` - if unsupported by the platform. - (Contributed by Victor Stinner in :issue:`25287`.) - -* The *verbose* and *rename* arguments for - :func:`~collections.namedtuple` are now keyword-only. - (Contributed by Raymond Hettinger in :issue:`25628`.) - -* On Linux, :func:`ctypes.util.find_library` now looks in - ``LD_LIBRARY_PATH`` for shared libraries. - (Contributed by Vinay Sajip in :issue:`9998`.) - -* The :class:`imaplib.IMAP4` class now handles flags containing the - ``']'`` character in messages sent from the server to improve - real-world compatibility. - (Contributed by Lita Cho in :issue:`21815`.) - -* The :func:`mmap.write() <mmap.write>` function now returns the number - of bytes written like other write methods. - (Contributed by Jakub Stasiak in :issue:`26335`.) - -* The :func:`pkgutil.iter_modules` and :func:`pkgutil.walk_packages` - functions now return :class:`~pkgutil.ModuleInfo` named tuples. - (Contributed by Ramchandra Apte in :issue:`17211`.) - -* :func:`re.sub` now raises an error for invalid numerical group - references in replacement templates even if the pattern is not - found in the string. The error message for invalid group references - now includes the group index and the position of the reference. - (Contributed by SilentGhost, Serhiy Storchaka in :issue:`25953`.) - -* :class:`zipfile.ZipFile` will now raise :exc:`NotImplementedError` for - unrecognized compression values. Previously a plain :exc:`RuntimeError` - was raised. Additionally, calling :class:`~zipfile.ZipFile` methods - on a closed ZipFile or calling the :meth:`~zipfile.ZipFile.write` method - on a ZipFile created with mode ``'r'`` will raise a :exc:`ValueError`. - Previously, a :exc:`RuntimeError` was raised in those scenarios. - -* when custom metaclasses are combined with zero-argument :func:`super` or - direct references from methods to the implicit ``__class__`` closure - variable, the implicit ``__classcell__`` namespace entry must now be passed - up to ``type.__new__`` for initialisation. Failing to do so will result in - a :exc:`DeprecationWarning` in Python 3.6 and a :exc:`RuntimeError` in - Python 3.8. - -* With the introduction of :exc:`ModuleNotFoundError`, import system consumers - may start expecting import system replacements to raise that more specific - exception when appropriate, rather than the less-specific :exc:`ImportError`. - To provide future compatibility with such consumers, implementors of - alternative import systems that completely replace :func:`__import__` will - need to update their implementations to raise the new subclass when a module - can't be found at all. Implementors of compliant plugins to the default - import system shouldn't need to make any changes, as the default import - system will raise the new subclass when appropriate. - - -Changes in the C API --------------------- - -* The :c:func:`PyMem_Malloc` allocator family now uses the :ref:`pymalloc allocator - <pymalloc>` rather than the system :c:func:`malloc`. Applications calling - :c:func:`PyMem_Malloc` without holding the GIL can now crash. Set the - :envvar:`PYTHONMALLOC` environment variable to ``debug`` to validate the - usage of memory allocators in your application. See :issue:`26249`. - -* :c:func:`Py_Exit` (and the main interpreter) now override the exit status - with 120 if flushing buffered data failed. See :issue:`5319`. - - -CPython bytecode changes ------------------------- - -There have been several major changes to the :term:`bytecode` in Python 3.6. - -* The Python interpreter now uses a 16-bit wordcode instead of bytecode. - (Contributed by Demur Rumed with input and reviews from - Serhiy Storchaka and Victor Stinner in :issue:`26647` and :issue:`28050`.) - -* The new :opcode:`FORMAT_VALUE` and :opcode:`BUILD_STRING` opcodes as part - of the :ref:`formatted string literal <whatsnew36-pep498>` implementation. - (Contributed by Eric Smith in :issue:`25483` and - Serhiy Storchaka in :issue:`27078`.) - -* The new :opcode:`BUILD_CONST_KEY_MAP` opcode to optimize the creation - of dictionaries with constant keys. - (Contributed by Serhiy Storchaka in :issue:`27140`.) - -* The function call opcodes have been heavily reworked for better performance - and simpler implementation. - The :opcode:`MAKE_FUNCTION`, :opcode:`CALL_FUNCTION`, - :opcode:`CALL_FUNCTION_KW` and :opcode:`BUILD_MAP_UNPACK_WITH_CALL` opcodes - have been modified, the new :opcode:`CALL_FUNCTION_EX` and - :opcode:`BUILD_TUPLE_UNPACK_WITH_CALL` have been added, and - ``CALL_FUNCTION_VAR``, ``CALL_FUNCTION_VAR_KW`` and ``MAKE_CLOSURE`` opcodes - have been removed. - (Contributed by Demur Rumed in :issue:`27095`, and Serhiy Storchaka in - :issue:`27213`, :issue:`28257`.) - -* The new :opcode:`SETUP_ANNOTATIONS` and :opcode:`STORE_ANNOTATION` opcodes - have been added to support the new :term:`variable annotation` syntax. - (Contributed by Ivan Levkivskyi in :issue:`27985`.) - - -Notable changes in Python 3.6.2 -=============================== - -New ``make regen-all`` build target ------------------------------------ - -To simplify cross-compilation, and to ensure that CPython can reliably be -compiled without requiring an existing version of Python to already be -available, the autotools-based build system no longer attempts to implicitly -recompile generated files based on file modification times. - -Instead, a new ``make regen-all`` command has been added to force regeneration -of these files when desired (e.g. after an initial version of Python has -already been built based on the pregenerated versions). - -More selective regeneration targets are also defined - see -:source:`Makefile.pre.in` for details. - -(Contributed by Victor Stinner in :issue:`23404`.) - -.. versionadded:: 3.6.2 - - -Removal of ``make touch`` build target --------------------------------------- - -The ``make touch`` build target previously used to request implicit regeneration -of generated files by updating their modification times has been removed. - -It has been replaced by the new ``make regen-all`` target. - -(Contributed by Victor Stinner in :issue:`23404`.) - -.. versionchanged:: 3.6.2 - - -Notable changes in Python 3.6.4 -=============================== - -The ``PyExc_RecursionErrorInst`` singleton that was part of the public API -has been removed as its members being never cleared may cause a segfault -during finalization of the interpreter. -(Contributed by Xavier de Gaye in :issue:`22898` and :issue:`30697`.) - - -Notable changes in Python 3.6.5 -=============================== - -The :func:`locale.localeconv` function now sets temporarily the ``LC_CTYPE`` -locale to the ``LC_NUMERIC`` locale in some cases. -(Contributed by Victor Stinner in :issue:`31900`.) - -Notable changes in Python 3.6.7 -=============================== - -In 3.6.7 the :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token -when provided with input that does not have a trailing new line. This behavior -now matches what the C tokenizer does internally. -(Contributed by Ammar Askar in :issue:`33899`.) diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst deleted file mode 100644 index 2197bd9..0000000 --- a/Doc/whatsnew/3.7.rst +++ /dev/null @@ -1,2543 +0,0 @@ -**************************** - What's New In Python 3.7 -**************************** - -:Editor: Elvis Pranskevichus <elvis@magic.io> - -.. Rules for maintenance: - - * Anyone can add text to this document. Do not spend very much time - on the wording of your changes, because your text will probably - get rewritten to some degree. - - * The maintainer will go through Misc/NEWS periodically and add - changes; it's therefore more important to add your changes to - Misc/NEWS than to this file. - - * This is not a complete list of every single change; completeness - is the purpose of Misc/NEWS. Some changes I consider too small - or esoteric to include. If such a change is added to the text, - I'll just remove it. (This is another reason you shouldn't spend - too much time on writing your addition.) - - * If you want to draw your new text to the attention of the - maintainer, add 'XXX' to the beginning of the paragraph or - section. - - * It's OK to just add a fragmentary note about a change. For - example: "XXX Describe the transmogrify() function added to the - socket module." The maintainer will research the change and - write the necessary text. - - * You can comment out your additions if you like, but it's not - necessary (especially when a final release is some months away). - - * Credit the author of a patch or bugfix. Just the name is - sufficient; the e-mail address isn't necessary. - - * It's helpful to add the bug/patch number as a comment: - - XXX Describe the transmogrify() function added to the socket - module. - (Contributed by P.Y. Developer in :issue:`12345`.) - - This saves the maintainer the effort of going through the Git log - when researching a change. - -This article explains the new features in Python 3.7, compared to 3.6. -Python 3.7 was released on June 27, 2018. -For full details, see the :ref:`changelog <changelog>`. - - -Summary -- Release Highlights -============================= - -.. This section singles out the most important changes in Python 3.7. - Brevity is key. - -New syntax features: - -* :ref:`PEP 563 <whatsnew37-pep563>`, postponed evaluation of type annotations. - -Backwards incompatible syntax changes: - -* :keyword:`async` and :keyword:`await` are now reserved keywords. - -New library modules: - -* :mod:`contextvars`: :ref:`PEP 567 -- Context Variables <whatsnew37-pep567>` -* :mod:`dataclasses`: :ref:`PEP 557 -- Data Classes <whatsnew37-pep557>` -* :ref:`whatsnew37_importlib_resources` - -New built-in features: - -* :ref:`PEP 553 <whatsnew37-pep553>`, the new :func:`breakpoint` function. - -Python data model improvements: - -* :ref:`PEP 562 <whatsnew37-pep562>`, customization of access to - module attributes. - -* :ref:`PEP 560 <whatsnew37-pep560>`, core support for typing module and - generic types. - -* the insertion-order preservation nature of :ref:`dict <typesmapping>` - objects `has been declared`_ to be an official - part of the Python language spec. - -.. _has been declared: https://mail.python.org/pipermail/python-dev/2017-December/151283.html - -Significant improvements in the standard library: - -* The :mod:`asyncio` module has received new features, significant - :ref:`usability and performance improvements <whatsnew37_asyncio>`. - -* The :mod:`time` module gained support for - :ref:`functions with nanosecond resolution <whatsnew37-pep564>`. - -CPython implementation improvements: - -* Avoiding the use of ASCII as a default text encoding: - - * :ref:`PEP 538 <whatsnew37-pep538>`, legacy C locale coercion - * :ref:`PEP 540 <whatsnew37-pep540>`, forced UTF-8 runtime mode -* :ref:`PEP 552 <whatsnew37-pep552>`, deterministic .pycs -* :ref:`the new development runtime mode <whatsnew37-devmode>` -* :ref:`PEP 565 <whatsnew37-pep565>`, improved :exc:`DeprecationWarning` - handling - -C API improvements: - -* :ref:`PEP 539 <whatsnew37-pep539>`, new C API for thread-local storage - -Documentation improvements: - -* :ref:`PEP 545 <whatsnew37-pep545>`, Python documentation translations -* New documentation translations: `Japanese <https://docs.python.org/ja/>`_, - `French <https://docs.python.org/fr/>`_, and - `Korean <https://docs.python.org/ko/>`_. - -This release features notable performance improvements in many areas. -The :ref:`whatsnew37-perf` section lists them in detail. - -For a list of changes that may affect compatibility with previous Python -releases please refer to the :ref:`porting-to-python-37` section. - - -New Features -============ - -.. _whatsnew37-pep563: - -PEP 563: Postponed Evaluation of Annotations --------------------------------------------- - -The advent of type hints in Python uncovered two glaring usability issues -with the functionality of annotations added in :pep:`3107` and refined -further in :pep:`526`: - -* annotations could only use names which were already available in the - current scope, in other words they didn't support forward references - of any kind; and - -* annotating source code had adverse effects on startup time of Python - programs. - -Both of these issues are fixed by postponing the evaluation of -annotations. Instead of compiling code which executes expressions in -annotations at their definition time, the compiler stores the annotation -in a string form equivalent to the AST of the expression in question. -If needed, annotations can be resolved at runtime using -:func:`typing.get_type_hints`. In the common case where this is not -required, the annotations are cheaper to store (since short strings -are interned by the interpreter) and make startup time faster. - -Usability-wise, annotations now support forward references, making the -following syntax valid:: - - class C: - @classmethod - def from_string(cls, source: str) -> C: - ... - - def validate_b(self, obj: B) -> bool: - ... - - class B: - ... - -Since this change breaks compatibility, the new behavior needs to be enabled -on a per-module basis in Python 3.7 using a :mod:`__future__` import:: - - from __future__ import annotations - -It will become the default in Python 4.0. - -.. seealso:: - - :pep:`563` -- Postponed evaluation of annotations - PEP written and implemented by Łukasz Langa. - - -.. _whatsnew37-pep538: - -PEP 538: Legacy C Locale Coercion ---------------------------------- - -An ongoing challenge within the Python 3 series has been determining a sensible -default strategy for handling the "7-bit ASCII" text encoding assumption -currently implied by the use of the default C or POSIX locale on non-Windows -platforms. - -:pep:`538` updates the default interpreter command line interface to -automatically coerce that locale to an available UTF-8 based locale as -described in the documentation of the new :envvar:`PYTHONCOERCECLOCALE` -environment variable. Automatically setting ``LC_CTYPE`` this way means that -both the core interpreter and locale-aware C extensions (such as -:mod:`readline`) will assume the use of UTF-8 as the default text encoding, -rather than ASCII. - -The platform support definition in :pep:`11` has also been updated to limit -full text handling support to suitably configured non-ASCII based locales. - -As part of this change, the default error handler for :data:`~sys.stdin` and -:data:`~sys.stdout` is now ``surrogateescape`` (rather than ``strict``) when -using any of the defined coercion target locales (currently ``C.UTF-8``, -``C.utf8``, and ``UTF-8``). The default error handler for :data:`~sys.stderr` -continues to be ``backslashreplace``, regardless of locale. - -Locale coercion is silent by default, but to assist in debugging potentially -locale related integration problems, explicit warnings (emitted directly on -:data:`~sys.stderr`) can be requested by setting ``PYTHONCOERCECLOCALE=warn``. -This setting will also cause the Python runtime to emit a warning if the -legacy C locale remains active when the core interpreter is initialized. - -While :pep:`538`'s locale coercion has the benefit of also affecting extension -modules (such as GNU ``readline``), as well as child processes (including those -running non-Python applications and older versions of Python), it has the -downside of requiring that a suitable target locale be present on the running -system. To better handle the case where no suitable target locale is available -(as occurs on RHEL/CentOS 7, for example), Python 3.7 also implements -:ref:`whatsnew37-pep540`. - -.. seealso:: - - :pep:`538` -- Coercing the legacy C locale to a UTF-8 based locale - PEP written and implemented by Nick Coghlan. - - -.. _whatsnew37-pep540: - -PEP 540: Forced UTF-8 Runtime Mode ------------------------------------ - -The new :option:`-X` ``utf8`` command line option and :envvar:`PYTHONUTF8` -environment variable can be used to enable the CPython *UTF-8 mode*. - -When in UTF-8 mode, CPython ignores the locale settings, and uses the -UTF-8 encoding by default. The error handlers for :data:`sys.stdin` and -:data:`sys.stdout` streams are set to ``surrogateescape``. - -The forced UTF-8 mode can be used to change the text handling behavior in -an embedded Python interpreter without changing the locale settings of -an embedding application. - -While :pep:`540`'s UTF-8 mode has the benefit of working regardless of which -locales are available on the running system, it has the downside of having no -effect on extension modules (such as GNU ``readline``), child processes running -non-Python applications, and child processes running older versions of Python. -To reduce the risk of corrupting text data when communicating with such -components, Python 3.7 also implements :ref:`whatsnew37-pep540`). - -The UTF-8 mode is enabled by default when the locale is ``C`` or ``POSIX``, and -the :pep:`538` locale coercion feature fails to change it to a UTF-8 based -alternative (whether that failure is due to ``PYTHONCOERCECLOCALE=0`` being set, -``LC_ALL`` being set, or the lack of a suitable target locale). - -.. seealso:: - - :pep:`540` -- Add a new UTF-8 mode - PEP written and implemented by Victor Stinner - - -.. _whatsnew37-pep553: - -PEP 553: Built-in ``breakpoint()`` ----------------------------------- - -Python 3.7 includes the new built-in :func:`breakpoint` function as -an easy and consistent way to enter the Python debugger. - -Built-in ``breakpoint()`` calls :func:`sys.breakpointhook`. By default, the -latter imports :mod:`pdb` and then calls ``pdb.set_trace()``, but by binding -``sys.breakpointhook()`` to the function of your choosing, ``breakpoint()`` can -enter any debugger. Additionally, the environment variable -:envvar:`PYTHONBREAKPOINT` can be set to the callable of your debugger of -choice. Set ``PYTHONBREAKPOINT=0`` to completely disable built-in -``breakpoint()``. - -.. seealso:: - - :pep:`553` -- Built-in breakpoint() - PEP written and implemented by Barry Warsaw - - -.. _whatsnew37-pep539: - -PEP 539: New C API for Thread-Local Storage -------------------------------------------- - -While Python provides a C API for thread-local storage support; the existing -:ref:`Thread Local Storage (TLS) API <thread-local-storage-api>` has used -:c:type:`int` to represent TLS keys across all platforms. This has not -generally been a problem for officially-support platforms, but that is neither -POSIX-compliant, nor portable in any practical sense. - -:pep:`539` changes this by providing a new :ref:`Thread Specific Storage (TSS) -API <thread-specific-storage-api>` to CPython which supersedes use of the -existing TLS API within the CPython interpreter, while deprecating the existing -API. The TSS API uses a new type :c:type:`Py_tss_t` instead of :c:type:`int` -to represent TSS keys--an opaque type the definition of which may depend on -the underlying TLS implementation. Therefore, this will allow to build CPython -on platforms where the native TLS key is defined in a way that cannot be safely -cast to :c:type:`int`. - -Note that on platforms where the native TLS key is defined in a way that cannot -be safely cast to :c:type:`int`, all functions of the existing TLS API will be -no-op and immediately return failure. This indicates clearly that the old API -is not supported on platforms where it cannot be used reliably, and that no -effort will be made to add such support. - -.. seealso:: - - :pep:`539` -- A New C-API for Thread-Local Storage in CPython - PEP written by Erik M. Bray; implementation by Masayuki Yamamoto. - - -.. _whatsnew37-pep562: - -PEP 562: Customization of Access to Module Attributes ------------------------------------------------------ - -Python 3.7 allows defining :meth:`__getattr__` on modules and will call -it whenever a module attribute is otherwise not found. Defining -:meth:`__dir__` on modules is now also allowed. - -A typical example of where this may be useful is module attribute deprecation -and lazy loading. - -.. seealso:: - - :pep:`562` -- Module ``__getattr__`` and ``__dir__`` - PEP written and implemented by Ivan Levkivskyi - - -.. _whatsnew37-pep564: - -PEP 564: New Time Functions With Nanosecond Resolution ------------------------------------------------------- - -The resolution of clocks in modern systems can exceed the limited precision -of a floating point number returned by the :func:`time.time` function -and its variants. To avoid loss of precision, :pep:`564` adds six new -"nanosecond" variants of the existing timer functions to the :mod:`time` -module: - -* :func:`time.clock_gettime_ns` -* :func:`time.clock_settime_ns` -* :func:`time.monotonic_ns` -* :func:`time.perf_counter_ns` -* :func:`time.process_time_ns` -* :func:`time.time_ns` - -The new functions return the number of nanoseconds as an integer value. - -`Measurements <https://www.python.org/dev/peps/pep-0564/#annex-clocks-resolution-in-python>`_ -show that on Linux and Windows the resolution of :func:`time.time_ns` is -approximately 3 times better than that of :func:`time.time`. - -.. seealso:: - - :pep:`564` -- Add new time functions with nanosecond resolution - PEP written and implemented by Victor Stinner - - -.. _whatsnew37-pep565: - -PEP 565: Show DeprecationWarning in ``__main__`` ------------------------------------------------- - -The default handling of :exc:`DeprecationWarning` has been changed such that -these warnings are once more shown by default, but only when the code -triggering them is running directly in the :mod:`__main__` module. As a result, -developers of single file scripts and those using Python interactively should -once again start seeing deprecation warnings for the APIs they use, but -deprecation warnings triggered by imported application, library and framework -modules will continue to be hidden by default. - -As a result of this change, the standard library now allows developers to choose -between three different deprecation warning behaviours: - -* :exc:`FutureWarning`: always displayed by default, recommended for warnings - intended to be seen by application end users (e.g. for deprecated application - configuration settings). -* :exc:`DeprecationWarning`: displayed by default only in :mod:`__main__` and when - running tests, recommended for warnings intended to be seen by other Python - developers where a version upgrade may result in changed behaviour or an - error. -* :exc:`PendingDeprecationWarning`: displayed by default only when running - tests, intended for cases where a future version upgrade will change the - warning category to :exc:`DeprecationWarning` or :exc:`FutureWarning`. - -Previously both :exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning` -were only visible when running tests, which meant that developers primarily -writing single file scripts or using Python interactively could be surprised -by breaking changes in the APIs they used. - -.. seealso:: - - :pep:`565` -- Show DeprecationWarning in ``__main__`` - PEP written and implemented by Nick Coghlan - - -.. _whatsnew37-pep560: - -PEP 560: Core Support for ``typing`` module and Generic Types -------------------------------------------------------------- - -Initially :pep:`484` was designed in such way that it would not introduce *any* -changes to the core CPython interpreter. Now type hints and the :mod:`typing` -module are extensively used by the community, so this restriction is removed. -The PEP introduces two special methods :meth:`__class_getitem__` and -``__mro_entries__``, these methods are now used by most classes and special -constructs in :mod:`typing`. As a result, the speed of various operations -with types increased up to 7 times, the generic types can be used without -metaclass conflicts, and several long standing bugs in :mod:`typing` module are -fixed. - -.. seealso:: - - :pep:`560` -- Core support for typing module and generic types - PEP written and implemented by Ivan Levkivskyi - - -.. _whatsnew37-pep552: - -PEP 552: Hash-based .pyc Files ------------------------------- - -Python has traditionally checked the up-to-dateness of bytecode cache files -(i.e., ``.pyc`` files) by comparing the source metadata (last-modified timestamp -and size) with source metadata saved in the cache file header when it was -generated. While effective, this invalidation method has its drawbacks. When -filesystem timestamps are too coarse, Python can miss source updates, leading to -user confusion. Additionally, having a timestamp in the cache file is -problematic for `build reproducibility <https://reproducible-builds.org/>`_ and -content-based build systems. - -:pep:`552` extends the pyc format to allow the hash of the source file to be -used for invalidation instead of the source timestamp. Such ``.pyc`` files are -called "hash-based". By default, Python still uses timestamp-based invalidation -and does not generate hash-based ``.pyc`` files at runtime. Hash-based ``.pyc`` -files may be generated with :mod:`py_compile` or :mod:`compileall`. - -Hash-based ``.pyc`` files come in two variants: checked and unchecked. Python -validates checked hash-based ``.pyc`` files against the corresponding source -files at runtime but doesn't do so for unchecked hash-based pycs. Unchecked -hash-based ``.pyc`` files are a useful performance optimization for environments -where a system external to Python (e.g., the build system) is responsible for -keeping ``.pyc`` files up-to-date. - -See :ref:`pyc-invalidation` for more information. - -.. seealso:: - - :pep:`552` -- Deterministic pycs - PEP written and implemented by Benjamin Peterson - - -.. _whatsnew37-pep545: - -PEP 545: Python Documentation Translations ------------------------------------------- - -:pep:`545` describes the process of creating and maintaining Python -documentation translations. - -Three new translations have been added: - -- Japanese: https://docs.python.org/ja/ -- French: https://docs.python.org/fr/ -- Korean: https://docs.python.org/ko/ - -.. seealso:: - - :pep:`545` -- Python Documentation Translations - PEP written and implemented by Julien Palard, Inada Naoki, and - Victor Stinner. - - -.. _whatsnew37-devmode: - -Development Runtime Mode: -X dev --------------------------------- - -The new :option:`-X` ``dev`` command line option or the new -:envvar:`PYTHONDEVMODE` environment variable can be used to enable -CPython's *development mode*. When in development mode, CPython performs -additional runtime checks that are too expensive to be enabled by default. -See :option:`-X` ``dev`` documentation for the full description of the effects -of this mode. - - -Other Language Changes -====================== - -* More than 255 arguments can now be passed to a function, and a function can - now have more than 255 parameters. (Contributed by Serhiy Storchaka in - :issue:`12844` and :issue:`18896`.) - -* :meth:`bytes.fromhex` and :meth:`bytearray.fromhex` now ignore all ASCII - whitespace, not only spaces. (Contributed by Robert Xiao in :issue:`28927`.) - -* :class:`str`, :class:`bytes`, and :class:`bytearray` gained support for - the new :meth:`isascii() <str.isascii>` method, which can be used to - test if a string or bytes contain only the ASCII characters. - (Contributed by INADA Naoki in :issue:`32677`.) - -* :exc:`ImportError` now displays module name and module ``__file__`` path when - ``from ... import ...`` fails. (Contributed by Matthias Bussonnier in - :issue:`29546`.) - -* Circular imports involving absolute imports with binding a submodule to - a name are now supported. - (Contributed by Serhiy Storchaka in :issue:`30024`.) - -* ``object.__format__(x, '')`` is now equivalent to ``str(x)`` rather than - ``format(str(self), '')``. - (Contributed by Serhiy Storchaka in :issue:`28974`.) - -* In order to better support dynamic creation of stack traces, - :class:`types.TracebackType` can now be instantiated from Python code, and - the ``tb_next`` attribute on :ref:`tracebacks <traceback-objects>` is now - writable. - (Contributed by Nathaniel J. Smith in :issue:`30579`.) - -* When using the :option:`-m` switch, ``sys.path[0]`` is now eagerly expanded - to the full starting directory path, rather than being left as the empty - directory (which allows imports from the *current* working directory at the - time when an import occurs) - (Contributed by Nick Coghlan in :issue:`33053`.) - -* The new :option:`-X` ``importtime`` option or the - :envvar:`PYTHONPROFILEIMPORTTIME` environment variable can be used to show - the timing of each module import. - (Contributed by Victor Stinner in :issue:`31415`.) - - -New Modules -=========== - -.. _whatsnew37-pep567: - -contextvars ------------ - -The new :mod:`contextvars` module and a set of -:ref:`new C APIs <contextvarsobjects>` introduce -support for *context variables*. Context variables are conceptually -similar to thread-local variables. Unlike TLS, context variables -support asynchronous code correctly. - -The :mod:`asyncio` and :mod:`decimal` modules have been updated to use -and support context variables out of the box. Particularly the active -decimal context is now stored in a context variable, which allows -decimal operations to work with the correct context in asynchronous code. - -.. seealso:: - - :pep:`567` -- Context Variables - PEP written and implemented by Yury Selivanov - - -.. _whatsnew37-pep557: - -dataclasses ------------ - -The new :func:`~dataclasses.dataclass` decorator provides a way to declare -*data classes*. A data class describes its attributes using class variable -annotations. Its constructor and other magic methods, such as -:meth:`~object.__repr__`, :meth:`~object.__eq__`, and -:meth:`~object.__hash__` are generated automatically. - -Example:: - - @dataclass - class Point: - x: float - y: float - z: float = 0.0 - - p = Point(1.5, 2.5) - print(p) # produces "Point(x=1.5, y=2.5, z=0.0)" - -.. seealso:: - - :pep:`557` -- Data Classes - PEP written and implemented by Eric V. Smith - - -.. _whatsnew37_importlib_resources: - -importlib.resources -------------------- - -The new :mod:`importlib.resources` module provides several new APIs and one -new ABC for access to, opening, and reading *resources* inside packages. -Resources are roughly similar to files inside packages, but they needn't -be actual files on the physical file system. Module loaders can provide a -:meth:`get_resource_reader()` function which returns -a :class:`importlib.abc.ResourceReader` instance to support this -new API. Built-in file path loaders and zip file loaders both support this. - -Contributed by Barry Warsaw and Brett Cannon in :issue:`32248`. - -.. seealso:: - - `importlib_resources <http://importlib-resources.readthedocs.io/en/latest/>`_ - -- a PyPI backport for earlier Python versions. - - -Improved Modules -================ - - -argparse --------- - -The new :meth:`ArgumentParser.parse_intermixed_args() -<argparse.ArgumentParser.parse_intermixed_args>` -method allows intermixing options and positional arguments. -(Contributed by paul.j3 in :issue:`14191`.) - - -.. _whatsnew37_asyncio: - -asyncio -------- - -The :mod:`asyncio` module has received many new features, usability and -:ref:`performance improvements <whatsnew37-asyncio-perf>`. Notable changes -include: - -* The new :term:`provisional <provisional api>` :func:`asyncio.run` function can - be used to run a coroutine from synchronous code by automatically creating and - destroying the event loop. - (Contributed by Yury Selivanov in :issue:`32314`.) - -* asyncio gained support for :mod:`contextvars`. - :meth:`loop.call_soon() <asyncio.loop.call_soon>`, - :meth:`loop.call_soon_threadsafe() <asyncio.loop.call_soon_threadsafe>`, - :meth:`loop.call_later() <asyncio.loop.call_later>`, - :meth:`loop.call_at() <asyncio.loop.call_at>`, and - :meth:`Future.add_done_callback() <asyncio.Future.add_done_callback>` - have a new optional keyword-only *context* parameter. - :class:`Tasks <asyncio.Task>` now track their context automatically. - See :pep:`567` for more details. - (Contributed by Yury Selivanov in :issue:`32436`.) - -* The new :func:`asyncio.create_task` function has been added as a shortcut - to ``asyncio.get_event_loop().create_task()``. - (Contributed by Andrew Svetlov in :issue:`32311`.) - -* The new :meth:`loop.start_tls() <asyncio.loop.start_tls>` - method can be used to upgrade an existing connection to TLS. - (Contributed by Yury Selivanov in :issue:`23749`.) - -* The new :meth:`loop.sock_recv_into() <asyncio.loop.sock_recv_into>` - method allows reading data from a socket directly into a provided buffer making - it possible to reduce data copies. - (Contributed by Antoine Pitrou in :issue:`31819`.) - -* The new :func:`asyncio.current_task` function returns the currently running - :class:`~asyncio.Task` instance, and the new :func:`asyncio.all_tasks` - function returns a set of all existing ``Task`` instances in a given loop. - The :meth:`Task.current_task() <asyncio.Task.current_task>` and - :meth:`Task.all_tasks() <asyncio.Task.all_tasks>` methods have been deprecated. - (Contributed by Andrew Svetlov in :issue:`32250`.) - -* The new *provisional* :class:`~asyncio.BufferedProtocol` class allows - implementing streaming protocols with manual control over the receive buffer. - (Contributed by Yury Selivanov in :issue:`32251`.) - -* The new :func:`asyncio.get_running_loop` function returns the currently - running loop, and raises a :exc:`RuntimeError` if no loop is running. - This is in contrast with :func:`asyncio.get_event_loop`, which will *create* - a new event loop if none is running. - (Contributed by Yury Selivanov in :issue:`32269`.) - -* The new :meth:`StreamWriter.wait_closed() <asyncio.StreamWriter.wait_closed>` - coroutine method allows waiting until the stream writer is closed. The new - :meth:`StreamWriter.is_closing() <asyncio.StreamWriter.is_closing>` method - can be used to determine if the writer is closing. - (Contributed by Andrew Svetlov in :issue:`32391`.) - -* The new :meth:`loop.sock_sendfile() <asyncio.loop.sock_sendfile>` - coroutine method allows sending files using :mod:`os.sendfile` when possible. - (Contributed by Andrew Svetlov in :issue:`32410`.) - -* The new :meth:`Future.get_loop() <asyncio.Future.get_loop>` and - ``Task.get_loop()`` methods return the instance of the loop on which a task or - a future were created. - :meth:`Server.get_loop() <asyncio.Server.get_loop>` allows doing the same for - :class:`asyncio.Server` objects. - (Contributed by Yury Selivanov in :issue:`32415` and - Srinivas Reddy Thatiparthy in :issue:`32418`.) - -* It is now possible to control how instances of :class:`asyncio.Server` begin - serving. Previously, the server would start serving immediately when created. - The new *start_serving* keyword argument to - :meth:`loop.create_server() <asyncio.loop.create_server>` and - :meth:`loop.create_unix_server() <asyncio.loop.create_unix_server>`, - as well as :meth:`Server.start_serving() <asyncio.Server.start_serving>`, and - :meth:`Server.serve_forever() <asyncio.Server.serve_forever>` - can be used to decouple server instantiation and serving. The new - :meth:`Server.is_serving() <asyncio.Server.is_serving>` method returns ``True`` - if the server is serving. :class:`~asyncio.Server` objects are now - asynchronous context managers:: - - srv = await loop.create_server(...) - - async with srv: - # some code - - # At this point, srv is closed and no longer accepts new connections. - - (Contributed by Yury Selivanov in :issue:`32662`.) - -* Callback objects returned by - :func:`loop.call_later() <asyncio.loop.call_later>` - gained the new :meth:`when() <asyncio.TimerHandle.when>` method which - returns an absolute scheduled callback timestamp. - (Contributed by Andrew Svetlov in :issue:`32741`.) - -* The :meth:`loop.create_datagram_endpoint() \ - <asyncio.loop.create_datagram_endpoint>` method - gained support for Unix sockets. - (Contributed by Quentin Dawans in :issue:`31245`.) - -* The :func:`asyncio.open_connection`, :func:`asyncio.start_server` functions, - :meth:`loop.create_connection() <asyncio.loop.create_connection>`, - :meth:`loop.create_server() <asyncio.loop.create_server>`, - :meth:`loop.create_accepted_socket() <asyncio.loop.connect_accepted_socket>` - methods and their corresponding UNIX socket variants now accept the - *ssl_handshake_timeout* keyword argument. - (Contributed by Neil Aspinall in :issue:`29970`.) - -* The new :meth:`Handle.cancelled() <asyncio.Handle.cancelled>` method returns - ``True`` if the callback was cancelled. - (Contributed by Marat Sharafutdinov in :issue:`31943`.) - -* The asyncio source has been converted to use the - :keyword:`async`/:keyword:`await` syntax. - (Contributed by Andrew Svetlov in :issue:`32193`.) - -* The new :meth:`ReadTransport.is_reading() <asyncio.ReadTransport.is_reading>` - method can be used to determine the reading state of the transport. - Additionally, calls to - :meth:`ReadTransport.resume_reading() <asyncio.ReadTransport.resume_reading>` - and :meth:`ReadTransport.pause_reading() <asyncio.ReadTransport.pause_reading>` - are now idempotent. - (Contributed by Yury Selivanov in :issue:`32356`.) - -* Loop methods which accept socket paths now support passing - :term:`path-like objects <path-like object>`. - (Contributed by Yury Selivanov in :issue:`32066`.) - -* In :mod:`asyncio` TCP sockets on Linux are now created with ``TCP_NODELAY`` - flag set by default. - (Contributed by Yury Selivanov and Victor Stinner in :issue:`27456`.) - -* Exceptions occurring in cancelled tasks are no longer logged. - (Contributed by Yury Selivanov in :issue:`30508`.) - -* New ``WindowsSelectorEventLoopPolicy`` and - ``WindowsProactorEventLoopPolicy`` classes. - (Contributed by Yury Selivanov in :issue:`33792`.) - -Several ``asyncio`` APIs have been -:ref:`deprecated <whatsnew37-asyncio-deprecated>`. - - -binascii --------- - -The :func:`~binascii.b2a_uu` function now accepts an optional *backtick* -keyword argument. When it's true, zeros are represented by ``'`'`` -instead of spaces. (Contributed by Xiang Zhang in :issue:`30103`.) - - -calendar --------- - -The :class:`~calendar.HTMLCalendar` class has new class attributes which ease -the customization of CSS classes in the produced HTML calendar. -(Contributed by Oz Tiram in :issue:`30095`.) - - -collections ------------ - -``collections.namedtuple()`` now supports default values. -(Contributed by Raymond Hettinger in :issue:`32320`.) - - -compileall ----------- - -:func:`compileall.compile_dir` learned the new *invalidation_mode* parameter, -which can be used to enable -:ref:`hash-based .pyc invalidation <whatsnew37-pep552>`. The invalidation -mode can also be specified on the command line using the new -``--invalidation-mode`` argument. -(Contributed by Benjamin Peterson in :issue:`31650`.) - - -concurrent.futures ------------------- - -:class:`ProcessPoolExecutor <concurrent.futures.ProcessPoolExecutor>` and -:class:`ThreadPoolExecutor <concurrent.futures.ThreadPoolExecutor>` now -support the new *initializer* and *initargs* constructor arguments. -(Contributed by Antoine Pitrou in :issue:`21423`.) - -The :class:`ProcessPoolExecutor <concurrent.futures.ProcessPoolExecutor>` -can now take the multiprocessing context via the new *mp_context* argument. -(Contributed by Thomas Moreau in :issue:`31540`.) - - -contextlib ----------- - -The new :func:`~contextlib.nullcontext` is a simpler and faster no-op -context manager than :class:`~contextlib.ExitStack`. -(Contributed by Jesse-Bakker in :issue:`10049`.) - -The new :func:`~contextlib.asynccontextmanager`, -:class:`~contextlib.AbstractAsyncContextManager`, and -:class:`~contextlib.AsyncExitStack` have been added to -complement their synchronous counterparts. (Contributed -by Jelle Zijlstra in :issue:`29679` and :issue:`30241`, -and by Alexander Mohr and Ilya Kulakov in :issue:`29302`.) - - -cProfile --------- - -The :mod:`cProfile` command line now accepts ``-m module_name`` as an -alternative to script path. (Contributed by Sanyam Khurana in :issue:`21862`.) - - -crypt ------ - -The :mod:`crypt` module now supports the Blowfish hashing method. -(Contributed by Serhiy Storchaka in :issue:`31664`.) - -The :func:`~crypt.mksalt` function now allows specifying the number of rounds -for hashing. (Contributed by Serhiy Storchaka in :issue:`31702`.) - - -datetime --------- - -The new :meth:`datetime.fromisoformat() <datetime.datetime.fromisoformat>` -method constructs a :class:`~datetime.datetime` object from a string -in one of the formats output by -:meth:`datetime.isoformat() <datetime.datetime.isoformat>`. -(Contributed by Paul Ganssle in :issue:`15873`.) - -The :class:`tzinfo <datetime.tzinfo>` class now supports sub-minute offsets. -(Contributed by Alexander Belopolsky in :issue:`5288`.) - - -dbm ---- - -:mod:`dbm.dumb` now supports reading read-only files and no longer writes the -index file when it is not changed. - - -decimal -------- - -The :mod:`decimal` module now uses :ref:`context variables <whatsnew37-pep567>` -to store the decimal context. -(Contributed by Yury Selivanov in :issue:`32630`.) - - -dis ---- - -The :func:`~dis.dis` function is now able to -disassemble nested code objects (the code of comprehensions, generator -expressions and nested functions, and the code used for building nested -classes). The maximum depth of disassembly recursion is controlled by -the new *depth* parameter. -(Contributed by Serhiy Storchaka in :issue:`11822`.) - - -distutils ---------- - -``README.rst`` is now included in the list of distutils standard READMEs and -therefore included in source distributions. -(Contributed by Ryan Gonzalez in :issue:`11913`.) - - -enum ----- - -The :class:`Enum <enum.Enum>` learned the new ``_ignore_`` class property, -which allows listing the names of properties which should not become -enum members. -(Contributed by Ethan Furman in :issue:`31801`.) - -In Python 3.8, attempting to check for non-Enum objects in :class:`Enum` -classes will raise a :exc:`TypeError` (e.g. ``1 in Color``); similarly, -attempting to check for non-Flag objects in a :class:`Flag` member will -raise :exc:`TypeError` (e.g. ``1 in Perm.RW``); currently, both operations -return :const:`False` instead and are deprecated. -(Contributed by Ethan Furman in :issue:`33217`.) - - -functools ---------- - -:func:`functools.singledispatch` now supports registering implementations -using type annotations. -(Contributed by Łukasz Langa in :issue:`32227`.) - - -gc --- - -The new :func:`gc.freeze` function allows freezing all objects tracked -by the garbage collector and excluding them from future collections. -This can be used before a POSIX ``fork()`` call to make the GC copy-on-write -friendly or to speed up collection. The new :func:`gc.unfreeze` functions -reverses this operation. Additionally, :func:`gc.get_freeze_count` can -be used to obtain the number of frozen objects. -(Contributed by Li Zekun in :issue:`31558`.) - - -hmac ----- - -The :mod:`hmac` module now has an optimized one-shot :func:`~hmac.digest` -function, which is up to three times faster than :func:`~hmac.HMAC`. -(Contributed by Christian Heimes in :issue:`32433`.) - - -http.client ------------ - -:class:`~http.client.HTTPConnection` and :class:`~http.client.HTTPSConnection` -now support the new *blocksize* argument for improved upload throughput. -(Contributed by Nir Soffer in :issue:`31945`.) - - -http.server ------------ - -:class:`~http.server.SimpleHTTPRequestHandler` now supports the HTTP -``If-Modified-Since`` header. The server returns the 304 response status if -the target file was not modified after the time specified in the header. -(Contributed by Pierre Quentel in :issue:`29654`.) - -:class:`~http.server.SimpleHTTPRequestHandler` accepts the new *directory* -argument, in addition to the new ``--directory`` command line argument. -With this parameter, the server serves the specified directory, by default it -uses the current working directory. -(Contributed by Stéphane Wirtel and Julien Palard in :issue:`28707`.) - -The new :class:`ThreadingHTTPServer <http.server.ThreadingHTTPServer>` class -uses threads to handle requests using :class:`~socketserver.ThreadingMixin`. -It is used when ``http.server`` is run with ``-m``. -(Contributed by Julien Palard in :issue:`31639`.) - - -idlelib and IDLE ----------------- - -Multiple fixes for autocompletion. (Contributed by Louie Lu in :issue:`15786`.) - -Module Browser (on the File menu, formerly called Class Browser), -now displays nested functions and classes in addition to top-level -functions and classes. -(Contributed by Guilherme Polo, Cheryl Sabella, and Terry Jan Reedy -in :issue:`1612262`.) - -The Settings dialog (Options, Configure IDLE) has been partly rewritten -to improve both appearance and function. -(Contributed by Cheryl Sabella and Terry Jan Reedy in multiple issues.) - -The font sample now includes a selection of non-Latin characters so that -users can better see the effect of selecting a particular font. -(Contributed by Terry Jan Reedy in :issue:`13802`.) -The sample can be edited to include other characters. -(Contributed by Serhiy Storchaka in :issue:`31860`.) - -The IDLE features formerly implemented as extensions have been reimplemented -as normal features. Their settings have been moved from the Extensions tab -to other dialog tabs. -(Contributed by Charles Wohlganger and Terry Jan Reedy in :issue:`27099`.) - -Editor code context option revised. Box displays all context lines up to -maxlines. Clicking on a context line jumps the editor to that line. Context -colors for custom themes is added to Highlights tab of Settings dialog. -(Contributed by Cheryl Sabella and Terry Jan Reedy in :issue:`33642`, -:issue:`33768`, and :issue:`33679`.) - -On Windows, a new API call tells Windows that tk scales for DPI. On Windows -8.1+ or 10, with DPI compatibility properties of the Python binary -unchanged, and a monitor resolution greater than 96 DPI, this should -make text and lines sharper. It should otherwise have no effect. -(Contributed by Terry Jan Reedy in :issue:`33656`.) - -New in 3.7.1: - -Output over N lines (50 by default) is squeezed down to a button. -N can be changed in the PyShell section of the General page of the -Settings dialog. Fewer, but possibly extra long, lines can be squeezed by -right clicking on the output. Squeezed output can be expanded in place -by double-clicking the button or into the clipboard or a separate window -by right-clicking the button. (Contributed by Tal Einat in :issue:`1529353`.) - -The changes above have been backported to 3.6 maintenance releases. - -NEW in 3.7.4: - -Add "Run Customized" to the Run menu to run a module with customized -settings. Any command line arguments entered are added to sys.argv. -They re-appear in the box for the next customized run. One can also -suppress the normal Shell main module restart. (Contributed by Cheryl -Sabella, Terry Jan Reedy, and others in :issue:`5680` and :issue:`37627`.) - -New in 3.7.5: - -Add optional line numbers for IDLE editor windows. Windows -open without line numbers unless set otherwise in the General -tab of the configuration dialog. Line numbers for an existing -window are shown and hidden in the Options menu. -(Contributed by Tal Einat and Saimadhav Heblikar in :issue:`17535`.) - - -importlib ---------- - -The :class:`importlib.abc.ResourceReader` ABC was introduced to -support the loading of resources from packages. See also -:ref:`whatsnew37_importlib_resources`. -(Contributed by Barry Warsaw, Brett Cannon in :issue:`32248`.) - -:func:`importlib.reload` now raises :exc:`ModuleNotFoundError` if the module -lacks a spec. -(Contributed by Garvit Khatri in :issue:`29851`.) - -:func:`importlib.find_spec` now raises :exc:`ModuleNotFoundError` instead of -:exc:`AttributeError` if the specified parent module is not a package (i.e. -lacks a ``__path__`` attribute). -(Contributed by Milan Oberkirch in :issue:`30436`.) - -The new :func:`importlib.source_hash` can be used to compute the hash of -the passed source. A :ref:`hash-based .pyc file <whatsnew37-pep552>` -embeds the value returned by this function. - - -io --- - -The new :meth:`TextIOWrapper.reconfigure() <io.TextIOWrapper.reconfigure>` -method can be used to reconfigure the text stream with the new settings. -(Contributed by Antoine Pitrou in :issue:`30526` and -INADA Naoki in :issue:`15216`.) - - -ipaddress ---------- - -The new ``subnet_of()`` and ``supernet_of()`` methods of -:class:`ipaddress.IPv6Network` and :class:`ipaddress.IPv4Network` can -be used for network containment tests. -(Contributed by Michel Albert and Cheryl Sabella in :issue:`20825`.) - - -itertools ---------- - -:func:`itertools.islice` now accepts -:meth:`integer-like objects <object.__index__>` as start, stop, -and slice arguments. -(Contributed by Will Roberts in :issue:`30537`.) - - -locale ------- - -The new *monetary* argument to :func:`locale.format_string` can be used -to make the conversion use monetary thousands separators and -grouping strings. (Contributed by Garvit in :issue:`10379`.) - -The :func:`locale.getpreferredencoding` function now always returns ``'UTF-8'`` -on Android or when in the :ref:`forced UTF-8 mode <whatsnew37-pep540>`. - - -logging -------- - -:class:`~logging.Logger` instances can now be pickled. -(Contributed by Vinay Sajip in :issue:`30520`.) - -The new :meth:`StreamHandler.setStream() <logging.StreamHandler.setStream>` -method can be used to replace the logger stream after handler creation. -(Contributed by Vinay Sajip in :issue:`30522`.) - -It is now possible to specify keyword arguments to handler constructors in -configuration passed to :func:`logging.config.fileConfig`. -(Contributed by Preston Landers in :issue:`31080`.) - - -math ----- - -The new :func:`math.remainder` function implements the IEEE 754-style remainder -operation. (Contributed by Mark Dickinson in :issue:`29962`.) - - -mimetypes ---------- - -The MIME type of .bmp has been changed from ``'image/x-ms-bmp'`` to -``'image/bmp'``. -(Contributed by Nitish Chandra in :issue:`22589`.) - - -msilib ------- - -The new :meth:`Database.Close() <msilib.Database.Close>` method can be used -to close the :abbr:`MSI` database. -(Contributed by Berker Peksag in :issue:`20486`.) - - -multiprocessing ---------------- - -The new :meth:`Process.close() <multiprocessing.Process.close>` method -explicitly closes the process object and releases all resources associated -with it. :exc:`ValueError` is raised if the underlying process is still -running. -(Contributed by Antoine Pitrou in :issue:`30596`.) - -The new :meth:`Process.kill() <multiprocessing.Process.kill>` method can -be used to terminate the process using the :data:`SIGKILL` signal on Unix. -(Contributed by Vitor Pereira in :issue:`30794`.) - -Non-daemonic threads created by :class:`~multiprocessing.Process` are now -joined on process exit. -(Contributed by Antoine Pitrou in :issue:`18966`.) - - -os --- - -:func:`os.fwalk` now accepts the *path* argument as :class:`bytes`. -(Contributed by Serhiy Storchaka in :issue:`28682`.) - -:func:`os.scandir` gained support for :ref:`file descriptors <path_fd>`. -(Contributed by Serhiy Storchaka in :issue:`25996`.) - -The new :func:`~os.register_at_fork` function allows registering Python -callbacks to be executed at process fork. -(Contributed by Antoine Pitrou in :issue:`16500`.) - -Added :func:`os.preadv` (combine the functionality of :func:`os.readv` and -:func:`os.pread`) and :func:`os.pwritev` functions (combine the functionality -of :func:`os.writev` and :func:`os.pwrite`). (Contributed by Pablo Galindo in -:issue:`31368`.) - -The mode argument of :func:`os.makedirs` no longer affects the file -permission bits of newly-created intermediate-level directories. -(Contributed by Serhiy Storchaka in :issue:`19930`.) - -:func:`os.dup2` now returns the new file descriptor. Previously, ``None`` -was always returned. -(Contributed by Benjamin Peterson in :issue:`32441`.) - -The structure returned by :func:`os.stat` now contains the -:attr:`~os.stat_result.st_fstype` attribute on Solaris and its derivatives. -(Contributed by Jesús Cea Avión in :issue:`32659`.) - - -pathlib -------- - -The new :meth:`Path.is_mount() <pathlib.Path.is_mount>` method is now available -on POSIX systems and can be used to determine whether a path is a mount point. -(Contributed by Cooper Ry Lees in :issue:`30897`.) - - -pdb ---- - -:func:`pdb.set_trace` now takes an optional *header* keyword-only -argument. If given, it is printed to the console just before debugging -begins. (Contributed by Barry Warsaw in :issue:`31389`.) - -:mod:`pdb` command line now accepts ``-m module_name`` as an alternative to -script file. (Contributed by Mario Corchero in :issue:`32206`.) - - -py_compile ----------- - -:func:`py_compile.compile` -- and by extension, :mod:`compileall` -- now -respects the :envvar:`SOURCE_DATE_EPOCH` environment variable by -unconditionally creating ``.pyc`` files for hash-based validation. -This allows for guaranteeing -`reproducible builds <https://reproducible-builds.org/>`_ of ``.pyc`` -files when they are created eagerly. (Contributed by Bernhard M. Wiedemann -in :issue:`29708`.) - - -pydoc ------ - -The pydoc server can now bind to an arbitrary hostname specified by the -new ``-n`` command-line argument. -(Contributed by Feanil Patel in :issue:`31128`.) - - -queue ------ - -The new :class:`~queue.SimpleQueue` class is an unbounded :abbr:`FIFO` queue. -(Contributed by Antoine Pitrou in :issue:`14976`.) - - -re --- - -The flags :const:`re.ASCII`, :const:`re.LOCALE` and :const:`re.UNICODE` -can be set within the scope of a group. -(Contributed by Serhiy Storchaka in :issue:`31690`.) - -:func:`re.split` now supports splitting on a pattern like ``r'\b'``, -``'^$'`` or ``(?=-)`` that matches an empty string. -(Contributed by Serhiy Storchaka in :issue:`25054`.) - -Regular expressions compiled with the :const:`re.LOCALE` flag no longer -depend on the locale at compile time. Locale settings are applied only -when the compiled regular expression is used. -(Contributed by Serhiy Storchaka in :issue:`30215`.) - -:exc:`FutureWarning` is now emitted if a regular expression contains -character set constructs that will change semantically in the future, -such as nested sets and set operations. -(Contributed by Serhiy Storchaka in :issue:`30349`.) - -Compiled regular expression and match objects can now be copied -using :func:`copy.copy` and :func:`copy.deepcopy`. -(Contributed by Serhiy Storchaka in :issue:`10076`.) - - -signal ------- - -The new *warn_on_full_buffer* argument to the :func:`signal.set_wakeup_fd` -function makes it possible to specify whether Python prints a warning on -stderr when the wakeup buffer overflows. -(Contributed by Nathaniel J. Smith in :issue:`30050`.) - - -socket ------- - -The new :func:`socket.getblocking() <socket.socket.getblocking>` method -returns ``True`` if the socket is in blocking mode and ``False`` otherwise. -(Contributed by Yury Selivanov in :issue:`32373`.) - -The new :func:`socket.close` function closes the passed socket file descriptor. -This function should be used instead of :func:`os.close` for better -compatibility across platforms. -(Contributed by Christian Heimes in :issue:`32454`.) - -The :mod:`socket` module now exposes the :data:`socket.TCP_CONGESTION` -(Linux 2.6.13), :data:`socket.TCP_USER_TIMEOUT` (Linux 2.6.37), and -:data:`socket.TCP_NOTSENT_LOWAT` (Linux 3.12) constants. -(Contributed by Omar Sandoval in :issue:`26273` and -Nathaniel J. Smith in :issue:`29728`.) - -Support for :data:`socket.AF_VSOCK` sockets has been added to allow -communication between virtual machines and their hosts. -(Contributed by Cathy Avery in :issue:`27584`.) - -Sockets now auto-detect family, type and protocol from file descriptor -by default. -(Contributed by Christian Heimes in :issue:`28134`.) - - -socketserver ------------- - -:meth:`socketserver.ThreadingMixIn.server_close` now waits until all non-daemon -threads complete. :meth:`socketserver.ForkingMixIn.server_close` now waits -until all child processes complete. - -Add a new :attr:`socketserver.ForkingMixIn.block_on_close` class attribute to -:class:`socketserver.ForkingMixIn` and :class:`socketserver.ThreadingMixIn` -classes. Set the class attribute to ``False`` to get the pre-3.7 behaviour. - - -sqlite3 -------- - -:class:`sqlite3.Connection` now exposes the :meth:`~sqlite3.Connection.backup` -method when the underlying SQLite library is at version 3.6.11 or higher. -(Contributed by Lele Gaifax in :issue:`27645`.) - -The *database* argument of :func:`sqlite3.connect` now accepts any -:term:`path-like object`, instead of just a string. -(Contributed by Anders Lorentsen in :issue:`31843`.) - - -ssl ---- - -The :mod:`ssl` module now uses OpenSSL's builtin API instead of -:func:`~ssl.match_hostname` to check a host name or an IP address. Values -are validated during TLS handshake. Any certificate validation error -including failing the host name check now raises -:exc:`~ssl.SSLCertVerificationError` and aborts the handshake with a proper -TLS Alert message. The new exception contains additional information. -Host name validation can be customized with -:attr:`SSLContext.hostname_checks_common_name <ssl.SSLContext.hostname_checks_common_name>`. -(Contributed by Christian Heimes in :issue:`31399`.) - -.. note:: - The improved host name check requires a *libssl* implementation compatible - with OpenSSL 1.0.2 or 1.1. Consequently, OpenSSL 0.9.8 and 1.0.1 are no - longer supported (see :ref:`37-platform-support-removals` for more details). - The ssl module is mostly compatible with LibreSSL 2.7.2 and newer. - -The ``ssl`` module no longer sends IP addresses in SNI TLS extension. -(Contributed by Christian Heimes in :issue:`32185`.) - -:func:`~ssl.match_hostname` no longer supports partial wildcards like -``www*.example.org``. -(Contributed by Mandeep Singh in :issue:`23033` and Christian Heimes in -:issue:`31399`.) - -The default cipher suite selection of the ``ssl`` module now uses a blacklist -approach rather than a hard-coded whitelist. Python no longer re-enables -ciphers that have been blocked by OpenSSL security updates. Default cipher -suite selection can be configured at compile time. -(Contributed by Christian Heimes in :issue:`31429`.) - -Validation of server certificates containing internationalized domain names -(IDNs) is now supported. As part of this change, the -:attr:`SSLSocket.server_hostname <ssl.SSLSocket.server_hostname>` attribute -now stores the expected hostname in A-label form (``"xn--pythn-mua.org"``), -rather than the U-label form (``"pythön.org"``). (Contributed by -Nathaniel J. Smith and Christian Heimes in :issue:`28414`.) - -The ``ssl`` module has preliminary and experimental support for TLS 1.3 and -OpenSSL 1.1.1. At the time of Python 3.7.0 release, OpenSSL 1.1.1 is still -under development and TLS 1.3 hasn't been finalized yet. The TLS 1.3 -handshake and protocol behaves slightly differently than TLS 1.2 and earlier, -see :ref:`ssl-tlsv1_3`. -(Contributed by Christian Heimes in :issue:`32947`, :issue:`20995`, -:issue:`29136`, :issue:`30622` and :issue:`33618`) - -:class:`~ssl.SSLSocket` and :class:`~ssl.SSLObject` no longer have a public -constructor. Direct instantiation was never a documented and supported -feature. Instances must be created with :class:`~ssl.SSLContext` methods -:meth:`~ssl.SSLContext.wrap_socket` and :meth:`~ssl.SSLContext.wrap_bio`. -(Contributed by Christian Heimes in :issue:`32951`) - -OpenSSL 1.1 APIs for setting the minimum and maximum TLS protocol version are -available as :attr:`SSLContext.minimum_version <ssl.SSLContext.minimum_version>` -and :attr:`SSLContext.maximum_version <ssl.SSLContext.maximum_version>`. -Supported protocols are indicated by several new flags, such as -:data:`~ssl.HAS_TLSv1_1`. -(Contributed by Christian Heimes in :issue:`32609`.) - - -string ------- - -:class:`string.Template` now lets you to optionally modify the regular -expression pattern for braced placeholders and non-braced placeholders -separately. (Contributed by Barry Warsaw in :issue:`1198569`.) - - -subprocess ----------- - -The :func:`subprocess.run` function accepts the new *capture_output* -keyword argument. When true, stdout and stderr will be captured. -This is equivalent to passing :data:`subprocess.PIPE` as *stdout* and -*stderr* arguments. -(Contributed by Bo Bayles in :issue:`32102`.) - -The ``subprocess.run`` function and the :class:`subprocess.Popen` constructor -now accept the *text* keyword argument as an alias -to *universal_newlines*. -(Contributed by Andrew Clegg in :issue:`31756`.) - -On Windows the default for *close_fds* was changed from ``False`` to -``True`` when redirecting the standard handles. It's now possible to set -*close_fds* to true when redirecting the standard handles. See -:class:`subprocess.Popen`. This means that *close_fds* now defaults to -``True`` on all supported platforms. -(Contributed by Segev Finer in :issue:`19764`.) - -The subprocess module is now more graceful when handling -:exc:`KeyboardInterrupt` during :func:`subprocess.call`, -:func:`subprocess.run`, or in a :class:`~subprocess.Popen` -context manager. It now waits a short amount of time for the child -to exit, before continuing the handling of the ``KeyboardInterrupt`` -exception. -(Contributed by Gregory P. Smith in :issue:`25942`.) - - -sys ---- - -The new :func:`sys.breakpointhook` hook function is called by the -built-in :func:`breakpoint`. -(Contributed by Barry Warsaw in :issue:`31353`.) - -On Android, the new :func:`sys.getandroidapilevel` returns the build-time -Android API version. -(Contributed by Victor Stinner in :issue:`28740`.) - -The new :func:`sys.get_coroutine_origin_tracking_depth` function returns -the current coroutine origin tracking depth, as set by -the new :func:`sys.set_coroutine_origin_tracking_depth`. :mod:`asyncio` -has been converted to use this new API instead of -the deprecated :func:`sys.set_coroutine_wrapper`. -(Contributed by Nathaniel J. Smith in :issue:`32591`.) - - -time ----- - -:pep:`564` adds six new functions with nanosecond resolution to the -:mod:`time` module: - -* :func:`time.clock_gettime_ns` -* :func:`time.clock_settime_ns` -* :func:`time.monotonic_ns` -* :func:`time.perf_counter_ns` -* :func:`time.process_time_ns` -* :func:`time.time_ns` - -New clock identifiers have been added: - -* :data:`time.CLOCK_BOOTTIME` (Linux): Identical to - :data:`time.CLOCK_MONOTONIC`, except it also includes any time that the - system is suspended. -* :data:`time.CLOCK_PROF` (FreeBSD, NetBSD and OpenBSD): High-resolution - per-process CPU timer. -* :data:`time.CLOCK_UPTIME` (FreeBSD, OpenBSD): Time whose absolute value is - the time the system has been running and not suspended, providing accurate - uptime measurement. - -The new :func:`time.thread_time` and :func:`time.thread_time_ns` functions -can be used to get per-thread CPU time measurements. -(Contributed by Antoine Pitrou in :issue:`32025`.) - -The new :func:`time.pthread_getcpuclockid` function returns the clock ID -of the thread-specific CPU-time clock. - - -tkinter -------- - -The new :class:`tkinter.ttk.Spinbox` class is now available. -(Contributed by Alan Moore in :issue:`32585`.) - - -tracemalloc ------------ - -:class:`tracemalloc.Traceback` behaves more like regular tracebacks, -sorting the frames from oldest to most recent. -:meth:`Traceback.format() <tracemalloc.Traceback.format>` -now accepts negative *limit*, truncating the result to the -``abs(limit)`` oldest frames. To get the old behaviour, use -the new *most_recent_first* argument to ``Traceback.format()``. -(Contributed by Jesse Bakker in :issue:`32121`.) - - -types ------ - -The new :class:`~types.WrapperDescriptorType`, -:class:`~types.MethodWrapperType`, :class:`~types.MethodDescriptorType`, -and :class:`~types.ClassMethodDescriptorType` classes are now available. -(Contributed by Manuel Krebber and Guido van Rossum in :issue:`29377`, -and Serhiy Storchaka in :issue:`32265`.) - -The new :func:`types.resolve_bases` function resolves MRO entries -dynamically as specified by :pep:`560`. -(Contributed by Ivan Levkivskyi in :issue:`32717`.) - - -unicodedata ------------ - -The internal :mod:`unicodedata` database has been upgraded to use `Unicode 11 -<http://www.unicode.org/versions/Unicode11.0.0/>`_. (Contributed by Benjamin -Peterson.) - - -unittest --------- - -The new ``-k`` command-line option allows filtering tests by a name -substring or a Unix shell-like pattern. -For example, ``python -m unittest -k foo`` runs -``foo_tests.SomeTest.test_something``, ``bar_tests.SomeTest.test_foo``, -but not ``bar_tests.FooTest.test_something``. -(Contributed by Jonas Haag in :issue:`32071`.) - - -unittest.mock -------------- - -The :const:`~unittest.mock.sentinel` attributes now preserve their identity -when they are :mod:`copied <copy>` or :mod:`pickled <pickle>`. (Contributed by -Serhiy Storchaka in :issue:`20804`.) - -The new :func:`~unittest.mock.seal` function allows sealing -:class:`~unittest.mock.Mock` instances, which will disallow further creation -of attribute mocks. The seal is applied recursively to all attributes that -are themselves mocks. -(Contributed by Mario Corchero in :issue:`30541`.) - - -urllib.parse ------------- - -:func:`urllib.parse.quote` has been updated from :rfc:`2396` to :rfc:`3986`, -adding ``~`` to the set of characters that are never quoted by default. -(Contributed by Christian Theune and Ratnadeep Debnath in :issue:`16285`.) - - -uu --- - -The :func:`uu.encode` function now accepts an optional *backtick* -keyword argument. When it's true, zeros are represented by ``'`'`` -instead of spaces. (Contributed by Xiang Zhang in :issue:`30103`.) - - -uuid ----- - -The new :attr:`UUID.is_safe <uuid.UUID.is_safe>` attribute relays information -from the platform about whether generated UUIDs are generated with a -multiprocessing-safe method. -(Contributed by Barry Warsaw in :issue:`22807`.) - -:func:`uuid.getnode` now prefers universally administered -MAC addresses over locally administered MAC addresses. -This makes a better guarantee for global uniqueness of UUIDs returned -from :func:`uuid.uuid1`. If only locally administered MAC addresses are -available, the first such one found is returned. -(Contributed by Barry Warsaw in :issue:`32107`.) - - -warnings --------- - -The initialization of the default warnings filters has changed as follows: - -* warnings enabled via command line options (including those for :option:`-b` - and the new CPython-specific :option:`-X` ``dev`` option) are always passed - to the warnings machinery via the :data:`sys.warnoptions` attribute. - -* warnings filters enabled via the command line or the environment now have the - following order of precedence: - - * the ``BytesWarning`` filter for :option:`-b` (or ``-bb``) - * any filters specified with the :option:`-W` option - * any filters specified with the :envvar:`PYTHONWARNINGS` environment - variable - * any other CPython specific filters (e.g. the ``default`` filter added - for the new ``-X dev`` mode) - * any implicit filters defined directly by the warnings machinery - -* in CPython debug builds, all warnings are now displayed by default (the - implicit filter list is empty) - -(Contributed by Nick Coghlan and Victor Stinner in :issue:`20361`, -:issue:`32043`, and :issue:`32230`.) - -Deprecation warnings are once again shown by default in single-file scripts and -at the interactive prompt. See :ref:`whatsnew37-pep565` for details. -(Contributed by Nick Coghlan in :issue:`31975`.) - - -xml.etree ---------- - -:ref:`ElementPath <elementtree-xpath>` predicates in the :meth:`find` -methods can now compare text of the current node with ``[. = "text"]``, -not only text in children. Predicates also allow adding spaces for -better readability. (Contributed by Stefan Behnel in :issue:`31648`.) - - -xmlrpc.server -------------- - -:meth:`SimpleXMLRPCDispatcher.register_function <xmlrpc.server.SimpleXMLRPCDispatcher>` -can now be used as a decorator. (Contributed by Xiang Zhang in -:issue:`7769`.) - - -zipapp ------- - -Function :func:`~zipapp.create_archive` now accepts an optional *filter* -argument to allow the user to select which files should be included in the -archive. (Contributed by Irmen de Jong in :issue:`31072`.) - -Function :func:`~zipapp.create_archive` now accepts an optional *compressed* -argument to generate a compressed archive. A command line option -``--compress`` has also been added to support compression. -(Contributed by Zhiming Wang in :issue:`31638`.) - - -zipfile -------- - -:class:`~zipfile.ZipFile` now accepts the new *compresslevel* parameter to -control the compression level. -(Contributed by Bo Bayles in :issue:`21417`.) - -Subdirectories in archives created by ``ZipFile`` are now stored in -alphabetical order. -(Contributed by Bernhard M. Wiedemann in :issue:`30693`.) - - -C API Changes -============= - -A new API for thread-local storage has been implemented. See -:ref:`whatsnew37-pep539` for an overview and -:ref:`thread-specific-storage-api` for a complete reference. -(Contributed by Masayuki Yamamoto in :issue:`25658`.) - -The new :ref:`context variables <whatsnew37-pep567>` functionality -exposes a number of :ref:`new C APIs <contextvarsobjects>`. - -The new :c:func:`PyImport_GetModule` function returns the previously -imported module with the given name. -(Contributed by Eric Snow in :issue:`28411`.) - -The new :c:macro:`Py_RETURN_RICHCOMPARE` macro eases writing rich -comparison functions. -(Contributed by Petr Victorin in :issue:`23699`.) - -The new :c:macro:`Py_UNREACHABLE` macro can be used to mark unreachable -code paths. -(Contributed by Barry Warsaw in :issue:`31338`.) - -The :mod:`tracemalloc` now exposes a C API through the new -:c:func:`PyTraceMalloc_Track` and :c:func:`PyTraceMalloc_Untrack` -functions. -(Contributed by Victor Stinner in :issue:`30054`.) - -The new :c:func:`import__find__load__start` and -:c:func:`import__find__load__done` static markers can be used to trace -module imports. -(Contributed by Christian Heimes in :issue:`31574`.) - -The fields :c:member:`name` and :c:member:`doc` of structures -:c:type:`PyMemberDef`, :c:type:`PyGetSetDef`, -:c:type:`PyStructSequence_Field`, :c:type:`PyStructSequence_Desc`, -and :c:type:`wrapperbase` are now of type ``const char *`` rather of -``char *``. (Contributed by Serhiy Storchaka in :issue:`28761`.) - -The result of :c:func:`PyUnicode_AsUTF8AndSize` and :c:func:`PyUnicode_AsUTF8` -is now of type ``const char *`` rather of ``char *``. (Contributed by Serhiy -Storchaka in :issue:`28769`.) - -The result of :c:func:`PyMapping_Keys`, :c:func:`PyMapping_Values` and -:c:func:`PyMapping_Items` is now always a list, rather than a list or a -tuple. (Contributed by Oren Milman in :issue:`28280`.) - -Added functions :c:func:`PySlice_Unpack` and :c:func:`PySlice_AdjustIndices`. -(Contributed by Serhiy Storchaka in :issue:`27867`.) - -:c:func:`PyOS_AfterFork` is deprecated in favour of the new functions -:c:func:`PyOS_BeforeFork`, :c:func:`PyOS_AfterFork_Parent` and -:c:func:`PyOS_AfterFork_Child`. (Contributed by Antoine Pitrou in -:issue:`16500`.) - -The ``PyExc_RecursionErrorInst`` singleton that was part of the public API -has been removed as its members being never cleared may cause a segfault -during finalization of the interpreter. Contributed by Xavier de Gaye in -:issue:`22898` and :issue:`30697`. - -Added C API support for timezones with timezone constructors -:c:func:`PyTimeZone_FromOffset` and :c:func:`PyTimeZone_FromOffsetAndName`, -and access to the UTC singleton with :c:data:`PyDateTime_TimeZone_UTC`. -Contributed by Paul Ganssle in :issue:`10381`. - -The type of results of :c:func:`PyThread_start_new_thread` and -:c:func:`PyThread_get_thread_ident`, and the *id* parameter of -:c:func:`PyThreadState_SetAsyncExc` changed from :c:type:`long` to -:c:type:`unsigned long`. -(Contributed by Serhiy Storchaka in :issue:`6532`.) - -:c:func:`PyUnicode_AsWideCharString` now raises a :exc:`ValueError` if the -second argument is ``NULL`` and the :c:type:`wchar_t*` string contains null -characters. (Contributed by Serhiy Storchaka in :issue:`30708`.) - -Changes to the startup sequence and the management of dynamic memory -allocators mean that the long documented requirement to call -:c:func:`Py_Initialize` before calling most C API functions is now -relied on more heavily, and failing to abide by it may lead to segfaults in -embedding applications. See the :ref:`porting-to-python-37` section in this -document and the :ref:`pre-init-safe` section in the C API documentation -for more details. - -The new :c:func:`PyInterpreterState_GetID` returns the unique ID for a -given interpreter. -(Contributed by Eric Snow in :issue:`29102`.) - -:c:func:`Py_DecodeLocale`, :c:func:`Py_EncodeLocale` now use the UTF-8 -encoding when the :ref:`UTF-8 mode <whatsnew37-pep540>` is enabled. -(Contributed by Victor Stinner in :issue:`29240`.) - -:c:func:`PyUnicode_DecodeLocaleAndSize` and :c:func:`PyUnicode_EncodeLocale` -now use the current locale encoding for ``surrogateescape`` error handler. -(Contributed by Victor Stinner in :issue:`29240`.) - -The *start* and *end* parameters of :c:func:`PyUnicode_FindChar` are -now adjusted to behave like string slices. -(Contributed by Xiang Zhang in :issue:`28822`.) - - -Build Changes -============= - -Support for building ``--without-threads`` has been removed. The -:mod:`threading` module is now always available. -(Contributed by Antoine Pitrou in :issue:`31370`.). - -A full copy of libffi is no longer bundled for use when building the -:mod:`_ctypes <ctypes>` module on non-OSX UNIX platforms. An installed copy -of libffi is now required when building ``_ctypes`` on such platforms. -(Contributed by Zachary Ware in :issue:`27979`.) - -The Windows build process no longer depends on Subversion to pull in external -sources, a Python script is used to download zipfiles from GitHub instead. -If Python 3.6 is not found on the system (via ``py -3.6``), NuGet is used to -download a copy of 32-bit Python for this purpose. (Contributed by Zachary -Ware in :issue:`30450`.) - -The :mod:`ssl` module requires OpenSSL 1.0.2 or 1.1 compatible libssl. -OpenSSL 1.0.1 has reached end of lifetime on 2016-12-31 and is no longer -supported. LibreSSL is temporarily not supported as well. LibreSSL releases -up to version 2.6.4 are missing required OpenSSL 1.0.2 APIs. - - -.. _whatsnew37-perf: - -Optimizations -============= - -The overhead of calling many methods of various standard library classes -implemented in C has been significantly reduced by porting more code -to use the ``METH_FASTCALL`` convention. -(Contributed by Victor Stinner in :issue:`29300`, :issue:`29507`, -:issue:`29452`, and :issue:`29286`.) - -Various optimizations have reduced Python startup time by 10% on Linux and -up to 30% on macOS. -(Contributed by Victor Stinner, INADA Naoki in :issue:`29585`, and -Ivan Levkivskyi in :issue:`31333`.) - -Method calls are now up to 20% faster due to the bytecode changes which -avoid creating bound method instances. -(Contributed by Yury Selivanov and INADA Naoki in :issue:`26110`.) - -.. _whatsnew37-asyncio-perf: - -The :mod:`asyncio` module received a number of notable optimizations for -commonly used functions: - -* The :func:`asyncio.get_event_loop` function has been reimplemented in C to - make it up to 15 times faster. - (Contributed by Yury Selivanov in :issue:`32296`.) - -* :class:`asyncio.Future` callback management has been optimized. - (Contributed by Yury Selivanov in :issue:`32348`.) - -* :func:`asyncio.gather` is now up to 15% faster. - (Contributed by Yury Selivanov in :issue:`32355`.) - -* :func:`asyncio.sleep` is now up to 2 times faster when the *delay* - argument is zero or negative. - (Contributed by Andrew Svetlov in :issue:`32351`.) - -* The performance overhead of asyncio debug mode has been reduced. - (Contributed by Antoine Pitrou in :issue:`31970`.) - -As a result of :ref:`PEP 560 work <whatsnew37-pep560>`, the import time -of :mod:`typing` has been reduced by a factor of 7, and many typing operations -are now faster. -(Contributed by Ivan Levkivskyi in :issue:`32226`.) - -:func:`sorted` and :meth:`list.sort` have been optimized for common cases -to be up to 40-75% faster. -(Contributed by Elliot Gorokhovsky in :issue:`28685`.) - -:meth:`dict.copy` is now up to 5.5 times faster. -(Contributed by Yury Selivanov in :issue:`31179`.) - -:func:`hasattr` and :func:`getattr` are now about 4 times faster when -*name* is not found and *obj* does not override :meth:`object.__getattr__` -or :meth:`object.__getattribute__`. -(Contributed by INADA Naoki in :issue:`32544`.) - -Searching for certain Unicode characters (like Ukrainian capital "Є") -in a string was up to 25 times slower than searching for other characters. -It is now only 3 times slower in the worst case. -(Contributed by Serhiy Storchaka in :issue:`24821`.) - -The :func:`collections.namedtuple` factory has been reimplemented to -make the creation of named tuples 4 to 6 times faster. -(Contributed by Jelle Zijlstra with further improvements by INADA Naoki, -Serhiy Storchaka, and Raymond Hettinger in :issue:`28638`.) - -:meth:`date.fromordinal` and :meth:`date.fromtimestamp` are now up to -30% faster in the common case. -(Contributed by Paul Ganssle in :issue:`32403`.) - -The :func:`os.fwalk` function is now up to 2 times faster thanks to -the use of :func:`os.scandir`. -(Contributed by Serhiy Storchaka in :issue:`25996`.) - -The speed of the :func:`shutil.rmtree` function has been improved by -20--40% thanks to the use of the :func:`os.scandir` function. -(Contributed by Serhiy Storchaka in :issue:`28564`.) - -Optimized case-insensitive matching and searching of :mod:`regular -expressions <re>`. Searching some patterns can now be up to 20 times faster. -(Contributed by Serhiy Storchaka in :issue:`30285`.) - -:func:`re.compile` now converts ``flags`` parameter to int object if -it is ``RegexFlag``. It is now as fast as Python 3.5, and faster than -Python 3.6 by about 10% depending on the pattern. -(Contributed by INADA Naoki in :issue:`31671`.) - -The :meth:`~selectors.BaseSelector.modify` methods of classes -:class:`selectors.EpollSelector`, :class:`selectors.PollSelector` -and :class:`selectors.DevpollSelector` may be around 10% faster under -heavy loads. (Contributed by Giampaolo Rodola' in :issue:`30014`) - -Constant folding has been moved from the peephole optimizer to the new AST -optimizer, which is able perform optimizations more consistently. -(Contributed by Eugene Toder and INADA Naoki in :issue:`29469` and -:issue:`11549`.) - -Most functions and methods in :mod:`abc` have been rewritten in C. -This makes creation of abstract base classes, and calling :func:`isinstance` -and :func:`issubclass` on them 1.5x faster. This also reduces Python -start-up time by up to 10%. (Contributed by Ivan Levkivskyi and INADA Naoki -in :issue:`31333`) - -Significant speed improvements to alternate constructors for -:class:`datetime.date` and :class:`datetime.datetime` by using fast-path -constructors when not constructing subclasses. (Contributed by Paul Ganssle -in :issue:`32403`) - -The speed of comparison of :class:`array.array` instances has been -improved considerably in certain cases. It is now from 10x to 70x faster -when comparing arrays holding values of the same integer type. -(Contributed by Adrian Wielgosik in :issue:`24700`.) - -The :func:`math.erf` and :func:`math.erfc` functions now use the (faster) -C library implementation on most platforms. -(Contributed by Serhiy Storchaka in :issue:`26121`.) - - -Other CPython Implementation Changes -==================================== - -* Trace hooks may now opt out of receiving the ``line`` and opt into receiving - the ``opcode`` events from the interpreter by setting the corresponding new - ``f_trace_lines`` and ``f_trace_opcodes`` attributes on the - frame being traced. (Contributed by Nick Coghlan in :issue:`31344`.) - -* Fixed some consistency problems with namespace package module attributes. - Namespace module objects now have an ``__file__`` that is set to ``None`` - (previously unset), and their ``__spec__.origin`` is also set to ``None`` - (previously the string ``"namespace"``). See :issue:`32305`. Also, the - namespace module object's ``__spec__.loader`` is set to the same value as - ``__loader__`` (previously, the former was set to ``None``). See - :issue:`32303`. - -* The :func:`locals` dictionary now displays in the lexical order that - variables were defined. Previously, the order was undefined. - (Contributed by Raymond Hettinger in :issue:`32690`.) - -* The :mod:`distutils` ``upload`` command no longer tries to change CR - end-of-line characters to CRLF. This fixes a corruption issue with sdists - that ended with a byte equivalent to CR. - (Contributed by Bo Bayles in :issue:`32304`.) - - -Deprecated Python Behavior -========================== - -Yield expressions (both ``yield`` and ``yield from`` clauses) are now deprecated -in comprehensions and generator expressions (aside from the iterable expression -in the leftmost :keyword:`!for` clause). This ensures that comprehensions -always immediately return a container of the appropriate type (rather than -potentially returning a :term:`generator iterator` object), while generator -expressions won't attempt to interleave their implicit output with the output -from any explicit yield expressions. In Python 3.7, such expressions emit -:exc:`DeprecationWarning` when compiled, in Python 3.8 this will be a -:exc:`SyntaxError`. -(Contributed by Serhiy Storchaka in :issue:`10544`.) - -Returning a subclass of :class:`complex` from :meth:`object.__complex__` is -deprecated and will be an error in future Python versions. This makes -``__complex__()`` consistent with :meth:`object.__int__` and -:meth:`object.__float__`. -(Contributed by Serhiy Storchaka in :issue:`28894`.) - - - -Deprecated Python modules, functions and methods -================================================ - -aifc ----- - -:func:`aifc.openfp` has been deprecated and will be removed in Python 3.9. -Use :func:`aifc.open` instead. -(Contributed by Brian Curtin in :issue:`31985`.) - - -.. _whatsnew37-asyncio-deprecated: - -asyncio -------- - -Support for directly ``await``-ing instances of :class:`asyncio.Lock` and -other asyncio synchronization primitives has been deprecated. An -asynchronous context manager must be used in order to acquire and release -the synchronization resource. -(Contributed by Andrew Svetlov in :issue:`32253`.) - -The :meth:`asyncio.Task.current_task` and :meth:`asyncio.Task.all_tasks` -methods have been deprecated. -(Contributed by Andrew Svetlov in :issue:`32250`.) - - -collections ------------ - -In Python 3.8, the abstract base classes in :mod:`collections.abc` will no -longer be exposed in the regular :mod:`collections` module. This will help -create a clearer distinction between the concrete classes and the abstract -base classes. -(Contributed by Serhiy Storchaka in :issue:`25988`.) - - -dbm ---- - -:mod:`dbm.dumb` now supports reading read-only files and no longer writes the -index file when it is not changed. A deprecation warning is now emitted -if the index file is missing and recreated in the ``'r'`` and ``'w'`` -modes (this will be an error in future Python releases). -(Contributed by Serhiy Storchaka in :issue:`28847`.) - - -enum ----- - -In Python 3.8, attempting to check for non-Enum objects in :class:`Enum` -classes will raise a :exc:`TypeError` (e.g. ``1 in Color``); similarly, -attempting to check for non-Flag objects in a :class:`Flag` member will -raise :exc:`TypeError` (e.g. ``1 in Perm.RW``); currently, both operations -return :const:`False` instead. -(Contributed by Ethan Furman in :issue:`33217`.) - - -gettext -------- - -Using non-integer value for selecting a plural form in :mod:`gettext` is -now deprecated. It never correctly worked. (Contributed by Serhiy Storchaka -in :issue:`28692`.) - - -importlib ---------- - -Methods -:meth:`MetaPathFinder.find_module() <importlib.abc.MetaPathFinder.find_module>` -(replaced by -:meth:`MetaPathFinder.find_spec() <importlib.abc.MetaPathFinder.find_spec>`) -and -:meth:`PathEntryFinder.find_loader() <importlib.abc.PathEntryFinder.find_loader>` -(replaced by -:meth:`PathEntryFinder.find_spec() <importlib.abc.PathEntryFinder.find_spec>`) -both deprecated in Python 3.4 now emit :exc:`DeprecationWarning`. -(Contributed by Matthias Bussonnier in :issue:`29576`) - -The :class:`importlib.abc.ResourceLoader` ABC has been deprecated in -favour of :class:`importlib.abc.ResourceReader`. - - -locale ------- - -:func:`locale.format` has been deprecated, use :meth:`locale.format_string` -instead. (Contributed by Garvit in :issue:`10379`.) - - -macpath -------- - -The :mod:`macpath` is now deprecated and will be removed in Python 3.8. -(Contributed by Chi Hsuan Yen in :issue:`9850`.) - - -threading ---------- - -:mod:`dummy_threading` and :mod:`_dummy_thread` have been deprecated. It is -no longer possible to build Python with threading disabled. -Use :mod:`threading` instead. -(Contributed by Antoine Pitrou in :issue:`31370`.) - - -socket ------- - -The silent argument value truncation in :func:`socket.htons` and -:func:`socket.ntohs` has been deprecated. In future versions of Python, -if the passed argument is larger than 16 bits, an exception will be raised. -(Contributed by Oren Milman in :issue:`28332`.) - - -ssl ---- - -:func:`ssl.wrap_socket` is deprecated. Use -:meth:`ssl.SSLContext.wrap_socket` instead. -(Contributed by Christian Heimes in :issue:`28124`.) - - -sunau ------ - -:func:`sunau.openfp` has been deprecated and will be removed in Python 3.9. -Use :func:`sunau.open` instead. -(Contributed by Brian Curtin in :issue:`31985`.) - - -sys ---- - -Deprecated :func:`sys.set_coroutine_wrapper` and -:func:`sys.get_coroutine_wrapper`. - -The undocumented ``sys.callstats()`` function has been deprecated and -will be removed in a future Python version. -(Contributed by Victor Stinner in :issue:`28799`.) - - -wave ----- - -:func:`wave.openfp` has been deprecated and will be removed in Python 3.9. -Use :func:`wave.open` instead. -(Contributed by Brian Curtin in :issue:`31985`.) - - -Deprecated functions and types of the C API -=========================================== - -Function :c:func:`PySlice_GetIndicesEx` is deprecated and replaced with -a macro if ``Py_LIMITED_API`` is not set or set to a value in the range -between ``0x03050400`` and ``0x03060000`` (not inclusive), or is ``0x03060100`` -or higher. (Contributed by Serhiy Storchaka in :issue:`27867`.) - -:c:func:`PyOS_AfterFork` has been deprecated. Use :c:func:`PyOS_BeforeFork`, -:c:func:`PyOS_AfterFork_Parent` or :c:func:`PyOS_AfterFork_Child()` instead. -(Contributed by Antoine Pitrou in :issue:`16500`.) - - -.. _37-platform-support-removals: - -Platform Support Removals -========================= - -* FreeBSD 9 and older are no longer officially supported. -* For full Unicode support, including within extension modules, \*nix platforms - are now expected to provide at least one of ``C.UTF-8`` (full locale), - ``C.utf8`` (full locale) or ``UTF-8`` (``LC_CTYPE``-only locale) as an - alternative to the legacy ``ASCII``-based ``C`` locale. -* OpenSSL 0.9.8 and 1.0.1 are no longer supported, which means building CPython - 3.7 with SSL/TLS support on older platforms still using these versions - requires custom build options that link to a more recent version of OpenSSL. - - Notably, this issue affects the Debian 8 (aka "jessie") and Ubuntu 14.04 - (aka "Trusty") LTS Linux distributions, as they still use OpenSSL 1.0.1 by - default. - - Debian 9 ("stretch") and Ubuntu 16.04 ("xenial"), as well as recent releases - of other LTS Linux releases (e.g. RHEL/CentOS 7.5, SLES 12-SP3), use OpenSSL - 1.0.2 or later, and remain supported in the default build configuration. - - CPython's own :source:`CI configuration file <.travis.yml>` provides an - example of using the SSL - :source:`compatibility testing infrastructure <Tools/ssl/multissltests.py>` in - CPython's test suite to build and link against OpenSSL 1.1.0 rather than an - outdated system provided OpenSSL. - - -API and Feature Removals -======================== - -The following features and APIs have been removed from Python 3.7: - -* The ``os.stat_float_times()`` function has been removed. It was introduced in - Python 2.3 for backward compatibility with Python 2.2, and was deprecated - since Python 3.1. - -* Unknown escapes consisting of ``'\'`` and an ASCII letter in replacement - templates for :func:`re.sub` were deprecated in Python 3.5, and will now - cause an error. - -* Removed support of the *exclude* argument in :meth:`tarfile.TarFile.add`. - It was deprecated in Python 2.7 and 3.2. Use the *filter* argument instead. - -* The ``splitunc()`` function in the :mod:`ntpath` module was deprecated in - Python 3.1, and has now been removed. Use the :func:`~os.path.splitdrive` - function instead. - -* :func:`collections.namedtuple` no longer supports the *verbose* parameter - or ``_source`` attribute which showed the generated source code for the - named tuple class. This was part of an optimization designed to speed-up - class creation. (Contributed by Jelle Zijlstra with further improvements - by INADA Naoki, Serhiy Storchaka, and Raymond Hettinger in :issue:`28638`.) - -* Functions :func:`bool`, :func:`float`, :func:`list` and :func:`tuple` no - longer take keyword arguments. The first argument of :func:`int` can now - be passed only as positional argument. - -* Removed previously deprecated in Python 2.4 classes ``Plist``, ``Dict`` and - ``_InternalDict`` in the :mod:`plistlib` module. Dict values in the result - of functions :func:`~plistlib.readPlist` and - :func:`~plistlib.readPlistFromBytes` are now normal dicts. You no longer - can use attribute access to access items of these dictionaries. - -* The ``asyncio.windows_utils.socketpair()`` function has been - removed. Use the :func:`socket.socketpair` function instead, - it is available on all platforms since Python 3.5. - ``asyncio.windows_utils.socketpair`` was just an alias to - ``socket.socketpair`` on Python 3.5 and newer. - -* :mod:`asyncio` no longer exports the :mod:`selectors` and - :mod:`_overlapped` modules as ``asyncio.selectors`` and - ``asyncio._overlapped``. Replace ``from asyncio import selectors`` with - ``import selectors``. - -* Direct instantiation of :class:`ssl.SSLSocket` and :class:`ssl.SSLObject` - objects is now prohibited. The constructors were never documented, tested, - or designed as public constructors. Users were supposed to use - :func:`ssl.wrap_socket` or :class:`ssl.SSLContext`. - (Contributed by Christian Heimes in :issue:`32951`.) - -* The unused :mod:`distutils` ``install_misc`` command has been removed. - (Contributed by Eric N. Vander Weele in :issue:`29218`.) - - -Module Removals -=============== - -The ``fpectl`` module has been removed. It was never enabled by -default, never worked correctly on x86-64, and it changed the Python -ABI in ways that caused unexpected breakage of C extensions. -(Contributed by Nathaniel J. Smith in :issue:`29137`.) - - -Windows-only Changes -==================== - -The python launcher, (py.exe), can accept 32 & 64 bit specifiers **without** -having to specify a minor version as well. So ``py -3-32`` and ``py -3-64`` -become valid as well as ``py -3.7-32``, also the -*m*-64 and -*m.n*-64 forms -are now accepted to force 64 bit python even if 32 bit would have otherwise -been used. If the specified version is not available py.exe will error exit. -(Contributed by Steve Barnes in :issue:`30291`.) - -The launcher can be run as ``py -0`` to produce a list of the installed pythons, -*with default marked with an asterisk*. Running ``py -0p`` will include the paths. -If py is run with a version specifier that cannot be matched it will also print -the *short form* list of available specifiers. -(Contributed by Steve Barnes in :issue:`30362`.) - - -.. _porting-to-python-37: - -Porting to Python 3.7 -===================== - -This section lists previously described changes and other bugfixes -that may require changes to your code. - - -Changes in Python Behavior --------------------------- - -* :keyword:`async` and :keyword:`await` names are now reserved keywords. - Code using these names as identifiers will now raise a :exc:`SyntaxError`. - (Contributed by Jelle Zijlstra in :issue:`30406`.) - -* :pep:`479` is enabled for all code in Python 3.7, meaning that - :exc:`StopIteration` exceptions raised directly or indirectly in - coroutines and generators are transformed into :exc:`RuntimeError` - exceptions. - (Contributed by Yury Selivanov in :issue:`32670`.) - -* :meth:`object.__aiter__` methods can no longer be declared as - asynchronous. (Contributed by Yury Selivanov in :issue:`31709`.) - -* Due to an oversight, earlier Python versions erroneously accepted the - following syntax:: - - f(1 for x in [1],) - - class C(1 for x in [1]): - pass - - Python 3.7 now correctly raises a :exc:`SyntaxError`, as a generator - expression always needs to be directly inside a set of parentheses - and cannot have a comma on either side, and the duplication of the - parentheses can be omitted only on calls. - (Contributed by Serhiy Storchaka in :issue:`32012` and :issue:`32023`.) - -* When using the :option:`-m` switch, the initial working directory is now added - to :data:`sys.path`, rather than an empty string (which dynamically denoted - the current working directory at the time of each import). Any programs that - are checking for the empty string, or otherwise relying on the previous - behaviour, will need to be updated accordingly (e.g. by also checking for - ``os.getcwd()`` or ``os.path.dirname(__main__.__file__)``, depending on why - the code was checking for the empty string in the first place). - - -Changes in the Python API -------------------------- - -* :meth:`socketserver.ThreadingMixIn.server_close` now waits until all - non-daemon threads complete. Set the new - :attr:`socketserver.ThreadingMixIn.block_on_close` class attribute to - ``False`` to get the pre-3.7 behaviour. - (Contributed by Victor Stinner in :issue:`31233` and :issue:`33540`.) - -* :meth:`socketserver.ForkingMixIn.server_close` now waits until all - child processes complete. Set the new - :attr:`socketserver.ForkingMixIn.block_on_close` class attribute to ``False`` - to get the pre-3.7 behaviour. - (Contributed by Victor Stinner in :issue:`31151` and :issue:`33540`.) - -* The :func:`locale.localeconv` function now temporarily sets the ``LC_CTYPE`` - locale to the value of ``LC_NUMERIC`` in some cases. - (Contributed by Victor Stinner in :issue:`31900`.) - -* :meth:`pkgutil.walk_packages` now raises a :exc:`ValueError` if *path* is - a string. Previously an empty list was returned. - (Contributed by Sanyam Khurana in :issue:`24744`.) - -* A format string argument for :meth:`string.Formatter.format` - is now :ref:`positional-only <positional-only_parameter>`. - Passing it as a keyword argument was deprecated in Python 3.5. (Contributed - by Serhiy Storchaka in :issue:`29193`.) - -* Attributes :attr:`~http.cookies.Morsel.key`, - :attr:`~http.cookies.Morsel.value` and - :attr:`~http.cookies.Morsel.coded_value` of class - :class:`http.cookies.Morsel` are now read-only. - Assigning to them was deprecated in Python 3.5. - Use the :meth:`~http.cookies.Morsel.set` method for setting them. - (Contributed by Serhiy Storchaka in :issue:`29192`.) - -* The *mode* argument of :func:`os.makedirs` no longer affects the file - permission bits of newly-created intermediate-level directories. - To set their file permission bits you can set the umask before invoking - ``makedirs()``. - (Contributed by Serhiy Storchaka in :issue:`19930`.) - -* The :attr:`struct.Struct.format` type is now :class:`str` instead of - :class:`bytes`. (Contributed by Victor Stinner in :issue:`21071`.) - -* :func:`~cgi.parse_multipart` now accepts the *encoding* and *errors* - arguments and returns the same results as - :class:`~FieldStorage`: for non-file fields, the value associated to a key - is a list of strings, not bytes. - (Contributed by Pierre Quentel in :issue:`29979`.) - -* Due to internal changes in :mod:`socket`, calling :func:`socket.fromshare` - on a socket created by :func:`socket.share <socket.socket.share>` in older - Python versions is not supported. - -* ``repr`` for :exc:`BaseException` has changed to not include the trailing - comma. Most exceptions are affected by this change. - (Contributed by Serhiy Storchaka in :issue:`30399`.) - -* ``repr`` for :class:`datetime.timedelta` has changed to include the keyword - arguments in the output. (Contributed by Utkarsh Upadhyay in :issue:`30302`.) - -* Because :func:`shutil.rmtree` is now implemented using the :func:`os.scandir` - function, the user specified handler *onerror* is now called with the first - argument ``os.scandir`` instead of ``os.listdir`` when listing the directory - is failed. - -* Support for nested sets and set operations in regular expressions as in - `Unicode Technical Standard #18`_ might be added in the future. This would - change the syntax. To facilitate this future change a :exc:`FutureWarning` - will be raised in ambiguous cases for the time being. - That include sets starting with a literal ``'['`` or containing literal - character sequences ``'--'``, ``'&&'``, ``'~~'``, and ``'||'``. To - avoid a warning, escape them with a backslash. - (Contributed by Serhiy Storchaka in :issue:`30349`.) - - .. _Unicode Technical Standard #18: https://unicode.org/reports/tr18/ - -* The result of splitting a string on a :mod:`regular expression <re>` - that could match an empty string has been changed. For example - splitting on ``r'\s*'`` will now split not only on whitespaces as it - did previously, but also on empty strings before all non-whitespace - characters and just before the end of the string. - The previous behavior can be restored by changing the pattern - to ``r'\s+'``. A :exc:`FutureWarning` was emitted for such patterns since - Python 3.5. - - For patterns that match both empty and non-empty strings, the result of - searching for all matches may also be changed in other cases. For example - in the string ``'a\n\n'``, the pattern ``r'(?m)^\s*?$'`` will not only - match empty strings at positions 2 and 3, but also the string ``'\n'`` at - positions 2--3. To match only blank lines, the pattern should be rewritten - as ``r'(?m)^[^\S\n]*$'``. - - :func:`re.sub()` now replaces empty matches adjacent to a previous - non-empty match. For example ``re.sub('x*', '-', 'abxd')`` returns now - ``'-a-b--d-'`` instead of ``'-a-b-d-'`` (the first minus between 'b' and - 'd' replaces 'x', and the second minus replaces an empty string between - 'x' and 'd'). - - (Contributed by Serhiy Storchaka in :issue:`25054` and :issue:`32308`.) - -* Change :func:`re.escape` to only escape regex special characters instead - of escaping all characters other than ASCII letters, numbers, and ``'_'``. - (Contributed by Serhiy Storchaka in :issue:`29995`.) - -* :class:`tracemalloc.Traceback` frames are now sorted from oldest to most - recent to be more consistent with :mod:`traceback`. - (Contributed by Jesse Bakker in :issue:`32121`.) - -* On OSes that support :const:`socket.SOCK_NONBLOCK` or - :const:`socket.SOCK_CLOEXEC` bit flags, the - :attr:`socket.type <socket.socket.type>` no longer has them applied. - Therefore, checks like ``if sock.type == socket.SOCK_STREAM`` - work as expected on all platforms. - (Contributed by Yury Selivanov in :issue:`32331`.) - -* On Windows the default for the *close_fds* argument of - :class:`subprocess.Popen` was changed from :const:`False` to :const:`True` - when redirecting the standard handles. If you previously depended on handles - being inherited when using :class:`subprocess.Popen` with standard io - redirection, you will have to pass ``close_fds=False`` to preserve the - previous behaviour, or use - :attr:`STARTUPINFO.lpAttributeList <subprocess.STARTUPINFO.lpAttributeList>`. - -* :meth:`importlib.machinery.PathFinder.invalidate_caches` -- which implicitly - affects :func:`importlib.invalidate_caches` -- now deletes entries - in :data:`sys.path_importer_cache` which are set to ``None``. - (Contributed by Brett Cannon in :issue:`33169`.) - -* In :mod:`asyncio`, - :meth:`loop.sock_recv() <asyncio.loop.sock_recv>`, - :meth:`loop.sock_sendall() <asyncio.loop.sock_sendall>`, - :meth:`loop.sock_accept() <asyncio.loop.sock_accept>`, - :meth:`loop.getaddrinfo() <asyncio.loop.getaddrinfo>`, - :meth:`loop.getnameinfo() <asyncio.loop.getnameinfo>` - have been changed to be proper coroutine methods to match their - documentation. Previously, these methods returned :class:`asyncio.Future` - instances. - (Contributed by Yury Selivanov in :issue:`32327`.) - -* :attr:`asyncio.Server.sockets` now returns a copy of the internal list - of server sockets, instead of returning it directly. - (Contributed by Yury Selivanov in :issue:`32662`.) - -* :attr:`Struct.format <struct.Struct.format>` is now a :class:`str` instance - instead of a :class:`bytes` instance. - (Contributed by Victor Stinner in :issue:`21071`.) - -* :mod:`argparse` subparsers can now be made mandatory by passing ``required=True`` - to :meth:`ArgumentParser.add_subparsers() <argparse.ArgumentParser.add_subparsers>`. - (Contributed by Anthony Sottile in :issue:`26510`.) - -* :meth:`ast.literal_eval()` is now stricter. Addition and subtraction of - arbitrary numbers are no longer allowed. - (Contributed by Serhiy Storchaka in :issue:`31778`.) - -* :meth:`Calendar.itermonthdates <calendar.Calendar.itermonthdates>` - will now consistently raise an exception when a date falls outside of the - ``0001-01-01`` through ``9999-12-31`` range. To support applications that - cannot tolerate such exceptions, the new - :meth:`Calendar.itermonthdays3 <calendar.Calendar.itermonthdays3>` and - :meth:`Calendar.itermonthdays4 <calendar.Calendar.itermonthdays4>` can be used. - The new methods return tuples and are not restricted by the range supported by - :class:`datetime.date`. - (Contributed by Alexander Belopolsky in :issue:`28292`.) - -* :class:`collections.ChainMap` now preserves the order of the underlying - mappings. (Contributed by Raymond Hettinger in :issue:`32792`.) - -* The ``submit()`` method of :class:`concurrent.futures.ThreadPoolExecutor` - and :class:`concurrent.futures.ProcessPoolExecutor` now raises - a :exc:`RuntimeError` if called during interpreter shutdown. - (Contributed by Mark Nemec in :issue:`33097`.) - -* The :class:`configparser.ConfigParser` constructor now uses ``read_dict()`` - to process the default values, making its behavior consistent with the - rest of the parser. Non-string keys and values in the defaults - dictionary are now being implicitly converted to strings. - (Contributed by James Tocknell in :issue:`23835`.) - -* Several undocumented internal imports were removed. - One example is that ``os.errno`` is no longer available; use ``import errno`` - directly instead. - Note that such undocumented internal imports may be removed any time without - notice, even in micro version releases. - - -Changes in the C API --------------------- - -The function :c:func:`PySlice_GetIndicesEx` is considered unsafe for -resizable sequences. If the slice indices are not instances of :class:`int`, -but objects that implement the :meth:`!__index__` method, the sequence can be -resized after passing its length to :c:func:`!PySlice_GetIndicesEx`. This -can lead to returning indices out of the length of the sequence. For -avoiding possible problems use new functions :c:func:`PySlice_Unpack` and -:c:func:`PySlice_AdjustIndices`. -(Contributed by Serhiy Storchaka in :issue:`27867`.) - - -CPython bytecode changes ------------------------- - -There are two new opcodes: :opcode:`LOAD_METHOD` and :opcode:`CALL_METHOD`. -(Contributed by Yury Selivanov and INADA Naoki in :issue:`26110`.) - -The :opcode:`STORE_ANNOTATION` opcode has been removed. -(Contributed by Mark Shannon in :issue:`32550`.) - - -Windows-only Changes --------------------- - -The file used to override :data:`sys.path` is now called -``<python-executable>._pth`` instead of ``'sys.path'``. -See :ref:`finding_modules` for more information. -(Contributed by Steve Dower in :issue:`28137`.) - - -Other CPython implementation changes ------------------------------------- - -In preparation for potential future changes to the public CPython runtime -initialization API (see :pep:`432` for an initial, but somewhat outdated, -draft), CPython's internal startup -and configuration management logic has been significantly refactored. While -these updates are intended to be entirely transparent to both embedding -applications and users of the regular CPython CLI, they're being mentioned -here as the refactoring changes the internal order of various operations -during interpreter startup, and hence may uncover previously latent defects, -either in embedding applications, or in CPython itself. -(Initially contributed by Nick Coghlan and Eric Snow as part of -:issue:`22257`, and further updated by Nick, Eric, and Victor Stinner in a -number of other issues). Some known details affected: - -* :c:func:`PySys_AddWarnOptionUnicode` is not currently usable by embedding - applications due to the requirement to create a Unicode object prior to - calling `Py_Initialize`. Use :c:func:`PySys_AddWarnOption` instead. - -* warnings filters added by an embedding application with - :c:func:`PySys_AddWarnOption` should now more consistently take precedence - over the default filters set by the interpreter - -Due to changes in the way the default warnings filters are configured, -setting :c:data:`Py_BytesWarningFlag` to a value greater than one is no longer -sufficient to both emit :exc:`BytesWarning` messages and have them converted -to exceptions. Instead, the flag must be set (to cause the warnings to be -emitted in the first place), and an explicit ``error::BytesWarning`` -warnings filter added to convert them to exceptions. - -Due to a change in the way docstrings are handled by the compiler, the -implicit ``return None`` in a function body consisting solely of a docstring -is now marked as occurring on the same line as the docstring, not on the -function's header line. - -The current exception state has been moved from the frame object to the co-routine. -This simplified the interpreter and fixed a couple of obscure bugs caused by -having swap exception state when entering or exiting a generator. -(Contributed by Mark Shannon in :issue:`25612`.) - -Notable changes in Python 3.7.1 -=============================== - -Starting in 3.7.1, :c:func:`Py_Initialize` now consistently reads and respects -all of the same environment settings as :c:func:`Py_Main` (in earlier Python -versions, it respected an ill-defined subset of those environment variables, -while in Python 3.7.0 it didn't read any of them due to :issue:`34247`). If -this behavior is unwanted, set :c:data:`Py_IgnoreEnvironmentFlag` to 1 before -calling :c:func:`Py_Initialize`. - -In 3.7.1 the C API for Context Variables -:ref:`was updated <contextvarsobjects_pointertype_change>` to use -:c:type:`PyObject` pointers. See also :issue:`34762`. - -In 3.7.1 the :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token -when provided with input that does not have a trailing new line. This behavior -now matches what the C tokenizer does internally. -(Contributed by Ammar Askar in :issue:`33899`.) - -Notable changes in Python 3.7.2 -=============================== - -In 3.7.2, :mod:`venv` on Windows no longer copies the original binaries, but -creates redirector scripts named ``python.exe`` and ``pythonw.exe`` instead. -This resolves a long standing issue where all virtual environments would have -to be upgraded or recreated with each Python update. However, note that this -release will still require recreation of virtual environments in order to get -the new scripts. diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst deleted file mode 100644 index c71c1ab..0000000 --- a/Doc/whatsnew/3.8.rst +++ /dev/null @@ -1,2206 +0,0 @@ -**************************** - What's New In Python 3.8 -**************************** - -.. Rules for maintenance: - - * Anyone can add text to this document. Do not spend very much time - on the wording of your changes, because your text will probably - get rewritten to some degree. - - * The maintainer will go through Misc/NEWS periodically and add - changes; it's therefore more important to add your changes to - Misc/NEWS than to this file. - - * This is not a complete list of every single change; completeness - is the purpose of Misc/NEWS. Some changes I consider too small - or esoteric to include. If such a change is added to the text, - I'll just remove it. (This is another reason you shouldn't spend - too much time on writing your addition.) - - * If you want to draw your new text to the attention of the - maintainer, add 'XXX' to the beginning of the paragraph or - section. - - * It's OK to just add a fragmentary note about a change. For - example: "XXX Describe the transmogrify() function added to the - socket module." The maintainer will research the change and - write the necessary text. - - * You can comment out your additions if you like, but it's not - necessary (especially when a final release is some months away). - - * Credit the author of a patch or bugfix. Just the name is - sufficient; the e-mail address isn't necessary. - - * It's helpful to add the bug/patch number as a comment: - - XXX Describe the transmogrify() function added to the socket - module. - (Contributed by P.Y. Developer in :issue:`12345`.) - - This saves the maintainer the effort of going through the Git log - when researching a change. - -:Editor: Raymond Hettinger - -This article explains the new features in Python 3.8, compared to 3.7. -For full details, see the :ref:`changelog <changelog>`. - -.. testsetup:: - - from datetime import date - from math import cos, radians - from unicodedata import normalize - import re - import math - - -Summary -- Release highlights -============================= - -.. This section singles out the most important changes in Python 3.8. - Brevity is key. - - -.. PEP-sized items next. - - - -New Features -============ - -Assignment expressions ----------------------- - -There is new syntax ``:=`` that assigns values to variables as part of a larger -expression. It is affectionately known as "the walrus operator" due to -its resemblance to `the eyes and tusks of a walrus -<https://en.wikipedia.org/wiki/Walrus#/media/File:Pacific_Walrus_-_Bull_(8247646168).jpg>`_. - -In this example, the assignment expression helps avoid calling -:func:`len` twice:: - - if (n := len(a)) > 10: - print(f"List is too long ({n} elements, expected <= 10)") - -A similar benefit arises during regular expression matching where -match objects are needed twice, once to test whether a match -occurred and another to extract a subgroup:: - - discount = 0.0 - if (mo := re.search(r'(\d+)% discount', advertisement)): - discount = float(mo.group(1)) / 100.0 - -The operator is also useful with while-loops that compute -a value to test loop termination and then need that same -value again in the body of the loop:: - - # Loop over fixed length blocks - while (block := f.read(256)) != '': - process(block) - -Another motivating use case arises in list comprehensions where -a value computed in a filtering condition is also needed in -the expression body:: - - [clean_name.title() for name in names - if (clean_name := normalize('NFC', name)) in allowed_names] - -Try to limit use of the walrus operator to clean cases that reduce -complexity and improve readability. - -See :pep:`572` for a full description. - -(Contributed by Emily Morehouse in :issue:`35224`.) - - -Positional-only parameters --------------------------- - -There is a new function parameter syntax ``/`` to indicate that some -function parameters must be specified positionally and cannot be used as -keyword arguments. This is the same notation shown by ``help()`` for C -functions annotated with Larry Hastings' `Argument Clinic -<https://docs.python.org/3/howto/clinic.html>`_ tool. - -In the following example, parameters *a* and *b* are positional-only, -while *c* or *d* can be positional or keyword, and *e* or *f* are -required to be keywords:: - - def f(a, b, /, c, d, *, e, f): - print(a, b, c, d, e, f) - -The following is a valid call:: - - f(10, 20, 30, d=40, e=50, f=60) - -However, these are invalid calls:: - - f(10, b=20, c=30, d=40, e=50, f=60) # b cannot be a keyword argument - f(10, 20, 30, 40, 50, f=60) # e must be a keyword argument - -One use case for this notation is that it allows pure Python functions -to fully emulate behaviors of existing C coded functions. For example, -the built-in :func:`pow` function does not accept keyword arguments:: - - def pow(x, y, z=None, /): - "Emulate the built in pow() function" - r = x ** y - return r if z is None else r%z - -Another use case is to preclude keyword arguments when the parameter -name is not helpful. For example, the builtin :func:`len` function has -the signature ``len(obj, /)``. This precludes awkward calls such as:: - - len(obj='hello') # The "obj" keyword argument impairs readability - -A further benefit of marking a parameter as positional-only is that it -allows the parameter name to be changed in the future without risk of -breaking client code. For example, in the :mod:`statistics` module, the -parameter name *dist* may be changed in the future. This was made -possible with the following function specification:: - - def quantiles(dist, /, *, n=4, method='exclusive') - ... - -Since the parameters to the left of ``/`` are not exposed as possible -keywords, the parameters names remain available for use in ``**kwargs``:: - - >>> def f(a, b, /, **kwargs): - ... print(a, b, kwargs) - ... - >>> f(10, 20, a=1, b=2, c=3) # a and b are used in two ways - 10 20 {'a': 1, 'b': 2, 'c': 3} - -This greatly simplifies the implementation of functions and methods -that need to accept arbitrary keyword arguments. For example, here -is an excerpt from code in the :mod:`collections` module:: - - class Counter(dict): - - def __init__(self, iterable=None, /, **kwds): - # Note "iterable" is a possible keyword argument - -See :pep:`570` for a full description. - -(Contributed by Pablo Galindo in :issue:`36540`.) - -.. TODO: Pablo will sprint on docs at PyCon US 2019. - - -Parallel filesystem cache for compiled bytecode files ------------------------------------------------------ - -The new :envvar:`PYTHONPYCACHEPREFIX` setting (also available as -:option:`-X` ``pycache_prefix``) configures the implicit bytecode -cache to use a separate parallel filesystem tree, rather than -the default ``__pycache__`` subdirectories within each source -directory. - -The location of the cache is reported in :data:`sys.pycache_prefix` -(:const:`None` indicates the default location in ``__pycache__`` -subdirectories). - -(Contributed by Carl Meyer in :issue:`33499`.) - - -Debug build uses the same ABI as release build ------------------------------------------------ - -Python now uses the same ABI whether it's built in release or debug mode. On -Unix, when Python is built in debug mode, it is now possible to load C -extensions built in release mode and C extensions built using the stable ABI. - -Release builds and debug builds are now ABI compatible: defining the -``Py_DEBUG`` macro no longer implies the ``Py_TRACE_REFS`` macro, which -introduces the only ABI incompatibility. The ``Py_TRACE_REFS`` macro, which -adds the :func:`sys.getobjects` function and the :envvar:`PYTHONDUMPREFS` -environment variable, can be set using the new ``./configure --with-trace-refs`` -build option. -(Contributed by Victor Stinner in :issue:`36465`.) - -On Unix, C extensions are no longer linked to libpython except on Android -and Cygwin. -It is now possible -for a statically linked Python to load a C extension built using a shared -library Python. -(Contributed by Victor Stinner in :issue:`21536`.) - -On Unix, when Python is built in debug mode, import now also looks for C -extensions compiled in release mode and for C extensions compiled with the -stable ABI. -(Contributed by Victor Stinner in :issue:`36722`.) - -To embed Python into an application, a new ``--embed`` option must be passed to -``python3-config --libs --embed`` to get ``-lpython3.8`` (link the application -to libpython). To support both 3.8 and older, try ``python3-config --libs ---embed`` first and fallback to ``python3-config --libs`` (without ``--embed``) -if the previous command fails. - -Add a pkg-config ``python-3.8-embed`` module to embed Python into an -application: ``pkg-config python-3.8-embed --libs`` includes ``-lpython3.8``. -To support both 3.8 and older, try ``pkg-config python-X.Y-embed --libs`` first -and fallback to ``pkg-config python-X.Y --libs`` (without ``--embed``) if the -previous command fails (replace ``X.Y`` with the Python version). - -On the other hand, ``pkg-config python3.8 --libs`` no longer contains -``-lpython3.8``. C extensions must not be linked to libpython (except on -Android and Cygwin, whose cases are handled by the script); -this change is backward incompatible on purpose. -(Contributed by Victor Stinner in :issue:`36721`.) - - -f-strings support ``=`` for self-documenting expressions and debugging ----------------------------------------------------------------------- - -Added an ``=`` specifier to :term:`f-string`\s. An f-string such as -``f'{expr=}'`` will expand to the text of the expression, an equal sign, -then the representation of the evaluated expression. For example: - - >>> user = 'eric_idle' - >>> member_since = date(1975, 7, 31) - >>> f'{user=} {member_since=}' - "user='eric_idle' member_since=datetime.date(1975, 7, 31)" - -The usual :ref:`f-string format specifiers <f-strings>` allow more -control over how the result of the expression is displayed:: - - >>> delta = date.today() - member_since - >>> f'{user=!s} {delta.days=:,d}' - 'user=eric_idle delta.days=16,075' - -The ``=`` specifier will display the whole expression so that -calculations can be shown:: - - >>> print(f'{theta=} {cos(radians(theta))=:.3f}') - theta=30 cos(radians(theta))=0.866 - -(Contributed by Eric V. Smith and Larry Hastings in :issue:`36817`.) - - -PEP 578: Python Runtime Audit Hooks ------------------------------------ - -The PEP adds an Audit Hook and Verified Open Hook. Both are available from -Python and native code, allowing applications and frameworks written in pure -Python code to take advantage of extra notifications, while also allowing -embedders or system administrators to deploy builds of Python where auditing is -always enabled. - -See :pep:`578` for full details. - - -PEP 587: Python Initialization Configuration --------------------------------------------- - -The :pep:`587` adds a new C API to configure the Python Initialization -providing finer control on the whole configuration and better error reporting. - -New structures: - -* :c:type:`PyConfig` -* :c:type:`PyPreConfig` -* :c:type:`PyStatus` -* :c:type:`PyWideStringList` - -New functions: - -* :c:func:`PyConfig_Clear` -* :c:func:`PyConfig_InitIsolatedConfig` -* :c:func:`PyConfig_InitPythonConfig` -* :c:func:`PyConfig_Read` -* :c:func:`PyConfig_SetArgv` -* :c:func:`PyConfig_SetBytesArgv` -* :c:func:`PyConfig_SetBytesString` -* :c:func:`PyConfig_SetString` -* :c:func:`PyPreConfig_InitIsolatedConfig` -* :c:func:`PyPreConfig_InitPythonConfig` -* :c:func:`PyStatus_Error` -* :c:func:`PyStatus_Exception` -* :c:func:`PyStatus_Exit` -* :c:func:`PyStatus_IsError` -* :c:func:`PyStatus_IsExit` -* :c:func:`PyStatus_NoMemory` -* :c:func:`PyStatus_Ok` -* :c:func:`PyWideStringList_Append` -* :c:func:`PyWideStringList_Insert` -* :c:func:`Py_BytesMain` -* :c:func:`Py_ExitStatusException` -* :c:func:`Py_InitializeFromConfig` -* :c:func:`Py_PreInitialize` -* :c:func:`Py_PreInitializeFromArgs` -* :c:func:`Py_PreInitializeFromBytesArgs` -* :c:func:`Py_RunMain` - -This PEP also adds ``_PyRuntimeState.preconfig`` (:c:type:`PyPreConfig` type) -and ``PyInterpreterState.config`` (:c:type:`PyConfig` type) fields to these -internal structures. ``PyInterpreterState.config`` becomes the new -reference configuration, replacing global configuration variables and -other private variables. - -See :ref:`Python Initialization Configuration <init-config>` for the -documentation. - -See :pep:`587` for a full description. - -(Contributed by Victor Stinner in :issue:`36763`.) - - -PEP 590: Vectorcall: a fast calling protocol for CPython --------------------------------------------------------- - -:ref:`vectorcall` is added to the Python/C API. -It is meant to formalize existing optimizations which were already done -for various classes. -Any static type implementing a callable can use this protocol. - -This is currently provisional. -The aim is to make it fully public in Python 3.9. - -See :pep:`590` for a full description. - -(Contributed by Jeroen Demeyer, Mark Shannon and Petr Viktorin in :issue:`36974`.) - - -Pickle protocol 5 with out-of-band data buffers ------------------------------------------------ - -When :mod:`pickle` is used to transfer large data between Python processes -in order to take advantage of multi-core or multi-machine processing, -it is important to optimize the transfer by reducing memory copies, and -possibly by applying custom techniques such as data-dependent compression. - -The :mod:`pickle` protocol 5 introduces support for out-of-band buffers -where :pep:`3118`-compatible data can be transmitted separately from the -main pickle stream, at the discretion of the communication layer. - -See :pep:`574` for a full description. - -(Contributed by Antoine Pitrou in :issue:`36785`.) - - -Other Language Changes -====================== - -* A :keyword:`continue` statement was illegal in the :keyword:`finally` clause - due to a problem with the implementation. In Python 3.8 this restriction - was lifted. - (Contributed by Serhiy Storchaka in :issue:`32489`.) - -* The :class:`bool`, :class:`int`, and :class:`fractions.Fraction` types - now have an :meth:`~int.as_integer_ratio` method like that found in - :class:`float` and :class:`decimal.Decimal`. This minor API extension - makes it possible to write ``numerator, denominator = - x.as_integer_ratio()`` and have it work across multiple numeric types. - (Contributed by Lisa Roach in :issue:`33073` and Raymond Hettinger in - :issue:`37819`.) - -* Constructors of :class:`int`, :class:`float` and :class:`complex` will now - use the :meth:`~object.__index__` special method, if available and the - corresponding method :meth:`~object.__int__`, :meth:`~object.__float__` - or :meth:`~object.__complex__` is not available. - (Contributed by Serhiy Storchaka in :issue:`20092`.) - -* Added support of ``\N{name}`` escapes in :mod:`regular expressions <re>`:: - - >>> notice = 'Copyright © 2019' - >>> copyright_year_pattern = re.compile(r'\N{copyright sign}\s*(\d{4})') - >>> int(copyright_year_pattern.search(notice).group(1)) - 2019 - - (Contributed by Jonathan Eunice and Serhiy Storchaka in :issue:`30688`.) - -* Dict and dictviews are now iterable in reversed insertion order using - :func:`reversed`. (Contributed by Rémi Lapeyre in :issue:`33462`.) - -* The syntax allowed for keyword names in function calls was further - restricted. In particular, ``f((keyword)=arg)`` is no longer allowed. It was - never intended to permit more than a bare name on the left-hand side of a - keyword argument assignment term. - (Contributed by Benjamin Peterson in :issue:`34641`.) - -* Generalized iterable unpacking in :keyword:`yield` and - :keyword:`return` statements no longer requires enclosing parentheses. - This brings the *yield* and *return* syntax into better agreement with - normal assignment syntax:: - - >>> def parse(family): - lastname, *members = family.split() - return lastname.upper(), *members - - >>> parse('simpsons homer marge bart lisa sally') - ('SIMPSONS', 'homer', 'marge', 'bart', 'lisa', 'sally') - - (Contributed by David Cuthbert and Jordan Chapman in :issue:`32117`.) - -* When a comma is missed in code such as ``[(10, 20) (30, 40)]``, the - compiler displays a :exc:`SyntaxWarning` with a helpful suggestion. - This improves on just having a :exc:`TypeError` indicating that the - first tuple was not callable. (Contributed by Serhiy Storchaka in - :issue:`15248`.) - -* Arithmetic operations between subclasses of :class:`datetime.date` or - :class:`datetime.datetime` and :class:`datetime.timedelta` objects now return - an instance of the subclass, rather than the base class. This also affects - the return type of operations whose implementation (directly or indirectly) - uses :class:`datetime.timedelta` arithmetic, such as - :meth:`~datetime.datetime.astimezone`. - (Contributed by Paul Ganssle in :issue:`32417`.) - -* When the Python interpreter is interrupted by Ctrl-C (SIGINT) and the - resulting :exc:`KeyboardInterrupt` exception is not caught, the Python process - now exits via a SIGINT signal or with the correct exit code such that the - calling process can detect that it died due to a Ctrl-C. Shells on POSIX - and Windows use this to properly terminate scripts in interactive sessions. - (Contributed by Google via Gregory P. Smith in :issue:`1054041`.) - -* Some advanced styles of programming require updating the - :class:`types.CodeType` object for an existing function. Since code - objects are immutable, a new code object needs to be created, one - that is modeled on the existing code object. With 19 parameters, - this was somewhat tedious. Now, the new ``replace()`` method makes - it possible to create a clone with a few altered parameters. - - Here's an example that alters the :func:`statistics.mean` function to - prevent the *data* parameter from being used as a keyword argument:: - - >>> from statistics import mean - >>> mean(data=[10, 20, 90]) - 40 - >>> mean.__code__ = mean.__code__.replace(co_posonlyargcount=1) - >>> mean(data=[10, 20, 90]) - Traceback (most recent call last): - ... - TypeError: mean() got some positional-only arguments passed as keyword arguments: 'data' - - (Contributed by Victor Stinner in :issue:`37032`.) - -* For integers, the three-argument form of the :func:`pow` function now - permits the exponent to be negative in the case where the base is - relatively prime to the modulus. It then computes a modular inverse to - the base when the exponent is ``-1``, and a suitable power of that - inverse for other negative exponents. For example, to compute the - `modular multiplicative inverse - <https://en.wikipedia.org/wiki/Modular_multiplicative_inverse>`_ of 38 - modulo 137, write:: - - >>> pow(38, -1, 137) - 119 - >>> 119 * 38 % 137 - 1 - - Modular inverses arise in the solution of `linear Diophantine - equations <https://en.wikipedia.org/wiki/Diophantine_equation>`_. - For example, to find integer solutions for ``4258𝑥 + 147𝑦 = 369``, - first rewrite as ``4258𝑥 ≡ 369 (mod 147)`` then solve: - - >>> x = 369 * pow(4258, -1, 147) % 147 - >>> y = (4258 * x - 369) // -147 - >>> 4258 * x + 147 * y - 369 - - (Contributed by Mark Dickinson in :issue:`36027`.) - -* Dict comprehensions have been synced-up with dict literals so that the - key is computed first and the value second:: - - >>> # Dict comprehension - >>> cast = {input('role? '): input('actor? ') for i in range(2)} - role? King Arthur - actor? Chapman - role? Black Knight - actor? Cleese - - >>> # Dict literal - >>> cast = {input('role? '): input('actor? ')} - role? Sir Robin - actor? Eric Idle - - The guaranteed execution order is helpful with assignment expressions - because variables assigned in the key expression will be available in - the value expression:: - - >>> names = ['Martin von Löwis', 'Łukasz Langa', 'Walter Dörwald'] - >>> {(n := normalize('NFC', name)).casefold() : n for name in names} - {'martin von löwis': 'Martin von Löwis', - 'łukasz langa': 'Łukasz Langa', - 'walter dörwald': 'Walter Dörwald'} - - (Contributed by Jörn Heissler in :issue:`35224`.) - -* The :meth:`object.__reduce__` method can now return a tuple from two to - six elements long. Formerly, five was the limit. The new, optional sixth - element is a callable with a ``(obj, state)`` signature. This allows the - direct control over the state-updating behavior of a specific object. If - not *None*, this callable will have priority over the object's - :meth:`~__setstate__` method. - (Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.) - -New Modules -=========== - -* The new :mod:`importlib.metadata` module provides (provisional) support for - reading metadata from third-party packages. For example, it can extract an - installed package's version number, list of entry points, and more:: - - >>> # Note following example requires that the popular "requests" - >>> # package has been installed. - >>> - >>> from importlib.metadata import version, requires, files - >>> version('requests') - '2.22.0' - >>> list(requires('requests')) - ['chardet (<3.1.0,>=3.0.2)'] - >>> list(files('requests'))[:5] - [PackagePath('requests-2.22.0.dist-info/INSTALLER'), - PackagePath('requests-2.22.0.dist-info/LICENSE'), - PackagePath('requests-2.22.0.dist-info/METADATA'), - PackagePath('requests-2.22.0.dist-info/RECORD'), - PackagePath('requests-2.22.0.dist-info/WHEEL')] - - (Contributed by Barry Warsaw and Jason R. Coombs in :issue:`34632`.) - - -Improved Modules -================ - -ast ---- - -AST nodes now have ``end_lineno`` and ``end_col_offset`` attributes, -which give the precise location of the end of the node. (This only -applies to nodes that have ``lineno`` and ``col_offset`` attributes.) - -New function :func:`ast.get_source_segment` returns the source code -for a specific AST node. - -(Contributed by Ivan Levkivskyi in :issue:`33416`.) - -The :func:`ast.parse` function has some new flags: - -* ``type_comments=True`` causes it to return the text of :pep:`484` and - :pep:`526` type comments associated with certain AST nodes; - -* ``mode='func_type'`` can be used to parse :pep:`484` "signature type - comments" (returned for function definition AST nodes); - -* ``feature_version=(3, N)`` allows specifying an earlier Python 3 - version. For example, ``feature_version=(3, 4)`` will treat - :keyword:`async` and :keyword:`await` as non-reserved words. - -(Contributed by Guido van Rossum in :issue:`35766`.) - - -asyncio -------- - -:func:`asyncio.run` has graduated from the provisional to stable API. This -function can be used to execute a :term:`coroutine` and return the result while -automatically managing the event loop. For example:: - - import asyncio - - async def main(): - await asyncio.sleep(0) - return 42 - - asyncio.run(main()) - -This is *roughly* equivalent to:: - - import asyncio - - async def main(): - await asyncio.sleep(0) - return 42 - - loop = asyncio.new_event_loop() - asyncio.set_event_loop(loop) - try: - loop.run_until_complete(main()) - finally: - asyncio.set_event_loop(None) - loop.close() - - -The actual implementation is significantly more complex. Thus, -:func:`asyncio.run` should be the preferred way of running asyncio programs. - -(Contributed by Yury Selivanov in :issue:`32314`.) - -Running ``python -m asyncio`` launches a natively async REPL. This allows rapid -experimentation with code that has a top-level :keyword:`await`. There is no -longer a need to directly call ``asyncio.run()`` which would spawn a new event -loop on every invocation: - -.. code-block:: none - - $ python -m asyncio - asyncio REPL 3.8.0 - Use "await" directly instead of "asyncio.run()". - Type "help", "copyright", "credits" or "license" for more information. - >>> import asyncio - >>> await asyncio.sleep(10, result='hello') - hello - -(Contributed by Yury Selivanov in :issue:`37028`.) - -The exception :class:`asyncio.CancelledError` now inherits from -:class:`BaseException` rather than :class:`Exception`. -(Contributed by Yury Selivanov in :issue:`32528`.) - -On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`. -(Contributed by Victor Stinner in :issue:`34687`.) - -:class:`~asyncio.ProactorEventLoop` now also supports UDP. -(Contributed by Adam Meily and Andrew Svetlov in :issue:`29883`.) - -:class:`~asyncio.ProactorEventLoop` can now be interrupted by -:exc:`KeyboardInterrupt` ("CTRL+C"). -(Contributed by Vladimir Matveev in :issue:`23057`.) - -Added :meth:`asyncio.Task.get_coro` for getting the wrapped coroutine -within an :class:`asyncio.Task`. -(Contributed by Alex Grönholm in :issue:`36999`.) - -Asyncio tasks can now be named, either by passing the ``name`` keyword -argument to :func:`asyncio.create_task` or -the :meth:`~asyncio.loop.create_task` event loop method, or by -calling the :meth:`~asyncio.Task.set_name` method on the task object. The -task name is visible in the ``repr()`` output of :class:`asyncio.Task` and -can also be retrieved using the :meth:`~asyncio.Task.get_name` method. -(Contributed by Alex Grönholm in :issue:`34270`.) - -Added support for -`Happy Eyeballs <https://en.wikipedia.org/wiki/Happy_Eyeballs>`_ to -:func:`asyncio.loop.create_connection`. To specify the behavior, two new -parameters have been added: *happy_eyeballs_delay* and *interleave*. The Happy -Eyeballs algorithm improves responsiveness in applications that support IPv4 -and IPv6 by attempting to simultaneously connect using both. -(Contributed by twisteroid ambassador in :issue:`33530`.) - - -builtins --------- - -The :func:`compile` built-in has been improved to accept the -``ast.PyCF_ALLOW_TOP_LEVEL_AWAIT`` flag. With this new flag passed, -:func:`compile` will allow top-level ``await``, ``async for`` and ``async with`` -constructs that are usually considered invalid syntax. Asynchronous code object -marked with the ``CO_COROUTINE`` flag may then be returned. -(Contributed by Matthias Bussonnier in :issue:`34616`) - - -collections ------------ - -The :meth:`~collections.somenamedtuple._asdict` method for -:func:`collections.namedtuple` now returns a :class:`dict` instead of a -:class:`collections.OrderedDict`. This works because regular dicts have -guaranteed ordering since Python 3.7. If the extra features of -:class:`OrderedDict` are required, the suggested remediation is to cast the -result to the desired type: ``OrderedDict(nt._asdict())``. -(Contributed by Raymond Hettinger in :issue:`35864`.) - - -cProfile --------- - -The :class:`cProfile.Profile <profile.Profile>` class can now be used as a context manager. -Profile a block of code by running:: - - import cProfile - - with cProfile.Profile() as profiler: - # code to be profiled - ... - -(Contributed by Scott Sanderson in :issue:`29235`.) - - -csv ---- - -The :class:`csv.DictReader` now returns instances of :class:`dict` instead of -a :class:`collections.OrderedDict`. The tool is now faster and uses less -memory while still preserving the field order. -(Contributed by Michael Seek in :issue:`34003`.) - - -curses -------- - -Added a new variable holding structured version information for the -underlying ncurses library: :data:`~curses.ncurses_version`. -(Contributed by Serhiy Storchaka in :issue:`31680`.) - - -ctypes ------- - -On Windows, :class:`~ctypes.CDLL` and subclasses now accept a *winmode* parameter -to specify flags for the underlying ``LoadLibraryEx`` call. The default flags are -set to only load DLL dependencies from trusted locations, including the path -where the DLL is stored (if a full or partial path is used to load the initial -DLL) and paths added by :func:`~os.add_dll_directory`. -(Contributed by Steve Dower in :issue:`36085`.) - - -datetime --------- - -Added new alternate constructors :meth:`datetime.date.fromisocalendar` and -:meth:`datetime.datetime.fromisocalendar`, which construct :class:`date` and -:class:`datetime` objects respectively from ISO year, week number, and weekday; -these are the inverse of each class's ``isocalendar`` method. -(Contributed by Paul Ganssle in :issue:`36004`.) - - -functools ---------- - -:func:`functools.lru_cache` can now be used as a straight decorator rather -than as a function returning a decorator. So both of these are now supported:: - - @lru_cache - def f(x): - ... - - @lru_cache(maxsize=256) - def f(x): - ... - -(Contributed by Raymond Hettinger in :issue:`36772`.) - -Added a new :func:`functools.cached_property` decorator, for computed properties -cached for the life of the instance. :: - - import functools - import statistics - - class Dataset: - def __init__(self, sequence_of_numbers): - self.data = sequence_of_numbers - - @functools.cached_property - def variance(self): - return statistics.variance(self.data) - -(Contributed by Carl Meyer in :issue:`21145`) - - -Added a new :func:`functools.singledispatchmethod` decorator that converts -methods into :term:`generic functions <generic function>` using -:term:`single dispatch`:: - - from functools import singledispatchmethod - from contextlib import suppress - - class TaskManager: - - def __init__(self, tasks): - self.tasks = list(tasks) - - @singledispatchmethod - def discard(self, value): - with suppress(ValueError): - self.tasks.remove(value) - - @discard.register(list) - def _(self, tasks): - targets = set(tasks) - self.tasks = [x for x in self.tasks if x not in targets] - -(Contributed by Ethan Smith in :issue:`32380`) - -gc --- - -:func:`~gc.get_objects` can now receive an optional *generation* parameter -indicating a generation to get objects from. -(Contributed by Pablo Galindo in :issue:`36016`.) - - -gettext -------- - -Added :func:`~gettext.pgettext` and its variants. -(Contributed by Franz Glasner, Éric Araujo, and Cheryl Sabella in :issue:`2504`.) - - -gzip ----- - -Added the *mtime* parameter to :func:`gzip.compress` for reproducible output. -(Contributed by Guo Ci Teo in :issue:`34898`.) - -A :exc:`~gzip.BadGzipFile` exception is now raised instead of :exc:`OSError` -for certain types of invalid or corrupt gzip files. -(Contributed by Filip Gruszczyński, Michele Orrù, and Zackery Spytz in -:issue:`6584`.) - - -IDLE and idlelib ----------------- - -Output over N lines (50 by default) is squeezed down to a button. -N can be changed in the PyShell section of the General page of the -Settings dialog. Fewer, but possibly extra long, lines can be squeezed by -right clicking on the output. Squeezed output can be expanded in place -by double-clicking the button or into the clipboard or a separate window -by right-clicking the button. (Contributed by Tal Einat in :issue:`1529353`.) - -Add "Run Customized" to the Run menu to run a module with customized -settings. Any command line arguments entered are added to sys.argv. -They also re-appear in the box for the next customized run. One can also -suppress the normal Shell main module restart. (Contributed by Cheryl -Sabella, Terry Jan Reedy, and others in :issue:`5680` and :issue:`37627`.) - -Added optional line numbers for IDLE editor windows. Windows -open without line numbers unless set otherwise in the General -tab of the configuration dialog. Line numbers for an existing -window are shown and hidden in the Options menu. -(Contributed by Tal Einat and Saimadhav Heblikar in :issue:`17535`.) - -OS native encoding is now used for converting between Python strings and Tcl -objects. This allows IDLE to work with emoji and other non-BMP characters. -These characters can be displayed or copied and pasted to or from the -clipboard. Converting strings from Tcl to Python and back now never fails. -(Many people worked on this for eight years but the problem was finally -solved by Serhiy Storchaka in :issue:`13153`.) - -The changes above have been backported to 3.7 maintenance releases. - - -inspect -------- - -The :func:`inspect.getdoc` function can now find docstrings for ``__slots__`` -if that attribute is a :class:`dict` where the values are docstrings. -This provides documentation options similar to what we already have -for :func:`property`, :func:`classmethod`, and :func:`staticmethod`:: - - class AudioClip: - __slots__ = {'bit_rate': 'expressed in kilohertz to one decimal place', - 'duration': 'in seconds, rounded up to an integer'} - def __init__(self, bit_rate, duration): - self.bit_rate = round(bit_rate / 1000.0, 1) - self.duration = ceil(duration) - -(Contributed by Raymond Hettinger in :issue:`36326`.) - - -io --- - -In development mode (:option:`-X` ``env``) and in debug build, the -:class:`io.IOBase` finalizer now logs the exception if the ``close()`` method -fails. The exception is ignored silently by default in release build. -(Contributed by Victor Stinner in :issue:`18748`.) - - -itertools ---------- - -The :func:`itertools.accumulate` function added an option *initial* keyword -argument to specify an initial value:: - - >>> from itertools import accumulate - >>> list(accumulate([10, 5, 30, 15], initial=1000)) - [1000, 1010, 1015, 1045, 1060] - -(Contributed by Lisa Roach in :issue:`34659`.) - - -json.tool ---------- - -Add option ``--json-lines`` to parse every input line as a separate JSON object. -(Contributed by Weipeng Hong in :issue:`31553`.) - - -logging -------- - -Added a *force* keyword argument to :func:`logging.basicConfig()` -When set to true, any existing handlers attached -to the root logger are removed and closed before carrying out the -configuration specified by the other arguments. - -This solves a long-standing problem. Once a logger or *basicConfig()* had -been called, subsequent calls to *basicConfig()* were silently ignored. -This made it difficult to update, experiment with, or teach the various -logging configuration options using the interactive prompt or a Jupyter -notebook. - -(Suggested by Raymond Hettinger, implemented by Dong-hee Na, and -reviewed by Vinay Sajip in :issue:`33897`.) - - -math ----- - -Added new function :func:`math.dist` for computing Euclidean distance -between two points. (Contributed by Raymond Hettinger in :issue:`33089`.) - -Expanded the :func:`math.hypot` function to handle multiple dimensions. -Formerly, it only supported the 2-D case. -(Contributed by Raymond Hettinger in :issue:`33089`.) - -Added new function, :func:`math.prod`, as analogous function to :func:`sum` -that returns the product of a 'start' value (default: 1) times an iterable of -numbers:: - - >>> prior = 0.8 - >>> likelihoods = [0.625, 0.84, 0.30] - >>> math.prod(likelihoods, start=prior) - 0.126 - -(Contributed by Pablo Galindo in :issue:`35606`.) - -Added two new combinatoric functions :func:`math.perm` and :func:`math.comb`:: - - >>> math.perm(10, 3) # Permutations of 10 things taken 3 at a time - 720 - >>> math.comb(10, 3) # Combinations of 10 things taken 3 at a time - 120 - -(Contributed by Yash Aggarwal, Keller Fuchs, Serhiy Storchaka, and Raymond -Hettinger in :issue:`37128`, :issue:`37178`, and :issue:`35431`.) - -Added a new function :func:`math.isqrt` for computing accurate integer square -roots without conversion to floating point. The new function supports -arbitrarily large integers. It is faster than ``floor(sqrt(n))`` but slower -than :func:`math.sqrt`:: - - >>> r = 650320427 - >>> s = r ** 2 - >>> isqrt(s - 1) # correct - 650320426 - >>> floor(sqrt(s - 1)) # incorrect - 650320427 - -(Contributed by Mark Dickinson in :issue:`36887`.) - -The function :func:`math.factorial` no longer accepts arguments that are not -int-like. (Contributed by Pablo Galindo in :issue:`33083`.) - - -mmap ----- - -The :class:`mmap.mmap` class now has an :meth:`~mmap.mmap.madvise` method to -access the ``madvise()`` system call. -(Contributed by Zackery Spytz in :issue:`32941`.) - - -multiprocessing ---------------- - -Added new :mod:`multiprocessing.shared_memory` module. -(Contributed by Davin Potts in :issue:`35813`.) - -On macOS, the *spawn* start method is now used by default. -(Contributed by Victor Stinner in :issue:`33725`.) - - -os --- - -Added new function :func:`~os.add_dll_directory` on Windows for providing -additional search paths for native dependencies when importing extension -modules or loading DLLs using :mod:`ctypes`. -(Contributed by Steve Dower in :issue:`36085`.) - -A new :func:`os.memfd_create` function was added to wrap the -``memfd_create()`` syscall. -(Contributed by Zackery Spytz and Christian Heimes in :issue:`26836`.) - -On Windows, much of the manual logic for handling reparse points (including -symlinks and directory junctions) has been delegated to the operating system. -Specifically, :func:`os.stat` will now traverse anything supported by the -operating system, while :func:`os.lstat` will only open reparse points that -identify as "name surrogates" while others are opened as for :func:`os.stat`. -In all cases, :attr:`stat_result.st_mode` will only have ``S_IFLNK`` set for -symbolic links and not other kinds of reparse points. To identify other kinds -of reparse point, check the new :attr:`stat_result.st_reparse_tag` attribute. - -On Windows, :func:`os.readlink` is now able to read directory junctions. Note -that :func:`~os.path.islink` will return ``False`` for directory junctions, -and so code that checks ``islink`` first will continue to treat junctions as -directories, while code that handles errors from :func:`os.readlink` may now -treat junctions as links. - -(Contributed by Steve Dower in :issue:`37834`.) - - -os.path -------- - -:mod:`os.path` functions that return a boolean result like -:func:`~os.path.exists`, :func:`~os.path.lexists`, :func:`~os.path.isdir`, -:func:`~os.path.isfile`, :func:`~os.path.islink`, and :func:`~os.path.ismount` -now return ``False`` instead of raising :exc:`ValueError` or its subclasses -:exc:`UnicodeEncodeError` and :exc:`UnicodeDecodeError` for paths that contain -characters or bytes unrepresentable at the OS level. -(Contributed by Serhiy Storchaka in :issue:`33721`.) - -:func:`~os.path.expanduser` on Windows now prefers the :envvar:`USERPROFILE` -environment variable and does not use :envvar:`HOME`, which is not normally set -for regular user accounts. -(Contributed by Anthony Sottile in :issue:`36264`.) - -:func:`~os.path.isdir` on Windows no longer returns ``True`` for a link to a -non-existent directory. - -:func:`~os.path.realpath` on Windows now resolves reparse points, including -symlinks and directory junctions. - -(Contributed by Steve Dower in :issue:`37834`.) - - -pathlib -------- - -:mod:`pathlib.Path` methods that return a boolean result like -:meth:`~pathlib.Path.exists()`, :meth:`~pathlib.Path.is_dir()`, -:meth:`~pathlib.Path.is_file()`, :meth:`~pathlib.Path.is_mount()`, -:meth:`~pathlib.Path.is_symlink()`, :meth:`~pathlib.Path.is_block_device()`, -:meth:`~pathlib.Path.is_char_device()`, :meth:`~pathlib.Path.is_fifo()`, -:meth:`~pathlib.Path.is_socket()` now return ``False`` instead of raising -:exc:`ValueError` or its subclass :exc:`UnicodeEncodeError` for paths that -contain characters unrepresentable at the OS level. -(Contributed by Serhiy Storchaka in :issue:`33721`.) - -Added :meth:`pathlib.Path.link_to()` which creates a hard link pointing -to a path. -(Contributed by Joannah Nanjekye in :issue:`26978`) - - -pickle ------- - -:mod:`pickle` extensions subclassing the C-optimized :class:`~pickle.Pickler` -can now override the pickling logic of functions and classes by defining the -special :meth:`~pickle.Pickler.reducer_override` method. -(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.) - - -plistlib --------- - -Added new :class:`plistlib.UID` and enabled support for reading and writing -NSKeyedArchiver-encoded binary plists. -(Contributed by Jon Janzen in :issue:`26707`.) - - -pprint ------- - -The :mod:`pprint` module added a *sort_dicts* parameter to several functions. -By default, those functions continue to sort dictionaries before rendering or -printing. However, if *sort_dicts* is set to false, the dictionaries retain -the order that keys were inserted. This can be useful for comparison to JSON -inputs during debugging. - -In addition, there is a convenience new function, :func:`pprint.pp` that is -like :func:`pprint.pprint` but with *sort_dicts* defaulting to ``False``:: - - >>> from pprint import pprint, pp - >>> d = dict(source='input.txt', operation='filter', destination='output.txt') - >>> pp(d, width=40) # Original order - {'source': 'input.txt', - 'operation': 'filter', - 'destination': 'output.txt'} - >>> pprint(d, width=40) # Keys sorted alphabetically - {'destination': 'output.txt', - 'operation': 'filter', - 'source': 'input.txt'} - -(Contributed by Rémi Lapeyre in :issue:`30670`.) - - -py_compile ----------- - -:func:`py_compile.compile` now supports silent mode. -(Contributed by Joannah Nanjekye in :issue:`22640`.) - - -shlex ------ - -The new :func:`shlex.join` function acts as the inverse of :func:`shlex.split`. -(Contributed by Bo Bayles in :issue:`32102`.) - - -shutil ------- - -:func:`shutil.copytree` now accepts a new ``dirs_exist_ok`` keyword argument. -(Contributed by Josh Bronson in :issue:`20849`.) - -:func:`shutil.make_archive` now defaults to the modern pax (POSIX.1-2001) -format for new archives to improve portability and standards conformance, -inherited from the corresponding change to the :mod:`tarfile` module. -(Contributed by C.A.M. Gerlach in :issue:`30661`.) - -:func:`shutil.rmtree` on Windows now removes directory junctions without -recursively removing their contents first. -(Contributed by Steve Dower in :issue:`37834`.) - - -socket ------- - -Added :meth:`~socket.create_server()` and :meth:`~socket.has_dualstack_ipv6()` -convenience functions to automate the necessary tasks usually involved when -creating a server socket, including accepting both IPv4 and IPv6 connections -on the same socket. (Contributed by Giampaolo Rodolà in :issue:`17561`.) - -The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and -:func:`socket.if_indextoname()` functions have been implemented on Windows. -(Contributed by Zackery Spytz in :issue:`37007`.) - - -ssl ---- - -Added :attr:`~ssl.SSLContext.post_handshake_auth` to enable and -:meth:`~ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3 -post-handshake authentication. -(Contributed by Christian Heimes in :issue:`34670`.) - - -statistics ----------- - -Added :func:`statistics.fmean` as a faster, floating point variant of -:func:`statistics.mean()`. (Contributed by Raymond Hettinger and -Steven D'Aprano in :issue:`35904`.) - -Added :func:`statistics.geometric_mean()` -(Contributed by Raymond Hettinger in :issue:`27181`.) - -Added :func:`statistics.multimode` that returns a list of the most -common values. (Contributed by Raymond Hettinger in :issue:`35892`.) - -Added :func:`statistics.quantiles` that divides data or a distribution -in to equiprobable intervals (e.g. quartiles, deciles, or percentiles). -(Contributed by Raymond Hettinger in :issue:`36546`.) - -Added :class:`statistics.NormalDist`, a tool for creating -and manipulating normal distributions of a random variable. -(Contributed by Raymond Hettinger in :issue:`36018`.) - -:: - - >>> temperature_feb = NormalDist.from_samples([4, 12, -3, 2, 7, 14]) - >>> temperature_feb.mean - 6.0 - >>> temperature_feb.stdev - 6.356099432828281 - - >>> temperature_feb.cdf(3) # Chance of being under 3 degrees - 0.3184678262814532 - >>> # Relative chance of being 7 degrees versus 10 degrees - >>> temperature_feb.pdf(7) / temperature_feb.pdf(10) - 1.2039930378537762 - - >>> el_niño = NormalDist(4, 2.5) - >>> temperature_feb += el_niño # Add in a climate effect - >>> temperature_feb - NormalDist(mu=10.0, sigma=6.830080526611674) - - >>> temperature_feb * (9/5) + 32 # Convert to Fahrenheit - NormalDist(mu=50.0, sigma=12.294144947901014) - >>> temperature_feb.samples(3) # Generate random samples - [7.672102882379219, 12.000027119750287, 4.647488369766392] - - -sys ---- - -Add new :func:`sys.unraisablehook` function which can be overridden to control -how "unraisable exceptions" are handled. It is called when an exception has -occurred but there is no way for Python to handle it. For example, when a -destructor raises an exception or during garbage collection -(:func:`gc.collect`). -(Contributed by Victor Stinner in :issue:`36829`.) - - -tarfile -------- - -The :mod:`tarfile` module now defaults to the modern pax (POSIX.1-2001) -format for new archives, instead of the previous GNU-specific one. -This improves cross-platform portability with a consistent encoding (UTF-8) -in a standardized and extensible format, and offers several other benefits. -(Contributed by C.A.M. Gerlach in :issue:`36268`.) - - -threading ---------- - -Add a new :func:`threading.excepthook` function which handles uncaught -:meth:`threading.Thread.run` exception. It can be overridden to control how -uncaught :meth:`threading.Thread.run` exceptions are handled. -(Contributed by Victor Stinner in :issue:`1230540`.) - -Add a new :func:`threading.get_native_id` function and -a :data:`~threading.Thread.native_id` -attribute to the :class:`threading.Thread` class. These return the native -integral Thread ID of the current thread assigned by the kernel. -This feature is only available on certain platforms, see -:func:`get_native_id <threading.get_native_id>` for more information. -(Contributed by Jake Tesler in :issue:`36084`.) - - -tokenize --------- - -The :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token when -provided with input that does not have a trailing new line. This behavior -now matches what the C tokenizer does internally. -(Contributed by Ammar Askar in :issue:`33899`.) - - -tkinter -------- - -Added methods :meth:`~tkinter.Spinbox.selection_from`, -:meth:`~tkinter.Spinbox.selection_present`, -:meth:`~tkinter.Spinbox.selection_range` and -:meth:`~tkinter.Spinbox.selection_to` -in the :class:`tkinter.Spinbox` class. -(Contributed by Juliette Monsel in :issue:`34829`.) - -Added method :meth:`~tkinter.Canvas.moveto` -in the :class:`tkinter.Canvas` class. -(Contributed by Juliette Monsel in :issue:`23831`.) - -The :class:`tkinter.PhotoImage` class now has -:meth:`~tkinter.PhotoImage.transparency_get` and -:meth:`~tkinter.PhotoImage.transparency_set` methods. (Contributed by -Zackery Spytz in :issue:`25451`.) - - -time ----- - -Added new clock :data:`~time.CLOCK_UPTIME_RAW` for macOS 10.12. -(Contributed by Joannah Nanjekye in :issue:`35702`.) - - -typing ------- - -The :mod:`typing` module incorporates several new features: - -* A dictionary type with per-key types. See :pep:`589` and - :class:`typing.TypedDict`. - TypedDict uses only string keys. By default, every key is required - to be present. Specify "total=False" to allow keys to be optional:: - - class Location(TypedDict, total=False): - lat_long: tuple - grid_square: str - xy_coordinate: tuple - -* Literal types. See :pep:`586` and :class:`typing.Literal`. - Literal types indicate that a parameter or return value - is constrained to one or more specific literal values:: - - def get_status(port: int) -> Literal['connected', 'disconnected']: - ... - -* "Final" variables, functions, methods and classes. See :pep:`591`, - :class:`typing.Final` and :func:`typing.final`. - The final qualifier instructs a static type checker to restrict - subclassing, overriding, or reassignment:: - - pi: Final[float] = 3.1415926536 - -* Protocol definitions. See :pep:`544`, :class:`typing.Protocol` and - :func:`typing.runtime_checkable`. Simple ABCs like - :class:`typing.SupportsInt` are now ``Protocol`` subclasses. - -* New protocol class :class:`typing.SupportsIndex`. - -* New functions :func:`typing.get_origin` and :func:`typing.get_args`. - - -unicodedata ------------ - -The :mod:`unicodedata` module has been upgraded to use the `Unicode 12.1.0 -<http://blog.unicode.org/2019/05/unicode-12-1-en.html>`_ release. - -New function :func:`~unicodedata.is_normalized` can be used to verify a string -is in a specific normal form, often much faster than by actually normalizing -the string. (Contributed by Max Belanger, David Euresti, and Greg Price in -:issue:`32285` and :issue:`37966`). - - -unittest --------- - -Added :class:`~unittest.mock.AsyncMock` to support an asynchronous version of -:class:`~unittest.mock.Mock`. Appropriate new assert functions for testing -have been added as well. -(Contributed by Lisa Roach in :issue:`26467`). - -Added :func:`~unittest.addModuleCleanup()` and -:meth:`~unittest.TestCase.addClassCleanup()` to unittest to support -cleanups for :func:`~unittest.setUpModule()` and -:meth:`~unittest.TestCase.setUpClass()`. -(Contributed by Lisa Roach in :issue:`24412`.) - -Several mock assert functions now also print a list of actual calls upon -failure. (Contributed by Petter Strandmark in :issue:`35047`.) - -:mod:`unittest` module gained support for coroutines to be used as test cases -with :class:`unittest.IsolatedAsyncioTestCase`. -(Contributed by Andrew Svetlov in :issue:`32972`.) - -Example:: - - import unittest - - - class TestRequest(unittest.IsolatedAsyncioTestCase): - - async def asyncSetUp(self): - self.connection = await AsyncConnection() - - async def test_get(self): - response = await self.connection.get("https://example.com") - self.assertEqual(response.status_code, 200) - - async def asyncTearDown(self): - await self.connection.close() - - - if __name__ == "__main__": - unittest.main() - - -venv ----- - -:mod:`venv` now includes an ``Activate.ps1`` script on all platforms for -activating virtual environments under PowerShell Core 6.1. -(Contributed by Brett Cannon in :issue:`32718`.) - - -weakref -------- - -The proxy objects returned by :func:`weakref.proxy` now support the matrix -multiplication operators ``@`` and ``@=`` in addition to the other -numeric operators. (Contributed by Mark Dickinson in :issue:`36669`.) - - -xml ---- - -As mitigation against DTD and external entity retrieval, the -:mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process -external entities by default. -(Contributed by Christian Heimes in :issue:`17239`.) - -The ``.find*()`` methods in the :mod:`xml.etree.ElementTree` module -support wildcard searches like ``{*}tag`` which ignores the namespace -and ``{namespace}*`` which returns all tags in the given namespace. -(Contributed by Stefan Behnel in :issue:`28238`.) - -The :mod:`xml.etree.ElementTree` module provides a new function -:func:`–xml.etree.ElementTree.canonicalize()` that implements C14N 2.0. -(Contributed by Stefan Behnel in :issue:`13611`.) - -The target object of :class:`xml.etree.ElementTree.XMLParser` can -receive namespace declaration events through the new callback methods -``start_ns()`` and ``end_ns()``. Additionally, the -:class:`xml.etree.ElementTree.TreeBuilder` target can be configured -to process events about comments and processing instructions to include -them in the generated tree. -(Contributed by Stefan Behnel in :issue:`36676` and :issue:`36673`.) - - -xmlrpc ------- - -:class:`xmlrpc.client.ServerProxy` now supports an optional *headers* keyword -argument for a sequence of HTTP headers to be sent with each request. Among -other things, this makes it possible to upgrade from default basic -authentication to faster session authentication. -(Contributed by Cédric Krier in :issue:`35153`.) - - -Optimizations -============= - -* The :mod:`subprocess` module can now use the :func:`os.posix_spawn` function - in some cases for better performance. Currently, it is only used on macOS - and Linux (using glibc 2.24 or newer) if all these conditions are met: - - * *close_fds* is false; - * *preexec_fn*, *pass_fds*, *cwd* and *start_new_session* parameters - are not set; - * the *executable* path contains a directory. - - (Contributed by Joannah Nanjekye and Victor Stinner in :issue:`35537`.) - -* :func:`shutil.copyfile`, :func:`shutil.copy`, :func:`shutil.copy2`, - :func:`shutil.copytree` and :func:`shutil.move` use platform-specific - "fast-copy" syscalls on Linux and macOS in order to copy the file - more efficiently. - "fast-copy" means that the copying operation occurs within the kernel, - avoiding the use of userspace buffers in Python as in - "``outfd.write(infd.read())``". - On Windows :func:`shutil.copyfile` uses a bigger default buffer size (1 MiB - instead of 16 KiB) and a :func:`memoryview`-based variant of - :func:`shutil.copyfileobj` is used. - The speedup for copying a 512 MiB file within the same partition is about - +26% on Linux, +50% on macOS and +40% on Windows. Also, much less CPU cycles - are consumed. - See :ref:`shutil-platform-dependent-efficient-copy-operations` section. - (Contributed by Giampaolo Rodolà in :issue:`33671`.) - -* :func:`shutil.copytree` uses :func:`os.scandir` function and all copy - functions depending from it use cached :func:`os.stat` values. The speedup - for copying a directory with 8000 files is around +9% on Linux, +20% on - Windows and +30% on a Windows SMB share. Also the number of :func:`os.stat` - syscalls is reduced by 38% making :func:`shutil.copytree` especially faster - on network filesystems. (Contributed by Giampaolo Rodolà in :issue:`33695`.) - -* The default protocol in the :mod:`pickle` module is now Protocol 4, - first introduced in Python 3.4. It offers better performance and smaller - size compared to Protocol 3 available since Python 3.0. - -* Removed one ``Py_ssize_t`` member from ``PyGC_Head``. All GC tracked - objects (e.g. tuple, list, dict) size is reduced 4 or 8 bytes. - (Contributed by Inada Naoki in :issue:`33597`.) - -* :class:`uuid.UUID` now uses ``__slots__`` to reduce its memory footprint. - (Contributed by Wouter Bolsterlee and Tal Einat in :issue:`30977`) - -* Improved performance of :func:`operator.itemgetter` by 33%. Optimized - argument handling and added a fast path for the common case of a single - non-negative integer index into a tuple (which is the typical use case in - the standard library). (Contributed by Raymond Hettinger in - :issue:`35664`.) - -* Sped-up field lookups in :func:`collections.namedtuple`. They are now more - than two times faster, making them the fastest form of instance variable - lookup in Python. (Contributed by Raymond Hettinger, Pablo Galindo, and - Joe Jevnik, Serhiy Storchaka in :issue:`32492`.) - -* The :class:`list` constructor does not overallocate the internal item buffer - if the input iterable has a known length (the input implements ``__len__``). - This makes the created list 12% smaller on average. (Contributed by - Raymond Hettinger and Pablo Galindo in :issue:`33234`.) - -* Doubled the speed of class variable writes. When a non-dunder attribute - was updated, there was an unnecessary call to update slots. - (Contributed by Stefan Behnel, Pablo Galindo Salgado, Raymond Hettinger, - Neil Schemenauer, and Serhiy Storchaka in :issue:`36012`.) - -* Reduced an overhead of converting arguments passed to many builtin functions - and methods. This sped up calling some simple builtin functions and - methods up to 20--50%. (Contributed by Serhiy Storchaka in :issue:`23867`, - :issue:`35582` and :issue:`36127`.) - -* ``LOAD_GLOBAL`` instruction now uses new "per opcode cache" mechanism. - It is about 40% faster now. (Contributed by Yury Selivanov and Inada Naoki in - :issue:`26219`.) - - -Build and C API Changes -======================= - -* Default :data:`sys.abiflags` became an empty string: the ``m`` flag for - pymalloc became useless (builds with and without pymalloc are ABI compatible) - and so has been removed. (Contributed by Victor Stinner in :issue:`36707`.) - - Example of changes: - - * Only ``python3.8`` program is installed, ``python3.8m`` program is gone. - * Only ``python3.8-config`` script is installed, ``python3.8m-config`` script - is gone. - * The ``m`` flag has been removed from the suffix of dynamic library - filenames: extension modules in the standard library as well as those - produced and installed by third-party packages, like those downloaded from - PyPI. On Linux, for example, the Python 3.7 suffix - ``.cpython-37m-x86_64-linux-gnu.so`` became - ``.cpython-38-x86_64-linux-gnu.so`` in Python 3.8. - -* The header files have been reorganized to better separate the different kinds - of APIs: - - * ``Include/*.h`` should be the portable public stable C API. - * ``Include/cpython/*.h`` should be the unstable C API specific to CPython; - public API, with some private API prefixed by ``_Py`` or ``_PY``. - * ``Include/internal/*.h`` is the private internal C API very specific to - CPython. This API comes with no backward compatibility warranty and should - not be used outside CPython. It is only exposed for very specific needs - like debuggers and profiles which has to access to CPython internals - without calling functions. This API is now installed by ``make install``. - - (Contributed by Victor Stinner in :issue:`35134` and :issue:`35081`, - work initiated by Eric Snow in Python 3.7.) - -* Some macros have been converted to static inline functions: parameter types - and return type are well defined, they don't have issues specific to macros, - variables have a local scopes. Examples: - - * :c:func:`Py_INCREF`, :c:func:`Py_DECREF` - * :c:func:`Py_XINCREF`, :c:func:`Py_XDECREF` - * :c:func:`PyObject_INIT`, :c:func:`PyObject_INIT_VAR` - * Private functions: :c:func:`_PyObject_GC_TRACK`, - :c:func:`_PyObject_GC_UNTRACK`, :c:func:`_Py_Dealloc` - - (Contributed by Victor Stinner in :issue:`35059`.) - -* The :c:func:`PyByteArray_Init` and :c:func:`PyByteArray_Fini` functions have - been removed. They did nothing since Python 2.7.4 and Python 3.2.0, were - excluded from the limited API (stable ABI), and were not documented. - (Contributed by Victor Stinner in :issue:`35713`.) - -* The result of :c:func:`PyExceptionClass_Name` is now of type - ``const char *`` rather of ``char *``. - (Contributed by Serhiy Storchaka in :issue:`33818`.) - -* The duality of ``Modules/Setup.dist`` and ``Modules/Setup`` has been - removed. Previously, when updating the CPython source tree, one had - to manually copy ``Modules/Setup.dist`` (inside the source tree) to - ``Modules/Setup`` (inside the build tree) in order to reflect any changes - upstream. This was of a small benefit to packagers at the expense of - a frequent annoyance to developers following CPython development, as - forgetting to copy the file could produce build failures. - - Now the build system always reads from ``Modules/Setup`` inside the source - tree. People who want to customize that file are encouraged to maintain - their changes in a git fork of CPython or as patch files, as they would do - for any other change to the source tree. - - (Contributed by Antoine Pitrou in :issue:`32430`.) - -* Functions that convert Python number to C integer like - :c:func:`PyLong_AsLong` and argument parsing functions like - :c:func:`PyArg_ParseTuple` with integer converting format units like ``'i'`` - will now use the :meth:`~object.__index__` special method instead of - :meth:`~object.__int__`, if available. The deprecation warning will be - emitted for objects with the ``__int__()`` method but without the - ``__index__()`` method (like :class:`~decimal.Decimal` and - :class:`~fractions.Fraction`). :c:func:`PyNumber_Check` will now return - ``1`` for objects implementing ``__index__()``. - :c:func:`PyNumber_Long`, :c:func:`PyNumber_Float` and - :c:func:`PyFloat_AsDouble` also now use the ``__index__()`` method if - available. - (Contributed by Serhiy Storchaka in :issue:`36048` and :issue:`20092`.) - -* Heap-allocated type objects will now increase their reference count - in :c:func:`PyObject_Init` (and its parallel macro ``PyObject_INIT``) - instead of in :c:func:`PyType_GenericAlloc`. Types that modify instance - allocation or deallocation may need to be adjusted. - (Contributed by Eddie Elizondo in :issue:`35810`.) - -* The new function :c:func:`PyCode_NewWithPosOnlyArgs` allows to create - code objects like :c:func:`PyCode_New`, but with an extra *posonlyargcount* - parameter for indicating the number of positional-only arguments. - (Contributed by Pablo Galindo in :issue:`37221`.) - -* :c:func:`Py_SetPath` now sets :data:`sys.executable` to the program full - path (:c:func:`Py_GetProgramFullPath`) rather than to the program name - (:c:func:`Py_GetProgramName`). - (Contributed by Victor Stinner in :issue:`38234`.) - - -Deprecated -========== - -* The distutils ``bdist_wininst`` command is now deprecated, use - ``bdist_wheel`` (wheel packages) instead. - (Contributed by Victor Stinner in :issue:`37481`.) - -* Deprecated methods ``getchildren()`` and ``getiterator()`` in - the :mod:`~xml.etree.ElementTree` module now emit a - :exc:`DeprecationWarning` instead of :exc:`PendingDeprecationWarning`. - They will be removed in Python 3.9. - (Contributed by Serhiy Storchaka in :issue:`29209`.) - -* Passing an object that is not an instance of - :class:`concurrent.futures.ThreadPoolExecutor` to - :meth:`loop.set_default_executor() <asyncio.loop.set_default_executor>` is - deprecated and will be prohibited in Python 3.9. - (Contributed by Elvis Pranskevichus in :issue:`34075`.) - -* The :meth:`__getitem__` methods of :class:`xml.dom.pulldom.DOMEventStream`, - :class:`wsgiref.util.FileWrapper` and :class:`fileinput.FileInput` have been - deprecated. - - Implementations of these methods have been ignoring their *index* parameter, - and returning the next item instead. - (Contributed by Berker Peksag in :issue:`9372`.) - -* The :class:`typing.NamedTuple` class has deprecated the ``_field_types`` - attribute in favor of the ``__annotations__`` attribute which has the same - information. (Contributed by Raymond Hettinger in :issue:`36320`.) - -* :mod:`ast` classes ``Num``, ``Str``, ``Bytes``, ``NameConstant`` and - ``Ellipsis`` are considered deprecated and will be removed in future Python - versions. :class:`~ast.Constant` should be used instead. - (Contributed by Serhiy Storchaka in :issue:`32892`.) - -* :class:`ast.NodeVisitor` methods ``visit_Num()``, ``visit_Str()``, - ``visit_Bytes()``, ``visit_NameConstant()`` and ``visit_Ellipsis()`` are - deprecated now and will not be called in future Python versions. - Add the :meth:`~ast.NodeVisitor.visit_Constant` method to handle all - constant nodes. - (Contributed by Serhiy Storchaka in :issue:`36917`.) - -* The :func:`asyncio.coroutine` :term:`decorator` is deprecated and will be - removed in version 3.10. Instead of ``@asyncio.coroutine``, use - :keyword:`async def` instead. - (Contributed by Andrew Svetlov in :issue:`36921`.) - -* In :mod:`asyncio`, the explicit passing of a *loop* argument has been - deprecated and will be removed in version 3.10 for the following: - :func:`asyncio.sleep`, :func:`asyncio.gather`, :func:`asyncio.shield`, - :func:`asyncio.wait_for`, :func:`asyncio.wait`, :func:`asyncio.as_completed`, - :class:`asyncio.Task`, :class:`asyncio.Lock`, :class:`asyncio.Event`, - :class:`asyncio.Condition`, :class:`asyncio.Semaphore`, - :class:`asyncio.BoundedSemaphore`, :class:`asyncio.Queue`, - :func:`asyncio.create_subprocess_exec`, and - :func:`asyncio.create_subprocess_shell`. - -* The explicit passing of coroutine objects to :func:`asyncio.wait` has been - deprecated and will be removed in version 3.11. - (Contributed by Yury Selivanov in :issue:`34790`.) - -* The following functions and methods are deprecated in the :mod:`gettext` - module: :func:`~gettext.lgettext`, :func:`~gettext.ldgettext`, - :func:`~gettext.lngettext` and :func:`~gettext.ldngettext`. - They return encoded bytes, and it's possible that you will get unexpected - Unicode-related exceptions if there are encoding problems with the - translated strings. It's much better to use alternatives which return - Unicode strings in Python 3. These functions have been broken for a long time. - - Function :func:`~gettext.bind_textdomain_codeset`, methods - :meth:`~gettext.NullTranslations.output_charset` and - :meth:`~gettext.NullTranslations.set_output_charset`, and the *codeset* - parameter of functions :func:`~gettext.translation` and - :func:`~gettext.install` are also deprecated, since they are only used for - for the ``l*gettext()`` functions. - (Contributed by Serhiy Storchaka in :issue:`33710`.) - -* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread` - has been deprecated. - (Contributed by Dong-hee Na in :issue:`35283`.) - -* Many builtin and extension functions that take integer arguments will - now emit a deprecation warning for :class:`~decimal.Decimal`\ s, - :class:`~fractions.Fraction`\ s and any other objects that can be converted - to integers only with a loss (e.g. that have the :meth:`~object.__int__` - method but do not have the :meth:`~object.__index__` method). In future - version they will be errors. - (Contributed by Serhiy Storchaka in :issue:`36048`.) - -* Deprecated passing the following arguments as keyword arguments: - - - *func* in :func:`functools.partialmethod`, :func:`weakref.finalize`, - :meth:`profile.Profile.runcall`, :meth:`cProfile.Profile.runcall`, - :meth:`bdb.Bdb.runcall`, :meth:`trace.Trace.runfunc` and - :func:`curses.wrapper`. - - *function* in :meth:`unittest.TestCase.addCleanup`. - - *fn* in the :meth:`~concurrent.futures.Executor.submit` method of - :class:`concurrent.futures.ThreadPoolExecutor` and - :class:`concurrent.futures.ProcessPoolExecutor`. - - *callback* in :meth:`contextlib.ExitStack.callback`, - :meth:`contextlib.AsyncExitStack.callback` and - :meth:`contextlib.AsyncExitStack.push_async_callback`. - - *c* and *typeid* in the :meth:`~multiprocessing.managers.Server.create` - method of :class:`multiprocessing.managers.Server` and - :class:`multiprocessing.managers.SharedMemoryServer`. - - *obj* in :func:`weakref.finalize`. - - In future releases of Python, they will be :ref:`positional-only - <positional-only_parameter>`. - (Contributed by Serhiy Storchaka in :issue:`36492`.) - - -API and Feature Removals -======================== - -The following features and APIs have been removed from Python 3.8: - -* Starting with Python 3.3, importing ABCs from :mod:`collections` was - deprecated, and importing should be done from :mod:`collections.abc`. Being - able to import from collections was marked for removal in 3.8, but has been - delayed to 3.9. (See :issue:`36952`.) - -* The :mod:`macpath` module, deprecated in Python 3.7, has been removed. - (Contributed by Victor Stinner in :issue:`35471`.) - -* The function :func:`platform.popen` has been removed, after having been - deprecated since Python 3.3: use :func:`os.popen` instead. - (Contributed by Victor Stinner in :issue:`35345`.) - -* The function :func:`time.clock` has been removed, after having been - deprecated since Python 3.3: use :func:`time.perf_counter` or - :func:`time.process_time` instead, depending - on your requirements, to have well-defined behavior. - (Contributed by Matthias Bussonnier in :issue:`36895`.) - -* The ``pyvenv`` script has been removed in favor of ``python3.8 -m venv`` - to help eliminate confusion as to what Python interpreter the ``pyvenv`` - script is tied to. (Contributed by Brett Cannon in :issue:`25427`.) - -* ``parse_qs``, ``parse_qsl``, and ``escape`` are removed from the :mod:`cgi` - module. They are deprecated in Python 3.2 or older. They should be imported - from the ``urllib.parse`` and ``html`` modules instead. - -* ``filemode`` function is removed from the :mod:`tarfile` module. - It is not documented and deprecated since Python 3.3. - -* The :class:`~xml.etree.ElementTree.XMLParser` constructor no longer accepts - the *html* argument. It never had an effect and was deprecated in Python 3.4. - All other parameters are now :ref:`keyword-only <keyword-only_parameter>`. - (Contributed by Serhiy Storchaka in :issue:`29209`.) - -* Removed the ``doctype()`` method of :class:`~xml.etree.ElementTree.XMLParser`. - (Contributed by Serhiy Storchaka in :issue:`29209`.) - -* "unicode_internal" codec is removed. - (Contributed by Inada Naoki in :issue:`36297`.) - -* The ``Cache`` and ``Statement`` objects of the :mod:`sqlite3` module are not - exposed to the user. - (Contributed by Aviv Palivoda in :issue:`30262`.) - -* The ``bufsize`` keyword argument of :func:`fileinput.input` and - :func:`fileinput.FileInput` which was ignored and deprecated since Python 3.6 - has been removed. :issue:`36952` (Contributed by Matthias Bussonnier.) - -* The functions :func:`sys.set_coroutine_wrapper` and - :func:`sys.get_coroutine_wrapper` deprecated in Python 3.7 have been removed; - :issue:`36933` (Contributed by Matthias Bussonnier.) - - -Porting to Python 3.8 -===================== - -This section lists previously described changes and other bugfixes -that may require changes to your code. - - -Changes in Python behavior --------------------------- - -* Yield expressions (both ``yield`` and ``yield from`` clauses) are now disallowed - in comprehensions and generator expressions (aside from the iterable expression - in the leftmost :keyword:`!for` clause). - (Contributed by Serhiy Storchaka in :issue:`10544`.) - -* The compiler now produces a :exc:`SyntaxWarning` when identity checks - (``is`` and ``is not``) are used with certain types of literals - (e.g. strings, numbers). These can often work by accident in CPython, - but are not guaranteed by the language spec. The warning advises users - to use equality tests (``==`` and ``!=``) instead. - (Contributed by Serhiy Storchaka in :issue:`34850`.) - -* The CPython interpreter can swallow exceptions in some circumstances. - In Python 3.8 this happens in fewer cases. In particular, exceptions - raised when getting the attribute from the type dictionary are no longer - ignored. (Contributed by Serhiy Storchaka in :issue:`35459`.) - -* Removed ``__str__`` implementations from builtin types :class:`bool`, - :class:`int`, :class:`float`, :class:`complex` and few classes from - the standard library. They now inherit ``__str__()`` from :class:`object`. - As result, defining the ``__repr__()`` method in the subclass of these - classes will affect their string representation. - (Contributed by Serhiy Storchaka in :issue:`36793`.) - -* On AIX, :attr:`sys.platform` doesn't contain the major version anymore. - It is always ``'aix'``, instead of ``'aix3'`` .. ``'aix7'``. Since - older Python versions include the version number, so it is recommended to - always use ``sys.platform.startswith('aix')``. - (Contributed by M. Felt in :issue:`36588`.) - -* :c:func:`PyEval_AcquireLock` and :c:func:`PyEval_AcquireThread` now - terminate the current thread if called while the interpreter is - finalizing, making them consistent with :c:func:`PyEval_RestoreThread`, - :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`. If this - behavior is not desired, guard the call by checking :c:func:`_Py_IsFinalizing` - or :c:func:`sys.is_finalizing`. - (Contributed by Joannah Nanjekye in :issue:`36475`.) - - -Changes in the Python API -------------------------- - -* The :func:`os.getcwdb` function now uses the UTF-8 encoding on Windows, - rather than the ANSI code page: see :pep:`529` for the rationale. The - function is no longer deprecated on Windows. - (Contributed by Victor Stinner in :issue:`37412`.) - -* :class:`subprocess.Popen` can now use :func:`os.posix_spawn` in some cases - for better performance. On Windows Subsystem for Linux and QEMU User - Emulation, the :class:`Popen` constructor using :func:`os.posix_spawn` no longer raises an - exception on errors like "missing program". Instead the child process fails with a - non-zero :attr:`~Popen.returncode`. - (Contributed by Joannah Nanjekye and Victor Stinner in :issue:`35537`.) - -* The *preexec_fn* argument of * :class:`subprocess.Popen` is no longer - compatible with subinterpreters. The use of the parameter in a - subinterpreter now raises :exc:`RuntimeError`. - (Contributed by Eric Snow in :issue:`34651`, modified by Christian Heimes - in :issue:`37951`.) - -* The :meth:`imap.IMAP4.logout` method no longer silently ignores arbitrary - exceptions. - (Contributed by Victor Stinner in :issue:`36348`.) - -* The function :func:`platform.popen` has been removed, after having been deprecated since - Python 3.3: use :func:`os.popen` instead. - (Contributed by Victor Stinner in :issue:`35345`.) - -* The :func:`statistics.mode` function no longer raises an exception - when given multimodal data. Instead, it returns the first mode - encountered in the input data. (Contributed by Raymond Hettinger - in :issue:`35892`.) - -* The :meth:`~tkinter.ttk.Treeview.selection` method of the - :class:`tkinter.ttk.Treeview` class no longer takes arguments. Using it with - arguments for changing the selection was deprecated in Python 3.6. Use - specialized methods like :meth:`~tkinter.ttk.Treeview.selection_set` for - changing the selection. (Contributed by Serhiy Storchaka in :issue:`31508`.) - -* The :meth:`writexml`, :meth:`toxml` and :meth:`toprettyxml` methods of - :mod:`xml.dom.minidom`, and the :meth:`write` method of :mod:`xml.etree`, - now preserve the attribute order specified by the user. - (Contributed by Diego Rojas and Raymond Hettinger in :issue:`34160`.) - -* A :mod:`dbm.dumb` database opened with flags ``'r'`` is now read-only. - :func:`dbm.dumb.open` with flags ``'r'`` and ``'w'`` no longer creates - a database if it does not exist. - (Contributed by Serhiy Storchaka in :issue:`32749`.) - -* The ``doctype()`` method defined in a subclass of - :class:`~xml.etree.ElementTree.XMLParser` will no longer be called and will - emit a :exc:`RuntimeWarning` instead of a :exc:`DeprecationWarning`. - Define the :meth:`doctype() <xml.etree.ElementTree.TreeBuilder.doctype>` - method on a target for handling an XML doctype declaration. - (Contributed by Serhiy Storchaka in :issue:`29209`.) - -* A :exc:`RuntimeError` is now raised when the custom metaclass doesn't - provide the ``__classcell__`` entry in the namespace passed to - ``type.__new__``. A :exc:`DeprecationWarning` was emitted in Python - 3.6--3.7. (Contributed by Serhiy Storchaka in :issue:`23722`.) - -* The :class:`cProfile.Profile` class can now be used as a context - manager. (Contributed by Scott Sanderson in :issue:`29235`.) - -* :func:`shutil.copyfile`, :func:`shutil.copy`, :func:`shutil.copy2`, - :func:`shutil.copytree` and :func:`shutil.move` use platform-specific - "fast-copy" syscalls (see - :ref:`shutil-platform-dependent-efficient-copy-operations` section). - -* :func:`shutil.copyfile` default buffer size on Windows was changed from - 16 KiB to 1 MiB. - -* The ``PyGC_Head`` struct has changed completely. All code that touched the - struct member should be rewritten. (See :issue:`33597`.) - -* The :c:type:`PyInterpreterState` struct has been moved into the "internal" - header files (specifically Include/internal/pycore_pystate.h). An - opaque ``PyInterpreterState`` is still available as part of the public - API (and stable ABI). The docs indicate that none of the struct's - fields are public, so we hope no one has been using them. However, - if you do rely on one or more of those private fields and have no - alternative then please open a BPO issue. We'll work on helping - you adjust (possibly including adding accessor functions to the - public API). (See :issue:`35886`.) - -* The :meth:`mmap.flush() <mmap.mmap.flush>` method now returns ``None`` on - success and raises an exception on error under all platforms. Previously, - its behavior was platform-dependent: a nonzero value was returned on success; - zero was returned on error under Windows. A zero value was returned on - success; an exception was raised on error under Unix. - (Contributed by Berker Peksag in :issue:`2122`.) - -* :mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process - external entities by default. - (Contributed by Christian Heimes in :issue:`17239`.) - -* Deleting a key from a read-only :mod:`dbm` database (:mod:`dbm.dumb`, - :mod:`dbm.gnu` or :mod:`dbm.ndbm`) raises :attr:`error` (:exc:`dbm.dumb.error`, - :exc:`dbm.gnu.error` or :exc:`dbm.ndbm.error`) instead of :exc:`KeyError`. - (Contributed by Xiang Zhang in :issue:`33106`.) - -* :func:`~os.path.expanduser` on Windows now prefers the :envvar:`USERPROFILE` - environment variable and does not use :envvar:`HOME`, which is not normally - set for regular user accounts. - (Contributed by Anthony Sottile in :issue:`36264`.) - -* The exception :class:`asyncio.CancelledError` now inherits from - :class:`BaseException` rather than :class:`Exception`. - (Contributed by Yury Selivanov in :issue:`32528`.) - -* The function :func:`asyncio.wait_for` now correctly waits for cancellation - when using an instance of :class:`asyncio.Task`. Previously, upon reaching - *timeout*, it was cancelled and immediately returned. - (Contributed by Elvis Pranskevichus in :issue:`32751`.) - -* The function :func:`asyncio.BaseTransport.get_extra_info` now returns a safe - to use socket object when 'socket' is passed to the *name* parameter. - (Contributed by Yury Selivanov in :issue:`37027`.) - -* :class:`asyncio.BufferedProtocol` has graduated to the stable API. - -.. _bpo-36085-whatsnew: - -* DLL dependencies for extension modules and DLLs loaded with :mod:`ctypes` on - Windows are now resolved more securely. Only the system paths, the directory - containing the DLL or PYD file, and directories added with - :func:`~os.add_dll_directory` are searched for load-time dependencies. - Specifically, :envvar:`PATH` and the current working directory are no longer - used, and modifications to these will no longer have any effect on normal DLL - resolution. If your application relies on these mechanisms, you should check - for :func:`~os.add_dll_directory` and if it exists, use it to add your DLLs - directory while loading your library. Note that Windows 7 users will need to - ensure that Windows Update KB2533623 has been installed (this is also verified - by the installer). - (Contributed by Steve Dower in :issue:`36085`.) - -* The header files and functions related to pgen have been removed after its - replacement by a pure Python implementation. (Contributed by Pablo Galindo - in :issue:`36623`.) - -* :class:`types.CodeType` has a new parameter in the second position of the - constructor (*posonlyargcount*) to support positional-only arguments defined - in :pep:`570`. The first argument (*argcount*) now represents the total - number of positional arguments (including positional-only arguments). The new - ``replace()`` method of :class:`types.CodeType` can be used to make the code - future-proof. - - -Changes in the C API --------------------- - -* The :c:type:`PyCompilerFlags` structure got a new *cf_feature_version* - field. It should be initialized to ``PY_MINOR_VERSION``. The field is ignored - by default, and is used if and only if ``PyCF_ONLY_AST`` flag is set in - *cf_flags*. - (Contributed by Guido van Rossum in :issue:`35766`.) - -* The :c:func:`PyEval_ReInitThreads` function has been removed from the C API. - It should not be called explicitly: use :c:func:`PyOS_AfterFork_Child` - instead. - (Contributed by Victor Stinner in :issue:`36728`.) - -* On Unix, C extensions are no longer linked to libpython except on Android - and Cygwin. When Python is embedded, ``libpython`` must not be loaded with - ``RTLD_LOCAL``, but ``RTLD_GLOBAL`` instead. Previously, using - ``RTLD_LOCAL``, it was already not possible to load C extensions which - were not linked to ``libpython``, like C extensions of the standard - library built by the ``*shared*`` section of ``Modules/Setup``. - (Contributed by Victor Stinner in :issue:`21536`.) - -* Use of ``#`` variants of formats in parsing or building value (e.g. - :c:func:`PyArg_ParseTuple`, :c:func:`Py_BuildValue`, :c:func:`PyObject_CallFunction`, - etc.) without ``PY_SSIZE_T_CLEAN`` defined raises ``DeprecationWarning`` now. - It will be removed in 3.10 or 4.0. Read :ref:`arg-parsing` for detail. - (Contributed by Inada Naoki in :issue:`36381`.) - -* Instances of heap-allocated types (such as those created with - :c:func:`PyType_FromSpec`) hold a reference to their type object. - Increasing the reference count of these type objects has been moved from - :c:func:`PyType_GenericAlloc` to the more low-level functions, - :c:func:`PyObject_Init` and :c:func:`PyObject_INIT`. - This makes types created through :c:func:`PyType_FromSpec` behave like - other classes in managed code. - - Statically allocated types are not affected. - - For the vast majority of cases, there should be no side effect. - However, types that manually increase the reference count after allocating - an instance (perhaps to work around the bug) may now become immortal. - To avoid this, these classes need to call Py_DECREF on the type object - during instance deallocation. - - To correctly port these types into 3.8, please apply the following - changes: - - * Remove :c:macro:`Py_INCREF` on the type object after allocating an - instance - if any. - This may happen after calling :c:func:`PyObject_New`, - :c:func:`PyObject_NewVar`, :c:func:`PyObject_GC_New`, - :c:func:`PyObject_GC_NewVar`, or any other custom allocator that uses - :c:func:`PyObject_Init` or :c:func:`PyObject_INIT`. - - Example: - - .. code-block:: c - - static foo_struct * - foo_new(PyObject *type) { - foo_struct *foo = PyObject_GC_New(foo_struct, (PyTypeObject *) type); - if (foo == NULL) - return NULL; - #if PY_VERSION_HEX < 0x03080000 - // Workaround for Python issue 35810; no longer necessary in Python 3.8 - PY_INCREF(type) - #endif - return foo; - } - - * Ensure that all custom ``tp_dealloc`` functions of heap-allocated types - decrease the type's reference count. - - Example: - - .. code-block:: c - - static void - foo_dealloc(foo_struct *instance) { - PyObject *type = Py_TYPE(instance); - PyObject_GC_Del(instance); - #if PY_VERSION_HEX >= 0x03080000 - // This was not needed before Python 3.8 (Python issue 35810) - Py_DECREF(type); - #endif - } - - (Contributed by Eddie Elizondo in :issue:`35810`.) - -* The :c:macro:`Py_DEPRECATED()` macro has been implemented for MSVC. - The macro now must be placed before the symbol name. - - Example: - - .. code-block:: c - - Py_DEPRECATED(3.8) PyAPI_FUNC(int) Py_OldFunction(void); - - (Contributed by Zackery Spytz in :issue:`33407`.) - -* The interpreter does not pretend to support binary compatibility of - extension types across feature releases, anymore. A :c:type:`PyTypeObject` - exported by a third-party extension module is supposed to have all the - slots expected in the current Python version, including - :c:member:`~PyTypeObject.tp_finalize` (:const:`Py_TPFLAGS_HAVE_FINALIZE` - is not checked anymore before reading :c:member:`~PyTypeObject.tp_finalize`). - - (Contributed by Antoine Pitrou in :issue:`32388`.) - -* The :c:func:`PyCode_New` has a new parameter in the second position (*posonlyargcount*) - to support :pep:`570`, indicating the number of positional-only arguments. - -* The functions :c:func:`PyNode_AddChild` and :c:func:`PyParser_AddToken` now accept - two additional ``int`` arguments *end_lineno* and *end_col_offset*. - -* The :file:`libpython38.a` file to allow MinGW tools to link directly against - :file:`python38.dll` is no longer included in the regular Windows distribution. - If you require this file, it may be generated with the ``gendef`` and - ``dlltool`` tools, which are part of the MinGW binutils package: - - .. code-block:: shell - - gendef python38.dll > tmp.def - dlltool --dllname python38.dll --def tmp.def --output-lib libpython38.a - - The location of an installed :file:`pythonXY.dll` will depend on the - installation options and the version and language of Windows. See - :ref:`using-on-windows` for more information. The resulting library should be - placed in the same directory as :file:`pythonXY.lib`, which is generally the - :file:`libs` directory under your Python installation. - - (Contributed by Steve Dower in :issue:`37351`.) - - -CPython bytecode changes ------------------------- - -* The interpreter loop has been simplified by moving the logic of unrolling - the stack of blocks into the compiler. The compiler emits now explicit - instructions for adjusting the stack of values and calling the - cleaning-up code for :keyword:`break`, :keyword:`continue` and - :keyword:`return`. - - Removed opcodes :opcode:`BREAK_LOOP`, :opcode:`CONTINUE_LOOP`, - :opcode:`SETUP_LOOP` and :opcode:`SETUP_EXCEPT`. Added new opcodes - :opcode:`ROT_FOUR`, :opcode:`BEGIN_FINALLY`, :opcode:`CALL_FINALLY` and - :opcode:`POP_FINALLY`. Changed the behavior of :opcode:`END_FINALLY` - and :opcode:`WITH_CLEANUP_START`. - - (Contributed by Mark Shannon, Antoine Pitrou and Serhiy Storchaka in - :issue:`17611`.) - -* Added new opcode :opcode:`END_ASYNC_FOR` for handling exceptions raised - when awaiting a next item in an :keyword:`async for` loop. - (Contributed by Serhiy Storchaka in :issue:`33041`.) - -* The :opcode:`MAP_ADD` now expects the value as the first element in the - stack and the key as the second element. This change was made so the key - is always evaluated before the value in dictionary comprehensions, as - proposed by :pep:`572`. (Contributed by Jörn Heissler in :issue:`35224`.) - - -Demos and Tools ---------------- - -Added a benchmark script for timing various ways to access variables: -``Tools/scripts/var_access_benchmark.py``. -(Contributed by Raymond Hettinger in :issue:`35884`.) - -Here's a summary of performance improvements since Python 3.3: - -.. code-block:: none - - Python version 3.3 3.4 3.5 3.6 3.7 3.8 - -------------- --- --- --- --- --- --- - - Variable and attribute read access: - read_local 4.0 7.1 7.1 5.4 5.1 3.9 - read_nonlocal 5.3 7.1 8.1 5.8 5.4 4.4 - read_global 13.3 15.5 19.0 14.3 13.6 7.6 - read_builtin 20.0 21.1 21.6 18.5 19.0 7.5 - read_classvar_from_class 20.5 25.6 26.5 20.7 19.5 18.4 - read_classvar_from_instance 18.5 22.8 23.5 18.8 17.1 16.4 - read_instancevar 26.8 32.4 33.1 28.0 26.3 25.4 - read_instancevar_slots 23.7 27.8 31.3 20.8 20.8 20.2 - read_namedtuple 68.5 73.8 57.5 45.0 46.8 18.4 - read_boundmethod 29.8 37.6 37.9 29.6 26.9 27.7 - - Variable and attribute write access: - write_local 4.6 8.7 9.3 5.5 5.3 4.3 - write_nonlocal 7.3 10.5 11.1 5.6 5.5 4.7 - write_global 15.9 19.7 21.2 18.0 18.0 15.8 - write_classvar 81.9 92.9 96.0 104.6 102.1 39.2 - write_instancevar 36.4 44.6 45.8 40.0 38.9 35.5 - write_instancevar_slots 28.7 35.6 36.1 27.3 26.6 25.7 - - Data structure read access: - read_list 19.2 24.2 24.5 20.8 20.8 19.0 - read_deque 19.9 24.7 25.5 20.2 20.6 19.8 - read_dict 19.7 24.3 25.7 22.3 23.0 21.0 - read_strdict 17.9 22.6 24.3 19.5 21.2 18.9 - - Data structure write access: - write_list 21.2 27.1 28.5 22.5 21.6 20.0 - write_deque 23.8 28.7 30.1 22.7 21.8 23.5 - write_dict 25.9 31.4 33.3 29.3 29.2 24.7 - write_strdict 22.9 28.4 29.9 27.5 25.2 23.1 - - Stack (or queue) operations: - list_append_pop 144.2 93.4 112.7 75.4 74.2 50.8 - deque_append_pop 30.4 43.5 57.0 49.4 49.2 42.5 - deque_append_popleft 30.8 43.7 57.3 49.7 49.7 42.8 - - Timing loop: - loop_overhead 0.3 0.5 0.6 0.4 0.3 0.3 - - (Measured from the macOS 64-bit builds found at python.org) diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst deleted file mode 100644 index 7cf49bf..0000000 --- a/Doc/whatsnew/3.9.rst +++ /dev/null @@ -1,374 +0,0 @@ -**************************** - What's New In Python 3.9 -**************************** - -:Release: |release| -:Date: |today| - -.. Rules for maintenance: - - * Anyone can add text to this document. Do not spend very much time - on the wording of your changes, because your text will probably - get rewritten to some degree. - - * The maintainer will go through Misc/NEWS periodically and add - changes; it's therefore more important to add your changes to - Misc/NEWS than to this file. - - * This is not a complete list of every single change; completeness - is the purpose of Misc/NEWS. Some changes I consider too small - or esoteric to include. If such a change is added to the text, - I'll just remove it. (This is another reason you shouldn't spend - too much time on writing your addition.) - - * If you want to draw your new text to the attention of the - maintainer, add 'XXX' to the beginning of the paragraph or - section. - - * It's OK to just add a fragmentary note about a change. For - example: "XXX Describe the transmogrify() function added to the - socket module." The maintainer will research the change and - write the necessary text. - - * You can comment out your additions if you like, but it's not - necessary (especially when a final release is some months away). - - * Credit the author of a patch or bugfix. Just the name is - sufficient; the e-mail address isn't necessary. - - * It's helpful to add the bug/patch number as a comment: - - XXX Describe the transmogrify() function added to the socket - module. - (Contributed by P.Y. Developer in :issue:`12345`.) - - This saves the maintainer the effort of going through the Mercurial log - when researching a change. - -This article explains the new features in Python 3.9, compared to 3.8. - -For full details, see the :ref:`changelog <changelog>`. - -.. note:: - - Prerelease users should be aware that this document is currently in draft - form. It will be updated substantially as Python 3.9 moves towards release, - so it's worth checking back even after reading earlier versions. - - -Summary -- Release highlights -============================= - -.. This section singles out the most important changes in Python 3.9. - Brevity is key. - - -.. PEP-sized items next. - - - -New Features -============ - - - -Other Language Changes -====================== - -* :func:`__import__` now raises :exc:`ImportError` instead of - :exc:`ValueError`, which used to occur when a relative import went past - its top-level package. - (Contributed by Ngalim Siregar in :issue:`37444`.) - - -* Python now gets the absolute path of the script filename specified on - the command line (ex: ``python3 script.py``): the ``__file__`` attribute of - the :mod:`__main__` module, ``sys.argv[0]`` and ``sys.path[0]`` become an - absolute path, rather than a relative path. These paths now remain valid - after the current directory is changed by :func:`os.chdir`. As a side effect, - a traceback also displays the absolute path for :mod:`__main__` module frames - in this case. - (Contributed by Victor Stinner in :issue:`20443`.) - -* In development mode and in debug build, *encoding* and *errors* arguments are - now checked on string encoding and decoding operations. Examples: - :func:`open`, :meth:`str.encode` and :meth:`bytes.decode`. - - By default, for best performance, the *errors* argument is only checked at - the first encoding/decoding error and the *encoding* argument is sometimes - ignored for empty strings. - (Contributed by Victor Stinner in :issue:`37388`.) - -* ``"".replace("", s, n)`` now returns ``s`` instead of an empty string for - all non-zero ``n``. It is now consistent with ``"".replace("", s)``. - There are similar changes for :class:`bytes` and :class:`bytearray` objects. - (Contributed by Serhiy Storchaka in :issue:`28029`.) - - -New Modules -=========== - -* None yet. - - -Improved Modules -================ - -ast ---- - -Added the *indent* option to :func:`~ast.dump` which allows it to produce a -multiline indented output. -(Contributed by Serhiy Storchaka in :issue:`37995`.) - -Added :func:`ast.unparse` as a function in the :mod:`ast` module that can -be used to unparse an :class:`ast.AST` object and produce a string with code -that would produce an equivalent :class:`ast.AST` object when parsed. -(Contributed by Pablo Galindo and Batuhan Taskaya in :issue:`38870`.) - -asyncio -------- - -Added a new :term:`coroutine` :meth:`~asyncio.loop.shutdown_default_executor` -that schedules a shutdown for the default executor that waits on the -:class:`~concurrent.futures.ThreadPoolExecutor` to finish closing. Also, -:func:`asyncio.run` has been updated to use the new :term:`coroutine`. -(Contributed by Kyle Stanley in :issue:`34037`.) - -Added :class:`asyncio.PidfdChildWatcher`, a Linux-specific child watcher -implementation that polls process file descriptors. (:issue:`38692`) - -curses ------- - -Add :func:`curses.get_escdelay`, :func:`curses.set_escdelay`, -:func:`curses.get_tabsize`, and :func:`curses.set_tabsize` functions. -(Contributed by Anthony Sottile in :issue:`38312`.) - -fcntl ------ - -Added constants :data:`~fcntl.F_OFD_GETLK`, :data:`~fcntl.F_OFD_SETLK` -and :data:`~fcntl.F_OFD_SETLKW`. -(Contributed by Dong-hee Na in :issue:`38602`.) - -os --- - -Added :data:`~os.CLD_KILLED` and :data:`~os.CLD_STOPPED` for :attr:`si_code`. -(Contributed by Dong-hee Na in :issue:`38493`.) - -Exposed the Linux-specific :func:`os.pidfd_open` (:issue:`38692`) and -:data:`os.P_PIDFD` (:issue:`38713`) for process management with file -descriptors. - -threading ---------- - -In a subinterpreter, spawning a daemon thread now raises a :exc:`RuntimeError`. Daemon -threads were never supported in subinterpreters. Previously, the subinterpreter -finalization crashed with a Python fatal error if a daemon thread was still -running. -(Contributed by Victor Stinner in :issue:`37266`.) - -venv ----- - -The activation scripts provided by :mod:`venv` now all specify their prompt -customization consistently by always using the value specified by -``__VENV_PROMPT__``. Previously some scripts unconditionally used -``__VENV_PROMPT__``, others only if it happened to be set (which was the default -case), and one used ``__VENV_NAME__`` instead. -(Contributed by Brett Cannon in :issue:`37663`.) - -pathlib -------- - -Added :meth:`pathlib.Path.readlink()` which acts similarly to -:func:`os.readlink`. -(Contributed by Girts Folkmanis in :issue:`30618`) - -pprint ------- - -:mod:`pprint` can now pretty-print :class:`types.SimpleNamespace`. -(Contributed by Carl Bordum Hansen in :issue:`37376`.) - -importlib ---------- - -To improve consistency with import statements, :func:`importlib.util.resolve_name` -now raises :exc:`ImportError` instead of :exc:`ValueError` for invalid relative -import attempts. -(Contributed by Ngalim Siregar in :issue:`37444`.) - -signal ------- - -Exposed the Linux-specific :func:`signal.pidfd_send_signal` for sending to -signals to a process using a file descriptor instead of a pid. (:issue:`38712`) - - -Optimizations -============= - - -Build and C API Changes -======================= - -* Provide :c:func:`Py_EnterRecursiveCall` and :c:func:`Py_LeaveRecursiveCall` - as regular functions for the limited API. Previously, there were defined as - macros, but these macros didn't work with the limited API which cannot access - ``PyThreadState.recursion_depth`` field. Remove ``_Py_CheckRecursionLimit`` - from the stable ABI. - (Contributed by Victor Stinner in :issue:`38644`.) - -* Add a new public :c:func:`PyObject_CallNoArgs` function to the C API, which - calls a callable Python object without any arguments. It is the most efficient - way to call a callable Python object without any argument. - (Contributed by Victor Stinner in :issue:`37194`.) - -* The global variable :c:data:`PyStructSequence_UnnamedField` is now a constant - and refers to a constant string. - (Contributed by Serhiy Storchaka in :issue:`38650`.) - -* Exclude ``PyFPE_START_PROTECT()`` and ``PyFPE_END_PROTECT()`` macros of - ``pyfpe.h`` from ``Py_LIMITED_API`` (stable API). - (Contributed by Victor Stinner in :issue:`38835`.) - -* Remove ``PyMethod_ClearFreeList()`` and ``PyCFunction_ClearFreeList()`` - functions: the free lists of bound method objects have been removed. - (Contributed by Inada Naoki and Victor Stinner in :issue:`37340`.) - -* Remove ``PyUnicode_ClearFreeList()`` function: the Unicode free list has been - removed in Python 3.3. - (Contributed by Victor Stinner in :issue:`38896`.) - - -Deprecated -========== - -* Currently :func:`math.factorial` accepts :class:`float` instances with - non-negative integer values (like ``5.0``). It raises a :exc:`ValueError` - for non-integral and negative floats. It is now deprecated. In future - Python versions it will raise a :exc:`TypeError` for all floats. - (Contributed by Serhiy Storchaka in :issue:`37315`.) - -* The :mod:`parser` module is deprecated and will be removed in future versions - of Python. For the majority of use cases, users can leverage the Abstract Syntax - Tree (AST) generation and compilation stage, using the :mod:`ast` module. - -* The :mod:`random` module currently accepts any hashable type as a - possible seed value. Unfortunately, some of those types are not - guaranteed to have a deterministic hash value. After Python 3.9, - the module will restrict its seeds to :const:`None`, :class:`int`, - :class:`float`, :class:`str`, :class:`bytes`, and :class:`bytearray`. - -* Opening the :class:`~gzip.GzipFile` file for writing without specifying - the *mode* argument is deprecated. In future Python versions it will always - be opened for reading by default. Specify the *mode* argument for opening - it for writing and silencing a warning. - (Contributed by Serhiy Storchaka in :issue:`28286`.) - -* Deprecated the ``split()`` method of :class:`_tkinter.TkappType` in - favour of the ``splitlist()`` method which has more consistent and - predicable behavior. - (Contributed by Serhiy Storchaka in :issue:`38371`.) - - -Removed -======= - -* :class:`array.array`: ``tostring()`` and ``fromstring()`` methods have been - removed. They were aliases to ``tobytes()`` and ``frombytes()``, deprecated - since Python 3.2. - (Contributed by Victor Stinner in :issue:`38916`.) - -* The abstract base classes in :mod:`collections.abc` no longer are - exposed in the regular :mod:`collections` module. This will help - create a clearer distinction between the concrete classes and the abstract - base classes. - -* The undocumented ``sys.callstats()`` function has been removed. Since Python - 3.7, it was deprecated and always returned :const:`None`. It required a special - build option ``CALL_PROFILE`` which was already removed in Python 3.7. - (Contributed by Victor Stinner in :issue:`37414`.) - -* The ``sys.getcheckinterval()`` and ``sys.setcheckinterval()`` functions have - been removed. They were deprecated since Python 3.2. Use - :func:`sys.getswitchinterval` and :func:`sys.setswitchinterval` instead. - (Contributed by Victor Stinner in :issue:`37392`.) - -* The C function ``PyImport_Cleanup()`` has been removed. It was documented as: - "Empty the module table. For internal use only." - (Contributed by Victor Stinner in :issue:`36710`.) - -* ``_dummy_thread`` and ``dummy_threading`` modules have been removed. These - modules were deprecated since Python 3.7 which requires threading support. - (Contributed by Victor Stinner in :issue:`37312`.) - -* ``aifc.openfp()`` alias to ``aifc.open()``, ``sunau.openfp()`` alias to - ``sunau.open()``, and ``wave.openfp()`` alias to :func:`wave.open()` have been - removed. They were deprecated since Python 3.7. - (Contributed by Victor Stinner in :issue:`37320`.) - -* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread` - has been removed. It was deprecated since Python 3.8. - Use :meth:`~threading.Thread.is_alive()` instead. - (Contributed by Dong-hee Na in :issue:`37804`.) - -* Methods ``getchildren()`` and ``getiterator()`` in the - :mod:`~xml.etree.ElementTree` module have been removed. They were - deprecated in Python 3.2. Use functions :func:`list` and :func:`iter` - instead. The ``xml.etree.cElementTree`` module has been removed. - (Contributed by Serhiy Storchaka in :issue:`36543`.) - -* The old :mod:`plistlib` API has been removed, it was deprecated since Python - 3.4. Use the :func:`~plistlib.load`, :func:`~plistlib.loads`, :func:`~plistlib.dump`, and - :func:`~plistlib.dumps` functions. Additionally, the *use_builtin_types* parameter was - removed, standard :class:`bytes` objects are always used instead. - (Contributed by Jon Janzen in :issue:`36409`.) - -* The C function ``PyThreadState_DeleteCurrent()`` has been removed. It was not documented. - (Contributed by Joannah Nanjekye in :issue:`37878`.) - -* The C function ``PyGen_NeedsFinalizing`` has been removed. It was not - documented, tested, or used anywhere within CPython after the implementation - of :pep:`442`. Patch by Joannah Nanjekye. - (Contributed by Joannah Nanjekye in :issue:`15088`) - - -Porting to Python 3.9 -===================== - -This section lists previously described changes and other bugfixes -that may require changes to your code. - - -Changes in the Python API -------------------------- - -* :func:`open`, :func:`io.open`, :func:`codecs.open` and - :class:`fileinput.FileInput` no longer accept ``'U'`` ("universal newline") - in the file mode. This flag was deprecated since Python 3.3. In Python 3, the - "universal newline" is used by default when a file is open in text mode. The - :ref:`newline parameter <open-newline-parameter>` of :func:`open` controls - how universal newlines works. - (Contributed by Victor Stinner in :issue:`37330`.) - -* :func:`__import__` and :func:`importlib.util.resolve_name` now raise - :exc:`ImportError` where it previously raised :exc:`ValueError`. Callers - catching the specific exception type and supporting both Python 3.9 and - earlier versions will need to catch both using ``except (ImportError, ValueError):``. - -* The :mod:`venv` activation scripts no longer special-case when - ``__VENV_PROMPT__`` is set to ``""``. - - -CPython bytecode changes ------------------------- - -* The :opcode:`LOAD_ASSERTION_ERROR` opcode was added for handling the - :keyword:`assert` statement. Previously, the assert statement would not work - correctly if the :exc:`AssertionError` exception was being shadowed. - (Contributed by Zackery Spytz in :issue:`34880`.) diff --git a/Doc/whatsnew/changelog.rst b/Doc/whatsnew/changelog.rst deleted file mode 100644 index b435614..0000000 --- a/Doc/whatsnew/changelog.rst +++ /dev/null @@ -1,7 +0,0 @@ -.. _changelog: - -+++++++++ -Changelog -+++++++++ - -.. miscnews:: ../build/NEWS diff --git a/Doc/whatsnew/index.rst b/Doc/whatsnew/index.rst index 954e38b..ec45756 100644 --- a/Doc/whatsnew/index.rst +++ b/Doc/whatsnew/index.rst @@ -11,16 +11,6 @@ anyone wishing to stay up-to-date after a new release. .. toctree:: :maxdepth: 2 - 3.9.rst - 3.8.rst - 3.7.rst - 3.6.rst - 3.5.rst - 3.4.rst - 3.3.rst - 3.2.rst - 3.1.rst - 3.0.rst 2.7.rst 2.6.rst 2.5.rst @@ -29,13 +19,3 @@ anyone wishing to stay up-to-date after a new release. 2.2.rst 2.1.rst 2.0.rst - -The "Changelog" is an HTML version of the `file built -<https://pypi.org/project/blurb>`_ from the contents of the -:source:`Misc/NEWS.d` directory tree, which contains *all* nontrivial changes -to Python for the current version. - -.. toctree:: - :maxdepth: 2 - - changelog.rst |