diff options
Diffstat (limited to 'Doc/whatsnew')
| -rw-r--r-- | Doc/whatsnew/3.5.rst | 2495 | ||||
| -rw-r--r-- | Doc/whatsnew/index.rst | 1 |
2 files changed, 2496 insertions, 0 deletions
diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst new file mode 100644 index 0000000..53cd602 --- /dev/null +++ b/Doc/whatsnew/3.5.rst @@ -0,0 +1,2495 @@ +**************************** + What's New In Python 3.5 +**************************** + +:Editors: Elvis Pranskevichus <elvis@magic.io>, Yury Selivanov <yury@magic.io> + +.. Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten to some degree. + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. + + * It's helpful to add the bug/patch number as a comment: + + XXX Describe the transmogrify() function added to the socket + module. + (Contributed by P.Y. Developer in :issue:`12345`.) + + This saves the maintainer the effort of going through the Mercurial log + when researching a change. + +This article explains the new features in Python 3.5, compared to 3.4. +Python 3.5 was released on September 13, 2015. See the +`changelog <https://docs.python.org/3.5/whatsnew/changelog.html>`_ for a full +list of changes. + +.. seealso:: + + :pep:`478` - Python 3.5 Release Schedule + + +Summary -- Release highlights +============================= + +New syntax features: + +* :ref:`PEP 492 <whatsnew-pep-492>`, coroutines with async and await syntax. +* :ref:`PEP 465 <whatsnew-pep-465>`, a new matrix multiplication operator: ``a @ b``. +* :ref:`PEP 448 <whatsnew-pep-448>`, additional unpacking generalizations. + + +New library modules: + +* :mod:`typing`: :ref:`PEP 484 -- Type Hints <whatsnew-pep-484>`. +* :mod:`zipapp`: :ref:`PEP 441 Improving Python ZIP Application Support + <whatsnew-zipapp>`. + + +New built-in features: + +* ``bytes % args``, ``bytearray % args``: :ref:`PEP 461 <whatsnew-pep-461>` -- + Adding ``%`` formatting to bytes and bytearray. + +* New :meth:`bytes.hex`, :meth:`bytearray.hex` and :meth:`memoryview.hex` + methods. (Contributed by Arnon Yaari in :issue:`9951`.) + +* :class:`memoryview` now supports tuple indexing (including multi-dimensional). + (Contributed by Antoine Pitrou in :issue:`23632`.) + +* Generators have a new ``gi_yieldfrom`` attribute, which returns the + object being iterated by ``yield from`` expressions. (Contributed + by Benno Leslie and Yury Selivanov in :issue:`24450`.) + +* A new :exc:`RecursionError` exception is now raised when maximum + recursion depth is reached. (Contributed by Georg Brandl + in :issue:`19235`.) + + +CPython implementation improvements: + +* When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), + :py:data:`sys.stdin` and :py:data:`sys.stdout` now use the + ``surrogateescape`` error handler, instead of the ``strict`` error handler. + (Contributed by Victor Stinner in :issue:`19977`.) + +* ``.pyo`` files are no longer used and have been replaced by a more flexible + scheme that includes the optimization level explicitly in ``.pyc`` name. + (See :ref:`PEP 488 overview <whatsnew-pep-488>`.) + +* Builtin and extension modules are now initialized in a multi-phase process, + which is similar to how Python modules are loaded. + (See :ref:`PEP 489 overview <whatsnew-pep-489>`.) + + +Significant improvements in the standard library: + +* :class:`collections.OrderedDict` is now + :ref:`implemented in C <whatsnew-ordereddict>`, which makes it + 4 to 100 times faster. + +* The :mod:`ssl` module gained + :ref:`support for Memory BIO <whatsnew-sslmemorybio>`, which decouples SSL + protocol handling from network IO. + +* The new :func:`os.scandir` function provides a + :ref:`better and significantly faster way <whatsnew-pep-471>` + of directory traversal. + +* :func:`functools.lru_cache` has been mostly + :ref:`reimplemented in C <whatsnew-lrucache>`, yielding much better + performance. + +* The new :func:`subprocess.run` function provides a + :ref:`streamlined way to run subprocesses <whatsnew-subprocess>`. + +* The :mod:`traceback` module has been significantly + :ref:`enhanced <whatsnew-traceback>` for improved + performance and developer convenience. + + +Security improvements: + +* SSLv3 is now disabled throughout the standard library. + It can still be enabled by instantiating a :class:`ssl.SSLContext` + manually. (See :issue:`22638` for more details; this change was + backported to CPython 3.4 and 2.7.) + +* HTTP cookie parsing is now stricter, in order to protect + against potential injection attacks. (Contributed by Antoine Pitrou + in :issue:`22796`.) + + +Windows improvements: + +* A new installer for Windows has replaced the old MSI. + See :ref:`using-on-windows` for more information. + +* Windows builds now use Microsoft Visual C++ 14.0, and extension modules + should use the same. + + +Please read on for a comprehensive list of user-facing changes, including many +other smaller improvements, CPython optimizations, deprecations, and potential +porting issues. + + +New Features +============ + +.. _whatsnew-pep-492: + +PEP 492 - Coroutines with async and await syntax +------------------------------------------------ + +:pep:`492` greatly improves support for asynchronous programming in Python +by adding :term:`awaitable objects <awaitable>`, +:term:`coroutine functions <coroutine function>`, +:term:`asynchronous iteration <asynchronous iterable>`, +and :term:`asynchronous context managers <asynchronous context manager>`. + +Coroutine functions are declared using the new :keyword:`async def` syntax:: + + >>> async def coro(): + ... return 'spam' + +Inside a coroutine function, the new :keyword:`await` expression can be used +to suspend coroutine execution until the result is available. Any object +can be *awaited*, as long as it implements the :term:`awaitable` protocol by +defining the :meth:`__await__` method. + +PEP 492 also adds :keyword:`async for` statement for convenient iteration +over asynchronous iterables. + +An example of a rudimentary HTTP client written using the new syntax:: + + import asyncio + + async def http_get(domain): + reader, writer = await asyncio.open_connection(domain, 80) + + writer.write(b'\r\n'.join([ + b'GET / HTTP/1.1', + b'Host: %b' % domain.encode('latin-1'), + b'Connection: close', + b'', b'' + ])) + + async for line in reader: + print('>>>', line) + + writer.close() + + loop = asyncio.get_event_loop() + try: + loop.run_until_complete(http_get('example.com')) + finally: + loop.close() + + +Similarly to asynchronous iteration, there is a new syntax for asynchronous +context managers. The following script:: + + import asyncio + + async def coro(name, lock): + print('coro {}: waiting for lock'.format(name)) + async with lock: + print('coro {}: holding the lock'.format(name)) + await asyncio.sleep(1) + print('coro {}: releasing the lock'.format(name)) + + loop = asyncio.get_event_loop() + lock = asyncio.Lock() + coros = asyncio.gather(coro(1, lock), coro(2, lock)) + try: + loop.run_until_complete(coros) + finally: + loop.close() + +will output:: + + coro 2: waiting for lock + coro 2: holding the lock + coro 1: waiting for lock + coro 2: releasing the lock + coro 1: holding the lock + coro 1: releasing the lock + +Note that both :keyword:`async for` and :keyword:`async with` can only +be used inside a coroutine function declared with :keyword:`async def`. + +Coroutine functions are intended to be run inside a compatible event loop, +such as the :ref:`asyncio loop <asyncio-event-loop>`. + +.. seealso:: + + :pep:`492` -- Coroutines with async and await syntax + PEP written and implemented by Yury Selivanov. + + +.. _whatsnew-pep-465: + +PEP 465 - A dedicated infix operator for matrix multiplication +-------------------------------------------------------------- + +:pep:`465` adds the ``@`` infix operator for matrix multiplication. +Currently, no builtin Python types implement the new operator, however, it +can be implemented by defining :meth:`__matmul__`, :meth:`__rmatmul__`, +and :meth:`__imatmul__` for regular, reflected, and in-place matrix +multiplication. The semantics of these methods is similar to that of +methods defining other infix arithmetic operators. + +Matrix multiplication is a notably common operation in many fields of +mathematics, science, engineering, and the addition of ``@`` allows writing +cleaner code:: + + S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r) + +instead of:: + + S = dot((dot(H, beta) - r).T, + dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r)) + +NumPy 1.10 has support for the new operator:: + + >>> import numpy + + >>> x = numpy.ones(3) + >>> x + array([ 1., 1., 1.]) + + >>> m = numpy.eye(3) + >>> m + array([[ 1., 0., 0.], + [ 0., 1., 0.], + [ 0., 0., 1.]]) + + >>> x @ m + array([ 1., 1., 1.]) + + +.. seealso:: + + :pep:`465` -- A dedicated infix operator for matrix multiplication + PEP written by Nathaniel J. Smith; implemented by Benjamin Peterson. + + +.. _whatsnew-pep-448: + +PEP 448 - Additional Unpacking Generalizations +---------------------------------------------- + +:pep:`448` extends the allowed uses of the ``*`` iterable unpacking +operator and ``**`` dictionary unpacking operator. It is now possible +to use an arbitrary number of unpackings in function calls:: + + >>> print(*[1], *[2], 3, *[4, 5]) + 1 2 3 4 5 + + >>> def fn(a, b, c, d): + ... print(a, b, c, d) + ... + + >>> fn(**{'a': 1, 'c': 3}, **{'b': 2, 'd': 4}) + 1 2 3 4 + +Similarly, tuple, list, set, and dictionary displays allow multiple +unpackings:: + + >>> *range(4), 4 + (0, 1, 2, 3, 4) + + >>> [*range(4), 4] + [0, 1, 2, 3, 4] + + >>> {*range(4), 4, *(5, 6, 7)} + {0, 1, 2, 3, 4, 5, 6, 7} + + >>> {'x': 1, **{'y': 2}} + {'x': 1, 'y': 2} + +.. seealso:: + + :pep:`448` -- Additional Unpacking Generalizations + PEP written by Joshua Landau; implemented by Neil Girdhar, + Thomas Wouters, and Joshua Landau. + + +.. _whatsnew-pep-461: + +PEP 461 - % formatting support for bytes and bytearray +------------------------------------------------------ + +:pep:`461` adds support for the ``%`` +:ref:`interpolation operator <bytes-formatting>` to :class:`bytes` +and :class:`bytearray`. + +While interpolation is usually thought of as a string operation, there are +cases where interpolation on ``bytes`` or ``bytearrays`` makes sense, and the +work needed to make up for this missing functionality detracts from the +overall readability of the code. This issue is particularly important when +dealing with wire format protocols, which are often a mixture of binary +and ASCII compatible text. + +Examples:: + + >>> b'Hello %b!' % b'World' + b'Hello World!' + + >>> b'x=%i y=%f' % (1, 2.5) + b'x=1 y=2.500000' + +Unicode is not allowed for ``%b``, but it is accepted by ``%a`` (equivalent of +``repr(obj).encode('ascii', 'backslashreplace')``):: + + >>> b'Hello %b!' % 'World' + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: %b requires bytes, or an object that implements __bytes__, not 'str' + + >>> b'price: %a' % '10€' + b"price: '10\\u20ac'" + +Note that ``%s`` and ``%r`` conversion types, although supported, should +only be used in codebases that need compatibility with Python 2. + +.. seealso:: + + :pep:`461` -- Adding % formatting to bytes and bytearray + PEP written by Ethan Furman; implemented by Neil Schemenauer and + Ethan Furman. + + +.. _whatsnew-pep-484: + +PEP 484 - Type Hints +-------------------- + +Function annotation syntax has been a Python feature since version 3.0 +(:pep:`3107`), however the semantics of annotations has been left undefined. + +Experience has shown that the majority of function annotation +uses were to provide type hints to function parameters and return values. It +became evident that it would be beneficial for Python users, if the +standard library included the base definitions and tools for type annotations. + +:pep:`484` introduces a :term:`provisional module <provisional api>` to +provide these standard definitions and tools, along with some conventions +for situations where annotations are not available. + +For example, here is a simple function whose argument and return type +are declared in the annotations:: + + def greeting(name: str) -> str: + return 'Hello ' + name + +While these annotations are available at runtime through the usual +:attr:`__annotations__` attribute, *no automatic type checking happens at +runtime*. Instead, it is assumed that a separate off-line type checker +(e.g. `mypy <http://mypy-lang.org>`_) will be used for on-demand +source code analysis. + +The type system supports unions, generic types, and a special type +named :class:`~typing.Any` which is consistent with (i.e. assignable to +and from) all types. + +.. seealso:: + + * :mod:`typing` module documentation + * :pep:`484` -- Type Hints + PEP written by Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa; + implemented by Guido van Rossum. + * :pep:`483` -- The Theory of Type Hints + PEP written by Guido van Rossum + + +.. _whatsnew-pep-471: + +PEP 471 - os.scandir() function -- a better and faster directory iterator +------------------------------------------------------------------------- + +:pep:`471` adds a new directory iteration function, :func:`os.scandir`, +to the standard library. Additionally, :func:`os.walk` is now +implemented using ``scandir``, which makes it 3 to 5 times faster +on POSIX systems and 7 to 20 times faster on Windows systems. This is +largely achieved by greatly reducing the number of calls to :func:`os.stat` +required to walk a directory tree. + +Additionally, ``scandir`` returns an iterator, as opposed to returning +a list of file names, which improves memory efficiency when iterating +over very large directories. + +The following example shows a simple use of :func:`os.scandir` to display all +the files (excluding directories) in the given *path* that don't start with +``'.'``. The :meth:`entry.is_file() <os.DirEntry.is_file>` call will generally +not make an additional system call:: + + for entry in os.scandir(path): + if not entry.name.startswith('.') and entry.is_file(): + print(entry.name) + +.. seealso:: + + :pep:`471` -- os.scandir() function -- a better and faster directory iterator + PEP written and implemented by Ben Hoyt with the help of Victor Stinner. + + +.. _whatsnew-pep-475: + +PEP 475: Retry system calls failing with EINTR +---------------------------------------------- + +An :py:data:`errno.EINTR` error code is returned whenever a system call, that +is waiting for I/O, is interrupted by a signal. Previously, Python would +raise :exc:`InterruptedError` in such cases. This meant that, when writing a +Python application, the developer had two choices: + +#. Ignore the ``InterruptedError``. +#. Handle the ``InterruptedError`` and attempt to restart the interrupted + system call at every call site. + +The first option makes an application fail intermittently. +The second option adds a large amount of boilerplate that makes the +code nearly unreadable. Compare:: + + print("Hello World") + +and:: + + while True: + try: + print("Hello World") + break + except InterruptedError: + continue + +:pep:`475` implements automatic retry of system calls on +``EINTR``. This removes the burden of dealing with ``EINTR`` +or :exc:`InterruptedError` in user code in most situations and makes +Python programs, including the standard library, more robust. Note that +the system call is only retried if the signal handler does not raise an +exception. + +Below is a list of functions which are now retried when interrupted +by a signal: + +* :func:`open` and :func:`io.open`; + +* functions of the :mod:`faulthandler` module; + +* :mod:`os` functions: :func:`~os.fchdir`, :func:`~os.fchmod`, + :func:`~os.fchown`, :func:`~os.fdatasync`, :func:`~os.fstat`, + :func:`~os.fstatvfs`, :func:`~os.fsync`, :func:`~os.ftruncate`, + :func:`~os.mkfifo`, :func:`~os.mknod`, :func:`~os.open`, + :func:`~os.posix_fadvise`, :func:`~os.posix_fallocate`, :func:`~os.pread`, + :func:`~os.pwrite`, :func:`~os.read`, :func:`~os.readv`, :func:`~os.sendfile`, + :func:`~os.wait3`, :func:`~os.wait4`, :func:`~os.wait`, + :func:`~os.waitid`, :func:`~os.waitpid`, :func:`~os.write`, + :func:`~os.writev`; + +* special cases: :func:`os.close` and :func:`os.dup2` now ignore + :py:data:`~errno.EINTR` errors; the syscall is not retried (see the PEP + for the rationale); + +* :mod:`select` functions: :func:`devpoll.poll() <select.devpoll.poll>`, + :func:`epoll.poll() <select.epoll.poll>`, + :func:`kqueue.control() <select.kqueue.control>`, + :func:`poll.poll() <select.poll.poll>`, :func:`~select.select`; + +* methods of the :class:`~socket.socket` class: :meth:`~socket.socket.accept`, + :meth:`~socket.socket.connect` (except for non-blocking sockets), + :meth:`~socket.socket.recv`, :meth:`~socket.socket.recvfrom`, + :meth:`~socket.socket.recvmsg`, :meth:`~socket.socket.send`, + :meth:`~socket.socket.sendall`, :meth:`~socket.socket.sendmsg`, + :meth:`~socket.socket.sendto`; + +* :func:`signal.sigtimedwait` and :func:`signal.sigwaitinfo`; + +* :func:`time.sleep`. + +.. seealso:: + + :pep:`475` -- Retry system calls failing with EINTR + PEP and implementation written by Charles-François Natali and + Victor Stinner, with the help of Antoine Pitrou (the French connection). + + +.. _whatsnew-pep-479: + +PEP 479: Change StopIteration handling inside generators +-------------------------------------------------------- + +The interaction of generators and :exc:`StopIteration` in Python 3.4 and +earlier was sometimes surprising, and could conceal obscure bugs. Previously, +``StopIteration`` raised accidentally inside a generator function was +interpreted as the end of the iteration by the loop construct driving the +generator. + +:pep:`479` changes the behavior of generators: when a ``StopIteration`` +exception is raised inside a generator, it is replaced with a +:exc:`RuntimeError` before it exits the generator frame. The main goal of +this change is to ease debugging in the situation where an unguarded +:func:`next` call raises ``StopIteration`` and causes the iteration controlled +by the generator to terminate silently. This is particularly pernicious in +combination with the ``yield from`` construct. + +This is a backwards incompatible change, so to enable the new behavior, +a :term:`__future__` import is necessary:: + + >>> from __future__ import generator_stop + + >>> def gen(): + ... next(iter([])) + ... yield + ... + >>> next(gen()) + Traceback (most recent call last): + File "<stdin>", line 2, in gen + StopIteration + + The above exception was the direct cause of the following exception: + + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + RuntimeError: generator raised StopIteration + +Without a ``__future__`` import, a :exc:`PendingDeprecationWarning` will be +raised whenever a ``StopIteration`` exception is raised inside a generator. + +.. seealso:: + + :pep:`479` -- Change StopIteration handling inside generators + PEP written by Chris Angelico and Guido van Rossum. Implemented by + Chris Angelico, Yury Selivanov and Nick Coghlan. + + +.. _whatsnew-pep-485: + +PEP 485: A function for testing approximate equality +---------------------------------------------------- + +:pep:`485` adds the :func:`math.isclose` and :func:`cmath.isclose` +functions which tell whether two values are approximately equal or +"close" to each other. Whether or not two values are considered +close is determined according to given absolute and relative tolerances. +Relative tolerance is the maximum allowed difference between ``isclose`` +arguments, relative to the larger absolute value:: + + >>> import math + >>> a = 5.0 + >>> b = 4.99998 + >>> math.isclose(a, b, rel_tol=1e-5) + True + >>> math.isclose(a, b, rel_tol=1e-6) + False + +It is also possible to compare two values using absolute tolerance, which +must be a non-negative value:: + + >>> import math + >>> a = 5.0 + >>> b = 4.99998 + >>> math.isclose(a, b, abs_tol=0.00003) + True + >>> math.isclose(a, b, abs_tol=0.00001) + False + +.. seealso:: + + :pep:`485` -- A function for testing approximate equality + PEP written by Christopher Barker; implemented by Chris Barker and + Tal Einat. + + +.. _whatsnew-pep-486: + +PEP 486: Make the Python Launcher aware of virtual environments +--------------------------------------------------------------- + +:pep:`486` makes the Windows launcher (see :pep:`397`) aware of an active +virtual environment. When the default interpreter would be used and the +``VIRTUAL_ENV`` environment variable is set, the interpreter in the virtual +environment will be used. + +.. seealso:: + + :pep:`486` -- Make the Python Launcher aware of virtual environments + PEP written and implemented by Paul Moore. + + +.. _whatsnew-pep-488: + +PEP 488: Elimination of PYO files +--------------------------------- + +:pep:`488` does away with the concept of ``.pyo`` files. This means that +``.pyc`` files represent both unoptimized and optimized bytecode. To prevent the +need to constantly regenerate bytecode files, ``.pyc`` files now have an +optional ``opt-`` tag in their name when the bytecode is optimized. This has the +side-effect of no more bytecode file name clashes when running under either +:option:`-O` or :option:`-OO`. Consequently, bytecode files generated from +:option:`-O`, and :option:`-OO` may now exist simultaneously. +:func:`importlib.util.cache_from_source` has an updated API to help with +this change. + +.. seealso:: + + :pep:`488` -- Elimination of PYO files + PEP written and implemented by Brett Cannon. + + +.. _whatsnew-pep-489: + +PEP 489: Multi-phase extension module initialization +---------------------------------------------------- + +:pep:`489` updates extension module initialization to take advantage of the +two step module loading mechanism introduced by :pep:`451` in Python 3.4. + +This change brings the import semantics of extension modules that opt-in to +using the new mechanism much closer to those of Python source and bytecode +modules, including the ability to use any valid identifier as a module name, +rather than being restricted to ASCII. + +.. seealso:: + + :pep:`489` -- Multi-phase extension module initialization + PEP written by Petr Viktorin, Stefan Behnel, and Nick Coghlan; + implemented by Petr Viktorin. + + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* Added the ``"namereplace"`` error handlers. The ``"backslashreplace"`` + error handlers now work with decoding and translating. + (Contributed by Serhiy Storchaka in :issue:`19676` and :issue:`22286`.) + +* The :option:`-b` option now affects comparisons of :class:`bytes` with + :class:`int`. (Contributed by Serhiy Storchaka in :issue:`23681`.) + +* New Kazakh ``kz1048`` and Tajik ``koi8_t`` :ref:`codecs <standard-encodings>`. + (Contributed by Serhiy Storchaka in :issue:`22682` and :issue:`22681`.) + +* Property docstrings are now writable. This is especially useful for + :func:`collections.namedtuple` docstrings. + (Contributed by Berker Peksag in :issue:`24064`.) + +* Circular imports involving relative imports are now supported. + (Contributed by Brett Cannon and Antoine Pitrou in :issue:`17636`.) + + +New Modules +=========== + +typing +------ + +The new :mod:`typing` :term:`provisional <provisional api>` module +provides standard definitions and tools for function type annotations. +See :ref:`Type Hints <whatsnew-pep-484>` for more information. + +.. _whatsnew-zipapp: + +zipapp +------ + +The new :mod:`zipapp` module (specified in :pep:`441`) provides an API and +command line tool for creating executable Python Zip Applications, which +were introduced in Python 2.6 in :issue:`1739468`, but which were not well +publicized, either at the time or since. + +With the new module, bundling your application is as simple as putting all +the files, including a ``__main__.py`` file, into a directory ``myapp`` +and running:: + + $ python -m zipapp myapp + $ python myapp.pyz + +The module implementation has been contributed by Paul Moore in +:issue:`23491`. + +.. seealso:: + + :pep:`441` -- Improving Python ZIP Application Support + + +Improved Modules +================ + +argparse +-------- + +The :class:`~argparse.ArgumentParser` class now allows to disable +:ref:`abbreviated usage <prefix-matching>` of long options by setting +:ref:`allow_abbrev` to ``False``. (Contributed by Jonathan Paugh, +Steven Bethard, paul j3 and Daniel Eriksson in :issue:`14910`.) + + +asyncio +------- + +Since the :mod:`asyncio` module is :term:`provisional <provisional api>`, +all changes introduced in Python 3.5 have also been backported to Python 3.4.x. + +Notable changes in the :mod:`asyncio` module since Python 3.4.0: + +* New debugging APIs: :meth:`loop.set_debug() <asyncio.BaseEventLoop.set_debug>` + and :meth:`loop.get_debug() <asyncio.BaseEventLoop.get_debug>` methods. + (Contributed by Victor Stinner.) + +* The proactor event loop now supports SSL. + (Contributed by Antoine Pitrou and Victor Stinner in :issue:`22560`.) + +* A new :meth:`loop.is_closed() <asyncio.BaseEventLoop.is_closed>` method to + check if the event loop is closed. + (Contributed by Victor Stinner in :issue:`21326`.) + +* A new :meth:`loop.create_task() <asyncio.BaseEventLoop.create_task>` + to conveniently create and schedule a new :class:`~asyncio.Task` + for a coroutine. The ``create_task`` method is also used by all + asyncio functions that wrap coroutines into tasks, such as + :func:`asyncio.wait`, :func:`asyncio.gather`, etc. + (Contributed by Victor Stinner.) + +* A new :meth:`transport.get_write_buffer_limits() <asyncio.WriteTransport.get_write_buffer_limits>` + method to inquire for *high-* and *low-* water limits of the flow + control. + (Contributed by Victor Stinner.) + +* The :func:`~asyncio.async` function is deprecated in favor of + :func:`~asyncio.ensure_future`. + (Contributed by Yury Selivanov.) + +* New :meth:`loop.set_task_factory() <asyncio.BaseEventLoop.set_task_factory>` + and :meth:`loop.set_task_factory() <asyncio.BaseEventLoop.get_task_factory>` + methods to customize the task factory that + :meth:`loop.create_task() <asyncio.BaseEventLoop.create_task>` method uses. + (Contributed by Yury Selivanov.) + +* New :meth:`Queue.join() <asyncio.Queue.join>` and + :meth:`Queue.task_done() <asyncio.Queue.task_done>` queue methods. + (Contributed by Victor Stinner.) + +* The ``JoinableQueue`` class was removed, in favor of the + :class:`asyncio.Queue` class. + (Contributed by Victor Stinner.) + +Updates in 3.5.1: + +* The :func:`~asyncio.ensure_future` function and all functions that + use it, such as :meth:`loop.run_until_complete() <asyncio.BaseEventLoop.run_until_complete>`, + now accept all kinds of :term:`awaitable objects <awaitable>`. + (Contributed by Yury Selivanov.) + +* New :func:`~asyncio.run_coroutine_threadsafe` function to submit + coroutines to event loops from other threads. + (Contributed by Vincent Michel.) + +* New :meth:`Transport.is_closing() <asyncio.BaseTransport.is_closing>` + method to check if the transport is closing or closed. + (Contributed by Yury Selivanov.) + +* The :meth:`loop.create_server() <asyncio.BaseEventLoop.create_server>` + method can now accept a list of hosts. + (Contributed by Yann Sionneau.) + + +bz2 +--- + +The :meth:`BZ2Decompressor.decompress <bz2.BZ2Decompressor.decompress>` +method now accepts an optional *max_length* argument to limit the maximum +size of decompressed data. (Contributed by Nikolaus Rath in :issue:`15955`.) + + +cgi +--- + +The :class:`~cgi.FieldStorage` class now supports the :term:`context manager` +protocol. (Contributed by Berker Peksag in :issue:`20289`.) + + +cmath +----- + +A new function :func:`~cmath.isclose` provides a way to test for approximate +equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) + + +code +---- + +The :func:`InteractiveInterpreter.showtraceback() <code.InteractiveInterpreter.showtraceback>` +method now prints the full chained traceback, just like the interactive +interpreter. (Contributed by Claudiu Popa in :issue:`17442`.) + + +collections +----------- + +.. _whatsnew-ordereddict: + +The :class:`~collections.OrderedDict` class is now implemented in C, which +makes it 4 to 100 times faster. (Contributed by Eric Snow in :issue:`16991`.) + +:meth:`OrderedDict.items() <collections.OrderedDict.items>`, +:meth:`OrderedDict.keys() <collections.OrderedDict.keys>`, +:meth:`OrderedDict.values() <collections.OrderedDict.values>` views now support +:func:`reversed` iteration. +(Contributed by Serhiy Storchaka in :issue:`19505`.) + +The :class:`~collections.deque` class now defines +:meth:`~collections.deque.index`, :meth:`~collections.deque.insert`, and +:meth:`~collections.deque.copy`, and supports the ``+`` and ``*`` operators. +This allows deques to be recognized as a :class:`~collections.abc.MutableSequence` +and improves their substitutability for lists. +(Contributed by Raymond Hettinger in :issue:`23704`.) + +Docstrings produced by :func:`~collections.namedtuple` can now be updated:: + + Point = namedtuple('Point', ['x', 'y']) + Point.__doc__ += ': Cartesian coodinate' + Point.x.__doc__ = 'abscissa' + Point.y.__doc__ = 'ordinate' + +(Contributed by Berker Peksag in :issue:`24064`.) + +The :class:`~collections.UserString` class now implements the +:meth:`__getnewargs__`, :meth:`__rmod__`, :meth:`~str.casefold`, +:meth:`~str.format_map`, :meth:`~str.isprintable`, and :meth:`~str.maketrans` +methods to match the corresponding methods of :class:`str`. +(Contributed by Joe Jevnik in :issue:`22189`.) + + +collections.abc +--------------- + +The :meth:`Sequence.index() <collections.abc.Sequence.index>` method now +accepts *start* and *stop* arguments to match the corresponding methods +of :class:`tuple`, :class:`list`, etc. +(Contributed by Devin Jeanpierre in :issue:`23086`.) + +A new :class:`~collections.abc.Generator` abstract base class. (Contributed +by Stefan Behnel in :issue:`24018`.) + +New :class:`~collections.abc.Awaitable`, :class:`~collections.abc.Coroutine`, +:class:`~collections.abc.AsyncIterator`, and +:class:`~collections.abc.AsyncIterable` abstract base classes. +(Contributed by Yury Selivanov in :issue:`24184`.) + +For earlier Python versions, a backport of the new ABCs is available in an +external `PyPI package <https://pypi.python.org/pypi/backports_abc>`_. + + +compileall +---------- + +A new :mod:`compileall` option, :samp:`-j {N}`, allows to run *N* workers +sumultaneously to perform parallel bytecode compilation. +The :func:`~compileall.compile_dir` function has a corresponding ``workers`` +parameter. (Contributed by Claudiu Popa in :issue:`16104`.) + +Another new option, ``-r``, allows to control the maximum recursion +level for subdirectories. (Contributed by Claudiu Popa in :issue:`19628`.) + +The ``-q`` command line option can now be specified more than once, in +which case all output, including errors, will be suppressed. The corresponding +``quiet`` parameter in :func:`~compileall.compile_dir`, +:func:`~compileall.compile_file`, and :func:`~compileall.compile_path` can now +accept an integer value indicating the level of output suppression. +(Contributed by Thomas Kluyver in :issue:`21338`.) + + +concurrent.futures +------------------ + +The :meth:`Executor.map() <concurrent.futures.Executor.map>` method now accepts a +*chunksize* argument to allow batching of tasks to improve performance when +:meth:`~concurrent.futures.ProcessPoolExecutor` is used. +(Contributed by Dan O'Reilly in :issue:`11271`.) + +The number of workers in the :class:`~concurrent.futures.ThreadPoolExecutor` +constructor is optional now. The default value is 5 times the number of CPUs. +(Contributed by Claudiu Popa in :issue:`21527`.) + + +configparser +------------ + +:mod:`configparser` now provides a way to customize the conversion +of values by specifying a dictionary of converters in the +:class:`~configparser.ConfigParser` constructor, or by defining them +as methods in ``ConfigParser`` subclasses. Converters defined in +a parser instance are inherited by its section proxies. + +Example:: + + >>> import configparser + >>> conv = {} + >>> conv['list'] = lambda v: [e.strip() for e in v.split() if e.strip()] + >>> cfg = configparser.ConfigParser(converters=conv) + >>> cfg.read_string(""" + ... [s] + ... list = a b c d e f g + ... """) + >>> cfg.get('s', 'list') + 'a b c d e f g' + >>> cfg.getlist('s', 'list') + ['a', 'b', 'c', 'd', 'e', 'f', 'g'] + >>> section = cfg['s'] + >>> section.getlist('list') + ['a', 'b', 'c', 'd', 'e', 'f', 'g'] + +(Contributed by Łukasz Langa in :issue:`18159`.) + + +contextlib +---------- + +The new :func:`~contextlib.redirect_stderr` :term:`context manager` (similar to +:func:`~contextlib.redirect_stdout`) makes it easier for utility scripts to +handle inflexible APIs that write their output to :data:`sys.stderr` and +don't provide any options to redirect it:: + + >>> import contextlib, io, logging + >>> f = io.StringIO() + >>> with contextlib.redirect_stderr(f): + ... logging.warning('warning') + ... + >>> f.getvalue() + 'WARNING:root:warning\n' + +(Contributed by Berker Peksag in :issue:`22389`.) + + +csv +--- + +The :meth:`~csv.csvwriter.writerow` method now supports arbitrary iterables, +not just sequences. (Contributed by Serhiy Storchaka in :issue:`23171`.) + + +curses +------ + +The new :func:`~curses.update_lines_cols` function updates the :envvar:`LINES` +and :envvar:`COLS` environment variables. This is useful for detecting +manual screen resizing. (Contributed by Arnon Yaari in :issue:`4254`.) + + +dbm +--- + +:func:`dumb.open <dbm.dumb.open>` always creates a new database when the flag +has the value ``"n"``. (Contributed by Claudiu Popa in :issue:`18039`.) + + +difflib +------- + +The charset of HTML documents generated by +:meth:`HtmlDiff.make_file() <difflib.HtmlDiff.make_file>` +can now be customized by using a new *charset* keyword-only argument. +The default charset of HTML document changed from ``"ISO-8859-1"`` +to ``"utf-8"``. +(Contributed by Berker Peksag in :issue:`2052`.) + +The :func:`~difflib.diff_bytes` function can now compare lists of byte +strings. This fixes a regression from Python 2. +(Contributed by Terry J. Reedy and Greg Ward in :issue:`17445`.) + + +distutils +--------- + +Both the ``build`` and ``build_ext`` commands now accept a ``-j`` option to +enable parallel building of extension modules. +(Contributed by Antoine Pitrou in :issue:`5309`.) + +The :mod:`distutils` module now supports ``xz`` compression, and can be +enabled by passing ``xztar`` as an argument to ``bdist --format``. +(Contributed by Serhiy Storchaka in :issue:`16314`.) + + +doctest +------- + +The :func:`~doctest.DocTestSuite` function returns an empty +:class:`unittest.TestSuite` if *module* contains no docstrings, instead of +raising :exc:`ValueError`. (Contributed by Glenn Jones in :issue:`15916`.) + + +email +----- + +A new policy option :attr:`Policy.mangle_from_ <email.policy.Policy.mangle_from_>` +controls whether or not lines that start with ``"From "`` in email bodies are +prefixed with a ``">"`` character by generators. The default is ``True`` for +:attr:`~email.policy.compat32` and ``False`` for all other policies. +(Contributed by Milan Oberkirch in :issue:`20098`.) + +A new +:meth:`Message.get_content_disposition() <email.message.Message.get_content_disposition>` +method provides easy access to a canonical value for the +:mailheader:`Content-Disposition` header. +(Contributed by Abhilash Raj in :issue:`21083`.) + +A new policy option :attr:`EmailPolicy.utf8 <email.policy.EmailPolicy.utf8>` +can be set to ``True`` to encode email headers using the UTF-8 charset instead +of using encoded words. This allows ``Messages`` to be formatted according to +:rfc:`6532` and used with an SMTP server that supports the :rfc:`6531` +``SMTPUTF8`` extension. (Contributed by R. David Murray in +:issue:`24211`.) + +The :class:`mime.text.MIMEText <email.mime.text.MIMEText>` constructor now +accepts a :class:`charset.Charset <email.charset.Charset>` instance. +(Contributed by Claude Paroz and Berker Peksag in :issue:`16324`.) + + +enum +---- + +The :class:`~enum.Enum` callable has a new parameter *start* to +specify the initial number of enum values if only *names* are provided:: + + >>> Animal = enum.Enum('Animal', 'cat dog', start=10) + >>> Animal.cat + <Animal.cat: 10> + >>> Animal.dog + <Animal.dog: 11> + +(Contributed by Ethan Furman in :issue:`21706`.) + + +faulthandler +------------ + +The :func:`~faulthandler.enable`, :func:`~faulthandler.register`, +:func:`~faulthandler.dump_traceback` and +:func:`~faulthandler.dump_traceback_later` functions now accept file +descriptors in addition to file-like objects. +(Contributed by Wei Wu in :issue:`23566`.) + + +functools +--------- + +.. _whatsnew-lrucache: + +Most of the :func:`~functools.lru_cache` machinery is now implemented in C, making +it significantly faster. (Contributed by Matt Joiner, Alexey Kachayev, and +Serhiy Storchaka in :issue:`14373`.) + + +glob +---- + +The :func:`~glob.iglob` and :func:`~glob.glob` functions now support recursive +search in subdirectories, using the ``"**"`` pattern. +(Contributed by Serhiy Storchaka in :issue:`13968`.) + + +gzip +---- + +The *mode* argument of the :class:`~gzip.GzipFile` constructor now +accepts ``"x"`` to request exclusive creation. +(Contributed by Tim Heaney in :issue:`19222`.) + + +heapq +----- + +Element comparison in :func:`~heapq.merge` can now be customized by +passing a :term:`key function` in a new optional *key* keyword argument, +and a new optional *reverse* keyword argument can be used to reverse element +comparison:: + + >>> import heapq + >>> a = ['9', '777', '55555'] + >>> b = ['88', '6666'] + >>> list(heapq.merge(a, b, key=len)) + ['9', '88', '777', '6666', '55555'] + >>> list(heapq.merge(reversed(a), reversed(b), key=len, reverse=True)) + ['55555', '6666', '777', '88', '9'] + +(Contributed by Raymond Hettinger in :issue:`13742`.) + + +http +---- + +A new :class:`HTTPStatus <http.HTTPStatus>` enum that defines a set of +HTTP status codes, reason phrases and long descriptions written in English. +(Contributed by Demian Brecht in :issue:`21793`.) + + +http.client +----------- + +:meth:`HTTPConnection.getresponse() <http.client.HTTPConnection.getresponse>` +now raises a :exc:`~http.client.RemoteDisconnected` exception when a +remote server connection is closed unexpectedly. Additionally, if a +:exc:`ConnectionError` (of which ``RemoteDisconnected`` +is a subclass) is raised, the client socket is now closed automatically, +and will reconnect on the next request:: + + import http.client + conn = http.client.HTTPConnection('www.python.org') + for retries in range(3): + try: + conn.request('GET', '/') + resp = conn.getresponse() + except http.client.RemoteDisconnected: + pass + +(Contributed by Martin Panter in :issue:`3566`.) + + +idlelib and IDLE +---------------- + +Since idlelib implements the IDLE shell and editor and is not intended for +import by other programs, it gets improvements with every release. See +:file:`Lib/idlelib/NEWS.txt` for a cumulative list of changes since 3.4.0, +as well as changes made in future 3.5.x releases. This file is also available +from the IDLE :menuselection:`Help --> About IDLE` dialog. + + +imaplib +------- + +The :class:`~imaplib.IMAP4` class now supports the :term:`context manager` protocol. +When used in a :keyword:`with` statement, the IMAP4 ``LOGOUT`` +command will be called automatically at the end of the block. +(Contributed by Tarek Ziadé and Serhiy Storchaka in :issue:`4972`.) + +The :mod:`imaplib` module now supports :rfc:`5161` (ENABLE Extension) +and :rfc:`6855` (UTF-8 Support) via the :meth:`IMAP4.enable() <imaplib.IMAP4.enable>` +method. A new :attr:`IMAP4.utf8_enabled <imaplib.IMAP4.utf8_enabled>` +attribute tracks whether or not :rfc:`6855` support is enabled. +(Contributed by Milan Oberkirch, R. David Murray, and Maciej Szulik in +:issue:`21800`.) + +The :mod:`imaplib` module now automatically encodes non-ASCII string usernames +and passwords using UTF-8, as recommended by the RFCs. (Contributed by Milan +Oberkirch in :issue:`21800`.) + + +imghdr +------ + +The :func:`~imghdr.what` function now recognizes the +`OpenEXR <http://www.openexr.com>`_ format +(contributed by Martin Vignali and Claudiu Popa in :issue:`20295`), +and the `WebP <https://en.wikipedia.org/wiki/WebP>`_ format +(contributed by Fabrice Aneche and Claudiu Popa in :issue:`20197`.) + + +importlib +--------- + +The :class:`util.LazyLoader <importlib.util.LazyLoader>` class allows for +lazy loading of modules in applications where startup time is important. +(Contributed by Brett Cannon in :issue:`17621`.) + +The :func:`abc.InspectLoader.source_to_code() <importlib.abc.InspectLoader.source_to_code>` +method is now a static method. This makes it easier to initialize a module +object with code compiled from a string by running +``exec(code, module.__dict__)``. +(Contributed by Brett Cannon in :issue:`21156`.) + +The new :func:`util.module_from_spec() <importlib.util.module_from_spec>` +function is now the preferred way to create a new module. As opposed to +creating a :class:`types.ModuleType` instance directly, this new function +will set the various import-controlled attributes based on the passed-in +spec object. (Contributed by Brett Cannon in :issue:`20383`.) + + +inspect +------- + +Both the :class:`~inspect.Signature` and :class:`~inspect.Parameter` classes are +now picklable and hashable. (Contributed by Yury Selivanov in :issue:`20726` +and :issue:`20334`.) + +A new +:meth:`BoundArguments.apply_defaults() <inspect.BoundArguments.apply_defaults>` +method provides a way to set default values for missing arguments:: + + >>> def foo(a, b='ham', *args): pass + >>> ba = inspect.signature(foo).bind('spam') + >>> ba.apply_defaults() + >>> ba.arguments + OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())]) + +(Contributed by Yury Selivanov in :issue:`24190`.) + +A new class method +:meth:`Signature.from_callable() <inspect.Signature.from_callable>` makes +subclassing of :class:`~inspect.Signature` easier. (Contributed +by Yury Selivanov and Eric Snow in :issue:`17373`.) + +The :func:`~inspect.signature` function now accepts a *follow_wrapped* +optional keyword argument, which, when set to ``False``, disables automatic +following of ``__wrapped__`` links. +(Contributed by Yury Selivanov in :issue:`20691`.) + +A set of new functions to inspect +:term:`coroutine functions <coroutine function>` and +:term:`coroutine objects <coroutine>` has been added: +:func:`~inspect.iscoroutine`, :func:`~inspect.iscoroutinefunction`, +:func:`~inspect.isawaitable`, :func:`~inspect.getcoroutinelocals`, +and :func:`~inspect.getcoroutinestate`. +(Contributed by Yury Selivanov in :issue:`24017` and :issue:`24400`.) + +The :func:`~inspect.stack`, :func:`~inspect.trace`, +:func:`~inspect.getouterframes`, and :func:`~inspect.getinnerframes` +functions now return a list of named tuples. +(Contributed by Daniel Shahaf in :issue:`16808`.) + + +io +-- + +A new :meth:`BufferedIOBase.readinto1() <io.BufferedIOBase.readinto1>` +method, that uses at most one call to the underlying raw stream's +:meth:`RawIOBase.read() <io.RawIOBase.read>` or +:meth:`RawIOBase.readinto() <io.RawIOBase.readinto>` methods. +(Contributed by Nikolaus Rath in :issue:`20578`.) + + +ipaddress +--------- + +Both the :class:`~ipaddress.IPv4Network` and :class:`~ipaddress.IPv6Network` classes +now accept an ``(address, netmask)`` tuple argument, so as to easily construct +network objects from existing addresses:: + + >>> import ipaddress + >>> ipaddress.IPv4Network(('127.0.0.0', 8)) + IPv4Network('127.0.0.0/8') + >>> ipaddress.IPv4Network(('127.0.0.0', '255.0.0.0')) + IPv4Network('127.0.0.0/8') + +(Contributed by Peter Moody and Antoine Pitrou in :issue:`16531`.) + +A new :attr:`~ipaddress.IPv4Network.reverse_pointer` attribute for the +:class:`~ipaddress.IPv4Network` and :class:`~ipaddress.IPv6Network` classes +returns the name of the reverse DNS PTR record:: + + >>> import ipaddress + >>> addr = ipaddress.IPv4Address('127.0.0.1') + >>> addr.reverse_pointer + '1.0.0.127.in-addr.arpa' + >>> addr6 = ipaddress.IPv6Address('::1') + >>> addr6.reverse_pointer + '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa' + +(Contributed by Leon Weber in :issue:`20480`.) + + +json +---- + +The :mod:`json.tool` command line interface now preserves the order of keys in +JSON objects passed in input. The new ``--sort-keys`` option can be used +to sort the keys alphabetically. (Contributed by Berker Peksag +in :issue:`21650`.) + +JSON decoder now raises :exc:`~json.JSONDecodeError` instead of +:exc:`ValueError` to provide better context information about the error. +(Contributed by Serhiy Storchaka in :issue:`19361`.) + + +linecache +--------- + +A new :func:`~linecache.lazycache` function can be used to capture information +about a non-file-based module to permit getting its lines later via +:func:`~linecache.getline`. This avoids doing I/O until a line is actually +needed, without having to carry the module globals around indefinitely. +(Contributed by Robert Collins in :issue:`17911`.) + + +locale +------ + +A new :func:`~locale.delocalize` function can be used to convert a string into +a normalized number string, taking the ``LC_NUMERIC`` settings into account:: + + >>> import locale + >>> locale.setlocale(locale.LC_NUMERIC, 'de_DE.UTF-8') + 'de_DE.UTF-8' + >>> locale.delocalize('1.234,56') + '1234.56' + >>> locale.setlocale(locale.LC_NUMERIC, 'en_US.UTF-8') + 'en_US.UTF-8' + >>> locale.delocalize('1,234.56') + '1234.56' + +(Contributed by Cédric Krier in :issue:`13918`.) + + +logging +------- + +All logging methods (:class:`~logging.Logger` :meth:`~logging.Logger.log`, +:meth:`~logging.Logger.exception`, :meth:`~logging.Logger.critical`, +:meth:`~logging.Logger.debug`, etc.), now accept exception instances +as an *exc_info* argument, in addition to boolean values and exception +tuples:: + + >>> import logging + >>> try: + ... 1/0 + ... except ZeroDivisionError as ex: + ... logging.error('exception', exc_info=ex) + ERROR:root:exception + +(Contributed by Yury Selivanov in :issue:`20537`.) + +The :class:`handlers.HTTPHandler <logging.handlers.HTTPHandler>` class now +accepts an optional :class:`ssl.SSLContext` instance to configure SSL +settings used in an HTTP connection. +(Contributed by Alex Gaynor in :issue:`22788`.) + +The :class:`handlers.QueueListener <logging.handlers.QueueListener>` class now +takes a *respect_handler_level* keyword argument which, if set to ``True``, +will pass messages to handlers taking handler levels into account. +(Contributed by Vinay Sajip.) + + +lzma +---- + +The :meth:`LZMADecompressor.decompress() <lzma.LZMADecompressor.decompress>` +method now accepts an optional *max_length* argument to limit the maximum +size of decompressed data. +(Contributed by Martin Panter in :issue:`15955`.) + + +math +---- + +Two new constants have been added to the :mod:`math` module: :data:`~math.inf` +and :data:`~math.nan`. (Contributed by Mark Dickinson in :issue:`23185`.) + +A new function :func:`~math.isclose` provides a way to test for approximate +equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) + +A new :func:`~math.gcd` function has been added. The :func:`fractions.gcd` +function is now deprecated. (Contributed by Mark Dickinson and Serhiy +Storchaka in :issue:`22486`.) + + +multiprocessing +--------------- + +:func:`sharedctypes.synchronized() <multiprocessing.sharedctypes.synchronized>` +objects now support the :term:`context manager` protocol. +(Contributed by Charles-François Natali in :issue:`21565`.) + + +operator +-------- + +:func:`~operator.attrgetter`, :func:`~operator.itemgetter`, +and :func:`~operator.methodcaller` objects now support pickling. +(Contributed by Josh Rosenberg and Serhiy Storchaka in :issue:`22955`.) + +New :func:`~operator.matmul` and :func:`~operator.imatmul` functions +to perform matrix multiplication. +(Contributed by Benjamin Peterson in :issue:`21176`.) + + +os +-- + +The new :func:`~os.scandir` function returning an iterator of +:class:`~os.DirEntry` objects has been added. If possible, :func:`~os.scandir` +extracts file attributes while scanning a directory, removing the need to +perform subsequent system calls to determine file type or attributes, which may +significantly improve performance. (Contributed by Ben Hoyt with the help +of Victor Stinner in :issue:`22524`.) + +On Windows, a new +:attr:`stat_result.st_file_attributes <os.stat_result.st_file_attributes>` +attribute is now available. It corresponds to the ``dwFileAttributes`` member +of the ``BY_HANDLE_FILE_INFORMATION`` structure returned by +``GetFileInformationByHandle()``. (Contributed by Ben Hoyt in :issue:`21719`.) + +The :func:`~os.urandom` function now uses the ``getrandom()`` syscall on Linux 3.17 +or newer, and ``getentropy()`` on OpenBSD 5.6 and newer, removing the need to +use ``/dev/urandom`` and avoiding failures due to potential file descriptor +exhaustion. (Contributed by Victor Stinner in :issue:`22181`.) + +New :func:`~os.get_blocking` and :func:`~os.set_blocking` functions allow to +get and set a file descriptor's blocking mode (:data:`~os.O_NONBLOCK`.) +(Contributed by Victor Stinner in :issue:`22054`.) + +The :func:`~os.truncate` and :func:`~os.ftruncate` functions are now supported +on Windows. (Contributed by Steve Dower in :issue:`23668`.) + +There is a new :func:`os.path.commonpath` function returning the longest +common sub-path of each passed pathname. Unlike the +:func:`os.path.commonprefix` function, it always returns a valid +path:: + + >>> os.path.commonprefix(['/usr/lib', '/usr/local/lib']) + '/usr/l' + + >>> os.path.commonpath(['/usr/lib', '/usr/local/lib']) + '/usr' + +(Contributed by Rafik Draoui and Serhiy Storchaka in :issue:`10395`.) + + +pathlib +------- + +The new :meth:`Path.samefile() <pathlib.Path.samefile>` method can be used +to check whether the path points to the same file as another path, which can +be either another :class:`~pathlib.Path` object, or a string:: + + >>> import pathlib + >>> p1 = pathlib.Path('/etc/hosts') + >>> p2 = pathlib.Path('/etc/../etc/hosts') + >>> p1.samefile(p2) + True + +(Contributed by Vajrasky Kok and Antoine Pitrou in :issue:`19775`.) + +The :meth:`Path.mkdir() <pathlib.Path.mkdir>` method now accepts a new optional +*exist_ok* argument to match ``mkdir -p`` and :func:`os.makedirs` +functionality. (Contributed by Berker Peksag in :issue:`21539`.) + +There is a new :meth:`Path.expanduser() <pathlib.Path.expanduser>` method to +expand ``~`` and ``~user`` prefixes. (Contributed by Serhiy Storchaka and +Claudiu Popa in :issue:`19776`.) + +A new :meth:`Path.home() <pathlib.Path.home>` class method can be used to get +a :class:`~pathlib.Path` instance representing the user’s home +directory. +(Contributed by Victor Salgado and Mayank Tripathi in :issue:`19777`.) + +New :meth:`Path.write_text() <pathlib.Path.write_text>`, +:meth:`Path.read_text() <pathlib.Path.read_text>`, +:meth:`Path.write_bytes() <pathlib.Path.write_bytes>`, +:meth:`Path.read_bytes() <pathlib.Path.read_bytes>` methods to simplify +read/write operations on files. + +The following code snippet will create or rewrite existing file +``~/spam42``:: + + >>> import pathlib + >>> p = pathlib.Path('~/spam42') + >>> p.expanduser().write_text('ham') + 3 + +(Contributed by Christopher Welborn in :issue:`20218`.) + + +pickle +------ + +Nested objects, such as unbound methods or nested classes, can now be pickled +using :ref:`pickle protocols <pickle-protocols>` older than protocol version 4. +Protocol version 4 already supports these cases. (Contributed by Serhiy +Storchaka in :issue:`23611`.) + + +poplib +------ + +A new :meth:`POP3.utf8() <poplib.POP3.utf8>` command enables :rfc:`6856` +(Internationalized Email) support, if a POP server supports it. +(Contributed by Milan OberKirch in :issue:`21804`.) + + +re +-- + +References and conditional references to groups with fixed length are now +allowed in lookbehind assertions:: + + >>> import re + >>> pat = re.compile(r'(a|b).(?<=\1)c') + >>> pat.match('aac') + <_sre.SRE_Match object; span=(0, 3), match='aac'> + >>> pat.match('bbc') + <_sre.SRE_Match object; span=(0, 3), match='bbc'> + +(Contributed by Serhiy Storchaka in :issue:`9179`.) + +The number of capturing groups in regular expressions is no longer limited to +100. (Contributed by Serhiy Storchaka in :issue:`22437`.) + +The :func:`~re.sub` and :func:`~re.subn` functions now replace unmatched +groups with empty strings instead of raising an exception. +(Contributed by Serhiy Storchaka in :issue:`1519638`.) + +The :class:`re.error` exceptions have new attributes, +:attr:`~re.error.msg`, :attr:`~re.error.pattern`, +:attr:`~re.error.pos`, :attr:`~re.error.lineno`, +and :attr:`~re.error.colno`, that provide better context +information about the error:: + + >>> re.compile(""" + ... (?x) + ... .++ + ... """) + Traceback (most recent call last): + ... + sre_constants.error: multiple repeat at position 16 (line 3, column 7) + +(Contributed by Serhiy Storchaka in :issue:`22578`.) + + +readline +-------- + +A new :func:`~readline.append_history_file` function can be used to append +the specified number of trailing elements in history to the given file. +(Contributed by Bruno Cauet in :issue:`22940`.) + + +selectors +--------- + +The new :class:`~selectors.DevpollSelector` supports efficient +``/dev/poll`` polling on Solaris. +(Contributed by Giampaolo Rodola' in :issue:`18931`.) + + +shutil +------ + +The :func:`~shutil.move` function now accepts a *copy_function* argument, +allowing, for example, the :func:`~shutil.copy` function to be used instead of +the default :func:`~shutil.copy2` if there is a need to ignore file metadata +when moving. +(Contributed by Claudiu Popa in :issue:`19840`.) + +The :func:`~shutil.make_archive` function now supports the *xztar* format. +(Contributed by Serhiy Storchaka in :issue:`5411`.) + + +signal +------ + +On Windows, the :func:`~signal.set_wakeup_fd` function now also supports +socket handles. (Contributed by Victor Stinner in :issue:`22018`.) + +Various ``SIG*`` constants in the :mod:`signal` module have been converted into +:mod:`Enums <enum>`. This allows meaningful names to be printed +during debugging, instead of integer "magic numbers". +(Contributed by Giampaolo Rodola' in :issue:`21076`.) + + +smtpd +----- + +Both the :class:`~smtpd.SMTPServer` and :class:`~smtpd.SMTPChannel` classes now +accept a *decode_data* keyword argument to determine if the ``DATA`` portion of +the SMTP transaction is decoded using the ``"utf-8"`` codec or is instead +provided to the +:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` +method as a byte string. The default is ``True`` for backward compatibility +reasons, but will change to ``False`` in Python 3.6. If *decode_data* is set +to ``False``, the ``process_message`` method must be prepared to accept keyword +arguments. +(Contributed by Maciej Szulik in :issue:`19662`.) + +The :class:`~smtpd.SMTPServer` class now advertises the ``8BITMIME`` extension +(:rfc:`6152`) if *decode_data* has been set ``True``. If the client +specifies ``BODY=8BITMIME`` on the ``MAIL`` command, it is passed to +:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` +via the *mail_options* keyword. +(Contributed by Milan Oberkirch and R. David Murray in :issue:`21795`.) + +The :class:`~smtpd.SMTPServer` class now also supports the ``SMTPUTF8`` +extension (:rfc:`6531`: Internationalized Email). If the client specified +``SMTPUTF8 BODY=8BITMIME`` on the ``MAIL`` command, they are passed to +:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` +via the *mail_options* keyword. It is the responsibility of the +``process_message`` method to correctly handle the ``SMTPUTF8`` data. +(Contributed by Milan Oberkirch in :issue:`21725`.) + +It is now possible to provide, directly or via name resolution, IPv6 +addresses in the :class:`~smtpd.SMTPServer` constructor, and have it +successfully connect. (Contributed by Milan Oberkirch in :issue:`14758`.) + + +smtplib +------- + +A new :meth:`SMTP.auth() <smtplib.SMTP.auth>` method provides a convenient way to +implement custom authentication mechanisms. (Contributed by Milan +Oberkirch in :issue:`15014`.) + +The :meth:`SMTP.set_debuglevel() <smtplib.SMTP.set_debuglevel>` method now +accepts an additional debuglevel (2), which enables timestamps in debug +messages. (Contributed by Gavin Chappell and Maciej Szulik in :issue:`16914`.) + +Both the :meth:`SMTP.sendmail() <smtplib.SMTP.sendmail>` and +:meth:`SMTP.send_message() <smtplib.SMTP.send_message>` methods now +support support :rfc:`6531` (SMTPUTF8). +(Contributed by Milan Oberkirch and R. David Murray in :issue:`22027`.) + + +sndhdr +------ + +The :func:`~sndhdr.what` and :func:`~sndhdr.whathdr` functions now return +a :func:`~collections.namedtuple`. (Contributed by Claudiu Popa in +:issue:`18615`.) + + +socket +------ + +Functions with timeouts now use a monotonic clock, instead of a system clock. +(Contributed by Victor Stinner in :issue:`22043`.) + +A new :meth:`socket.sendfile() <socket.socket.sendfile>` method allows to +send a file over a socket by using the high-performance :func:`os.sendfile` +function on UNIX, resulting in uploads being from 2 to 3 times faster than when +using plain :meth:`socket.send() <socket.socket.send>`. +(Contributed by Giampaolo Rodola' in :issue:`17552`.) + +The :meth:`socket.sendall() <socket.socket.sendall>` method no longer resets the +socket timeout every time bytes are received or sent. The socket timeout is +now the maximum total duration to send all data. +(Contributed by Victor Stinner in :issue:`23853`.) + +The *backlog* argument of the :meth:`socket.listen() <socket.socket.listen>` +method is now optional. By default it is set to +:data:`SOMAXCONN <socket.SOMAXCONN>` or to ``128``, whichever is less. +(Contributed by Charles-François Natali in :issue:`21455`.) + + +ssl +--- + +.. _whatsnew-sslmemorybio: + +Memory BIO Support +~~~~~~~~~~~~~~~~~~ + +(Contributed by Geert Jansen in :issue:`21965`.) + +The new :class:`~ssl.SSLObject` class has been added to provide SSL protocol +support for cases when the network I/O capabilities of :class:`~ssl.SSLSocket` +are not necessary or are suboptimal. ``SSLObject`` represents +an SSL protocol instance, but does not implement any network I/O methods, and +instead provides a memory buffer interface. The new :class:`~ssl.MemoryBIO` +class can be used to pass data between Python and an SSL protocol instance. + +The memory BIO SSL support is primarily intended to be used in frameworks +implementing asynchronous I/O for which :class:`~ssl.SSLSocket`'s readiness +model ("select/poll") is inefficient. + +A new :meth:`SSLContext.wrap_bio() <ssl.SSLContext.wrap_bio>` method can be used +to create a new ``SSLObject`` instance. + + +Application-Layer Protocol Negotiation Support +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +(Contributed by Benjamin Peterson in :issue:`20188`.) + +Where OpenSSL support is present, the :mod:`ssl` module now implements +the *Application-Layer Protocol Negotiation* TLS extension as described +in :rfc:`7301`. + +The new :meth:`SSLContext.set_alpn_protocols() <ssl.SSLContext.set_alpn_protocols>` +can be used to specify which protocols a socket should advertise during +the TLS handshake. + +The new +:meth:`SSLSocket.selected_alpn_protocol() <ssl.SSLSocket.selected_alpn_protocol>` +returns the protocol that was selected during the TLS handshake. +The :data:`~ssl.HAS_ALPN` flag indicates whether ALPN support is present. + + +Other Changes +~~~~~~~~~~~~~ + +There is a new :meth:`SSLSocket.version() <ssl.SSLSocket.version>` method to +query the actual protocol version in use. +(Contributed by Antoine Pitrou in :issue:`20421`.) + +The :class:`~ssl.SSLSocket` class now implements +a :meth:`SSLSocket.sendfile() <ssl.SSLSocket.sendfile>` method. +(Contributed by Giampaolo Rodola' in :issue:`17552`.) + +The :meth:`SSLSocket.send() <ssl.SSLSocket.send>` method now raises either +the :exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError` exception on a +non-blocking socket if the operation would block. Previously, it would return +``0``. (Contributed by Nikolaus Rath in :issue:`20951`.) + +The :func:`~ssl.cert_time_to_seconds` function now interprets the input time +as UTC and not as local time, per :rfc:`5280`. Additionally, the return +value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.) + +New :meth:`SSLObject.shared_ciphers() <ssl.SSLObject.shared_ciphers>` and +:meth:`SSLSocket.shared_ciphers() <ssl.SSLSocket.shared_ciphers>` methods return +the list of ciphers sent by the client during the handshake. +(Contributed by Benjamin Peterson in :issue:`23186`.) + +The :meth:`SSLSocket.do_handshake() <ssl.SSLSocket.do_handshake>`, +:meth:`SSLSocket.read() <ssl.SSLSocket.read>`, +:meth:`SSLSocket.shutdown() <ssl.SSLSocket.shutdown>`, and +:meth:`SSLSocket.write() <ssl.SSLSocket.write>` methods of the :class:`~ssl.SSLSocket` +class no longer reset the socket timeout every time bytes are received or sent. +The socket timeout is now the maximum total duration of the method. +(Contributed by Victor Stinner in :issue:`23853`.) + +The :func:`~ssl.match_hostname` function now supports matching of IP addresses. +(Contributed by Antoine Pitrou in :issue:`23239`.) + + +sqlite3 +------- + +The :class:`~sqlite3.Row` class now fully supports the sequence protocol, +in particular :func:`reversed` iteration and slice indexing. +(Contributed by Claudiu Popa in :issue:`10203`; by Lucas Sinclair, +Jessica McKellar, and Serhiy Storchaka in :issue:`13583`.) + + +.. _whatsnew-subprocess: + +subprocess +---------- + +The new :func:`~subprocess.run` function has been added. +It runs the specified command and returns a +:class:`~subprocess.CompletedProcess` object, which describes a finished +process. The new API is more consistent and is the recommended approach +to invoking subprocesses in Python code that does not need to maintain +compatibility with earlier Python versions. +(Contributed by Thomas Kluyver in :issue:`23342`.) + +Examples:: + + >>> subprocess.run(["ls", "-l"]) # doesn't capture output + CompletedProcess(args=['ls', '-l'], returncode=0) + + >>> subprocess.run("exit 1", shell=True, check=True) + Traceback (most recent call last): + ... + subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 + + >>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE) + CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0, + stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n') + + +sys +--- + +A new :func:`~sys.set_coroutine_wrapper` function allows setting a global +hook that will be called whenever a :term:`coroutine object <coroutine>` +is created by an :keyword:`async def` function. A corresponding +:func:`~sys.get_coroutine_wrapper` can be used to obtain a currently set +wrapper. Both functions are :term:`provisional <provisional api>`, +and are intended for debugging purposes only. (Contributed by Yury Selivanov +in :issue:`24017`.) + +A new :func:`~sys.is_finalizing` function can be used to check if the Python +interpreter is :term:`shutting down <interpreter shutdown>`. +(Contributed by Antoine Pitrou in :issue:`22696`.) + + +sysconfig +--------- + +The name of the user scripts directory on Windows now includes the first +two components of the Python version. (Contributed by Paul Moore +in :issue:`23437`.) + + +tarfile +------- + +The *mode* argument of the :func:`~tarfile.open` function now accepts ``"x"`` +to request exclusive creation. (Contributed by Berker Peksag in :issue:`21717`.) + +The :meth:`TarFile.extractall() <tarfile.TarFile.extractall>` and +:meth:`TarFile.extract() <tarfile.TarFile.extract>` methods now take a keyword +argument *numeric_only*. If set to ``True``, the extracted files and +directories will be owned by the numeric ``uid`` and ``gid`` from the tarfile. +If set to ``False`` (the default, and the behavior in versions prior to 3.5), +they will be owned by the named user and group in the tarfile. +(Contributed by Michael Vogt and Eric Smith in :issue:`23193`.) + +The :meth:`TarFile.list() <tarfile.TarFile.list>` now accepts an optional +*members* keyword argument that can be set to a subset of the list returned +by :meth:`TarFile.getmembers() <tarfile.TarFile.getmembers>`. +(Contributed by Serhiy Storchaka in :issue:`21549`.) + + +threading +--------- + +Both the :meth:`Lock.acquire() <threading.Lock.acquire>` and +:meth:`RLock.acquire() <threading.RLock.acquire>` methods +now use a monotonic clock for timeout management. +(Contributed by Victor Stinner in :issue:`22043`.) + + +time +---- + +The :func:`~time.monotonic` function is now always available. +(Contributed by Victor Stinner in :issue:`22043`.) + + +timeit +------ + +A new command line option ``-u`` or :samp:`--unit={U}` can be used to specify the time +unit for the timer output. Supported options are ``usec``, ``msec``, +or ``sec``. (Contributed by Julian Gindi in :issue:`18983`.) + +The :func:`~timeit.timeit` function has a new *globals* parameter for +specifying the namespace in which the code will be running. +(Contributed by Ben Roberts in :issue:`2527`.) + + +tkinter +------- + +The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment +on Windows has been replaced by a private function in the :mod:`_tkinter` +module which makes no permanent changes to environment variables. +(Contributed by Zachary Ware in :issue:`20035`.) + + +.. _whatsnew-traceback: + +traceback +--------- + +New :func:`~traceback.walk_stack` and :func:`~traceback.walk_tb` +functions to conveniently traverse frame and traceback objects. +(Contributed by Robert Collins in :issue:`17911`.) + +New lightweight classes: :class:`~traceback.TracebackException`, +:class:`~traceback.StackSummary`, and :class:`~traceback.FrameSummary`. +(Contributed by Robert Collins in :issue:`17911`.) + +Both the :func:`~traceback.print_tb` and :func:`~traceback.print_stack` functions +now support negative values for the *limit* argument. +(Contributed by Dmitry Kazakov in :issue:`22619`.) + + +types +----- + +A new :func:`~types.coroutine` function to transform +:term:`generator <generator iterator>` and +:class:`generator-like <collections.abc.Generator>` objects into +:term:`awaitables <awaitable>`. +(Contributed by Yury Selivanov in :issue:`24017`.) + +A new type called :class:`~types.CoroutineType`, which is used for +:term:`coroutine` objects created by :keyword:`async def` functions. +(Contributed by Yury Selivanov in :issue:`24400`.) + + +unicodedata +----------- + +The :mod:`unicodedata` module now uses data from `Unicode 8.0.0 +<http://unicode.org/versions/Unicode8.0.0/>`_. + + +unittest +-------- + +The :meth:`TestLoader.loadTestsFromModule() <unittest.TestLoader.loadTestsFromModule>` +method now accepts a keyword-only argument *pattern* which is passed to +``load_tests`` as the third argument. Found packages are now checked for +``load_tests`` regardless of whether their path matches *pattern*, because it +is impossible for a package name to match the default pattern. +(Contributed by Robert Collins and Barry A. Warsaw in :issue:`16662`.) + +Unittest discovery errors now are exposed in the +:data:`TestLoader.errors <unittest.TestLoader.errors>` attribute of the +:class:`~unittest.TestLoader` instance. +(Contributed by Robert Collins in :issue:`19746`.) + +A new command line option ``--locals`` to show local variables in +tracebacks. (Contributed by Robert Collins in :issue:`22936`.) + + +unittest.mock +------------- + +The :class:`~unittest.mock.Mock` class has the following improvements: + +* The class constructor has a new *unsafe* parameter, which causes mock + objects to raise :exc:`AttributeError` on attribute names starting + with ``"assert"``. + (Contributed by Kushal Das in :issue:`21238`.) + +* A new :meth:`Mock.assert_not_called() <unittest.mock.Mock.assert_not_called>` + method to check if the mock object was called. + (Contributed by Kushal Das in :issue:`21262`.) + +The :class:`~unittest.mock.MagicMock` class now supports :meth:`__truediv__`, +:meth:`__divmod__` and :meth:`__matmul__` operators. +(Contributed by Johannes Baiter in :issue:`20968`, and Håkan Lövdahl +in :issue:`23581` and :issue:`23568`.) + +It is no longer necessary to explicitly pass ``create=True`` to the +:func:`~unittest.mock.patch` function when patching builtin names. +(Contributed by Kushal Das in :issue:`17660`.) + + +urllib +------ + +A new +:class:`request.HTTPPasswordMgrWithPriorAuth <urllib.request.HTTPPasswordMgrWithPriorAuth>` +class allows HTTP Basic Authentication credentials to be managed so as to +eliminate unnecessary ``401`` response handling, or to unconditionally send +credentials on the first request in order to communicate with servers that +return a ``404`` response instead of a ``401`` if the ``Authorization`` header +is not sent. (Contributed by Matej Cepl in :issue:`19494` and Akshit Khurana in +:issue:`7159`.) + +A new *quote_via* argument for the +:func:`parse.urlencode() <urllib.parse.urlencode>` +function provides a way to control the encoding of query parts if needed. +(Contributed by Samwyse and Arnon Yaari in :issue:`13866`.) + +The :func:`request.urlopen() <urllib.request.urlopen>` function accepts an +:class:`ssl.SSLContext` object as a *context* argument, which will be used for +the HTTPS connection. (Contributed by Alex Gaynor in :issue:`22366`.) + +The :func:`parse.urljoin() <urllib.parse.urljoin>` was updated to use the +:rfc:`3986` semantics for the resolution of relative URLs, rather than +:rfc:`1808` and :rfc:`2396`. +(Contributed by Demian Brecht and Senthil Kumaran in :issue:`22118`.) + + +wsgiref +------- + +The *headers* argument of the :class:`headers.Headers <wsgiref.headers.Headers>` +class constructor is now optional. +(Contributed by Pablo Torres Navarrete and SilentGhost in :issue:`5800`.) + + +xmlrpc +------ + +The :class:`client.ServerProxy <xmlrpc.client.ServerProxy>` class now supports +the :term:`context manager` protocol. +(Contributed by Claudiu Popa in :issue:`20627`.) + +The :class:`client.ServerProxy <xmlrpc.client.ServerProxy>` constructor now accepts +an optional :class:`ssl.SSLContext` instance. +(Contributed by Alex Gaynor in :issue:`22960`.) + + +xml.sax +------- + +SAX parsers now support a character stream of the +:class:`xmlreader.InputSource <xml.sax.xmlreader.InputSource>` object. +(Contributed by Serhiy Storchaka in :issue:`2175`.) + +:func:`~xml.sax.parseString` now accepts a :class:`str` instance. +(Contributed by Serhiy Storchaka in :issue:`10590`.) + + +zipfile +------- + +ZIP output can now be written to unseekable streams. +(Contributed by Serhiy Storchaka in :issue:`23252`.) + +The *mode* argument of :meth:`ZipFile.open() <zipfile.ZipFile.open>` method now +accepts ``"x"`` to request exclusive creation. +(Contributed by Serhiy Storchaka in :issue:`21717`.) + + +Other module-level changes +========================== + +Many functions in the :mod:`mmap`, :mod:`ossaudiodev`, :mod:`socket`, +:mod:`ssl`, and :mod:`codecs` modules now accept writable +:term:`bytes-like objects <bytes-like object>`. +(Contributed by Serhiy Storchaka in :issue:`23001`.) + + +Optimizations +============= + +The :func:`os.walk` function has been sped up by 3 to 5 times on POSIX systems, +and by 7 to 20 times on Windows. This was done using the new :func:`os.scandir` +function, which exposes file information from the underlying ``readdir`` or +``FindFirstFile``/``FindNextFile`` system calls. (Contributed by +Ben Hoyt with help from Victor Stinner in :issue:`23605`.) + +Construction of ``bytes(int)`` (filled by zero bytes) is faster and uses less +memory for large objects. ``calloc()`` is used instead of ``malloc()`` to +allocate memory for these objects. +(Contributed by Victor Stinner in :issue:`21233`.) + +Some operations on :mod:`ipaddress` :class:`~ipaddress.IPv4Network` and +:class:`~ipaddress.IPv6Network` have been massively sped up, such as +:meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`, +:func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`. +The speed up can range from 3 to 15 times. +(Contributed by Antoine Pitrou, Michel Albert, and Markus in +:issue:`21486`, :issue:`21487`, :issue:`20826`, :issue:`23266`.) + +Pickling of :mod:`ipaddress` objects was optimized to produce significantly +smaller output. (Contributed by Serhiy Storchaka in :issue:`23133`.) + +Many operations on :class:`io.BytesIO` are now 50% to 100% faster. +(Contributed by Serhiy Storchaka in :issue:`15381` and David Wilson in +:issue:`22003`.) + +The :func:`marshal.dumps` function is now faster: 65-85% with versions 3 +and 4, 20-25% with versions 0 to 2 on typical data, and up to 5 times in +best cases. +(Contributed by Serhiy Storchaka in :issue:`20416` and :issue:`23344`.) + +The UTF-32 encoder is now 3 to 7 times faster. +(Contributed by Serhiy Storchaka in :issue:`15027`.) + +Regular expressions are now parsed up to 10% faster. +(Contributed by Serhiy Storchaka in :issue:`19380`.) + +The :func:`json.dumps` function was optimized to run with +``ensure_ascii=False`` as fast as with ``ensure_ascii=True``. +(Contributed by Naoki Inada in :issue:`23206`.) + +The :c:func:`PyObject_IsInstance` and :c:func:`PyObject_IsSubclass` +functions have been sped up in the common case that the second argument +has :class:`type` as its metaclass. +(Contributed Georg Brandl by in :issue:`22540`.) + +Method caching was slightly improved, yielding up to 5% performance +improvement in some benchmarks. +(Contributed by Antoine Pitrou in :issue:`22847`.) + +Objects from the :mod:`random` module now use 50% less memory on 64-bit +builds. (Contributed by Serhiy Storchaka in :issue:`23488`.) + +The :func:`property` getter calls are up to 25% faster. +(Contributed by Joe Jevnik in :issue:`23910`.) + +Instantiation of :class:`fractions.Fraction` is now up to 30% faster. +(Contributed by Stefan Behnel in :issue:`22464`.) + +String methods :meth:`~str.find`, :meth:`~str.rfind`, :meth:`~str.split`, +:meth:`~str.partition` and the :keyword:`in` string operator are now significantly +faster for searching 1-character substrings. +(Contributed by Serhiy Storchaka in :issue:`23573`.) + + +Build and C API Changes +======================= + +New ``calloc`` functions were added: + +* :c:func:`PyMem_RawCalloc`, +* :c:func:`PyMem_Calloc`, +* :c:func:`PyObject_Calloc`, +* :c:func:`_PyObject_GC_Calloc`. + +(Contributed by Victor Stinner in :issue:`21233`.) + +New encoding/decoding helper functions: + +* :c:func:`Py_DecodeLocale` (replaced ``_Py_char2wchar()``), +* :c:func:`Py_EncodeLocale` (replaced ``_Py_wchar2char()``). + +(Contributed by Victor Stinner in :issue:`18395`.) + +A new :c:func:`PyCodec_NameReplaceErrors` function to replace the unicode +encode error with ``\N{...}`` escapes. +(Contributed by Serhiy Storchaka in :issue:`19676`.) + +A new :c:func:`PyErr_FormatV` function similar to :c:func:`PyErr_Format`, +but accepts a ``va_list`` argument. +(Contributed by Antoine Pitrou in :issue:`18711`.) + +A new :c:data:`PyExc_RecursionError` exception. +(Contributed by Georg Brandl in :issue:`19235`.) + +New :c:func:`PyModule_FromDefAndSpec`, :c:func:`PyModule_FromDefAndSpec2`, +and :c:func:`PyModule_ExecDef` functions introduced by :pep:`489` -- +multi-phase extension module initialization. +(Contributed by Petr Viktorin in :issue:`24268`.) + +New :c:func:`PyNumber_MatrixMultiply` and +:c:func:`PyNumber_InPlaceMatrixMultiply` functions to perform matrix +multiplication. +(Contributed by Benjamin Peterson in :issue:`21176`. See also :pep:`465` +for details.) + +The :c:member:`PyTypeObject.tp_finalize` slot is now part of the stable ABI. + +Windows builds now require Microsoft Visual C++ 14.0, which +is available as part of `Visual Studio 2015 <http://www.visualstudio.com>`_. + +Extension modules now include a platform information tag in their filename on +some platforms (the tag is optional, and CPython will import extensions without +it, although if the tag is present and mismatched, the extension won't be +loaded): + +* On Linux, extension module filenames end with + ``.cpython-<major><minor>m-<architecture>-<os>.pyd``: + + * ``<major>`` is the major number of the Python version; + for Python 3.5 this is ``3``. + + * ``<minor>`` is the minor number of the Python version; + for Python 3.5 this is ``5``. + + * ``<architecture>`` is the hardware architecture the extension module + was built to run on. It's most commonly either ``i386`` for 32-bit Intel + platforms or ``x86_64`` for 64-bit Intel (and AMD) platforms. + + * ``<os>`` is always ``linux-gnu``, except for extensions built to + talk to the 32-bit ABI on 64-bit platforms, in which case it is + ``linux-gnu32`` (and ``<architecture>`` will be ``x86_64``). + +* On Windows, extension module filenames end with + ``<debug>.cp<major><minor>-<platform>.pyd``: + + * ``<major>`` is the major number of the Python version; + for Python 3.5 this is ``3``. + + * ``<minor>`` is the minor number of the Python version; + for Python 3.5 this is ``5``. + + * ``<platform>`` is the platform the extension module was built for, + either ``win32`` for Win32, ``win_amd64`` for Win64, ``win_ia64`` for + Windows Itanium 64, and ``win_arm`` for Windows on ARM. + + * If built in debug mode, ``<debug>`` will be ``_d``, + otherwise it will be blank. + +* On OS X platforms, extension module filenames now end with ``-darwin.so``. + +* On all other platforms, extension module filenames are the same as they were + with Python 3.4. + + +Deprecated +========== + +New Keywords +------------ + +``async`` and ``await`` are not recommended to be used as variable, class, +function or module names. Introduced by :pep:`492` in Python 3.5, they will +become proper keywords in Python 3.7. + + +Deprecated Python Behavior +-------------------------- + +Raising the :exc:`StopIteration` exception inside a generator will now generate a silent +:exc:`PendingDeprecationWarning`, which will become a non-silent deprecation +warning in Python 3.6 and will trigger a :exc:`RuntimeError` in Python 3.7. +See :ref:`PEP 479: Change StopIteration handling inside generators <whatsnew-pep-479>` +for details. + + +Unsupported Operating Systems +----------------------------- + +Windows XP is no longer supported by Microsoft, thus, per :PEP:`11`, CPython +3.5 is no longer officially supported on this OS. + + +Deprecated Python modules, functions and methods +------------------------------------------------ + +The :mod:`formatter` module has now graduated to full deprecation and is still +slated for removal in Python 3.6. + +The :func:`asyncio.async` function is deprecated in favor of +:func:`~asyncio.ensure_future`. + +The :mod:`smtpd` module has in the past always decoded the DATA portion of +email messages using the ``utf-8`` codec. This can now be controlled by the +new *decode_data* keyword to :class:`~smtpd.SMTPServer`. The default value is +``True``, but this default is deprecated. Specify the *decode_data* keyword +with an appropriate value to avoid the deprecation warning. + +Directly assigning values to the :attr:`~http.cookies.Morsel.key`, +:attr:`~http.cookies.Morsel.value` and +:attr:`~http.cookies.Morsel.coded_value` of :class:`http.cookies.Morsel` +objects is deprecated. Use the :meth:`~http.cookies.Morsel.set` method +instead. In addition, the undocumented *LegalChars* parameter of +:meth:`~http.cookies.Morsel.set` is deprecated, and is now ignored. + +Passing a format string as keyword argument *format_string* to the +:meth:`~string.Formatter.format` method of the :class:`string.Formatter` +class has been deprecated. +(Contributed by Serhiy Storchaka in :issue:`23671`.) + +The :func:`platform.dist` and :func:`platform.linux_distribution` functions +are now deprecated and will be removed in Python 3.7. Linux distributions use +too many different ways of describing themselves, so the functionality is +left to a package. +(Contributed by Vajrasky Kok and Berker Peksag in :issue:`1322`.) + +The previously undocumented ``from_function`` and ``from_builtin`` methods of +:class:`inspect.Signature` are deprecated. Use the new +:meth:`Signature.from_callable() <inspect.Signature.from_callable>` +method instead. (Contributed by Yury Selivanov in :issue:`24248`.) + +The :func:`inspect.getargspec` function is deprecated and scheduled to be +removed in Python 3.6. (See :issue:`20438` for details.) + +The :mod:`inspect` :func:`~inspect.getfullargspec`, +:func:`~inspect.getargvalues`, :func:`~inspect.getcallargs`, +:func:`~inspect.getargvalues`, :func:`~inspect.formatargspec`, and +:func:`~inspect.formatargvalues` functions are deprecated in favor of +the :func:`inspect.signature` API. +(Contributed by Yury Selivanov in :issue:`20438`.) + +Use of :const:`re.LOCALE` flag with str patterns or :const:`re.ASCII` is now +deprecated. (Contributed by Serhiy Storchaka in :issue:`22407`.) + +Use of unrecognized special sequences consisting of ``'\'`` and an ASCII letter +in regular expression patterns and replacement patterns now raises a +deprecation warning and will be forbidden in Python 3.6. +(Contributed by Serhiy Storchaka in :issue:`23622`.) + +The undocumented and unofficial *use_load_tests* default argument of the +:meth:`unittest.TestLoader.loadTestsFromModule` method now is +deprecated and ignored. +(Contributed by Robert Collins and Barry A. Warsaw in :issue:`16662`.) + + +Removed +======= + +API and Feature Removals +------------------------ + +The following obsolete and previously deprecated APIs and features have been +removed: + +* The ``__version__`` attribute has been dropped from the email package. The + email code hasn't been shipped separately from the stdlib for a long time, + and the ``__version__`` string was not updated in the last few releases. + +* The internal ``Netrc`` class in the :mod:`ftplib` module was deprecated in + 3.4, and has now been removed. + (Contributed by Matt Chaput in :issue:`6623`.) + +* The concept of ``.pyo`` files has been removed. + +* The JoinableQueue class in the provisional :mod:`asyncio` module was + deprecated in 3.4.4 and is now removed. + (Contributed by A. Jesse Jiryu Davis in :issue:`23464`.) + + +Porting to Python 3.5 +===================== + +This section lists previously described changes and other bugfixes +that may require changes to your code. + + +Changes in Python behavior +-------------------------- + +* Due to an oversight, earlier Python versions erroneously accepted the + following syntax:: + + f(1 for x in [1], *args) + f(1 for x in [1], **kwargs) + + Python 3.5 now correctly raises a :exc:`SyntaxError`, as generator + expressions must be put in parentheses if not a sole argument to a function. + + +Changes in the Python API +------------------------- + +* :pep:`475`: System calls are now retried when interrupted by a signal instead + of raising :exc:`InterruptedError` if the Python signal handler does not + raise an exception. + +* Before Python 3.5, a :class:`datetime.time` object was considered to be false + if it represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. + +* The :meth:`ssl.SSLSocket.send()` method now raises either + :exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError` + on a non-blocking socket if the operation would block. Previously, + it would return ``0``. (Contributed by Nikolaus Rath in :issue:`20951`.) + +* The ``__name__`` attribute of generators is now set from the function name, + instead of being set from the code name. Use ``gen.gi_code.co_name`` to + retrieve the code name. Generators also have a new ``__qualname__`` + attribute, the qualified name, which is now used for the representation + of a generator (``repr(gen)``). + (Contributed by Victor Stinner in :issue:`21205`.) + +* The deprecated "strict" mode and argument of :class:`~html.parser.HTMLParser`, + :meth:`HTMLParser.error`, and the :exc:`HTMLParserError` exception have been + removed. (Contributed by Ezio Melotti in :issue:`15114`.) + The *convert_charrefs* argument of :class:`~html.parser.HTMLParser` is + now ``True`` by default. (Contributed by Berker Peksag in :issue:`21047`.) + +* Although it is not formally part of the API, it is worth noting for porting + purposes (ie: fixing tests) that error messages that were previously of the + form "'sometype' does not support the buffer protocol" are now of the form "a + :term:`bytes-like object` is required, not 'sometype'". + (Contributed by Ezio Melotti in :issue:`16518`.) + +* If the current directory is set to a directory that no longer exists then + :exc:`FileNotFoundError` will no longer be raised and instead + :meth:`~importlib.machinery.FileFinder.find_spec` will return ``None`` + **without** caching ``None`` in :data:`sys.path_importer_cache`, which is + different than the typical case (:issue:`22834`). + +* HTTP status code and messages from :mod:`http.client` and :mod:`http.server` + were refactored into a common :class:`~http.HTTPStatus` enum. The values in + :mod:`http.client` and :mod:`http.server` remain available for backwards + compatibility. (Contributed by Demian Brecht in :issue:`21793`.) + +* When an import loader defines :meth:`importlib.machinery.Loader.exec_module` + it is now expected to also define + :meth:`~importlib.machinery.Loader.create_module` (raises a + :exc:`DeprecationWarning` now, will be an error in Python 3.6). If the loader + inherits from :class:`importlib.abc.Loader` then there is nothing to do, else + simply define :meth:`~importlib.machinery.Loader.create_module` to return + ``None``. (Contributed by Brett Cannon in :issue:`23014`.) + +* The :func:`re.split` function always ignored empty pattern matches, so the + ``"x*"`` pattern worked the same as ``"x+"``, and the ``"\b"`` pattern never + worked. Now :func:`re.split` raises a warning if the pattern could match + an empty string. For compatibility, use patterns that never match an empty + string (e.g. ``"x+"`` instead of ``"x*"``). Patterns that could only match + an empty string (such as ``"\b"``) now raise an error. + (Contributed by Serhiy Storchaka in :issue:`22818`.) + +* The :class:`http.cookies.Morsel` dict-like interface has been made self + consistent: morsel comparison now takes the :attr:`~http.cookies.Morsel.key` + and :attr:`~http.cookies.Morsel.value` into account, + :meth:`~http.cookies.Morsel.copy` now results in a + :class:`~http.cookies.Morsel` instance rather than a :class:`dict`, and + :meth:`~http.cookies.Morsel.update` will now raise an exception if any of the + keys in the update dictionary are invalid. In addition, the undocumented + *LegalChars* parameter of :func:`~http.cookies.Morsel.set` is deprecated and + is now ignored. (Contributed by Demian Brecht in :issue:`2211`.) + +* :pep:`488` has removed ``.pyo`` files from Python and introduced the optional + ``opt-`` tag in ``.pyc`` file names. The + :func:`importlib.util.cache_from_source` has gained an *optimization* + parameter to help control the ``opt-`` tag. Because of this, the + *debug_override* parameter of the function is now deprecated. `.pyo` files + are also no longer supported as a file argument to the Python interpreter and + thus serve no purpose when distributed on their own (i.e. sourcless code + distribution). Due to the fact that the magic number for bytecode has changed + in Python 3.5, all old `.pyo` files from previous versions of Python are + invalid regardless of this PEP. + +* The :mod:`socket` module now exports the :data:`~socket.CAN_RAW_FD_FRAMES` + constant on linux 3.6 and greater. + +* The :func:`ssl.cert_time_to_seconds` function now interprets the input time + as UTC and not as local time, per :rfc:`5280`. Additionally, the return + value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.) + +* The ``pygettext.py`` Tool now uses the standard +NNNN format for timezones in + the POT-Creation-Date header. + +* The :mod:`smtplib` module now uses :data:`sys.stderr` instead of the previous + module-level :data:`stderr` variable for debug output. If your (test) + program depends on patching the module-level variable to capture the debug + output, you will need to update it to capture sys.stderr instead. + +* The :meth:`str.startswith` and :meth:`str.endswith` methods no longer return + ``True`` when finding the empty string and the indexes are completely out of + range. (Contributed by Serhiy Storchaka in :issue:`24284`.) + +* The :func:`inspect.getdoc` function now returns documentation strings + inherited from base classes. Documentation strings no longer need to be + duplicated if the inherited documentation is appropriate. To suppress an + inherited string, an empty string must be specified (or the documentation + may be filled in). This change affects the output of the :mod:`pydoc` + module and the :func:`help` function. + (Contributed by Serhiy Storchaka in :issue:`15582`.) + +* Nested :func:`functools.partial` calls are now flattened. If you were + relying on the previous behavior, you can now either add an attribute to a + :func:`functools.partial` object or you can create a subclass of + :func:`functools.partial`. + (Contributed by Alexander Belopolsky in :issue:`7830`.) + +Changes in the C API +-------------------- + +* The undocumented :c:member:`~PyMemoryViewObject.format` member of the + (non-public) :c:type:`PyMemoryViewObject` structure has been removed. + All extensions relying on the relevant parts in ``memoryobject.h`` + must be rebuilt. + +* The :c:type:`PyMemAllocator` structure was renamed to + :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. + +* Removed non-documented macro :c:macro:`PyObject_REPR` which leaked references. + Use format character ``%R`` in :c:func:`PyUnicode_FromFormat`-like functions + to format the :func:`repr` of the object. + (Contributed by Serhiy Storchaka in :issue:`22453`.) + +* Because the lack of the :attr:`__module__` attribute breaks pickling and + introspection, a deprecation warning is now raised for builtin types without + the :attr:`__module__` attribute. This would be an AttributeError in + the future. + (Contributed by Serhiy Storchaka in :issue:`20204`.) + +* As part of the :pep:`492` implementation, the ``tp_reserved`` slot of + :c:type:`PyTypeObject` was replaced with a + :c:member:`tp_as_async` slot. Refer to :ref:`coro-objects` for + new types, structures and functions. + diff --git a/Doc/whatsnew/index.rst b/Doc/whatsnew/index.rst index 29902e4..edb5502 100644 --- a/Doc/whatsnew/index.rst +++ b/Doc/whatsnew/index.rst @@ -11,6 +11,7 @@ anyone wishing to stay up-to-date after a new release. .. toctree:: :maxdepth: 2 + 3.5.rst 3.4.rst 3.3.rst 3.2.rst |