path: root/Doc/whatsnew/3.9.rst

****************************
  What's New In Python 3.9
****************************

:Release: |release|
:Date: |today|

.. Rules for maintenance:

   * Anyone can add text to this document.  Do not spend very much time
   on the wording of your changes, because your text will probably
   get rewritten to some degree.

   * The maintainer will go through Misc/NEWS periodically and add
   changes; it's therefore more important to add your changes to
   Misc/NEWS than to this file.

   * This is not a complete list of every single change; completeness
   is the purpose of Misc/NEWS.  Some changes I consider too small
   or esoteric to include.  If such a change is added to the text,
   I'll just remove it.  (This is another reason you shouldn't spend
   too much time on writing your addition.)

   * If you want to draw your new text to the attention of the
   maintainer, add 'XXX' to the beginning of the paragraph or
   section.

   * It's OK to just add a fragmentary note about a change.  For
   example: "XXX Describe the transmogrify() function added to the
   socket module."  The maintainer will research the change and
   write the necessary text.

   * You can comment out your additions if you like, but it's not
   necessary (especially when a final release is some months away).

   * Credit the author of a patch or bugfix.   Just the name is
   sufficient; the e-mail address isn't necessary.

   * It's helpful to add the bug/patch number as a comment:

   XXX Describe the transmogrify() function added to the socket
   module.
   (Contributed by P.Y. Developer in :issue:`12345`.)

   This saves the maintainer the effort of going through the Mercurial log
   when researching a change.

This article explains the new features in Python 3.9, compared to 3.8.

For full details, see the :ref:`changelog <changelog>`.

.. note::

   Prerelease users should be aware that this document is currently in draft
   form. It will be updated substantially as Python 3.9 moves towards release,
   so it's worth checking back even after reading earlier versions.


Summary -- Release highlights
=============================

.. This section singles out the most important changes in Python 3.9.
   Brevity is key.


.. PEP-sized items next.



New Features
============



Other Language Changes
======================

* :func:`__import__` now raises :exc:`ImportError` instead of
  :exc:`ValueError`, which used to occur when a relative import went past
  its top-level package.
  (Contributed by Ngalim Siregar in :issue:`37444`.)


* Python now gets the absolute path of the script filename specified on
  the command line (ex: ``python3 script.py``): the ``__file__`` attribute of
  the :mod:`__main__` module and ``sys.path[0]`` become an
  absolute path, rather than a relative path. These paths now remain valid
  after the current directory is changed by :func:`os.chdir`. As a side effect,
  a traceback also displays the absolute path for :mod:`__main__` module frames
  in this case.
  (Contributed by Victor Stinner in :issue:`20443`.)

* In the :ref:`Python Development Mode <devmode>` and in debug build, the
  *encoding* and *errors* arguments are now checked for string encoding and
  decoding operations. Examples: :func:`open`, :meth:`str.encode` and
  :meth:`bytes.decode`.

  By default, for best performance, the *errors* argument is only checked at
  the first encoding/decoding error and the *encoding* argument is sometimes
  ignored for empty strings.
  (Contributed by Victor Stinner in :issue:`37388`.)

* ``"".replace("", s, n)`` now returns ``s`` instead of an empty string for
  all non-zero ``n``.  It is now consistent with ``"".replace("", s)``.
  There are similar changes for :class:`bytes` and :class:`bytearray` objects.
  (Contributed by Serhiy Storchaka in :issue:`28029`.)


New Modules
===========

* None yet.


Improved Modules
================

ast
---

Added the *indent* option to :func:`~ast.dump` which allows it to produce a
multiline indented output.
(Contributed by Serhiy Storchaka in :issue:`37995`.)

Added :func:`ast.unparse` as a function in the :mod:`ast` module that can
be used to unparse an :class:`ast.AST` object and produce a string with code
that would produce an equivalent :class:`ast.AST` object when parsed.
(Contributed by Pablo Galindo and Batuhan Taskaya in :issue:`38870`.)

asyncio
-------

Due to significant security concerns, the *reuse_address* parameter of
:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is
because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more
details, see the documentation for ``loop.create_datagram_endpoint()``.
(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in
:issue:`37228`.)

Added a new :term:`coroutine` :meth:`~asyncio.loop.shutdown_default_executor`
that schedules a shutdown for the default executor that waits on the
:class:`~concurrent.futures.ThreadPoolExecutor` to finish closing. Also,
:func:`asyncio.run` has been updated to use the new :term:`coroutine`.
(Contributed by Kyle Stanley in :issue:`34037`.)

Added :class:`asyncio.PidfdChildWatcher`, a Linux-specific child watcher
implementation that polls process file descriptors. (:issue:`38692`)

concurrent.futures
------------------

Added a new *cancel_futures* parameter to
:meth:`concurrent.futures.Executor.shutdown` that cancels all pending futures
which have not started running, instead of waiting for them to complete before
shutting down the executor.
(Contributed by Kyle Stanley in :issue:`39349`.)

curses
------

Add :func:`curses.get_escdelay`, :func:`curses.set_escdelay`,
:func:`curses.get_tabsize`, and :func:`curses.set_tabsize` functions.
(Contributed by Anthony Sottile in :issue:`38312`.)

fcntl
-----

Added constants :data:`~fcntl.F_OFD_GETLK`, :data:`~fcntl.F_OFD_SETLK`
and :data:`~fcntl.F_OFD_SETLKW`.
(Contributed by Dong-hee Na in :issue:`38602`.)

ftplib
-------

:class:`~ftplib.FTP` and :class:`~ftplib.FTP_TLS` now raise a :class:`ValueError`
if the given timeout for their constructor is zero to prevent the creation of
a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.)

functools
---------

Add the :class:`functools.TopologicalSorter` class to offer functionality to perform
topological sorting of graphs. (Contributed by Pablo Galindo, Tim Peters and Larry
Hastings in :issue:`17005`.)

gc
--

When the garbage collector makes a collection in which some objects resurrect
(they are reachable from outside the isolated cycles after the finalizers have
been executed), do not block the collection of all objects that are still
unreachable. (Contributed by Pablo Galindo and Tim Peters in :issue:`38379`.)

Added a new function :func:`gc.is_finalized` to check if an object has been
finalized by the garbage collector. (Contributed by Pablo Galindo in
:issue:`39322`.)

imaplib
-------

:class:`~imaplib.IMAP4` and :class:`~imaplib.IMAP4_SSL` now have
an optional *timeout* parameter for their constructors.
Also, the :meth:`~imaplib.IMAP4.open` method now has an optional *timeout* parameter
with this change. The overridden methods of :class:`~imaplib.IMAP4_SSL` and
:class:`~imaplib.IMAP4_stream` were applied to this change.
(Contributed by Dong-hee Na in :issue:`38615`.)

importlib
---------

To improve consistency with import statements, :func:`importlib.util.resolve_name`
now raises :exc:`ImportError` instead of :exc:`ValueError` for invalid relative
import attempts.
(Contributed by Ngalim Siregar in :issue:`37444`.)

math
----

Add :func:`math.nextafter`: return the next floating-point value after *x*
towards *y*.
(Contributed by Victor Stinner in :issue:`39288`.)

Add :func:`math.ulp`: return the value of the least significant bit
of a float.
(Contributed by Victor Stinner in :issue:`39310`.)

nntplib
-------

:class:`~nntplib.NNTP` and :class:`~nntplib.NNTP_SSL` now raise a :class:`ValueError`
if the given timeout for their constructor is zero to prevent the creation of
a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.)

os
--

Added :data:`~os.CLD_KILLED` and :data:`~os.CLD_STOPPED` for :attr:`si_code`.
(Contributed by Dong-hee Na in :issue:`38493`.)

Exposed the Linux-specific :func:`os.pidfd_open` (:issue:`38692`) and
:data:`os.P_PIDFD` (:issue:`38713`) for process management with file
descriptors.

The :func:`os.unsetenv` function is now also available on Windows.
(Contributed by Victor Stinner in :issue:`39413`.)

The :func:`os.putenv` and :func:`os.unsetenv` functions are now always
available.
(Contributed by Victor Stinner in :issue:`39395`.)

pathlib
-------

Added :meth:`pathlib.Path.readlink()` which acts similarly to
:func:`os.readlink`.
(Contributed by Girts Folkmanis in :issue:`30618`)

poplib
------

:class:`~poplib.POP3` and :class:`~poplib.POP3_SSL` now raise a :class:`ValueError`
if the given timeout for their constructor is zero to prevent the creation of
a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.)

pprint
------

:mod:`pprint` can now pretty-print :class:`types.SimpleNamespace`.
(Contributed by Carl Bordum Hansen in :issue:`37376`.)

signal
------

Exposed the Linux-specific :func:`signal.pidfd_send_signal` for sending to
signals to a process using a file descriptor instead of a pid. (:issue:`38712`)

smtplib
-------

:class:`~smtplib.SMTP` and :class:`~smtplib.SMTP_SSL` now raise a :class:`ValueError`
if the given timeout for their constructor is zero to prevent the creation of
a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.)

:class:`~smtplib.LMTP` constructor  now has an optional *timeout* parameter.
(Contributed by Dong-hee Na in :issue:`39329`.)

threading
---------

In a subinterpreter, spawning a daemon thread now raises a :exc:`RuntimeError`. Daemon
threads were never supported in subinterpreters. Previously, the subinterpreter
finalization crashed with a Python fatal error if a daemon thread was still
running.
(Contributed by Victor Stinner in :issue:`37266`.)

typing
------

:pep:`593` introduced an :data:`typing.Annotated` type to decorate existing
types with context-specific metadata and new ``include_extras`` parameter to
:func:`typing.get_type_hints` to access the metadata at runtime. (Contributed
by Till Varoquaux and Konstantin Kashin.)

venv
----

The activation scripts provided by :mod:`venv` now all specify their prompt
customization consistently by always using the value specified by
``__VENV_PROMPT__``. Previously some scripts unconditionally used
``__VENV_PROMPT__``, others only if it happened to be set (which was the default
case), and one used ``__VENV_NAME__`` instead.
(Contributed by Brett Cannon in :issue:`37663`.)


Optimizations
=============

* Optimized the idiom for assignment a temporary variable in comprehensions.
  Now ``for y in [expr]`` in comprehensions is as fast as a simple assignment
  ``y = expr``.  For example:

     sums = [s for s in [0] for x in data for s in [s + x]]

  Unlike to the ``:=`` operator this idiom does not leak a variable to the
  outer scope.

  (Contributed by Serhiy Storchaka in :issue:`32856`.)


Build and C API Changes
=======================

* Add a new public :c:func:`PyObject_CallNoArgs` function to the C API, which
  calls a callable Python object without any arguments. It is the most efficient
  way to call a callable Python object without any argument.
  (Contributed by Victor Stinner in :issue:`37194`.)

* The global variable :c:data:`PyStructSequence_UnnamedField` is now a constant
  and refers to a constant string.
  (Contributed by Serhiy Storchaka in :issue:`38650`.)

* Exclude ``PyFPE_START_PROTECT()`` and ``PyFPE_END_PROTECT()`` macros of
  ``pyfpe.h`` from ``Py_LIMITED_API`` (stable API).
  (Contributed by Victor Stinner in :issue:`38835`.)

* Remove ``PyMethod_ClearFreeList()`` and ``PyCFunction_ClearFreeList()``
  functions: the free lists of bound method objects have been removed.
  (Contributed by Inada Naoki and Victor Stinner in :issue:`37340`.)

* Remove ``PyUnicode_ClearFreeList()`` function: the Unicode free list has been
  removed in Python 3.3.
  (Contributed by Victor Stinner in :issue:`38896`.)

* The ``tp_print`` slot of :ref:`PyTypeObject <type-structs>` has been removed.
  It was used for printing objects to files in Python 2.7 and before. Since
  Python 3.0, it has been ignored and unused.
  (Contributed by Jeroen Demeyer in :issue:`36974`.)

* On non-Windows platforms, the :c:func:`setenv` and :c:func:`unsetenv`
  functions are now required to build Python.
  (Contributed by Victor Stinner in :issue:`39395`.)

* The ``COUNT_ALLOCS`` special build macro has been removed.
  (Contributed by Victor Stinner in :issue:`39489`.)

* Changes in the limited C API (if ``Py_LIMITED_API`` macro is defined):

  * Provide :c:func:`Py_EnterRecursiveCall` and :c:func:`Py_LeaveRecursiveCall`
    as regular functions for the limited API. Previously, there were defined as
    macros, but these macros didn't compile with the limited C API which cannot
    access ``PyThreadState.recursion_depth`` field (the structure is opaque in
    the limited C API).

  * Exclude the following functions from the limited C API:

    * ``_Py_CheckRecursionLimit``
    * ``_Py_NewReference()``
    * ``_Py_ForgetReference()``
    * ``_PyTraceMalloc_NewReference()``
    * ``_Py_GetRefTotal()``
    * The trashcan mechanism which never worked in the limited C API.
    * ``PyTrash_UNWIND_LEVEL``
    * ``Py_TRASHCAN_BEGIN_CONDITION``
    * ``Py_TRASHCAN_BEGIN``
    * ``Py_TRASHCAN_END``
    * ``Py_TRASHCAN_SAFE_BEGIN``
    * ``Py_TRASHCAN_SAFE_END``

  * The following static inline functions or macros become regular "opaque"
    function to hide implementation details:

    * ``_Py_NewReference()``
    * ``PyObject_INIT()`` and ``PyObject_INIT_VAR()``  become aliases to
      :c:func:`PyObject_Init` and :c:func:`PyObject_InitVar` in the limited C
      API, but are overriden with static inline function otherwise. Thanks to
      that, it was possible to exclude ``_Py_NewReference()`` from the limited
      C API.

  * Move following functions and definitions to the internal C API:

    * ``_PyDebug_PrintTotalRefs()``
    * ``_Py_PrintReferences()``
    * ``_Py_PrintReferenceAddresses()``
    * ``_Py_tracemalloc_config``
    * ``_Py_AddToAllObjects()`` (specific to ``Py_TRACE_REFS`` build)

  (Contributed by Victor Stinner in :issue:`38644` and :issue:`39542`.)


Deprecated
==========

* The distutils ``bdist_msi`` command is now deprecated, use
  ``bdist_wheel`` (wheel packages) instead.
  (Contributed by Hugo van Kemenade in :issue:`39586`.)

* Currently :func:`math.factorial` accepts :class:`float` instances with
  non-negative integer values (like ``5.0``).  It raises a :exc:`ValueError`
  for non-integral and negative floats.  It is now deprecated.  In future
  Python versions it will raise a :exc:`TypeError` for all floats.
  (Contributed by Serhiy Storchaka in :issue:`37315`.)

* The :mod:`parser` module is deprecated and will be removed in future versions
  of Python. For the majority of use cases, users can leverage the Abstract Syntax
  Tree (AST) generation and compilation stage, using the :mod:`ast` module.

* The :mod:`random` module currently accepts any hashable type as a
  possible seed value.  Unfortunately, some of those types are not
  guaranteed to have a deterministic hash value.  After Python 3.9,
  the module will restrict its seeds to :const:`None`, :class:`int`,
  :class:`float`, :class:`str`, :class:`bytes`, and :class:`bytearray`.

* Opening the :class:`~gzip.GzipFile` file for writing without specifying
  the *mode* argument is deprecated.  In future Python versions it will always
  be opened for reading by default.  Specify the *mode* argument for opening
  it for writing and silencing a warning.
  (Contributed by Serhiy Storchaka in :issue:`28286`.)

* Deprecated the ``split()`` method of :class:`_tkinter.TkappType` in
  favour of the ``splitlist()`` method which has more consistent and
  predicable behavior.
  (Contributed by Serhiy Storchaka in :issue:`38371`.)

* The explicit passing of coroutine objects to :func:`asyncio.wait` has been
  deprecated and will be removed in version 3.11.
  (Contributed by Yury Selivanov and Kyle Stanley in :issue:`34790`.)

* binhex4 and hexbin4 standards are now deprecated. The :`binhex` module
  and the following :mod:`binascii` functions are now deprecated:

  * :func:`~binascii.b2a_hqx`, :func:`~binascii.a2b_hqx`
  * :func:`~binascii.rlecode_hqx`, :func:`~binascii.rledecode_hqx`

  (Contributed by Victor Stinner in :issue:`39353`.)


Removed
=======

* The erroneous version at :data:`unittest.mock.__version__` has been removed.

* :class:`nntplib.NNTP`: ``xpath()`` and ``xgtitle()`` methods have been removed.
  These methods are deprecated since Python 3.3. Generally, these extensions
  are not supported or not enabled by NNTP server administrators.
  For ``xgtitle()``, please use :meth:`nntplib.NNTP.descriptions` or
  :meth:`nntplib.NNTP.description` instead.
  (Contributed by Dong-hee Na in :issue:`39366`.)

* :class:`array.array`: ``tostring()`` and ``fromstring()`` methods have been
  removed. They were aliases to ``tobytes()`` and ``frombytes()``, deprecated
  since Python 3.2.
  (Contributed by Victor Stinner in :issue:`38916`.)

* The abstract base classes in :mod:`collections.abc` no longer are
  exposed in the regular :mod:`collections` module.  This will help
  create a clearer distinction between the concrete classes and the abstract
  base classes.

* The undocumented ``sys.callstats()`` function has been removed. Since Python
  3.7, it was deprecated and always returned :const:`None`. It required a special
  build option ``CALL_PROFILE`` which was already removed in Python 3.7.
  (Contributed by Victor Stinner in :issue:`37414`.)

* The ``sys.getcheckinterval()`` and ``sys.setcheckinterval()`` functions have
  been removed. They were deprecated since Python 3.2. Use
  :func:`sys.getswitchinterval` and :func:`sys.setswitchinterval` instead.
  (Contributed by Victor Stinner in :issue:`37392`.)

* The C function ``PyImport_Cleanup()`` has been removed. It was documented as:
  "Empty the module table.  For internal use only."
  (Contributed by Victor Stinner in :issue:`36710`.)

* ``_dummy_thread`` and ``dummy_threading`` modules have been removed. These
  modules were deprecated since Python 3.7 which requires threading support.
  (Contributed by Victor Stinner in :issue:`37312`.)

* ``aifc.openfp()`` alias to ``aifc.open()``, ``sunau.openfp()`` alias to
  ``sunau.open()``, and ``wave.openfp()`` alias to :func:`wave.open()` have been
  removed. They were deprecated since Python 3.7.
  (Contributed by Victor Stinner in :issue:`37320`.)

* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread`
  has been removed. It was deprecated since Python 3.8.
  Use :meth:`~threading.Thread.is_alive()` instead.
  (Contributed by Dong-hee Na in :issue:`37804`.)

* Methods ``getchildren()`` and ``getiterator()`` in the
  :mod:`~xml.etree.ElementTree` module have been removed.  They were
  deprecated in Python 3.2.  Use functions :func:`list` and :func:`iter`
  instead.  The ``xml.etree.cElementTree`` module has been removed.
  (Contributed by Serhiy Storchaka in :issue:`36543`.)

* The old :mod:`plistlib` API has been removed, it was deprecated since Python
  3.4. Use the :func:`~plistlib.load`, :func:`~plistlib.loads`, :func:`~plistlib.dump`, and
  :func:`~plistlib.dumps` functions. Additionally, the *use_builtin_types* parameter was
  removed, standard :class:`bytes` objects are always used instead.
  (Contributed by Jon Janzen in :issue:`36409`.)

* The C function ``PyThreadState_DeleteCurrent()`` has been removed. It was not documented.
  (Contributed by Joannah Nanjekye in :issue:`37878`.)

* The C function ``PyGen_NeedsFinalizing`` has been removed. It was not
  documented, tested, or used anywhere within CPython after the implementation
  of :pep:`442`. Patch by Joannah Nanjekye.
  (Contributed by Joannah Nanjekye in :issue:`15088`)

* ``base64.encodestring()`` and ``base64.decodestring()``, aliases deprecated
  since Python 3.1, have been removed: use :func:`base64.encodebytes` and
  :func:`base64.decodebytes` instead.
  (Contributed by Victor Stinner in :issue:`39351`.)

* ``fractions.gcd()`` function has been removed, it was deprecated since Python
  3.5 (:issue:`22486`): use :func:`math.gcd` instead.
  (Contributed by Victor Stinner in :issue:`39350`.)

* The *buffering* parameter of :class:`bz2.BZ2File` has been removed. Since
  Python 3.0, it was ignored and using it emitted a :exc:`DeprecationWarning`.
  Pass an open file object to control how the file is opened.
  (Contributed by Victor Stinner in :issue:`39357`.)

* The *encoding* parameter of :func:`json.loads` has been removed.
  As of Python 3.1, it was deprecated and ignored; using it has emitted a
  :exc:`DeprecationWarning` since Python 3.8.
  (Contributed by Inada Naoki in :issue:`39377`)

* ``with (await asyncio.lock):`` and ``with (yield from asyncio.lock):`` statements are
  not longer supported, use ``async with lock`` instead.  The same is correct for
  ``asyncio.Condition`` and ``asyncio.Semaphore``.
  (Contributed by Andrew Svetlov in :issue:`34793`.)

* The :func:`sys.getcounts` function, the ``-X showalloccount`` command line
  option and the ``show_alloc_count`` field of the C structure
  :c:type:`PyConfig` have been removed. They required a special Python build by
  defining ``COUNT_ALLOCS`` macro.
  (Contributed by Victor Stinner in :issue:`39489`.)


Porting to Python 3.9
=====================

This section lists previously described changes and other bugfixes
that may require changes to your code.


Changes in the Python API
-------------------------

* :func:`open`, :func:`io.open`, :func:`codecs.open` and
  :class:`fileinput.FileInput` no longer accept ``'U'`` ("universal newline")
  in the file mode. This flag was deprecated since Python 3.3. In Python 3, the
  "universal newline" is used by default when a file is open in text mode.  The
  :ref:`newline parameter <open-newline-parameter>` of :func:`open` controls
  how universal newlines works.
  (Contributed by Victor Stinner in :issue:`37330`.)

* :func:`__import__` and :func:`importlib.util.resolve_name` now raise
  :exc:`ImportError` where it previously raised :exc:`ValueError`. Callers
  catching the specific exception type and supporting both Python 3.9 and
  earlier versions will need to catch both using ``except (ImportError, ValueError):``.

* The :mod:`venv` activation scripts no longer special-case when
  ``__VENV_PROMPT__`` is set to ``""``.

* The :meth:`select.epoll.unregister` method no longer ignores the
  :data:`~errno.EBADF` error.
  (Contributed by Victor Stinner in :issue:`39239`.)

* The *compresslevel* parameter of :class:`bz2.BZ2File` became keyword-only,
  since the *buffering* parameter has been removed.
  (Contributed by Victor Stinner in :issue:`39357`.)


CPython bytecode changes
------------------------

* The :opcode:`LOAD_ASSERTION_ERROR` opcode was added for handling the
  :keyword:`assert` statement. Previously, the assert statement would not work
  correctly if the :exc:`AssertionError` exception was being shadowed.
  (Contributed by Zackery Spytz in :issue:`34880`.)