bpo-33649: Fix asyncio-dev (GH-9324)

1 files changed, 138 insertions, 316 deletions

diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst
index 87abe92..e815318 100644
--- a/Doc/library/asyncio-dev.rst
+++ b/Doc/library/asyncio-dev.rst
@@ -2,415 +2,237 @@
 
 .. _asyncio-dev:
 
-Develop with asyncio
-====================
+=======================
+Developing with asyncio
+=======================
 
-Asynchronous programming is different than classical "sequential" programming.
-This page lists common traps and explains how to avoid them.
+Asynchronous programming is different from classical "sequential"
+programming.
+
+This page lists common mistakes and traps and explains how
+to avoid them.
 
 
 .. _asyncio-debug-mode:
 
-Debug mode of asyncio
----------------------
-
-The implementation of :mod:`asyncio` has been written for performance.
-In order to ease the development of asynchronous code, you may wish to
-enable *debug mode*.
-
-To enable all debug checks for an application:
-
-* Enable the asyncio debug mode globally by setting the environment variable
-  :envvar:`PYTHONASYNCIODEBUG` to ``1``, using ``-X dev`` command line option
-  (see the :option:`-X` option), or by calling
-  :meth:`loop.set_debug`.
-* Set the log level of the :ref:`asyncio logger <asyncio-logger>` to
-  :py:data:`logging.DEBUG`. For example, call
-  ``logging.basicConfig(level=logging.DEBUG)`` at startup.
-* Configure the :mod:`warnings` module to display :exc:`ResourceWarning`
-  warnings. For example, use the ``-Wdefault`` command line option of Python to
-  display them.
-
-Examples debug checks:
-
-* Log :ref:`coroutines defined but never "yielded from"
-  <asyncio-coroutine-not-scheduled>`
-* :meth:`loop.call_soon` and :meth:`loop.call_at` methods
-  raise an exception if they are called from the wrong thread.
-* Log the execution time of the selector
-* Log callbacks taking more than 100 ms to be executed. The
-  :attr:`loop.slow_callback_duration` attribute is the minimum
-  duration in seconds of "slow" callbacks.
-* :exc:`ResourceWarning` warnings are emitted when transports and event loops
-  are :ref:`not closed explicitly <asyncio-close-transports>`.
+Debug Mode
+==========
 
-.. versionchanged:: 3.7
+By default asyncio runs in production mode.  In order to ease
+the development asyncio has a *debug mode*.
 
-   The new ``-X dev`` command line option can now also be used to enable
-   the debug mode.
+To enable debugging for an application:
 
-.. seealso::
+* Enable the debug mode globally by setting the environment variable
+  :envvar:`PYTHONASYNCIODEBUG` to ``1``.
 
-   The :meth:`loop.set_debug` method and the :ref:`asyncio logger
-   <asyncio-logger>`.
+* Alternatively, the debug mode can be enabled by using the ``-X dev``
+  command line option for Python (see the :option:`-X` option).
 
+* Yet another way to enable the debug mode is by calling
+  :meth:`loop.set_debug` or by passing ``debug=True`` to
+  :func:`asyncio.run`.
 
-Cancellation
-------------
+In addition to enabling debug mode, consider also:
 
-Cancellation of tasks is not common in classic programming. In asynchronous
-programming, not only is it something common, but you have to prepare your
-code to handle it.
+* setting the log level of the :ref:`asyncio logger <asyncio-logger>` to
+  :py:data:`logging.DEBUG`, for example the following snippet of code
+  can be run at startup of the application::
 
-Futures and tasks can be cancelled explicitly with their :meth:`Future.cancel`
-method. The :func:`wait_for` function cancels the waited task when the timeout
-occurs. There are many other cases where a task can be cancelled indirectly.
+    logging.basicConfig(level=logging.DEBUG)
 
-Don't call :meth:`~Future.set_result` or :meth:`~Future.set_exception` method
-of :class:`Future` if the future is cancelled: it would fail with an exception.
-For example, write::
+* configuring the :mod:`warnings` module to display
+  :exc:`ResourceWarning` warnings.  One way of doing that is by
+  using the ``-Wdefault`` command line option.
 
-    if not fut.cancelled():
-        fut.set_result('done')
 
-Don't schedule directly a call to the :meth:`~Future.set_result` or the
-:meth:`~Future.set_exception` method of a future with
-:meth:`loop.call_soon`: the future can be cancelled before its method
-is called.
+In asyncio debug mode the following checks are performed:
 
-If you wait for a future, you should check early if the future was cancelled to
-avoid useless operations. Example::
+* Log :ref:`coroutines that were not awaited
+  <asyncio-coroutine-not-scheduled>`; this mitigates the "forgotten
+  await" pitfall.
 
-    async def slow_operation(fut):
-        if fut.cancelled():
-            return
-        # ... slow computation ...
-        await fut
-        # ...
+* Many non-treadsafe asyncio APIs (such as :meth:`loop.call_soon` and
+  :meth:`loop.call_at` methods) raise an exception if they are called
+  from a wrong thread.
 
-The :func:`shield` function can also be used to ignore cancellation.
+* Log the execution time of the IO selector if it takes too long to
+  perform an IO operation.
+
+* Log callbacks taking longer than 100 ms to be executed.  The
+  :attr:`loop.slow_callback_duration` attribute is the minimum
+  duration in seconds of "slow" callbacks.
 
 
 .. _asyncio-multithreading:
 
-Concurrency and multithreading
-------------------------------
+Concurrency and Multithreading
+==============================
 
 An event loop runs in a thread (typically the main thread) and executes
-all callbacks and tasks in its thread.  While a task is running in the
-event loop, no other tasks may run in the same thread. When a task
-executes an ``await`` expression, the running task gets suspended, and the
-event loop executes the next task. Prior to suspending the task, the awaiting
-chain is checked, and if the chain ends with a future, the running task is
-not suspended.
-
-To schedule a callback from a different thread, the
+all callbacks and Tasks in its thread.  While a Task is running in the
+event loop, no other Tasks can run in the same thread.  When a Task
+executes an ``await`` expression, the running Task gets suspended, and
+the event loop executes the next Task.
+
+To schedule a callback from a different OS thread, the
 :meth:`loop.call_soon_threadsafe` method should be used. Example::
 
     loop.call_soon_threadsafe(callback, *args)
 
-Most asyncio objects are not thread safe. You should only worry if you access
-objects outside the event loop. For example, to cancel a future, don't call
-directly its :meth:`Future.cancel` method, but::
+Almost all asyncio objects are not thread safe, which is typically
+not a problem unless there is code that works with them from outside
+of a Task or a callback.  If there's a need for such code to call a
+low-level asyncio API, the :meth:`loop.call_soon_threadsafe` method
+should be used, e.g.::
 
     loop.call_soon_threadsafe(fut.cancel)
 
-To handle signals and to execute subprocesses, the event loop must be run in
-the main thread.
-
-To schedule a coroutine object from a different thread, the
+To schedule a coroutine object from a different OS thread, the
 :func:`run_coroutine_threadsafe` function should be used. It returns a
 :class:`concurrent.futures.Future` to access the result::
 
-     future = asyncio.run_coroutine_threadsafe(coro_func(), loop)
-     result = future.result(timeout)  # Wait for the result with a timeout
-
-The :meth:`loop.run_in_executor` method can be used with a
-:class:`concurrent.futures.ThreadPoolExecutor` to execute a callback in
-different thread so as not to block the event loop's main thread.
+     async def coro_func():
+          return await asyncio.sleep(1, 42)
 
-.. seealso::
+     # Later in another OS thread:
 
-   The :ref:`Synchronization primitives <asyncio-sync>` section describes ways
-   to synchronize tasks.
-
-   The :ref:`Subprocess and threads <asyncio-subprocess-threads>` section lists
-   asyncio limitations to run subprocesses from different threads.
+     future = asyncio.run_coroutine_threadsafe(coro_func(), loop)
+     # Wait for the result:
+     result = future.result()
 
+To handle signals and to execute subprocesses, the event loop must be
+run in the main thread.
 
+The :meth:`loop.run_in_executor` method can be used with a
+:class:`concurrent.futures.ThreadPoolExecutor` to execute
+blocking code in a different OS thread without blocking the OS thread
+that the event loop runs in.
 
 
 .. _asyncio-handle-blocking:
 
-Handle blocking functions correctly
------------------------------------
-
-Blocking functions should not be called directly. For example, if a function
-blocks for 1 second, other tasks are delayed by 1 second which can have an
-important impact on reactivity.
-
-For networking and subprocesses, the :mod:`asyncio` module provides high-level
-APIs like :ref:`protocols <asyncio-protocol>`.
+Running Blocking Code
+=====================
 
-An executor can be used to run a task in a different thread or even in a
-different process, to not block the thread of the event loop. See the
-:meth:`loop.run_in_executor` method.
+Blocking (CPU-bound) code should not be called directly.  For example,
+if a function performs a CPU-intensive calculation for 1 second,
+all concurrent asyncio Tasks and IO operations would be delayed
+by 1 second.
 
-.. seealso::
-
-   The :ref:`Delayed calls <asyncio-delayed-calls>` section details how the
-   event loop handles time.
+An executor can be used to run a task in a different thread or even in
+a different process to avoid blocking block the OS thread with the
+event loop.  See the :meth:`loop.run_in_executor` method for more
+details.
 
 
 .. _asyncio-logger:
 
 Logging
--------
-
-The :mod:`asyncio` module logs information with the :mod:`logging` module in
-the logger ``'asyncio'``.
+=======
 
-The default log level for the :mod:`asyncio` module is :py:data:`logging.INFO`.
-For those not wanting such verbosity from :mod:`asyncio` the log level can
-be changed.  For example, to change the level to :py:data:`logging.WARNING`:
+asyncio uses the :mod:`logging` module and all logging is performed
+via the ``"asyncio"`` logger.
 
-.. code-block:: none
+The default log level is :py:data:`logging.INFO`, which can easily be
+adjusted::
 
-   logging.getLogger('asyncio').setLevel(logging.WARNING)
+   logging.getLogger("asyncio").setLevel(logging.WARNING)
 
 
 .. _asyncio-coroutine-not-scheduled:
 
-Detect coroutine objects never scheduled
-----------------------------------------
-
-When a coroutine function is called and its result is not passed to
-:meth:`asyncio.create_task` the execution of the coroutine object will
-never be scheduled which is probably a bug. Using ``asyncio.create_task`` is
-preferred to the low level :func:`ensure_future` and :meth:`loop.create_task`
-methods. :ref:`Enable the debug mode of asyncio <asyncio-debug-mode>`
-to :ref:`log a warning <asyncio-logger>` to detect it.
+Detect never awaited coroutines
+===============================
 
-Example with the bug::
+When a coroutine is called (e.g. ``coro()`` instead of ``await coro()``)
+the call is not wrapped with :meth:`asyncio.create_task`, the execution
+of the coroutine object will never be scheduled.  For example::
 
     import asyncio
 
     async def test():
         print("never scheduled")
 
+    async def main():
+        test()
+
+    asyncio.run(main())
+
+Output::
+
+  test.py:7: RuntimeWarning: coroutine 'test' was never awaited
     test()
 
 Output in debug mode::
 
-    Coroutine test() at test.py:3 was never yielded from
-    Coroutine object created at (most recent call last):
-      File "test.py", line 7, in <module>
-        test()
+  test.py:7: RuntimeWarning: coroutine 'test' was never awaited
+  Coroutine created at (most recent call last)
+    File "../t.py", line 9, in <module>
+      asyncio.run(main(), debug=True)
+
+    < .. >
 
-The fix is to call the :meth:`asyncio.create_task` function. Using
-``asyncio.create_task`` is preferred to the low level :func:`ensure_future` and
-:meth:`loop.create_task` methods.
+    File "../t.py", line 7, in main
+      test()
+    test()
 
-.. seealso::
+The usual fix is to either await the coroutine or call the
+:meth:`asyncio.create_task` function::
 
-   :ref:`Pending task destroyed <asyncio-pending-task-destroyed>`.
+    async def main():
+        await test()
 
 
-Detect exceptions never consumed
---------------------------------
+Detect never consumed exceptions
+================================
 
-Python usually calls :func:`sys.excepthook` on unhandled exceptions. If
-:meth:`Future.set_exception` is called, but the exception is never consumed,
-:func:`sys.excepthook` is not called. Instead, :ref:`a log is emitted
-<asyncio-logger>` when the future is deleted by the garbage collector, with the
-traceback where the exception was raised.
+If a :meth:`Future.set_exception` is called but the Future object is
+never awaited on, the exception would never be propagated to the
+user code.  In this case, asyncio would emit a log message when the
+Future object is garbage collected.
 
-Example of unhandled exception::
+Example of an unhandled exception::
 
     import asyncio
 
-    @asyncio.coroutine
-    def bug():
+    async def bug():
         raise Exception("not consumed")
 
-    loop = asyncio.get_event_loop()
-    asyncio.ensure_future(bug())
-    loop.run_forever()
-    loop.close()
+    async def main():
+        asyncio.create_task(bug())
+
+    asyncio.run(main())
 
 Output::
 
     Task exception was never retrieved
-    future: <Task finished coro=<coro() done, defined at asyncio/coroutines.py:139> exception=Exception('not consumed',)>
-    Traceback (most recent call last):
-      File "asyncio/tasks.py", line 237, in _step
-        result = next(coro)
-      File "asyncio/coroutines.py", line 141, in coro
-        res = func(*args, **kw)
-      File "test.py", line 5, in bug
-        raise Exception("not consumed")
-    Exception: not consumed
+    future: <Task finished coro=<bug() done, defined at test.py:3>
+      exception=Exception('not consumed')>
 
-:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to get the
-traceback where the task was created. Output in debug mode::
-
-    Task exception was never retrieved
-    future: <Task finished coro=<bug() done, defined at test.py:3> exception=Exception('not consumed',) created at test.py:8>
-    source_traceback: Object created at (most recent call last):
-      File "test.py", line 8, in <module>
-        asyncio.ensure_future(bug())
     Traceback (most recent call last):
-      File "asyncio/tasks.py", line 237, in _step
-        result = next(coro)
-      File "asyncio/coroutines.py", line 79, in __next__
-        return next(self.gen)
-      File "asyncio/coroutines.py", line 141, in coro
-        res = func(*args, **kw)
-      File "test.py", line 5, in bug
+      File "test.py", line 4, in bug
         raise Exception("not consumed")
     Exception: not consumed
 
-There are different options to fix this issue. The first option is to chain the
-coroutine in another coroutine and use classic try/except::
-
-    async def handle_exception():
-        try:
-            await bug()
-        except Exception:
-            print("exception consumed")
-
-    loop = asyncio.get_event_loop()
-    asyncio.ensure_future(handle_exception())
-    loop.run_forever()
-    loop.close()
-
-Another option is to use the :meth:`asyncio.run` function::
-
-    asyncio.run(bug())
-
-.. seealso::
-
-   The :meth:`Future.exception` method.
-
+:ref:`Enable the debug mode <asyncio-debug-mode>` to get the
+traceback where the task was created::
 
-Chain coroutines correctly
---------------------------
+    asyncio.run(main(), debug=True)
 
-When a coroutine function calls other coroutine functions and tasks, they
-should be chained explicitly with ``await``. Otherwise, the execution is
-not guaranteed to be sequential.
-
-Example with different bugs using :func:`asyncio.sleep` to simulate slow
-operations::
-
-    import asyncio
-
-    async def create():
-        await asyncio.sleep(3.0)
-        print("(1) create file")
-
-    async def write():
-        await asyncio.sleep(1.0)
-        print("(2) write into file")
-
-    async def close():
-        print("(3) close file")
-
-    async def test():
-        asyncio.ensure_future(create())
-        asyncio.ensure_future(write())
-        asyncio.ensure_future(close())
-        await asyncio.sleep(2.0)
-        loop.stop()
-
-    loop = asyncio.get_event_loop()
-    asyncio.ensure_future(test())
-    loop.run_forever()
-    print("Pending tasks at exit: %s" % asyncio.Task.all_tasks(loop))
-    loop.close()
-
-Expected output:
-
-.. code-block:: none
-
-    (1) create file
-    (2) write into file
-    (3) close file
-    Pending tasks at exit: set()
-
-Actual output:
-
-.. code-block:: none
-
-    (3) close file
-    (2) write into file
-    Pending tasks at exit: {<Task pending create() at test.py:7 wait_for=<Future pending cb=[Task._wakeup()]>>}
-    Task was destroyed but it is pending!
-    task: <Task pending create() done at test.py:5 wait_for=<Future pending cb=[Task._wakeup()]>>
-
-The loop stopped before the ``create()`` finished, ``close()`` has been called
-before ``write()``, whereas coroutine functions were called in this order:
-``create()``, ``write()``, ``close()``.
-
-To fix the example, tasks must be marked with ``await``::
-
-    async def test():
-        await asyncio.ensure_future(create())
-        await asyncio.ensure_future(write())
-        await asyncio.ensure_future(close())
-        await asyncio.sleep(2.0)
-        loop.stop()
-
-Or without ``asyncio.ensure_future()``::
-
-    async def test():
-        await create()
-        await write()
-        await close()
-        await asyncio.sleep(2.0)
-        loop.stop()
-
-
-.. _asyncio-pending-task-destroyed:
-
-Pending task destroyed
-----------------------
-
-If a pending task is destroyed, the execution of its wrapped :ref:`coroutine
-<coroutine>` did not complete. It is probably a bug and so a warning is logged.
-
-Example of log:
-
-.. code-block:: none
-
-    Task was destroyed but it is pending!
-    task: <Task pending coro=<kill_me() done, defined at test.py:5> wait_for=<Future pending cb=[Task._wakeup()]>>
-
-:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to get the
-traceback where the task was created. Example of log in debug mode:
+Output in debug mode::
 
-.. code-block:: none
+    Task exception was never retrieved
+    future: <Task finished coro=<bug() done, defined at test.py:3>
+        exception=Exception('not consumed') created at asyncio/tasks.py:321>
 
-    Task was destroyed but it is pending!
     source_traceback: Object created at (most recent call last):
-      File "test.py", line 15, in <module>
-        task = asyncio.ensure_future(coro, loop=loop)
-    task: <Task pending coro=<kill_me() done, defined at test.py:5> wait_for=<Future pending cb=[Task._wakeup()] created at test.py:7> created at test.py:15>
-
-
-.. seealso::
-
-   :ref:`Detect coroutine objects never scheduled <asyncio-coroutine-not-scheduled>`.
+      File "../t.py", line 9, in <module>
+        asyncio.run(main(), debug=True)
 
-.. _asyncio-close-transports:
+    < .. >
 
-Close transports and event loops
---------------------------------
-
-When a transport is no longer needed, call its ``close()`` method to release
-resources. Event loops must also be closed explicitly.
-
-If a transport or an event loop is not closed explicitly, a
-:exc:`ResourceWarning` warning will be emitted in its destructor. By default,
-:exc:`ResourceWarning` warnings are ignored. The :ref:`Debug mode of asyncio
-<asyncio-debug-mode>` section explains how to display them.
+    Traceback (most recent call last):
+      File "../t.py", line 4, in bug
+        raise Exception("not consumed")
+    Exception: not consumed