Issue #24439: Improve PEP 492 related docs.

Patch by Martin Panter.

11 files changed, 179 insertions, 85 deletions

diff --git a/Doc/glossary.rst b/Doc/glossary.rst
index bdbb272..d00185e 100644
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -169,18 +169,19 @@ Glossary
       statement by defining :meth:`__enter__` and :meth:`__exit__` methods.
       See :pep:`343`.
 
-   coroutine function
-      A function which returns a :term:`coroutine` object.  It is defined
-      with an :keyword:`async def` keyword, and may contain :keyword:`await`,
-      :keyword:`async for`, and :keyword:`async with` keywords.  Introduced
-      by :pep:`492`.
-
    coroutine
       Coroutines is a more generalized form of subroutines. Subroutines are
-      entered at one point and exited at another point.  Coroutines, can be
-      entered, exited, and resumed at many different points.  See
-      :keyword:`await` expressions, and :keyword:`async for` and
-      :keyword:`async with` statements. See also :pep:`492`.
+      entered at one point and exited at another point.  Coroutines can be
+      entered, exited, and resumed at many different points.  They can be
+      implemented with the :keyword:`async def` statement.  See also
+      :pep:`492`.
+
+   coroutine function
+      A function which returns a :term:`coroutine` object.  A coroutine
+      function may be defined with the :keyword:`async def` statement,
+      and may contain :keyword:`await`, :keyword:`async for`, and
+      :keyword:`async with` keywords.  These were introduced
+      by :pep:`492`.
 
    CPython
       The canonical implementation of the Python programming language, as
diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst
index e7ff7d2..fa9e96a 100644
--- a/Doc/library/asyncio-task.rst
+++ b/Doc/library/asyncio-task.rst
@@ -8,17 +8,23 @@ Tasks and coroutines
 Coroutines
 ----------
 
-A coroutine is a generator that follows certain conventions.  For
-documentation purposes, all coroutines should be decorated with
-``@asyncio.coroutine``, but this cannot be strictly enforced.
-
-Coroutines use the ``yield from`` syntax introduced in :pep:`380`,
+Coroutines used with :mod:`asyncio` may be implemented using the
+:keyword:`async def` statement, or by using :term:`generators <generator>`.
+The :keyword:`async def` type of coroutine was added in Python 3.5, and
+is recommended if there is no need to support older Python versions.
+
+Generator-based coroutines should be decorated with :func:`@asyncio.coroutine
+<asyncio.coroutine>`, although this is not strictly enforced.
+The decorator enables compatibility with :keyword:`async def` coroutines,
+and also serves as documentation.  Generator-based
+coroutines use the ``yield from`` syntax introduced in :pep:`380`,
 instead of the original ``yield`` syntax.
 
 The word "coroutine", like the word "generator", is used for two
 different (though related) concepts:
 
-- The function that defines a coroutine (a function definition
+- The function that defines a coroutine
+  (a function definition using :keyword:`async def` or
   decorated with ``@asyncio.coroutine``).  If disambiguation is needed
   we will call this a *coroutine function* (:func:`iscoroutinefunction`
   returns ``True``).
@@ -30,27 +36,28 @@ different (though related) concepts:
 
 Things a coroutine can do:
 
-- ``result = yield from future`` -- suspends the coroutine until the
+- ``result = await future`` or ``result = yield from future`` --
+  suspends the coroutine until the
   future is done, then returns the future's result, or raises an
   exception, which will be propagated.  (If the future is cancelled,
   it will raise a ``CancelledError`` exception.)  Note that tasks are
   futures, and everything said about futures also applies to tasks.
 
-- ``result = yield from coroutine`` -- wait for another coroutine to
+- ``result = await coroutine`` or ``result = yield from coroutine`` --
+  wait for another coroutine to
   produce a result (or raise an exception, which will be propagated).
   The ``coroutine`` expression must be a *call* to another coroutine.
 
 - ``return expression`` -- produce a result to the coroutine that is
-  waiting for this one using ``yield from``.
+  waiting for this one using :keyword:`await` or ``yield from``.
 
 - ``raise exception`` -- raise an exception in the coroutine that is
-  waiting for this one using ``yield from``.
+  waiting for this one using :keyword:`await` or ``yield from``.
 
-Calling a coroutine does not start its code running -- it is just a
-generator, and the coroutine object returned by the call is really a
-generator object, which doesn't do anything until you iterate over it.
-In the case of a coroutine object, there are two basic ways to start
-it running: call ``yield from coroutine`` from another coroutine
+Calling a coroutine does not start its code running --
+the coroutine object returned by the call doesn't do anything until you
+schedule its execution.  There are two basic ways to start it running:
+call ``await coroutine`` or ``yield from coroutine`` from another coroutine
 (assuming the other coroutine is already running!), or schedule its execution
 using the :func:`async` function or the :meth:`BaseEventLoop.create_task`
 method.
@@ -60,9 +67,15 @@ Coroutines (and tasks) can only run when the event loop is running.
 
 .. decorator:: coroutine
 
-    Decorator to mark coroutines.
+    Decorator to mark generator-based coroutines.  This enables
+    the generator use :keyword:`!yield from` to call :keyword:`async
+    def` coroutines, and also enables the generator to be called by
+    :keyword:`async def` coroutines, for instance using an
+    :keyword:`await` expression.
+
+    There is no need to decorate :keyword:`async def` coroutines themselves.
 
-    If the coroutine is not yielded from before it is destroyed, an error
+    If the generator is not yielded from before it is destroyed, an error
     message is logged. See :ref:`Detect coroutines never scheduled
     <asyncio-coroutine-not-scheduled>`.
 
@@ -84,8 +97,7 @@ Example of coroutine displaying ``"Hello World"``::
 
     import asyncio
 
-    @asyncio.coroutine
-    def hello_world():
+    async def hello_world():
         print("Hello World!")
 
     loop = asyncio.get_event_loop()
@@ -111,20 +123,30 @@ using the :meth:`sleep` function::
     import asyncio
     import datetime
 
-    @asyncio.coroutine
-    def display_date(loop):
+    async def display_date(loop):
         end_time = loop.time() + 5.0
         while True:
             print(datetime.datetime.now())
             if (loop.time() + 1.0) >= end_time:
                 break
-            yield from asyncio.sleep(1)
+            await asyncio.sleep(1)
 
     loop = asyncio.get_event_loop()
     # Blocking call which returns when the display_date() coroutine is done
     loop.run_until_complete(display_date(loop))
     loop.close()
 
+The same coroutine implemented using a generator::
+
+    @asyncio.coroutine
+    def display_date(loop):
+        end_time = loop.time() + 5.0
+        while True:
+            print(datetime.datetime.now())
+            if (loop.time() + 1.0) >= end_time:
+                break
+            yield from asyncio.sleep(1)
+
 .. seealso::
 
    The :ref:`display the current date with call_later()
@@ -139,15 +161,13 @@ Example chaining coroutines::
 
     import asyncio
 
-    @asyncio.coroutine
-    def compute(x, y):
+    async def compute(x, y):
         print("Compute %s + %s ..." % (x, y))
-        yield from asyncio.sleep(1.0)
+        await asyncio.sleep(1.0)
         return x + y
 
-    @asyncio.coroutine
-    def print_sum(x, y):
-        result = yield from compute(x, y)
+    async def print_sum(x, y):
+        result = await compute(x, y)
         print("%s + %s = %s" % (x, y, result))
 
     loop = asyncio.get_event_loop()
@@ -550,12 +570,14 @@ Task functions
 
 .. function:: iscoroutine(obj)
 
-   Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`.
+   Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`,
+   which may be based on a generator or an :keyword:`async def` coroutine.
 
-.. function:: iscoroutinefunction(obj)
+.. function:: iscoroutinefunction(func)
 
-   Return ``True`` if *func* is a decorated :ref:`coroutine function
-   <coroutine>`.
+   Return ``True`` if *func* is determined to be a :ref:`coroutine function
+   <coroutine>`, which may be a decorated generator function or an
+   :keyword:`async def` function.
 
 .. coroutinefunction:: sleep(delay, result=None, \*, loop=None)
 
diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst
index 8c710ef..0653d4e 100644
--- a/Doc/library/collections.abc.rst
+++ b/Doc/library/collections.abc.rst
@@ -154,21 +154,22 @@ ABC                        Inherits from          Abstract Methods        Mixin
 
 .. class:: Awaitable
 
-   ABC for classes that provide ``__await__`` method.  Instances
-   of such classes can be used in ``await`` expression.
+   ABC for :term:`awaitable` objects, which can be used in :keyword:`await`
+   expressions.  Custom implementations must provide the :meth:`__await__`
+   method.
 
-   :term:`coroutine` objects and instances of
-   :class:`~collections.abc.Coroutine` are too instances of this ABC.
+   :term:`Coroutine` objects and instances of the
+   :class:`~collections.abc.Coroutine` ABC are all instances of this ABC.
 
    .. versionadded:: 3.5
 
 .. class:: Coroutine
 
-   ABC for coroutine compatible classes that implement a subset of
-   generator methods defined in :pep:`342`, namely:
-   :meth:`~generator.send`, :meth:`~generator.throw`,
-   :meth:`~generator.close` methods.  :meth:`__await__` must also be
-   implemented.  All :class:`Coroutine` instances are also instances of
+   ABC for coroutine compatible classes.  These implement the
+   following methods, defined in :ref:`coroutine-objects`:
+   :meth:`~coroutine.send`, :meth:`~coroutine.throw`, and
+   :meth:`~coroutine.close`.  Custom implementations must also implement
+   :meth:`__await__`.  All :class:`Coroutine` instances are also instances of
    :class:`Awaitable`.  See also the definition of :term:`coroutine`.
 
    .. versionadded:: 3.5
diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst
index 836c4c1..7a214ed 100644
--- a/Doc/library/dis.rst
+++ b/Doc/library/dis.rst
@@ -516,12 +516,14 @@ the original TOS1.
    Implements ``del TOS1[TOS]``.
 
 
-**Coroutines opcodes**
+**Coroutine opcodes**
 
 .. opcode:: GET_AWAITABLE
 
-   Implements ``TOS = get_awaitable(TOS)``; where ``get_awaitable(o)``
-   returns ``o`` if ``o`` is a coroutine object; or resolved ``o.__await__``.
+   Implements ``TOS = get_awaitable(TOS)``, where ``get_awaitable(o)``
+   returns ``o`` if ``o`` is a coroutine object or a generator object with
+   the CO_ITERABLE_COROUTINE flag, or resolves
+   ``o.__await__``.
 
 
 .. opcode:: GET_AITER
diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst
index 2209f16..1a9d029 100644
--- a/Doc/library/exceptions.rst
+++ b/Doc/library/exceptions.rst
@@ -162,7 +162,8 @@ The following exceptions are the exceptions that are usually raised.
 
 .. exception:: GeneratorExit
 
-   Raised when a :term:`generator`\'s :meth:`close` method is called.  It
+   Raised when a :term:`generator` or :term:`coroutine` is closed;
+   see :meth:`generator.close` and :meth:`coroutine.close`.  It
    directly inherits from :exc:`BaseException` instead of :exc:`Exception` since
    it is technically not an error.
 
@@ -306,7 +307,8 @@ The following exceptions are the exceptions that are usually raised.
    given as an argument when constructing the exception, and defaults
    to :const:`None`.
 
-   When a generator function returns, a new :exc:`StopIteration` instance is
+   When a :term:`generator` or :term:`coroutine` function
+   returns, a new :exc:`StopIteration` instance is
    raised, and the value returned by the function is used as the
    :attr:`value` parameter to the constructor of the exception.
 
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index 144c986..545e674 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -1080,7 +1080,7 @@ always available.
    :func:`types.coroutine` or :func:`asyncio.coroutine` will not be
    intercepted).
 
-   *wrapper* must be either:
+   The *wrapper* argument must be either:
 
    * a callable that accepts one argument (a coroutine object);
    * ``None``, to reset the wrapper.
@@ -1096,7 +1096,8 @@ always available.
             return wrap(coro)
         sys.set_coroutine_wrapper(wrapper)
 
-        async def foo(): pass
+        async def foo():
+            pass
 
         # The following line will fail with a RuntimeError, because
         # `wrapper` creates a `wrap(coro)` coroutine:
diff --git a/Doc/library/tulip_coro.png b/Doc/library/tulip_coro.png
index 65b6951..36ced8d 100644
--- a/Doc/library/tulip_coro.png
+++ b/Doc/library/tulip_coro.png
diff --git a/Doc/library/types.rst b/Doc/library/types.rst
index d7b14e7..ff75de1 100644
--- a/Doc/library/types.rst
+++ b/Doc/library/types.rst
@@ -281,15 +281,23 @@ Additional Utility Classes and Functions
    .. versionadded:: 3.4
 
 
-Coroutines Utility Functions
-----------------------------
+Coroutine Utility Functions
+---------------------------
 
 .. function:: coroutine(gen_func)
 
-   The function transforms a generator function to a :term:`coroutine function`,
-   so that it returns a :term:`coroutine` object.
+   This function transforms a :term:`generator` function into a
+   :term:`coroutine function` which returns a generator-based coroutine.
+   The generator-based coroutine is still a :term:`generator iterator`,
+   but is also considered to be a :term:`coroutine` object and is
+   :term:`awaitable`.  However, it may not necessarily implement
+   the :meth:`__await__` method.
 
-   *gen_func* is modified in-place, hence the function can be used as a
-   decorator.
+   If *gen_func* is a generator function, it will be modified in-place.
+
+   If *gen_func* is not a generator function, it will be wrapped. If it
+   returns an instance of :class:`collections.abc.Generator`, the instance
+   will be wrapped in an *awaitable* proxy object.  All other types
+   of objects will be returned as is.
 
    .. versionadded:: 3.5
diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst
index c73e886..76b3850 100644
--- a/Doc/reference/compound_stmts.rst
+++ b/Doc/reference/compound_stmts.rst
@@ -666,15 +666,9 @@ can be used to create instance variables with different implementation details.
 Coroutines
 ==========
 
-.. index::
-   statement: async def
-   statement: async for
-   statement: async with
-   keyword: async
-   keyword: await
-
 .. versionadded:: 3.5
 
+.. index:: statement: async def
 .. _`async def`:
 
 Coroutine function definition
@@ -683,14 +677,23 @@ Coroutine function definition
 .. productionlist::
    async_funcdef: "async" `funcdef`
 
+.. index::
+   keyword: async
+   keyword: await
+
 Execution of Python coroutines can be suspended and resumed at many points
-(see :term:`coroutine`.)  :keyword:`await` expressions, :keyword:`async for`
-and :keyword:`async with` can only be used in their bodies.
+(see :term:`coroutine`).  In the body of a coroutine, any ``await`` and
+``async`` identifiers become reserved keywords; :keyword:`await` expressions,
+:keyword:`async for` and :keyword:`async with` can only be used in
+coroutine bodies.  However, to simplify the parser, these keywords cannot
+be used on the same line as a function or coroutine (:keyword:`def`
+statement) header.
 
 Functions defined with ``async def`` syntax are always coroutine functions,
 even if they do not contain ``await`` or ``async`` keywords.
 
-It is a :exc:`SyntaxError` to use :keyword:`yield` expressions in coroutines.
+It is a :exc:`SyntaxError` to use :keyword:`yield` expressions in
+``async def`` coroutines.
 
 An example of a coroutine function::
 
@@ -699,6 +702,7 @@ An example of a coroutine function::
         await some_coroutine()
 
 
+.. index:: statement: async for
 .. _`async for`:
 
 The :keyword:`async for` statement
@@ -742,6 +746,7 @@ It is a :exc:`SyntaxError` to use ``async for`` statement outside of an
 :keyword:`async def` function.
 
 
+.. index:: statement: async with
 .. _`async with`:
 
 The :keyword:`async with` statement
diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
index 508b4b5..2ca1194 100644
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -624,7 +624,7 @@ Callable types
       a :dfn:`coroutine function`.  Such a function, when called, returns a
       :term:`coroutine` object.  It may contain :keyword:`await` expressions,
       as well as :keyword:`async with` and :keyword:`async for` statements. See
-      also :ref:`coroutines` section.
+      also the :ref:`coroutine-objects` section.
 
    Built-in functions
       .. index::
@@ -2264,26 +2264,25 @@ special methods (the special method *must* be set on the class
 object itself in order to be consistently invoked by the interpreter).
 
 
-.. _coroutines:
+.. index::
+   single: coroutine
 
 Coroutines
 ==========
 
-.. index::
-   single: coroutine
-
 
 Awaitable Objects
 -----------------
 
-An *awaitable* object can be one of the following:
-
-* A :term:`coroutine` object returned from a :term:`coroutine function`.
+An :term:`awaitable` object generally implements an :meth:`__await__` method.
+:term:`Coroutine` objects returned from :keyword:`async def` functions
+are awaitable.
 
-* A :term:`generator` decorated with :func:`types.coroutine`
-  (or :func:`asyncio.coroutine`) decorator.
+.. note::
 
-* An object that implements an ``__await__`` method.
+   The :term:`generator iterator` objects returned from generators
+   decorated with :func:`types.coroutine` or :func:`asyncio.coroutine`
+   are also awaitable, but they do not implement :meth:`__await__`.
 
 .. method:: object.__await__(self)
 
@@ -2296,6 +2295,58 @@ An *awaitable* object can be one of the following:
 .. seealso:: :pep:`492` for additional information about awaitable objects.
 
 
+.. _coroutine-objects:
+
+Coroutine Objects
+-----------------
+
+:term:`Coroutine` objects are :term:`awaitable` objects.
+A coroutine's execution can be controlled by calling :meth:`__await__` and
+iterating over the result.  When the coroutine has finished executing and
+returns, the iterator raises :exc:`StopIteration`, and the exception's
+:attr:`~StopIteration.value` attribute holds the return value.  If the
+coroutine raises an exception, it is propagated by the iterator.  Coroutines
+should not directly raise unhandled :exc:`StopIteration` exceptions.
+
+Coroutines also have the methods listed below, which are analogous to
+those of generators (see :ref:`generator-methods`).  However, unlike
+generators, coroutines do not directly support iteration.
+
+.. method:: coroutine.send(value)
+
+   Starts or resumes execution of the coroutine.  If *value* is ``None``,
+   this is equivalent to advancing the iterator returned by
+   :meth:`__await__`.  If *value* is not ``None``, this method delegates
+   to the :meth:`~generator.send` method of the iterator that caused
+   the coroutine to suspend.  The result (return value,
+   :exc:`StopIteration`, or other exception) is the same as when
+   iterating over the :meth:`__await__` return value, described above.
+
+.. method:: coroutine.throw(type[, value[, traceback]])
+
+   Raises the specified exception in the coroutine.  This method delegates
+   to the :meth:`~generator.throw` method of the iterator that caused
+   the coroutine to suspend, if it has such a method.  Otherwise,
+   the exception is raised at the suspension point.  The result
+   (return value, :exc:`StopIteration`, or other exception) is the same as
+   when iterating over the :meth:`__await__` return value, described
+   above.  If the exception is not caught in the coroutine, it propagates
+   back to the caller.
+
+.. method:: coroutine.close()
+
+   Causes the coroutine to clean itself up and exit.  If the coroutine
+   is suspended, this method first delegates to the :meth:`~generator.close`
+   method of the iterator that caused the coroutine to suspend, if it
+   has such a method.  Then it raises :exc:`GeneratorExit` at the
+   suspension point, causing the coroutine to immediately clean itself up.
+   Finally, the coroutine is marked as having finished executing, even if
+   it was never started.
+
+   Coroutine objects are automatically closed using the above process when
+   they are about to be destroyed.
+
+
 Asynchronous Iterators
 ----------------------
 
diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst
index 9d3fdf2..d3face7 100644
--- a/Doc/reference/expressions.rst
+++ b/Doc/reference/expressions.rst
@@ -390,6 +390,7 @@ on the right hand side of an assignment statement.
       to sub-generators easy.
 
 .. index:: object: generator
+.. _generator-methods:
 
 Generator-iterator methods
 ^^^^^^^^^^^^^^^^^^^^^^^^^^