Issue #28091: Document PEP 525 & PEP 530.

Patch by Eric Appelt.

4 files changed, 187 insertions, 4 deletions

diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst
index 4fc6af0..4b425a4 100644
--- a/Doc/reference/compound_stmts.rst
+++ b/Doc/reference/compound_stmts.rst
@@ -697,7 +697,7 @@ coroutine bodies.
 Functions defined with ``async def`` syntax are always coroutine functions,
 even if they do not contain ``await`` or ``async`` keywords.
 
-It is a :exc:`SyntaxError` to use :keyword:`yield` expressions in
+It is a :exc:`SyntaxError` to use ``yield from`` expressions in
 ``async def`` coroutines.
 
 An example of a coroutine function::
diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
index 36a9037..82e35e5 100644
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -627,6 +627,25 @@ Callable types
       as well as :keyword:`async with` and :keyword:`async for` statements. See
       also the :ref:`coroutine-objects` section.
 
+   Asynchronous generator functions
+      .. index::
+         single: asynchronous generator; function
+         single: asynchronous generator; asynchronous iterator
+
+      A function or method which is defined using :keyword:`async def` and
+      which uses the :keyword:`yield` statement is called a
+      :dfn:`asynchronous generator function`.  Such a function, when called,
+      returns an asynchronous iterator object which can be used in an
+      :keyword:`async for` statement to execute the body of the function.
+
+      Calling the asynchronous iterator's :meth:`aiterator.__anext__` method
+      will return an :term:`awaitable` which when awaited
+      will execute until it provides a value using the :keyword:`yield`
+      expression.  When the function executes an empty :keyword:`return`
+      statement or falls off the end, a :exc:`StopAsyncIteration` exception
+      is raised and the asynchronous iterator will have reached the end of
+      the set of values to be yielded.
+
    Built-in functions
       .. index::
          object: built-in function
diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst
index 08938b2..39c33bc 100644
--- a/Doc/reference/expressions.rst
+++ b/Doc/reference/expressions.rst
@@ -172,7 +172,7 @@ Common syntax elements for comprehensions are:
 
 .. productionlist::
    comprehension: `expression` `comp_for`
-   comp_for: "for" `target_list` "in" `or_test` [`comp_iter`]
+   comp_for: [ASYNC] "for" `target_list` "in" `or_test` [`comp_iter`]
    comp_iter: `comp_for` | `comp_if`
    comp_if: "if" `expression_nocond` [`comp_iter`]
 
@@ -186,6 +186,17 @@ each time the innermost block is reached.
 Note that the comprehension is executed in a separate scope, so names assigned
 to in the target list don't "leak" into the enclosing scope.
 
+Since Python 3.6, in an :keyword:`async def` function, an :keyword:`async for`
+clause may be used to iterate over a :term:`asynchronous iterator`.
+A comprehension in an :keyword:`async def` function may consist of either a
+:keyword:`for` or :keyword:`async for` clause following the leading
+expression, may contan additonal :keyword:`for` or :keyword:`async for`
+clauses, and may also use :keyword:`await` expressions.
+If a comprehension contains either :keyword:`async for` clauses
+or :keyword:`await` expressions it is called an
+:dfn:`asynchronous comprehension`.  An asynchronous comprehension may
+suspend the execution of the coroutine function in which it appears.
+See also :pep:`530`.
 
 .. _lists:
 
@@ -315,6 +326,14 @@ range(10) for y in bar(x))``.
 The parentheses can be omitted on calls with only one argument.  See section
 :ref:`calls` for details.
 
+Since Python 3.6, if the generator appears in an :keyword:`async def` function,
+then :keyword:`async for` clauses and :keyword:`await` expressions are permitted
+as with an asynchronous comprehension.  If a generator expression
+contains either :keyword:`async for` clauses or :keyword:`await` expressions
+it is called an :dfn:`asynchronous generator expression`.
+An asynchronous generator expression yields a new asynchronous
+generator object, which is an asynchronous iterator
+(see :ref:`async-iterators`).
 
 .. _yieldexpr:
 
@@ -330,9 +349,22 @@ Yield expressions
    yield_atom: "(" `yield_expression` ")"
    yield_expression: "yield" [`expression_list` | "from" `expression`]
 
-The yield expression is only used when defining a :term:`generator` function and
+The yield expression is used when defining a :term:`generator` function
+or an :term:`asynchronous generator` function and
 thus can only be used in the body of a function definition.  Using a yield
-expression in a function's body causes that function to be a generator.
+expression in a function's body causes that function to be a generator,
+and using it in an :keyword:`async def` function's body causes that
+coroutine function to be an asynchronous generator. For example::
+
+    def gen():  # defines a generator function
+        yield 123
+
+    async def agen(): # defines an asynchronous generator function (PEP 525)
+        yield 123
+
+Generator functions are described below, while asynchronous generator
+functions are described separately in section
+:ref:`asynchronous-generator-functions`.
 
 When a generator function is called, it returns an iterator known as a
 generator.  That generator then controls the execution of the generator function.
@@ -496,6 +528,134 @@ generator functions::
 For examples using ``yield from``, see :ref:`pep-380` in "What's New in
 Python."
 
+.. _asynchronous-generator-functions:
+
+Asynchronous generator functions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The presence of a yield expression in a function or method defined using
+:keyword:`async def` further defines the function as a
+:term:`asynchronous generator` function.
+
+When an asynchronous generator function is called, it returns an
+asynchronous iterator known as an asynchronous generator object.
+That object then controls the execution of the generator function.
+An asynchronous generator object is typically used in an
+:keyword:`async for` statement in a coroutine function analogously to
+how a generator object would be used in a :keyword:`for` statement.
+
+Calling one of the asynchronous generator's methods returns an
+:term:`awaitable` object, and the execution starts when this object
+is awaited on. At that time, the execution proceeds to the first yield
+expression, where it is suspended again, returning the value of
+:token:`expression_list` to the awaiting coroutine. As with a generator,
+suspension means that all local state is retained, including the
+current bindings of local variables, the instruction pointer, the internal
+evaluation stack, and the state of any exception handling.  When the execution
+is resumed by awaiting on the next object returned by the asynchronous
+generator's methods, the function can proceed exactly as if the yield
+expression were just another external call. The value of the yield expression
+after resuming depends on the method which resumed the execution.  If
+:meth:`~agen.__anext__` is used then the result is :const:`None`. Otherwise, if
+:meth:`~agen.asend` is used, then the result will be the value passed in to
+that method.
+
+In an asynchronous generator function, yield expressions are allowed anywhere
+in a :keyword:`try` construct. However, if an asynchronous generator is not
+resumed before it is finalized (by reaching a zero reference count or by
+being garbage collected), then a yield expression within a :keyword:`try`
+construct could result in a failure to execute pending :keyword:`finally`
+clauses.  In this case, it is the responsibility of the event loop or
+scheduler running the asynchronous generator to call the asynchronous
+generator-iterator's :meth:`~agen.aclose` method and run the resulting
+coroutine object, thus allowing any pending :keyword:`finally` clauses
+to execute.
+
+To take care of finalization, an event loop should define
+a *finalizer* function which takes an asynchronous generator-iterator
+and presumably calls :meth:`~agen.aclose` and executes the coroutine.
+This  *finalizer* may be registered by calling :func:`sys.set_asyncgen_hooks`.
+When first iterated over, an asynchronous generator-iterator will store the
+registered *finalizer* to be called upon finalization. For a reference example
+of a *finalizer* method see the implementation of
+``asyncio.Loop.shutdown_asyncgens`` in :source:`Lib/asyncio/base_events.py`.
+
+The expression ``yield from <expr>`` is a syntax error when used in an
+asynchronous generator function.
+
+.. index:: object: asynchronous-generator
+.. _asynchronous-generator-methods:
+
+Asynchronous generator-iterator methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This subsection describes the methods of an asynchronous generator iterator,
+which are used to control the execution of a generator function.
+
+
+.. index:: exception: StopAsyncIteration
+
+.. coroutinemethod:: agen.__anext__()
+
+   Returns an awaitable which when run starts to execute the asynchronous
+   generator or resumes it at the last executed yield expression.  When an
+   asynchronous generator function is resumed with a :meth:`~agen.__anext__`
+   method, the current yield expression always evaluates to :const:`None` in
+   the returned awaitable, which when run will continue to the next yield
+   expression. The value of the :token:`expression_list` of the yield
+   expression is the value of the :exc:`StopIteration` exception raised by
+   the completing coroutine.  If the asynchronous generator exits without
+   yielding another value, the awaitable instead raises an
+   :exc:`StopAsyncIteration` exception, signalling that the asynchronous
+   iteration has completed.
+
+   This method is normally called implicitly by a :keyword:`async for` loop.
+
+
+.. coroutinemethod:: agen.asend(value)
+
+   Returns an awaitable which when run resumes the execution of the
+   asynchronous generator. As with the :meth:`~generator.send()` method for a
+   generator, this "sends" a value into the asynchronous generator function,
+   and the *value* argument becomes the result of the current yield expression.
+   The awaitable returned by the :meth:`asend` method will return the next
+   value yielded by the generator as the value of the raised
+   :exc:`StopIteration`, or raises :exc:`StopAsyncIteration` if the
+   asynchronous generator exits without yielding another value.  When
+   :meth:`asend` is called to start the asynchronous
+   generator, it must be called with :const:`None` as the argument,
+   because there is no yield expression that could receive the value.
+
+
+.. coroutinemethod:: agen.athrow(type[, value[, traceback]])
+
+   Returns an awaitable that raises an exception of type ``type`` at the point
+   where the asynchronous generator was paused, and returns the next value
+   yielded by the generator function as the value of the raised
+   :exc:`StopIteration` exception.  If the asynchronous generator exits
+   without yielding another value, an :exc:`StopAsyncIteration` exception is
+   raised by the awaitable.
+   If the generator function does not catch the passed-in exception, or
+   raises a different exception, then when the awaitalbe is run that exception
+   propagates to the caller of the awaitable.
+
+.. index:: exception: GeneratorExit
+
+
+.. coroutinemethod:: agen.aclose()
+
+   Returns an awaitable that when run will throw a :exc:`GeneratorExit` into
+   the asynchronous generator function at the point where it was paused.
+   If the asynchronous generator function then exits gracefully, is already
+   closed, or raises :exc:`GeneratorExit` (by not catching the exception),
+   then the returned awaitable will raise a :exc:`StopIteration` exception.
+   Any further awaitables returned by subsequent calls to the asynchronous
+   generator will raise a :exc:`StopAsyncIteration` exception.  If the
+   asynchronous generator yields a value, a :exc:`RuntimeError` is raised
+   by the awaitable.  If the asynchronous generator raises any other exception,
+   it is propagated to the caller of the awaitable.  If the asynchronous
+   generator has already exited due to an exception or normal exit, then
+   further calls to :meth:`aclose` will return an awaitable that does nothing.
 
 .. _primaries:
 
diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst
index 3dc4418..e152b16 100644
--- a/Doc/reference/simple_stmts.rst
+++ b/Doc/reference/simple_stmts.rst
@@ -492,6 +492,10 @@ generator is done and will cause :exc:`StopIteration` to be raised. The returned
 value (if any) is used as an argument to construct :exc:`StopIteration` and
 becomes the :attr:`StopIteration.value` attribute.
 
+In an asynchronous generator function, an empty :keyword:`return` statement
+indicates that the asynchronous generator is done and will cause
+:exc:`StopAsyncIteration` to be raised.  A non-empty :keyword:`return`
+statement is a syntax error in an asynchronous generator function.
 
 .. _yield: