Issue 24180: Documentation for PEP 492 changes.

11 files changed, 483 insertions, 8 deletions

diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst
index eb63705..ee33a67 100644
--- a/Doc/c-api/typeobj.rst
+++ b/Doc/c-api/typeobj.rst
@@ -220,9 +220,16 @@ type objects) *must* have the :attr:`ob_size` field.
    the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both *NULL*.
 
 
-.. c:member:: void* PyTypeObject.tp_reserved
+.. c:member:: void* PyTypeObject.tp_as_async
 
-   Reserved slot, formerly known as tp_compare.
+   Pointer to an additional structure that contains fields relevant only to
+   objects which implement :term:`awaitable` and :term:`asynchronous iterator`
+   protocols at the C-level.  See :ref:`async-structs` for details.
+
+   .. versionadded:: 3.5
+
+   .. note::
+      Formerly known as tp_compare and tp_reserved.
 
 
 .. c:member:: reprfunc PyTypeObject.tp_repr
@@ -1332,3 +1339,57 @@ Buffer Object Structures
 
    :c:func:`PyBuffer_Release` is the interface for the consumer that
    wraps this function.
+
+
+.. _async-structs:
+
+
+Async Object Structures
+=======================
+
+.. sectionauthor:: Yury Selivanov <yselivanov@sprymix.com>
+
+
+.. c:type:: PyAsyncMethods
+
+   This structure holds pointers to the functions required to implement
+   :term:`awaitable` and :term:`asynchronous iterator` objects.
+
+   Here is the structure definition::
+
+        typedef struct {
+            getawaitablefunc am_await;
+            getaiterfunc am_aiter;
+            aiternextfunc am_anext;
+        } PyAsyncMethods;
+
+.. c:member:: getawaitablefunc PyAsyncMethods.am_await
+
+   The signature of this function is::
+
+      PyObject *am_await(PyObject *self)
+
+   The returned object must be an iterator, i.e. :c:func:`PyIter_Check` must
+   return ``1`` for it.
+
+   This slot may be set to *NULL* if an object is not an :term:`awaitable`.
+
+.. c:member:: getaiterfunc PyAsyncMethods.am_aiter
+
+   The signature of this function is::
+
+      PyObject *am_aiter(PyObject *self)
+
+   Must return an :term:`awaitable` object.  See :meth:`__anext__` for details.
+
+   This slot may be set to *NULL* if an object does not implement
+   asynchronous iteration protocol.
+
+.. c:member:: aiternextfunc PyAsyncMethods.am_anext
+
+   The signature of this function is::
+
+      PyObject *am_anext(PyObject *self)
+
+   Must return an :term:`awaitable` object.  See :meth:`__anext__` for details.
+   This slot may be set to *NULL*.
diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst
index aaa37b8..b7e35f4 100644
--- a/Doc/extending/newtypes.rst
+++ b/Doc/extending/newtypes.rst
@@ -80,7 +80,7 @@ Moving on, we come to the crunch --- the type object. ::
        0,                         /* tp_print */
        0,                         /* tp_getattr */
        0,                         /* tp_setattr */
-       0,                         /* tp_reserved */
+       0,                         /* tp_as_async */
        0,                         /* tp_repr */
        0,                         /* tp_as_number */
        0,                         /* tp_as_sequence */
diff --git a/Doc/glossary.rst b/Doc/glossary.rst
index 57c6ddd..6158a57 100644
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -69,11 +69,42 @@ Glossary
       :ref:`the difference between arguments and parameters
       <faq-argument-vs-parameter>`, and :pep:`362`.
 
+   asynchronous context manager
+      An object which controls the environment seen in an
+      :keyword:`async with` statement by defining :meth:`__aenter__` and
+      :meth:`__aexit__` methods.  Introduced by :pep:`492`.
+
+      .. versionadded:: 3.5
+
+   asynchronous iterable
+      An object, that can be used in an :keyword:`async for` statement.
+      Must return an :term:`awaitable` from its :meth:`__aiter__` method,
+      which should in turn be resolved in an :term:`asynchronous iterator`
+      object.  Introduced by :pep:`492`.
+
+      .. versionadded:: 3.5
+
+   asynchronous iterator
+      An object that implements :meth:`__aiter__` and :meth:`__anext__`
+      methods, that must return :term:`awaitable` objects.
+      :keyword:`async for` resolves awaitable returned from asynchronous
+      iterator's :meth:`__anext__` method until it raises
+      :exc:`StopAsyncIteration` exception.  Introduced by :pep:`492`.
+
+      .. versionadded:: 3.5
+
    attribute
       A value associated with an object which is referenced by name using
       dotted expressions.  For example, if an object *o* has an attribute
       *a* it would be referenced as *o.a*.
 
+   awaitable
+      An object that can be used in an :keyword:`await` expression.  Can be
+      a :term:`coroutine` or an object with an :meth:`__await__` method.
+      See also :pep:`492`.
+
+      .. versionadded:: 3.5
+
    BDFL
       Benevolent Dictator For Life, a.k.a. `Guido van Rossum
       <https://www.python.org/~guido/>`_, Python's creator.
@@ -146,6 +177,23 @@ Glossary
       statement by defining :meth:`__enter__` and :meth:`__exit__` methods.
       See :pep:`343`.
 
+   coroutine function
+      A function which returns a :term:`coroutine` object.  It is defined
+      with an :keyword:`async def` keyword, and may contain :keyword:`await`,
+      :keyword:`async for`, and :keyword:`async with` keywords.  Introduced
+      by :pep:`492`.
+
+      .. versionadded:: 3.5
+
+   coroutine
+      Coroutines is a more generalized form of subroutines. Subroutines are
+      entered at one point and exited at another point.  Coroutines, can be
+      entered, exited, and resumed at many different points.  See
+      :keyword:`await` expressions, and :keyword:`async for` and
+      :keyword:`async with` statements. See also :pep:`492`.
+
+      .. versionadded:: 3.5
+
    CPython
       The canonical implementation of the Python programming language, as
       distributed on `python.org <https://www.python.org>`_.  The term "CPython"
diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst
index 8a71259..28ba430 100644
--- a/Doc/library/collections.abc.rst
+++ b/Doc/library/collections.abc.rst
@@ -33,9 +33,9 @@ The collections module offers the following :term:`ABCs <abstract base class>`:
 
 .. tabularcolumns:: |l|L|L|L|
 
-=========================  =====================  ======================  ====================================================
+========================== ====================== ======================= ====================================================
 ABC                        Inherits from          Abstract Methods        Mixin Methods
-=========================  =====================  ======================  ====================================================
+========================== ====================== ======================= ====================================================
 :class:`Container`                                ``__contains__``
 :class:`Hashable`                                 ``__hash__``
 :class:`Iterable`                                 ``__iter__``
@@ -81,7 +81,11 @@ ABC                        Inherits from          Abstract Methods        Mixin
 :class:`KeysView`          :class:`MappingView`,                          ``__contains__``,
                            :class:`Set`                                   ``__iter__``
 :class:`ValuesView`        :class:`MappingView`                           ``__contains__``, ``__iter__``
-=========================  =====================  ======================  ====================================================
+:class:`Awaitable`                                ``__await__``
+:class:`Coroutine`                                ``send``, ``throw``     ``close``
+:class:`AsyncIterable`                            ``__aiter__``
+:class:`AsyncIterator`     :class:`AsyncIterable` ``__anext__``           ``__aiter__``
+========================== ====================== ======================= ====================================================
 
 
 .. class:: Container
@@ -134,6 +138,41 @@ ABC                        Inherits from          Abstract Methods        Mixin
 
    ABCs for mapping, items, keys, and values :term:`views <view>`.
 
+.. class:: Awaitable
+
+   ABC for classes that provide ``__await__`` method.  Instances
+   of such classes can be used in ``await`` expression.
+
+   :term:`coroutine` objects and instances of
+   :class:`~collections.abc.Coroutine` are too instances of this ABC.
+
+   .. versionadded:: 3.5
+
+.. class:: Coroutine
+
+   ABC for coroutine compatible classes that implement a subset of
+   generator methods defined in :pep:`342`, namely:
+   :meth:`~generator.send`, :meth:`~generator.throw` and
+   :meth:`~generator.close` methods.  All :class:`Coroutine` instances
+   are also instances of :class:`Awaitable`.  See also the definition
+   of :term:`coroutine`.
+
+   .. versionadded:: 3.5
+
+.. class:: AsyncIterable
+
+   ABC for classes that provide ``__aiter__`` method.  See also the
+   definition of :term:`asynchronous iterable`.
+
+   .. versionadded:: 3.5
+
+.. class:: AsyncIterator
+
+   ABC for classes that provide ``__aiter__`` and ``__anext__``
+   methods.  See also the definition of :term:`asynchronous iterator`.
+
+   .. versionadded:: 3.5
+
 
 These ABCs allow us to ask classes or instances if they provide
 particular functionality, for example::
diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst
index e7768be..6162068 100644
--- a/Doc/library/exceptions.rst
+++ b/Doc/library/exceptions.rst
@@ -322,6 +322,14 @@ The following exceptions are the exceptions that are usually raised.
    .. versionchanged:: 3.5
       Introduced the RuntimeError transformation.
 
+.. exception:: StopAsyncIteration
+
+   Must be raised by :meth:`__anext__` method of an
+   :term:`asynchronous iterator` object to stop the iteration.
+
+   .. versionadded:: 3.5
+      See also :pep:`492`.
+
 .. exception:: SyntaxError
 
    Raised when the parser encounters a syntax error.  This may occur in an
diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst
index d3cd1a8..b3b4dd8 100644
--- a/Doc/library/inspect.rst
+++ b/Doc/library/inspect.rst
@@ -266,6 +266,47 @@ attributes:
    Return true if the object is a generator.
 
 
+.. function:: iscoroutinefunction(object)
+
+   Return true if the object is a coroutine function.
+
+   Coroutine functions are defined with an ``async def`` syntax,
+   or are generators decorated with :func:`types.coroutine`
+   or :func:`asyncio.coroutine`.
+
+   The function will return false for plain python generator
+   functions.
+
+   See also :pep:`492`.
+
+   .. versionadded:: 3.5
+
+
+.. function:: iscoroutine(object)
+
+   Return true if the object is a coroutine.
+
+   Coroutines are results of calls of coroutine functions or
+   generator functions decorated with :func:`types.coroutine`
+   or :func:`asyncio.coroutine`.
+
+   The function will return false for plain python generators.
+
+   See also :class:`collections.abc.Coroutine` and :pep:`492`.
+
+   .. versionadded:: 3.5
+
+
+.. function:: isawaitable(object)
+
+   Return true if the object can be used in :keyword:`await`
+   expression.
+
+   See also :class:`collections.abc.Awaitable` and :pep:`492`.
+
+   .. versionadded:: 3.5
+
+
 .. function:: istraceback(object)
 
    Return true if the object is a traceback.
diff --git a/Doc/library/types.rst b/Doc/library/types.rst
index 34fffe6..3fb0c9c 100644
--- a/Doc/library/types.rst
+++ b/Doc/library/types.rst
@@ -271,3 +271,17 @@ Additional Utility Classes and Functions
    attributes on the class with the same name (see Enum for an example).
 
    .. versionadded:: 3.4
+
+
+Coroutines Utility Functions
+----------------------------
+
+.. function:: coroutine(gen_func)
+
+   The function transforms a generator function to a :term:`coroutine function`,
+   so that it returns a :term:`coroutine` object.
+
+   *gen_func* is modified in-place, hence the function can be used as a
+   decorator.
+
+   .. versionadded:: 3.5
diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst
index 5dd17eb..58b6d71 100644
--- a/Doc/reference/compound_stmts.rst
+++ b/Doc/reference/compound_stmts.rst
@@ -51,6 +51,9 @@ Summarizing:
                 : | `with_stmt`
                 : | `funcdef`
                 : | `classdef`
+                : | `async_with_stmt`
+                : | `async_for_stmt`
+                : | `async_funcdef`
    suite: `stmt_list` NEWLINE | NEWLINE INDENT `statement`+ DEDENT
    statement: `stmt_list` NEWLINE | `compound_stmt`
    stmt_list: `simple_stmt` (";" `simple_stmt`)* [";"]
@@ -660,6 +663,112 @@ can be used to create instance variables with different implementation details.
    :pep:`3129` - Class Decorators
 
 
+Coroutines
+==========
+
+.. _`async def`:
+
+Coroutine function definition
+-----------------------------
+
+.. productionlist::
+   async_funcdef: "async" `funcdef`
+
+Execution of Python coroutines can be suspended and resumed at many points
+(see :term:`coroutine`.)  :keyword:`await` expressions, :keyword:`async for`
+and :keyword:`async with` can only be used in their bodies.
+
+Functions defined with ``async def`` syntax are always coroutine functions,
+even if they do not contain ``await`` or ``async`` keywords.
+
+It is a :exc:`SyntaxError` to use :keyword:`yield` expressions in coroutines.
+
+.. versionadded:: 3.5
+
+
+.. _`async for`:
+
+The :keyword:`async for` statement
+----------------------------------
+
+.. productionlist::
+   async_for_stmt: "async" `for_stmt`
+
+An :term:`asynchronous iterable` is able to call asynchronous code in its
+*iter* implementation, and :term:`asynchronous iterator` can call asynchronous
+code in its *next* method.
+
+The ``async for`` statement allows convenient iteration over asynchronous
+iterators.
+
+The following code::
+
+    async for TARGET in ITER:
+        BLOCK
+    else:
+        BLOCK2
+
+Is semantically equivalent to::
+
+    iter = (ITER)
+    iter = await type(iter).__aiter__(iter)
+    running = True
+    while running:
+        try:
+            TARGET = await type(iter).__anext__(iter)
+        except StopAsyncIteration:
+            running = False
+        else:
+            BLOCK
+    else:
+        BLOCK2
+
+See also :meth:`__aiter__` and :meth:`__anext__` for details.
+
+.. versionadded:: 3.5
+
+
+.. _`async with`:
+
+The :keyword:`async with` statement
+-----------------------------------
+
+.. productionlist::
+   async_with_stmt: "async" `with_stmt`
+
+An :term:`asynchronous context manager` is a :term:`context manager` that is
+able to suspend execution in its *enter* and *exit* methods.
+
+The following code::
+
+    async with EXPR as VAR:
+        BLOCK
+
+Is semantically equivalent to::
+
+    mgr = (EXPR)
+    aexit = type(mgr).__aexit__
+    aenter = type(mgr).__aenter__(mgr)
+    exc = True
+
+    VAR = await aenter
+    try:
+        BLOCK
+    except:
+        if not await aexit(mgr, *sys.exc_info()):
+            raise
+    else:
+        await aexit(mgr, None, None, None)
+
+See also :meth:`__aenter__` and :meth:`__aexit__` for details.
+
+.. versionadded:: 3.5
+
+.. seealso::
+
+   :pep:`492` - Coroutines with async and await syntax
+
+
 .. rubric:: Footnotes
 
 .. [#] The exception is propagated to the invocation stack unless
diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
index df8b245..508b4b5 100644
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -616,6 +616,16 @@ Callable types
       exception is raised and the iterator will have reached the end of the set of
       values to be returned.
 
+   Coroutine functions
+      .. index::
+         single: coroutine; function
+
+      A function or method which is defined using :keyword:`async def` is called
+      a :dfn:`coroutine function`.  Such a function, when called, returns a
+      :term:`coroutine` object.  It may contain :keyword:`await` expressions,
+      as well as :keyword:`async with` and :keyword:`async for` statements. See
+      also :ref:`coroutines` section.
+
    Built-in functions
       .. index::
          object: built-in function
@@ -2254,6 +2264,104 @@ special methods (the special method *must* be set on the class
 object itself in order to be consistently invoked by the interpreter).
 
 
+.. _coroutines:
+
+Coroutines
+==========
+
+.. index::
+   single: coroutine
+
+
+Awaitable Objects
+-----------------
+
+An *awaitable* object can be one of the following:
+
+* A :term:`coroutine` object returned from a :term:`coroutine function`.
+
+* A :term:`generator` decorated with :func:`types.coroutine`
+  (or :func:`asyncio.coroutine`) decorator.
+
+* An object that implements an ``__await__`` method.
+
+.. method:: object.__await__(self)
+
+   Must return an :term:`iterator`.  Should be used to implement
+   :term:`awaitable` objects.  For instance, :class:`asyncio.Future` implements
+   this method to be compatible with the :keyword:`await` expression.
+
+.. versionadded:: 3.5
+
+.. seealso:: :pep:`492` for additional information about awaitable objects.
+
+
+Asynchronous Iterators
+----------------------
+
+An *asynchronous iterable* is able to call asynchronous code in its
+``__aiter__`` implementation, and an *asynchronous iterator* can call
+asynchronous code in its ``__anext__`` method.
+
+Asynchronous iterators can be used in a :keyword:`async for` statement.
+
+.. method:: object.__aiter__(self)
+
+   Must return an *awaitable* resulting in an *asynchronous iterator* object.
+
+.. method:: object.__anext__(self)
+
+   Must return an *awaitable* resulting in a next value of the iterator.  Should
+   raise a :exc:`StopAsyncIteration` error when the iteration is over.
+
+An example of an asynchronous iterable object::
+
+    class Reader:
+        async def readline(self):
+            ...
+
+        async def __aiter__(self):
+            return self
+
+        async def __anext__(self):
+            val = await self.readline()
+            if val == b'':
+                raise StopAsyncIteration
+            return val
+
+.. versionadded:: 3.5
+
+
+Asynchronous Context Managers
+-----------------------------
+
+An *asynchronous context manager* is a *context manager* that is able to
+suspend execution in its ``__aenter__`` and ``__aexit__`` methods.
+
+Asynchronous context managers can be used in a :keyword:`async with` statement.
+
+.. method:: object.__aenter__(self)
+
+   This method is semantically similar to the :meth:`__enter__`, with only
+   difference that it must return an *awaitable*.
+
+.. method:: object.__aexit__(self, exc_type, exc_value, traceback)
+
+   This method is semantically similar to the :meth:`__exit__`, with only
+   difference that it must return an *awaitable*.
+
+An example of an asynchronous context manager class::
+
+    class AsyncContextManager:
+        async def __aenter__(self):
+            await log('entering context')
+
+        async def __aexit__(self, exc_type, exc, tb):
+            await log('exiting context')
+
+.. versionadded:: 3.5
+
+
 .. rubric:: Footnotes
 
 .. [#] It *is* possible in some cases to change an object's type, under certain
diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst
index b252e26..b1cf4a3 100644
--- a/Doc/reference/expressions.rst
+++ b/Doc/reference/expressions.rst
@@ -811,6 +811,20 @@ a class instance:
    if that method was called.
 
 
+.. _await:
+
+Await expression
+================
+
+Suspend the execution of :term:`coroutine` on an :term:`awaitable` object.
+Can only be used inside a :term:`coroutine function`.
+
+.. productionlist::
+   await: ["await"] `primary`
+
+.. versionadded:: 3.5
+
+
 .. _power:
 
 The power operator
@@ -820,7 +834,7 @@ The power operator binds more tightly than unary operators on its left; it binds
 less tightly than unary operators on its right.  The syntax is:
 
 .. productionlist::
-   power: `primary` ["**" `u_expr`]
+   power: `await` ["**" `u_expr`]
 
 Thus, in an unparenthesized sequence of power and unary operators, the operators
 are evaluated from right to left (this does not constrain the evaluation order
@@ -1362,6 +1376,8 @@ precedence and have a left-to-right chaining feature as described in the
 +-----------------------------------------------+-------------------------------------+
 | ``**``                                        | Exponentiation [#]_                 |
 +-----------------------------------------------+-------------------------------------+
+| ``await`` ``x``                               | Await expression                    |
++-----------------------------------------------+-------------------------------------+
 | ``x[index]``, ``x[index:index]``,             | Subscription, slicing,              |
 | ``x(arguments...)``, ``x.attribute``          | call, attribute reference           |
 +-----------------------------------------------+-------------------------------------+
diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst
index 3a7b5d6..b9ea627 100644
--- a/Doc/whatsnew/3.5.rst
+++ b/Doc/whatsnew/3.5.rst
@@ -121,6 +121,26 @@ Please read on for a comprehensive list of user-facing changes.
        PEP written by Carl Meyer
 
 
+PEP 492 - Coroutines with async and await syntax
+------------------------------------------------
+
+The PEP added dedicated syntax for declaring :term:`coroutines <coroutine>`,
+:keyword:`await` expressions, new asynchronous :keyword:`async for`
+and :keyword:`async with` statements.
+
+Example::
+
+    async def read_data(db):
+        async with db.transaction():
+            data = await db.fetch('SELECT ...')
+
+PEP written and implemented by Yury Selivanov.
+
+.. seealso::
+
+   :pep:`492` -- Coroutines with async and await syntax
+
+
 PEP 461 - Adding formatting to bytes and bytearray
 --------------------------------------------------
 
@@ -433,6 +453,10 @@ inspect
 * New argument ``follow_wrapped`` for :func:`inspect.signature`.
   (Contributed by Yury Selivanov in :issue:`20691`.)
 
+* New :func:`~inspect.iscoroutine`, :func:`~inspect.iscoroutinefunction`,
+  and :func:`~inspect.isawaitable` functions.  (Contributed by Yury Selivanov
+  in :issue:`24017`.)
+
 ipaddress
 ---------
 
@@ -614,6 +638,12 @@ time
 * The :func:`time.monotonic` function is now always available.  (Contributed by
   Victor Stinner in :issue:`22043`.)
 
+types
+-----
+
+* New :func:`~types.coroutine` function.  (Contributed by Yury Selivanov
+  in :issue:`24017`.)
+
 urllib
 ------
 
@@ -961,4 +991,5 @@ Changes in the C API
   (:issue:`20204`)
 
 * As part of PEP 492 implementation, ``tp_reserved`` slot of
-  ``PyTypeObject`` was replaced with ``tp_as_async`` slot.
+  :c:type:`PyTypeObject` was replaced with a
+  :c:member:`PyTypeObject.tp_as_async` slot.