bpo-43987: Add "Annotations Best Practices" HOWTO doc. (#25746)

Add "Annotations Best Practices" HOWTO doc.

7 files changed, 298 insertions, 21 deletions

diff --git a/Doc/glossary.rst b/Doc/glossary.rst
index 0661c82..29c68ed 100644
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -57,6 +57,8 @@ Glossary
 
       See :term:`variable annotation`, :term:`function annotation`, :pep:`484`
       and :pep:`526`, which describe this functionality.
+      Also see :ref:`annotations-howto`
+      for best practices on working with annotations.
 
    argument
       A value passed to a :term:`function` (or :term:`method`) when calling the
@@ -455,6 +457,8 @@ Glossary
 
       See :term:`variable annotation` and :pep:`484`,
       which describe this functionality.
+      Also see :ref:`annotations-howto`
+      for best practices on working with annotations.
 
    __future__
       A pseudo-module which programmers can use to enable new language features
@@ -1211,6 +1215,8 @@ Glossary
 
       See :term:`function annotation`, :pep:`484`
       and :pep:`526`, which describe this functionality.
+      Also see :ref:`annotations-howto`
+      for best practices on working with annotations.
 
    virtual environment
       A cooperatively isolated runtime environment that allows Python users
diff --git a/Doc/howto/annotations.rst b/Doc/howto/annotations.rst
new file mode 100644
index 0000000..3e61103
--- /dev/null
+++ b/Doc/howto/annotations.rst
@@ -0,0 +1,226 @@
+.. _annotations-howto:
+
+**************************
+Annotations Best Practices
+**************************
+
+:author: Larry Hastings
+
+.. topic:: Abstract
+
+  This document is designed to encapsulate the best practices
+  for working with annotations dicts.  If you write Python code
+  that examines ``__annotations__`` on Python objects, we
+  encourage you to follow the guidelines described below.
+
+  The document is organized into four sections:
+  best practices for accessing the annotations of an object
+  in Python versions 3.10 and newer,
+  best practices for accessing the annotations of an object
+  in Python versions 3.9 and older,
+  other best practices
+  for ``__annotations__`` that apply to any Python version,
+  and
+  quirks of ``__annotations__``.
+
+  Note that this document is specifically about working with
+  ``__annotations__``, not uses *for* annotations.
+  If you're looking for information on how to use "type hints"
+  in your code, please see the :mod:`typing` module.
+
+
+Accessing The Annotations Dict Of An Object In Python 3.10 And Newer
+====================================================================
+
+  Python 3.10 adds a new function to the standard library:
+  :func:`inspect.get_annotations`.  In Python versions 3.10
+  and newer, calling this function is the best practice for
+  accessing the annotations dict of any object that supports
+  annotations.  This function can also "un-stringize"
+  stringized annotations for you.
+
+  If for some reason :func:`inspect.get_annotations` isn't
+  viable for your use case, you may access the
+  ``__annotations__`` data member manually.  Best practice
+  for this changed in Python 3.10 as well: as of Python 3.10,
+  ``o.__annotations__`` is guaranteed to *always* work
+  on Python functions, classes, and modules.  If you're
+  certain the object you're examining is one of these three
+  *specific* objects, you may simply use ``o.__annotations__``
+  to get at the object's annotations dict.
+
+  However, other types of callables--for example,
+  callables created by :func:`functools.partial`--may
+  not have an ``__annotations__`` attribute defined.  When
+  accessing the ``__annotations__`` of a possibly unknown
+  object,  best practice in Python versions 3.10 and
+  newer is to call :func:`getattr` with three arguments,
+  for example ``getattr(o, '__annotations__', None)``.
+
+
+Accessing The Annotations Dict Of An Object In Python 3.9 And Older
+===================================================================
+
+  In Python 3.9 and older, accessing the annotations dict
+  of an object is much more complicated than in newer versions.
+  The problem is a design flaw in these older versions of Python,
+  specifically to do with class annotations.
+
+  Best practice for accessing the annotations dict of other
+  objects--functions, other callables, and modules--is the same
+  as best practice for 3.10, assuming you aren't calling
+  :func:`inspect.get_annotations`: you should use three-argument
+  :func:`getattr` to access the object's ``__annotations__``
+  attribute.
+
+  Unfortunately, this isn't best practice for classes.  The problem
+  is that, since ``__annotations__`` is optional on classes, and
+  because classes can inherit attributes from their base classes,
+  accessing the ``__annotations__`` attribute of a class may
+  inadvertently return the annotations dict of a *base class.*
+  As an example::
+
+      class Base:
+          a: int = 3
+          b: str = 'abc'
+
+      class Derived(Base):
+          pass
+
+      print(Derived.__annotations__)
+
+  This will print the annotations dict from ``Base``, not
+  ``Derived``.
+
+  Your code will have to have a separate code path if the object
+  you're examining is a class (``isinstance(o, type)``).
+  In that case, best practice relies on an implementation detail
+  of Python 3.9 and before: if a class has annotations defined,
+  they are stored in the class's ``__dict__`` dictionary.  Since
+  the class may or may not have annotations defined, best practice
+  is to call the ``get`` method on the class dict.
+
+  To put it all together, here is some sample code that safely
+  accesses the ``__annotations__`` attribute on an arbitrary
+  object in Python 3.9 and before::
+
+      if isinstance(o, type):
+          ann = o.__dict__.get('__annotations__', None)
+      else:
+          ann = getattr(o, '__annotations__', None)
+
+  After running this code, ``ann`` should be either a
+  dictionary or ``None``.  You're encouraged to double-check
+  the type of ``ann`` using :func:`isinstance` before further
+  examination.
+
+  Note that some exotic or malformed type objects may not have
+  a ``__dict__`` attribute, so for extra safety you may also wish
+  to use :func:`getattr` to access ``__dict__``.
+
+
+Manually Un-Stringizing Stringized Annotations
+==============================================
+
+  In situations where some annotations may be "stringized",
+  and you wish to evaluate those strings to produce the
+  Python values they represent, it really is best to
+  call :func:`inspect.get_annotations` to do this work
+  for you.
+
+  If you're using Python 3.9 or older, or if for some reason
+  you can't use :func:`inspect.get_annotations`, you'll need
+  to duplicate its logic.  You're encouraged to examine the
+  implementation of :func:`inspect.get_annotations` in the
+  current Python version and follow a similar approach.
+
+  In a nutshell, if you wish to evaluate a stringized annotation
+  on an arbitrary object ``o``:
+
+  * If ``o`` is a module, use ``o.__dict__`` as the
+    ``globals`` when calling :func:`eval`.
+  * If ``o`` is a class, use ``sys.modules[o.__module__].__dict__``
+    as the ``globals``, and ``dict(vars(o))`` as the ``locals``,
+    when calling :func:`eval`.
+  * If ``o`` is a wrapped callable using :func:`functools.update_wrapper`,
+    :func:`functools.wraps`, or :func:`functools.partial`, iteratively
+    unwrap it by accessing either ``o.__wrapped__`` or ``o.func`` as
+    appropriate, until you have found the root unwrapped function.
+  * If ``o`` is a callable (but not a class), use
+    ``o.__globals__`` as the globals when calling :func:`eval`.
+
+  However, not all string values used as annotations can
+  be successfully turned into Python values by :func:`eval`.
+  String values could theoretically contain any valid string,
+  and in practice there are valid use cases for type hints that
+  require annotating with string values that specifically
+  *can't* be evaluated.  For example:
+
+  * :pep:`604` union types using `|`, before support for this
+    was added to Python 3.10.
+  * Definitions that aren't needed at runtime, only imported
+    when :const:`typing.TYPE_CHECKING` is true.
+
+  If :func:`eval` attempts to evaluate such values, it will
+  fail and raise an exception.  So, when designing a library
+  API that works with annotations, it's recommended to only
+  attempt to evaluate string values when explicitly requested
+  to by the caller.
+
+
+Best Practices For ``__annotations__`` In Any Python Version
+============================================================
+
+  * You should avoid assigning to the ``__annotations__`` member
+    of objects directly.  Let Python manage setting ``__annotations__``.
+
+  * If you do assign directly to the ``__annotations__`` member
+    of an object, you should always set it to a ``dict`` object.
+
+  * If you directly access the ``__annotations__`` member
+    of an object, you should ensure that it's a
+    dictionary before attempting to examine its contents.
+
+  * You should avoid modifying ``__annotations__`` dicts.
+
+  * You should avoid deleting the ``__annotations__`` attribute
+    of an object.
+
+
+``__annotations__`` Quirks
+==========================
+
+  In all versions of Python 3, function
+  objects lazy-create an annotations dict if no annotations
+  are defined on that object.  You can delete the ``__annotations__``
+  attribute using ``del fn.__annotations__``, but if you then
+  access ``fn.__annotations__`` the object will create a new empty dict
+  that it will store and return as its annotations.  Deleting the
+  annotations on a function before it has lazily created its annotations
+  dict will throw an ``AttributeError``; using ``del fn.__annotations__``
+  twice in a row is guaranteed to always throw an ``AttributeError``.
+
+  Everything in the above paragraph also applies to class and module
+  objects in Python 3.10 and newer.
+
+  In all versions of Python 3, you can set ``__annotations__``
+  on a function object to ``None``.  However, subsequently
+  accessing the annotations on that object using ``fn.__annotations__``
+  will lazy-create an empty dictionary as per the first paragraph of
+  this section.  This is *not* true of modules and classes, in any Python
+  version; those objects permit setting ``__annotations__`` to any
+  Python value, and will retain whatever value is set.
+
+  If Python stringizes your annotations for you
+  (using ``from __future__ import annotations``), and you
+  specify a string as an annotation, the string will
+  itself be quoted.  In effect the annotation is quoted
+  *twice.*  For example::
+
+       from __future__ import annotations
+       def foo(a: "str"): pass
+
+       print(foo.__annotations__)
+
+  This prints ``{'a': "'str'"}``.  This shouldn't really be considered
+  a "quirk"; it's mentioned here simply because it might be surprising.
diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst
index e0dacd2..eae8f14 100644
--- a/Doc/howto/index.rst
+++ b/Doc/howto/index.rst
@@ -30,4 +30,5 @@ Currently, the HOWTOs are:
    ipaddress.rst
    clinic.rst
    instrumentation.rst
+   annotations.rst
 
diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst
index 56c2f767..b9e8be1 100644
--- a/Doc/library/inspect.rst
+++ b/Doc/library/inspect.rst
@@ -1149,6 +1149,9 @@ Classes and functions
    with the result of calling :func:`eval()` on those values:
 
    * If eval_str is true, :func:`eval()` is called on values of type ``str``.
+     (Note that ``get_annotations`` doesn't catch exceptions; if :func:`eval()`
+     raises an exception, it will unwind the stack past the ``get_annotations``
+     call.)
    * If eval_str is false (the default), values of type ``str`` are unchanged.
 
    ``globals`` and ``locals`` are passed in to :func:`eval()`; see the documentation
@@ -1164,6 +1167,10 @@ Classes and functions
      although if ``obj`` is a wrapped function (using
      ``functools.update_wrapper()``) it is first unwrapped.
 
+   Calling ``get_annotations`` is best practice for accessing the
+   annotations dict of any object.  See :ref:`annotations-howto` for
+   more information on annotations best practices.
+
    .. versionadded:: 3.10
 
 
diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
index 3a812eb..eefdc3d 100644
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -553,7 +553,10 @@ Callable types
       |                         | the dict are the parameter    |           |
       |                         | names, and ``'return'`` for   |           |
       |                         | the return annotation, if     |           |
-      |                         | provided.                     |           |
+      |                         | provided.  For more           |           |
+      |                         | information on working with   |           |
+      |                         | this attribute, see           |           |
+      |                         | :ref:`annotations-howto`.     |           |
       +-------------------------+-------------------------------+-----------+
       | :attr:`__kwdefaults__`  | A dict containing defaults    | Writable  |
       |                         | for keyword-only parameters.  |           |
@@ -748,16 +751,29 @@ Modules
       single: __annotations__ (module attribute)
       pair: module; namespace
 
-   Predefined (writable) attributes: :attr:`__name__` is the module's name;
-   :attr:`__doc__` is the module's documentation string, or ``None`` if
-   unavailable; :attr:`__annotations__` (optional) is a dictionary containing
-   :term:`variable annotations <variable annotation>` collected during module
-   body execution; :attr:`__file__` is the pathname of the file from which the
-   module was loaded, if it was loaded from a file. The :attr:`__file__`
-   attribute may be missing for certain types of modules, such as C modules
-   that are statically linked into the interpreter; for extension modules
-   loaded dynamically from a shared library, it is the pathname of the shared
-   library file.
+   Predefined (writable) attributes:
+
+      :attr:`__name__`
+         The module's name.
+
+      :attr:`__doc__`
+         The module's documentation string, or ``None`` if
+         unavailable.
+
+      :attr:`__file__`
+         The pathname of the file from which the
+         module was loaded, if it was loaded from a file.
+         The :attr:`__file__`
+         attribute may be missing for certain types of modules, such as C modules
+         that are statically linked into the interpreter.  For extension modules
+         loaded dynamically from a shared library, it's the pathname of the shared
+         library file.
+
+      :attr:`__annotations__`
+         A dictionary containing
+         :term:`variable annotations <variable annotation>` collected during
+         module body execution.  For best practices on working
+         with :attr:`__annotations__`, please see :ref:`annotations-howto`.
 
    .. index:: single: __dict__ (module attribute)
 
@@ -821,14 +837,30 @@ Custom classes
       single: __doc__ (class attribute)
       single: __annotations__ (class attribute)
 
-   Special attributes: :attr:`~definition.__name__` is the class name; :attr:`__module__` is
-   the module name in which the class was defined; :attr:`~object.__dict__` is the
-   dictionary containing the class's namespace; :attr:`~class.__bases__` is a
-   tuple containing the base classes, in the order of their occurrence in the
-   base class list; :attr:`__doc__` is the class's documentation string,
-   or ``None`` if undefined; :attr:`__annotations__` (optional) is a dictionary
-   containing :term:`variable annotations <variable annotation>` collected during
-   class body execution.
+   Special attributes:
+
+      :attr:`~definition.__name__`
+         The class name.
+
+      :attr:`__module__`
+         The name of the module in which the class was defined.
+
+      :attr:`~object.__dict__`
+         The dictionary containing the class's namespace.
+
+      :attr:`~class.__bases__`
+         A tuple containing the base classes, in the order of
+         their occurrence in the base class list.
+
+      :attr:`__doc__`
+         The class's documentation string, or ``None`` if undefined.
+
+      :attr:`__annotations__`
+         A dictionary containing
+         :term:`variable annotations <variable annotation>`
+         collected during class body execution.  For best practices on
+         working with :attr:`__annotations__`, please see
+         :ref:`annotations-howto`.
 
 Class instances
    .. index::
diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst
index a59e2e5..679522b 100644
--- a/Doc/whatsnew/3.10.rst
+++ b/Doc/whatsnew/3.10.rst
@@ -807,7 +807,9 @@ Other Language Changes
 
 * Class and module objects now lazy-create empty annotations dicts on demand.
   The annotations dicts are stored in the object’s ``__dict__`` for
-  backwards compatibility.
+  backwards compatibility.  This improves the best practices for working
+  with ``__annotations__``; for more information, please see
+  :ref:`annotations-howto`.
   (Contributed by Larry Hastings in :issue:`43901`.)
 
 New Modules
@@ -996,7 +998,9 @@ defined on an object. It works around the quirks of accessing the annotations
 on various types of objects, and makes very few assumptions about the object
 it examines. :func:`inspect.get_annotations` can also correctly un-stringize
 stringized annotations.  :func:`inspect.get_annotations` is now considered
-best practice for accessing the annotations dict defined on any Python object.
+best practice for accessing the annotations dict defined on any Python object;
+for more information on best practices for working with annotations, please see
+:ref:`annotations-howto`.
 Relatedly, :func:`inspect.signature`,
 :func:`inspect.Signature.from_callable`, and ``inspect.Signature.from_function``
 now call :func:`inspect.get_annotations` to retrieve annotations. This means
diff --git a/Misc/NEWS.d/next/Documentation/2021-04-30-04-27-02.bpo-43987.1DftVa.rst b/Misc/NEWS.d/next/Documentation/2021-04-30-04-27-02.bpo-43987.1DftVa.rst
new file mode 100644
index 0000000..158259e
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2021-04-30-04-27-02.bpo-43987.1DftVa.rst
@@ -0,0 +1 @@
+Add "Annotations Best Practices" document as a new HOWTO.