closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575)

Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.

3 files changed, 84 insertions, 4 deletions

diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst
index c1af02b..7b3963d 100644
--- a/Doc/library/compileall.rst
+++ b/Doc/library/compileall.rst
@@ -83,6 +83,16 @@ compile Python sources.
    If ``0`` is used, then the result of :func:`os.cpu_count()`
    will be used.
 
+.. cmdoption:: --invalidation-mode [timestamp|checked-hash|unchecked-hash]
+
+   Control how the generated pycs will be invalidated at runtime. The default
+   setting, ``timestamp``, means that ``.pyc`` files with the source timestamp
+   and size embedded will be generated. The ``checked-hash`` and
+   ``unchecked-hash`` values cause hash-based pycs to be generated. Hash-based
+   pycs embed a hash of the source file contents rather than a timestamp. See
+   :ref:`pyc-invalidation` for more information on how Python validates bytecode
+   cache files at runtime.
+
 .. versionchanged:: 3.2
    Added the ``-i``, ``-b`` and ``-h`` options.
 
@@ -91,6 +101,9 @@ compile Python sources.
    was changed to a multilevel value.  ``-b`` will always produce a
    byte-code file ending in ``.pyc``, never ``.pyo``.
 
+.. versionchanged:: 3.7
+   Added the ``--invalidation-mode`` parameter.
+
 
 There is no command-line option to control the optimization level used by the
 :func:`compile` function, because the Python interpreter itself already
@@ -99,7 +112,7 @@ provides the option: :program:`python -O -m compileall`.
 Public functions
 ----------------
 
-.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1)
+.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP)
 
    Recursively descend the directory tree named by *dir*, compiling all :file:`.py`
    files along the way. Return a true value if all the files compiled successfully,
@@ -140,6 +153,10 @@ Public functions
    then sequential compilation will be used as a fallback.  If *workers* is
    lower than ``0``, a :exc:`ValueError` will be raised.
 
+   *invalidation_mode* should be a member of the
+   :class:`py_compile.PycInvalidationMode` enum and controls how the generated
+   pycs are invalidated at runtime.
+
    .. versionchanged:: 3.2
       Added the *legacy* and *optimize* parameter.
 
@@ -156,7 +173,10 @@ Public functions
    .. versionchanged:: 3.6
       Accepts a :term:`path-like object`.
 
-.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1)
+   .. versionchanged:: 3.7
+      The *invalidation_mode* parameter was added.
+
+.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP)
 
    Compile the file with path *fullname*. Return a true value if the file
    compiled successfully, and a false value otherwise.
@@ -184,6 +204,10 @@ Public functions
    *optimize* specifies the optimization level for the compiler.  It is passed to
    the built-in :func:`compile` function.
 
+   *invalidation_mode* should be a member of the
+   :class:`py_compile.PycInvalidationMode` enum and controls how the generated
+   pycs are invalidated at runtime.
+
    .. versionadded:: 3.2
 
    .. versionchanged:: 3.5
@@ -193,7 +217,10 @@ Public functions
       The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
       no matter what the value of *optimize* is.
 
-.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1)
+   .. versionchanged:: 3.7
+      The *invalidation_mode* parameter was added.
+
+.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP)
 
    Byte-compile all the :file:`.py` files found along ``sys.path``. Return a
    true value if all the files compiled successfully, and a false value otherwise.
@@ -213,6 +240,9 @@ Public functions
       The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
       no matter what the value of *optimize* is.
 
+   .. versionchanged:: 3.7
+      The *invalidation_mode* parameter was added.
+
 To force a recompile of all the :file:`.py` files in the :file:`Lib/`
 subdirectory and all its subdirectories::
 
diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
index 3d350e8..3cafb41 100644
--- a/Doc/library/importlib.rst
+++ b/Doc/library/importlib.rst
@@ -67,6 +67,9 @@ generically as an :term:`importer`) to participate in the import process.
     :pep:`489`
         Multi-phase extension module initialization
 
+    :pep:`552`
+        Deterministic pycs
+
     :pep:`3120`
         Using UTF-8 as the Default Source Encoding
 
@@ -1327,6 +1330,14 @@ an :term:`importer`.
    .. versionchanged:: 3.6
       Accepts a :term:`path-like object`.
 
+.. function:: source_hash(source_bytes)
+
+   Return the hash of *source_bytes* as bytes. A hash-based ``.pyc`` file embeds
+   the :func:`source_hash` of the corresponding source file's contents in its
+   header.
+
+   .. versionadded:: 3.7
+
 .. class:: LazyLoader(loader)
 
    A class which postpones the execution of the loader of a module until the
diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst
index 0af8fb1..a4f06de 100644
--- a/Doc/library/py_compile.rst
+++ b/Doc/library/py_compile.rst
@@ -27,7 +27,7 @@ byte-code cache files in the directory containing the source code.
    Exception raised when an error occurs while attempting to compile the file.
 
 
-.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1)
+.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1, invalidation_mode=PycInvalidationMode.TIMESTAMP)
 
    Compile a source file to byte-code and write out the byte-code cache file.
    The source code is loaded from the file named *file*.  The byte-code is
@@ -53,6 +53,10 @@ byte-code cache files in the directory containing the source code.
    :func:`compile` function.  The default of ``-1`` selects the optimization
    level of the current interpreter.
 
+   *invalidation_mode* should be a member of the :class:`PycInvalidationMode`
+   enum and controls how the generated ``.pyc`` files are invalidated at
+   runtime.
+
    .. versionchanged:: 3.2
       Changed default value of *cfile* to be :PEP:`3147`-compliant.  Previous
       default was *file* + ``'c'`` (``'o'`` if optimization was enabled).
@@ -65,6 +69,41 @@ byte-code cache files in the directory containing the source code.
       caveat that :exc:`FileExistsError` is raised if *cfile* is a symlink or
       non-regular file.
 
+   .. versionchanged:: 3.7
+      The *invalidation_mode* parameter was added as specified in :pep:`552`.
+
+
+.. class:: PycInvalidationMode
+
+   A enumeration of possible methods the interpreter can use to determine
+   whether a bytecode file is up to date with a source file. The ``.pyc`` file
+   indicates the desired invalidation mode in its header. See
+   :ref:`pyc-invalidation` for more information on how Python invalidates
+   ``.pyc`` files at runtime.
+
+   .. versionadded:: 3.7
+
+   .. attribute:: TIMESTAMP
+
+      The ``.pyc`` file includes the timestamp and size of the source file,
+      which Python will compare against the metadata of the source file at
+      runtime to determine if the ``.pyc`` file needs to be regenerated.
+
+   .. attribute:: CHECKED_HASH
+
+      The ``.pyc`` file includes a hash of the source file content, which Python
+      will compare against the source at runtime to determine if the ``.pyc``
+      file needs to be regenerated.
+
+   .. attribute:: UNCHECKED_HASH
+
+      Like :attr:`CHECKED_HASH`, the ``.pyc`` file includes a hash of the source
+      file content. However, Python will at runtime assume the ``.pyc`` file is
+      up to date and not validate the ``.pyc`` against the source file at all.
+
+      This option is useful when the ``.pycs`` are kept up to date by some
+      system external to Python like a build system.
+
 
 .. function:: main(args=None)