Issue #23857: Implement PEP 493

Adds a Python-2-only ssl module API and environment variable to
configure the default handling of SSL/TLS certificates for
HTTPS connections.

6 files changed, 156 insertions, 10 deletions

diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst
index c18d2a0..417cfff 100644
--- a/Doc/library/ssl.rst
+++ b/Doc/library/ssl.rst
@@ -280,6 +280,44 @@ purposes.
 
      RC4 was dropped from the default cipher string.
 
+.. function:: _https_verify_certificates(enable=True)
+
+   Specifies whether or not server certificates are verified when creating
+   client HTTPS connections without specifying a particular SSL context.
+
+   Starting with Python 2.7.9, :mod:`httplib` and modules which use it, such as
+   :mod:`urllib2` and :mod:`xmlrpclib`, default to verifying remote server
+   certificates received when establishing client HTTPS connections. This
+   default verification checks that the certificate is signed by a Certificate
+   Authority in the system trust store and that the Common Name (or Subject
+   Alternate Name) on the presented certificate matches the requested host.
+
+   Setting *enable* to :const:`True` ensures this default behaviour is in
+   effect.
+
+   Setting *enable* to :const:`False` reverts the default HTTPS certificate
+   handling to that of Python 2.7.8 and earlier, allowing connections to
+   servers using self-signed certificates, servers using certificates signed
+   by a Certicate Authority not present in the system trust store, and servers
+   where the hostname does not match the presented server certificate.
+
+   The leading underscore on this function denotes that it intentionally does
+   not exist in any implementation of Python 3 and may not be present in all
+   Python 2.7 implementations. The portable approach to bypassing certificate
+   checks or the system trust store when necessary is for tools to enable that
+   on a case-by-case basis by explicitly passing in a suitably configured SSL
+   context, rather than reverting the default behaviour of the standard library
+   client modules.
+
+   .. versionadded:: 2.7.12
+
+   .. seealso::
+
+      * `CVE-2014-9365 <http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-9365>`_
+        -- HTTPS man-in-the-middle attack against Python clients using default settings
+      * :pep:`476` -- Enabling certificate verification by default for HTTPS
+      * :pep:`493` -- HTTPS verification migration tools for Python 2.7
+
 
 Random generation
 ^^^^^^^^^^^^^^^^^
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
index 3f314b7..bcfcbd4 100644
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -613,6 +613,17 @@ conflict.
    times.
 
 
+.. envvar:: PYTHONHTTPSVERIFY
+
+   If this environment variable is set specifically to ``0``, then it is
+   equivalent to implicitly calling :func:`ssl._https_verify_certificates` with
+   ``enable=False`` when :mod:`ssl` is first imported.
+
+   Refer to the documentation of :func:`ssl._https_verify_certificates` for
+   details.
+
+   .. versionadded:: 2.7.12
+
 Debug-mode variables
 ~~~~~~~~~~~~~~~~~~~~
 
diff --git a/Doc/whatsnew/2.7.rst b/Doc/whatsnew/2.7.rst
index e2560c7..f4b9148 100644
--- a/Doc/whatsnew/2.7.rst
+++ b/Doc/whatsnew/2.7.rst
@@ -2588,7 +2588,7 @@ PEP 477: Backport ensurepip (PEP 453) to Python 2.7
 
 :pep:`477` approves the inclusion of the :pep:`453` ensurepip module and the
 improved documentation that was enabled by it in the Python 2.7 maintenance
-releases, appearing first in the the Python 2.7.9 release.
+releases, appearing first in the Python 2.7.9 release.
 
 
 Bootstrapping pip By Default
@@ -2649,11 +2649,12 @@ and :ref:`distutils-index`.
 PEP 476: Enabling certificate verification by default for stdlib http clients
 -----------------------------------------------------------------------------
 
-:mod:`httplib` and modules which use it, such as :mod:`urllib2` and
-:mod:`xmlrpclib`, will now verify that the server presents a certificate
-which is signed by a CA in the platform trust store and whose hostname matches
-the hostname being requested by default, significantly improving security for
-many applications.
+:pep:`476` updated :mod:`httplib` and modules which use it, such as
+:mod:`urllib2` and :mod:`xmlrpclib`, to now verify that the server
+presents a certificate which is signed by a Certificate Authority in the
+platform trust store and whose hostname matches the hostname being requested
+by default, significantly improving security for many applications. This
+change was made in the Python 2.7.9 release.
 
 For applications which require the old previous behavior, they can pass an
 alternate context::
@@ -2670,6 +2671,29 @@ alternate context::
 
     urllib2.urlopen("https://invalid-cert", context=context)
 
+
+PEP 493: HTTPS verification migration tools for Python 2.7
+----------------------------------------------------------
+
+:pep:`493` provides additional migration tools to support a more incremental
+infrastructure upgrade process for environments containing applications and
+services relying on the historically permissive processing of server
+certificates when establishing client HTTPS connections.  These additions were
+made in the Python 2.7.12 release.
+
+These tools are intended for use in cases where affected applications and
+services can't be modified to explicitly pass a more permissive SSL context
+when establishing the connection.
+
+For applications and services which can't be modified at all, the new
+``PYTHONHTTPSVERIFY`` environment variable may be set to ``0`` to revert an
+entire Python process back to the default permissive behaviour of Python 2.7.8
+and earlier.
+
+For cases where the connection establishment code can't be modified, but the
+overall application can be, the new :func:`ssl._https_verify_certificates`
+function can be used to adjust the default behaviour at runtime.
+
 .. ======================================================================
 
 .. _acks27:
diff --git a/Lib/ssl.py b/Lib/ssl.py
index 34f7aaa..febc547 100644
--- a/Lib/ssl.py
+++ b/Lib/ssl.py
@@ -482,13 +482,30 @@ def _create_unverified_context(protocol=PROTOCOL_SSLv23, cert_reqs=None,
 
     return context
 
-# Used by http.client if no context is explicitly passed.
-_create_default_https_context = create_default_context
-
-
 # Backwards compatibility alias, even though it's not a public name.
 _create_stdlib_context = _create_unverified_context
 
+# PEP 493: Verify HTTPS by default, but allow envvar to override that
+_https_verify_envvar = 'PYTHONHTTPSVERIFY'
+
+def _get_https_context_factory():
+    if not sys.flags.ignore_environment:
+        config_setting = os.environ.get(_https_verify_envvar)
+        if config_setting == '0':
+            return _create_unverified_context
+    return create_default_context
+
+_create_default_https_context = _get_https_context_factory()
+
+# PEP 493: "private" API to configure HTTPS defaults without monkeypatching
+def _https_verify_certificates(enable=True):
+    """Verify server HTTPS certificates by default?"""
+    global _create_default_https_context
+    if enable:
+        _create_default_https_context = create_default_context
+    else:
+        _create_default_https_context = _create_unverified_context
+
 
 class SSLSocket(socket):
     """This class implements a subtype of socket.socket that wraps
diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
index e9723a7..86ba655 100644
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -4,6 +4,7 @@
 import sys
 import unittest
 from test import test_support as support
+from test.script_helper import assert_python_ok
 import asyncore
 import socket
 import select
@@ -1174,6 +1175,57 @@ class ContextTests(unittest.TestCase):
         self.assertEqual(ctx.verify_mode, ssl.CERT_NONE)
         self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2)
 
+    def test__https_verify_certificates(self):
+        # Unit test to check the contect factory mapping
+        # The factories themselves are tested above
+        # This test will fail by design if run under PYTHONHTTPSVERIFY=0
+        # (as will various test_httplib tests)
+
+        # Uses a fresh SSL module to avoid affecting the real one
+        local_ssl = support.import_fresh_module("ssl")
+        # Certificate verification is enabled by default
+        self.assertIs(local_ssl._create_default_https_context,
+                      local_ssl.create_default_context)
+        # Turn default verification off
+        local_ssl._https_verify_certificates(enable=False)
+        self.assertIs(local_ssl._create_default_https_context,
+                      local_ssl._create_unverified_context)
+        # And back on
+        local_ssl._https_verify_certificates(enable=True)
+        self.assertIs(local_ssl._create_default_https_context,
+                      local_ssl.create_default_context)
+        # The default behaviour is to enable
+        local_ssl._https_verify_certificates(enable=False)
+        local_ssl._https_verify_certificates()
+        self.assertIs(local_ssl._create_default_https_context,
+                      local_ssl.create_default_context)
+
+    def test__https_verify_envvar(self):
+        # Unit test to check the PYTHONHTTPSVERIFY handling
+        # Need to use a subprocess so it can still be run under -E
+        https_is_verified = """import ssl, sys; \
+            status = "Error: _create_default_https_context does not verify certs" \
+                       if ssl._create_default_https_context is \
+                          ssl._create_unverified_context \
+                     else None; \
+            sys.exit(status)"""
+        https_is_not_verified = """import ssl, sys; \
+            status = "Error: _create_default_https_context verifies certs" \
+                       if ssl._create_default_https_context is \
+                          ssl.create_default_context \
+                     else None; \
+            sys.exit(status)"""
+        extra_env = {}
+        # Omitting it leaves verification on
+        assert_python_ok("-c", https_is_verified, **extra_env)
+        # Setting it to zero turns verification off
+        extra_env[ssl._https_verify_envvar] = "0"
+        assert_python_ok("-c", https_is_not_verified, **extra_env)
+        # Any other value should also leave it on
+        for setting in ("", "1", "enabled", "foo"):
+            extra_env[ssl._https_verify_envvar] = setting
+            assert_python_ok("-c", https_is_verified, **extra_env)
+
     def test_check_hostname(self):
         ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
         self.assertFalse(ctx.check_hostname)
diff --git a/Misc/NEWS b/Misc/NEWS
index 18df317..56e2cad 100644
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -58,6 +58,10 @@ Core and Builtins
 Library
 -------
 
+- Issue #23857: Implement PEP 493, adding a Python-2-only ssl module API and
+  environment variable to configure the default handling of SSL/TLS certificates
+  for HTTPS connections.
+
 - Issue #26313: ssl.py _load_windows_store_certs fails if windows cert store
   is empty. Patch by Baji.