Forward port macOS installer updates from 3.7/3.8 (GH-21132) (GH-21135)

- fix installer builds when using latest versions of Python 3
- fix installer builds on newer macOS releases with SIP
- Python Launcher app factory defaults now use python3

2 files changed, 123 insertions, 232 deletions

diff --git a/Mac/BuildScript/README.rst b/Mac/BuildScript/README.rst
index f8b25fa..94a6bb2 100644
--- a/Mac/BuildScript/README.rst
+++ b/Mac/BuildScript/README.rst
@@ -7,216 +7,84 @@ framework-based Python out-of-tree, installs it in a funny place with
 $DESTROOT, massages that installation to remove .pyc files and such, creates
 an Installer package from the installation plus other files in ``resources``
 and ``scripts`` and placed that on a ``.dmg`` disk image.
-
-For Python 3.4.0, PSF practice is to build two installer variants
-for each release.
-
-1.  32-bit-only, i386 and PPC universal, capable on running on all machines
-    supported by Mac OS X 10.5 through (at least) 10.9::
-
-        /path/to/bootstrap/python2.7 build-installer.py \
-            --sdk-path=/Developer/SDKs/MacOSX10.5.sdk \
-            --universal-archs=32-bit \
-            --dep-target=10.5
-
-    - builds the following third-party libraries
-
-        * NCurses 5.9 (http://bugs.python.org/issue15037)
-        * SQLite 3.8.11
-        * XZ 5.0.5
-
-    - uses system-supplied versions of third-party libraries
-
-        * readline module links with Apple BSD editline (libedit)
-
-    - requires ActiveState ``Tcl/Tk 8.4`` (currently 8.4.20) to be installed for building
-
-    - recommended build environment:
-
-        * Mac OS X 10.5.8 Intel or PPC
-        * Xcode 3.1.4
-        * ``MacOSX10.5`` SDK
-        * ``MACOSX_DEPLOYMENT_TARGET=10.5``
-        * Apple ``gcc-4.2``
-        * bootstrap non-framework Python 2.7 for documentation build with
-          Sphinx (as of 3.4.1)
-
-    - alternate build environments:
-
-        * Mac OS X 10.6.8 with Xcode 3.2.6
-            - need to change ``/System/Library/Frameworks/{Tcl,Tk}.framework/Version/Current`` to ``8.4``
-        * Note Xcode 4.* does not support building for PPC so cannot be used for this build
-
-2.  64-bit / 32-bit, x86_64 and i386 universal, for OS X 10.6 (and later)::
+The installer package built on the dmg is a macOS bundle format installer
+package. This format is deprecated and is no longer supported by modern
+macOS systems; it is usable on macOS 10.6 and earlier systems.
+To be usable on newer versions of macOS, the bits in the bundle package
+must be assembled in a macOS flat installer package, using current
+versions of the pkgbuild and productbuild utilities. To pass macoS
+Gatekeeper download quarantine, the final package must be signed
+with a valid Apple Developer ID certificate using productsign.
+Starting with macOS 10.15 Catalina, Gatekeeper now also requires
+that installer packages are submitted to and pass Apple's automated
+notarization service using the altool command.  To pass notarization,
+the binaries included in the package must be built with at least
+the macOS 10.9 SDK, mout now be signed with the codesign utility
+and executables must opt in to the hardened run time option with
+any necessary entitlements.  Details of these processes are
+available in the on-line Apple Developer Documentation and man pages.
+
+As of 3.8.0 and 3.7.7, PSF practice is to build one installer variants
+for each release.  Note that as of this writing, no Pythons support
+building on a newer version of macOS that will run on older versions
+by setting MACOSX_DEPLOYMENT_TARGET. This is because the various
+Python C modules do not yet support runtime testing of macOS
+feature availability (for example, by using macOS AvailabilityMacros.h
+and weak-linking).  To build a Python that is to be used on a
+range of macOS releases, always build on the oldest release to be
+supported; the necessary shared libraries for that release will
+normally also be available on later systems, with the occasional
+exception such as the removal of 32-bit libraries in macOS 10.15.
+
+build-installer requires Apple Developer tools, either from the
+Command Line Tools package or from a full Xcode installation.
+You should use the most recent version of either for the operating
+system version in use.  (One notable exception: on macOS 10.6,
+Snow Leopard, use Xcode 3, not Xcode 4 which was released later
+in the 10.6 support cycle.)
+
+1.  64-bit, x86_64, for OS X 10.9 (and later)::
 
         /path/to/bootstrap/python2.7 build-installer.py \
-            --sdk-path=/Developer/SDKs/MacOSX10.6.sdk \
-            --universal-archs=intel \
-            --dep-target=10.6
+            --universal-archs=intel-64 \
+            --dep-target=10.9
 
     - builds the following third-party libraries
 
-        * NCurses 5.9 (http://bugs.python.org/issue15037)
-        * SQLite 3.8.11
-        * XZ 5.0.5
+        * OpenSSL 1.1.1
+        * Tcl/Tk 8.6
+        * NCurses
+        * SQLite
+        * XZ
+        * libffi
 
     - uses system-supplied versions of third-party libraries
 
         * readline module links with Apple BSD editline (libedit)
-
-    - requires ActiveState Tcl/Tk 8.5.15.1 (or later) to be installed for building
-
-    - recommended build environment:
-
-        * Mac OS X 10.6.8 (or later)
-        * Xcode 3.2.6
-        * ``MacOSX10.6`` SDK
-        * ``MACOSX_DEPLOYMENT_TARGET=10.6``
-        * Apple ``gcc-4.2``
-        * bootstrap non-framework Python 2.7 for documentation build with
-          Sphinx (as of 3.4.1)
-
-    - alternate build environments:
-
-        * none.  Xcode 4.x currently supplies two C compilers.
-          ``llvm-gcc-4.2.1`` has been found to miscompile Python 3.3.x and
-          produce a non-functional Python executable.  As it appears to be
-          considered a migration aid by Apple and is not likely to be fixed,
-          its use should be avoided.  The other compiler, ``clang``, has been
-          undergoing rapid development.  While it appears to have become
-          production-ready in the most recent Xcode 5 releases, the versions
-          available on the deprecated Xcode 4.x for 10.6 were early releases
-          and did not receive the level of exposure in production environments
-          that the Xcode 3 gcc-4.2 compiler has had.
-
-
-*   For Python 2.7.x and 3.2.x, the 32-bit-only installer was configured to
-    support Mac OS X 10.3.9 through (at least) 10.6.  Because it is
-    believed that there are few systems still running OS X 10.3 or 10.4
-    and because it has become increasingly difficult to test and
-    support the differences in these earlier systems, as of Python 3.3.0 the PSF
-    32-bit installer no longer supports them.  For reference in building such
-    an installer yourself, the details are::
-
-        /usr/bin/python build-installer.py \
-            --sdk-path=/Developer/SDKs/MacOSX10.4u.sdk \
-            --universal-archs=32-bit \
-            --dep-target=10.3
-
-    - builds the following third-party libraries
-
-        * Bzip2
-        * NCurses
-        * GNU Readline (GPL)
-        * SQLite 3
-        * XZ
-        * Zlib 1.2.3
-        * Oracle Sleepycat DB 4.8 (Python 2.x only)
-
-    - requires ActiveState ``Tcl/Tk 8.4`` (currently 8.4.20) to be installed for building
+        * zlib
+        * bz2
 
     - recommended build environment:
 
-        * Mac OS X 10.5.8 PPC or Intel
-        * Xcode 3.1.4 (or later)
-        * ``MacOSX10.4u`` SDK (later SDKs do not support PPC G3 processors)
-        * ``MACOSX_DEPLOYMENT_TARGET=10.3``
-        * Apple ``gcc-4.0``
-        * system Python 2.5 for documentation build with Sphinx
-
-    - alternate build environments:
-
-        * Mac OS X 10.6.8 with Xcode 3.2.6
-            - need to change ``/System/Library/Frameworks/{Tcl,Tk}.framework/Version/Current`` to ``8.4``
-
+        * Mac OS X 10.9.5
+        * Xcode Command Line Tools 6.2
+        * ``MacOSX10.9`` SDK
+        * ``MACOSX_DEPLOYMENT_TARGET=10.9``
+        * Apple ``clang``
 
 
 General Prerequisites
 ---------------------
 
-* No Fink (in ``/sw``) or MacPorts (in ``/opt/local``) or other local
-  libraries or utilities (in ``/usr/local``) as they could
+* No Fink (in ``/sw``) or MacPorts (in ``/opt/local``) or Homebrew or
+  other local libraries or utilities (in ``/usr/local``) as they could
   interfere with the build.
 
-* The documentation for the release is built using Sphinx
-  because it is included in the installer.  For 2.7.x and 3.x.x up to and
-  including 3.4.0, the ``Doc/Makefile`` uses ``svn`` to download repos of
-  ``Sphinx`` and its dependencies.  Beginning with 3.4.1, the ``Doc/Makefile``
-  assumes there is an externally-provided ``sphinx-build`` and requires at
-  least Python 2.6 to run.  Because of this, it is no longer possible to
-  build a 3.4.1 or later installer on OS X 10.5 using the Apple-supplied
-  Python 2.5.
-
 * It is safest to start each variant build with an empty source directory
-  populated with a fresh copy of the untarred source.
+  populated with a fresh copy of the untarred source or a source repo.
 
 * It is recommended that you remove any existing installed version of the
   Python being built::
 
       sudo rm -rf /Library/Frameworks/Python.framework/Versions/n.n
 
-
-The Recipe
-----------
-
-Here are the steps you need to follow to build a Python installer:
-
-* Run ``build-installer.py``. Optionally you can pass a number of arguments
-  to specify locations of various files. Please see the top of
-  ``build-installer.py`` for its usage.
-
-  Running this script takes some time, it will not only build Python itself
-  but also some 3th-party libraries that are needed for extensions.
-
-* When done the script will tell you where the DMG image is (by default
-  somewhere in ``/tmp/_py``).
-
-Building other universal installers
-...................................
-
-It is also possible to build a 4-way universal installer that runs on
-OS X 10.5 Leopard or later::
-
-    /usr/bin/python /build-installer.py \
-        --dep-target=10.5
-        --universal-archs=all
-        --sdk-path=/Developer/SDKs/MacOSX10.5.sdk
-
-This requires that the deployment target is 10.5, and hence
-also that you are building on at least OS X 10.5.  4-way includes
-``i386``, ``x86_64``, ``ppc``, and ``ppc64`` (G5).  ``ppc64`` executable
-variants can only be run on G5 machines running 10.5.  Note that,
-while OS X 10.6 is only supported on Intel-based machines, it is possible
-to run ``ppc`` (32-bit) executables unmodified thanks to the Rosetta ppc
-emulation in OS X 10.5 and 10.6.  The 4-way installer variant must be
-built with Xcode 3.  It is not regularly built or tested.
-
-Other ``--universal-archs`` options are ``64-bit`` (``x86_64``, ``ppc64``),
-and ``3-way`` (``ppc``, ``i386``, ``x86_64``).  None of these options
-are regularly exercised; use at your own risk.
-
-
-Testing
--------
-
-Ideally, the resulting binaries should be installed and the test suite run
-on all supported OS X releases and architectures.  As a practical matter,
-that is generally not possible.  At a minimum, variant 1 should be run on
-a PPC G4 system with OS X 10.5 and at least one Intel system running OS X
-10.9, 10.8, 10.7, 10.6, or 10.5.  Variant 2 should be run on 10.9, 10.8,
-10.7, and 10.6 systems in both 32-bit and 64-bit modes.::
-
-    /usr/local/bin/pythonn.n -m test -w -u all,-largefile
-    /usr/local/bin/pythonn.n-32 -m test -w -u all
-
-Certain tests will be skipped and some cause the interpreter to fail
-which will likely generate ``Python quit unexpectedly`` alert messages
-to be generated at several points during a test run.  These are normal
-during testing and can be ignored.
-
-It is also recommend to launch IDLE and verify that it is at least
-functional.  Double-click on the IDLE app icon in ``/Applications/Python n.n``.
-It should also be tested from the command line::
-
-    /usr/local/bin/idlen.n
-
diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py
index a2cba32..4fab488 100755
--- a/Mac/BuildScript/build-installer.py
+++ b/Mac/BuildScript/build-installer.py
@@ -2,6 +2,20 @@
 """
 This script is used to build "official" universal installers on macOS.
 
+NEW for 3.9.0 and backports:
+- 2.7 end-of-life issues:
+    - Python 3 installs now update the Current version link
+      in /Library/Frameworks/Python.framework/Versions
+- fully support running under Python 3 as well as 2.7
+- support building on newer macOS systems with SIP
+- fully support building on macOS 10.9+
+- support 10.6+ on best effort
+- support bypassing docs build by supplying a prebuilt
+    docs html tarball in the third-party source library,
+    in the format and filename conventional of those
+    downloadable from python.org:
+        python-3.x.y-docs-html.tar.bz2
+
 NEW for 3.7.0:
 - support Intel 64-bit-only () and 32-bit-only installer builds
 - build and use internal Tcl/Tk 8.6 for 10.6+ builds
@@ -14,28 +28,7 @@ NEW for 3.7.0:
 - use generic "gcc" as compiler (CC env var) rather than "gcc-4.2"
 
 TODO:
-- support SDKROOT and DEVELOPER_DIR xcrun env variables
-- test with 10.5 and 10.4 and determine support status
-
-Please ensure that this script keeps working with Python 2.5, to avoid
-bootstrap issues (/usr/bin/python is Python 2.5 on OSX 10.5).  Doc builds
-use current versions of Sphinx and require a reasonably current python3.
-Sphinx and dependencies are installed into a venv using the python3's pip
-so will fetch them from PyPI if necessary.  Since python3 is now used for
-Sphinx, build-installer.py should also be converted to use python3!
-
-For 3.7.0, when building for a 10.6 or higher deployment target,
-build-installer builds and links with its own copy of Tcl/Tk 8.6.
-Otherwise, it requires an installed third-party version of
-Tcl/Tk 8.4 (for OS X 10.4 and 10.5 deployment targets), Tcl/TK 8.5
-(for 10.6 or later), or Tcl/TK 8.6 (for 10.9 or later)
-installed in /Library/Frameworks.  When installed,
-the Python built by this script will attempt to dynamically link first to
-Tcl and Tk frameworks in /Library/Frameworks if available otherwise fall
-back to the ones in /System/Library/Framework.  For the build, we recommend
-installing the most recent ActiveTcl 8.6. 8.5, or 8.4 version, depending
-on the deployment target.  The actual version linked to depends on the
-path of /Library/Frameworks/{Tcl,Tk}.framework/Versions/Current.
+- test building with SDKROOT and DEVELOPER_DIR xcrun env variables
 
 Usage: see USAGE variable in the script.
 """
@@ -56,14 +49,15 @@ STAT_0o775 = ( stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR
 INCLUDE_TIMESTAMP = 1
 VERBOSE = 1
 
-from plistlib import Plist
+RUNNING_ON_PYTHON2 = sys.version_info.major == 2
 
-try:
+if RUNNING_ON_PYTHON2:
     from plistlib import writePlist
-except ImportError:
-    # We're run using python2.3
-    def writePlist(plist, path):
-        plist.write(path)
+else:
+    from plistlib import dump
+    def writePlist(path, plist):
+        with open(plist, 'wb') as fp:
+            dump(path, fp)
 
 def shellQuote(value):
     """
@@ -1096,7 +1090,7 @@ def buildPythonDocs():
     if not os.path.exists(htmlDir):
         # Create virtual environment for docs builds with blurb and sphinx
         runCommand('make venv')
-        runCommand('venv/bin/python3 -m pip install -U Sphinx==2.2.0')
+        runCommand('venv/bin/python3 -m pip install -U Sphinx==2.3.1')
         runCommand('make html PYTHON=venv/bin/python')
     os.rename(htmlDir, docdir)
     os.chdir(curDir)
@@ -1125,8 +1119,7 @@ def buildPython():
     # Since the extra libs are not in their installed framework location
     # during the build, augment the library path so that the interpreter
     # will find them during its extension import sanity checks.
-    os.environ['DYLD_LIBRARY_PATH'] = os.path.join(WORKDIR,
-                                        'libraries', 'usr', 'local', 'lib')
+
     print("Running configure...")
     runCommand("%s -C --enable-framework --enable-universalsdk=/ "
                "--with-universal-archs=%s "
@@ -1134,12 +1127,15 @@ def buildPython():
                "%s "
                "%s "
                "%s "
+               "%s "
                "LDFLAGS='-g -L%s/libraries/usr/local/lib' "
                "CFLAGS='-g -I%s/libraries/usr/local/include' 2>&1"%(
         shellQuote(os.path.join(SRCDIR, 'configure')),
         UNIVERSALARCHS,
         (' ', '--with-computed-gotos ')[PYTHON_3],
         (' ', '--without-ensurepip ')[PYTHON_3],
+        (' ', "--with-openssl='%s/libraries/usr/local'"%(
+                            shellQuote(WORKDIR)[1:-1],))[PYTHON_3],
         (' ', "--with-tcltk-includes='-I%s/libraries/usr/local/include'"%(
                             shellQuote(WORKDIR)[1:-1],))[internalTk()],
         (' ', "--with-tcltk-libs='-L%s/libraries/usr/local/lib -ltcl8.6 -ltk8.6'"%(
@@ -1147,6 +1143,24 @@ def buildPython():
         shellQuote(WORKDIR)[1:-1],
         shellQuote(WORKDIR)[1:-1]))
 
+    # As of macOS 10.11 with SYSTEM INTEGRITY PROTECTION, DYLD_*
+    # environment variables are no longer automatically inherited
+    # by child processes from their parents. We used to just set
+    # DYLD_LIBRARY_PATH, pointing to the third-party libs,
+    # in build-installer.py's process environment and it was
+    # passed through the make utility into the environment of
+    # setup.py. Instead, we now append DYLD_LIBRARY_PATH to
+    # the existing RUNSHARED configuration value when we call
+    # make for extension module builds.
+
+    runshared_for_make = "".join([
+            " RUNSHARED=",
+            "'",
+            grepValue("Makefile", "RUNSHARED"),
+            ' DYLD_LIBRARY_PATH=',
+            os.path.join(WORKDIR, 'libraries', 'usr', 'local', 'lib'),
+            "'" ])
+
     # Look for environment value BUILDINSTALLER_BUILDPYTHON_MAKE_EXTRAS
     # and, if defined, append its value to the make command.  This allows
     # us to pass in version control tags, like GITTAG, to a build from a
@@ -1161,21 +1175,24 @@ def buildPython():
 
     make_extras = os.getenv("BUILDINSTALLER_BUILDPYTHON_MAKE_EXTRAS")
     if make_extras:
-        make_cmd = "make " + make_extras
+        make_cmd = "make " + make_extras + runshared_for_make
     else:
-        make_cmd = "make"
+        make_cmd = "make" + runshared_for_make
     print("Running " + make_cmd)
     runCommand(make_cmd)
 
-    print("Running make install")
-    runCommand("make install DESTDIR=%s"%(
-        shellQuote(rootDir)))
+    make_cmd = "make install DESTDIR=%s %s"%(
+        shellQuote(rootDir),
+        runshared_for_make)
+    print("Running " + make_cmd)
+    runCommand(make_cmd)
 
-    print("Running make frameworkinstallextras")
-    runCommand("make frameworkinstallextras DESTDIR=%s"%(
-        shellQuote(rootDir)))
+    make_cmd = "make frameworkinstallextras DESTDIR=%s %s"%(
+        shellQuote(rootDir),
+        runshared_for_make)
+    print("Running " + make_cmd)
+    runCommand(make_cmd)
 
-    del os.environ['DYLD_LIBRARY_PATH']
     print("Copying required shared libraries")
     if os.path.exists(os.path.join(WORKDIR, 'libraries', 'Library')):
         build_lib_dir = os.path.join(
@@ -1304,7 +1321,13 @@ def buildPython():
     data = fp.read()
     fp.close()
     # create build_time_vars dict
-    exec(data)
+    if RUNNING_ON_PYTHON2:
+        exec(data)
+    else:
+        g_dict = {}
+        l_dict = {}
+        exec(data, g_dict, l_dict)
+        build_time_vars = l_dict['build_time_vars']
     vars = {}
     for k, v in build_time_vars.items():
         if type(v) == type(''):
@@ -1421,7 +1444,7 @@ def packageFromRecipe(targetDir, recipe):
 
         vers = getFullVersion()
         major, minor = getVersionMajorMinor()
-        pl = Plist(
+        pl = dict(
                 CFBundleGetInfoString="Python.%s %s"%(pkgname, vers,),
                 CFBundleIdentifier='org.python.Python.%s'%(pkgname,),
                 CFBundleName='Python.%s'%(pkgname,),
@@ -1443,7 +1466,7 @@ def packageFromRecipe(targetDir, recipe):
             )
         writePlist(pl, os.path.join(packageContents, 'Info.plist'))
 
-        pl = Plist(
+        pl = dict(
                     IFPkgDescriptionDescription=readme,
                     IFPkgDescriptionTitle=recipe.get('long_name', "Python.%s"%(pkgname,)),
                     IFPkgDescriptionVersion=vers,
@@ -1459,7 +1482,7 @@ def makeMpkgPlist(path):
     vers = getFullVersion()
     major, minor = getVersionMajorMinor()
 
-    pl = Plist(
+    pl = dict(
             CFBundleGetInfoString="Python %s"%(vers,),
             CFBundleIdentifier='org.python.Python',
             CFBundleName='Python',
@@ -1512,7 +1535,7 @@ def buildInstaller():
     os.mkdir(rsrcDir)
 
     makeMpkgPlist(os.path.join(pkgroot, 'Info.plist'))
-    pl = Plist(
+    pl = dict(
                 IFPkgDescriptionTitle="Python",
                 IFPkgDescriptionVersion=getVersion(),
             )