Merged revisions 88004,88006,88235 via svnmerge from

svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r88004 | ned.deily | 2011-01-14 20:44:12 -0800 (Fri, 14 Jan 2011) | 4 lines

  #10907: Update OS X installer build README to better reflect
          current build practices.
........
  r88006 | ned.deily | 2011-01-14 21:29:12 -0800 (Fri, 14 Jan 2011) | 6 lines

  #10843: Update third-party library versions used in OS X 32-bit
          installer builds: bzip2 1.0.6, readline 6.1.2, SQLite 3.7.4
          (with FTS3/FTS4 and RTREE enabled), and ncursesw 5.5 (wide-char
          support enabled).
........
  r88235 | ned.deily | 2011-01-29 10:56:28 -0800 (Sat, 29 Jan 2011) | 5 lines

  Issue #11054: Allow Mac OS X installer builds to again work on 10.5 with
  the system-provided Python.  Also, properly guard a new Python 3 only
  installer build step so that build-installer.py can stay compatible
  with the 2.7 version.  (with release manager approval for 3.2rc2)
........

2 files changed, 212 insertions, 95 deletions

diff --git a/Mac/BuildScript/README.txt b/Mac/BuildScript/README.txt
index 0f1aeb5..de2f5cb 100644
--- a/Mac/BuildScript/README.txt
+++ b/Mac/BuildScript/README.txt
@@ -1,78 +1,147 @@
-Building a MacPython distribution
-=================================
+Building a Python Mac OS X distribution
+=======================================
 
-The ``build-install.py`` script creates MacPython distributions, including
-sleepycat db4, sqlite3 and readline support.  It builds a complete 
+The ``build-install.py`` script creates Python distributions, including
+certain third-party libraries as necessary.  It builds a complete 
 framework-based Python out-of-tree, installs it in a funny place with 
 $DESTROOT, massages that installation to remove .pyc files and such, creates 
 an Installer package from the installation plus other files in ``resources`` 
 and ``scripts`` and placed that on a ``.dmg`` disk image.
 
-Prerequisites
--------------
+As of Python 2.7.x and 3.2, PSF practice is to build two installer variants
+for each release:
 
-* A MacOS X 10.4 (or later)
+1.  32-bit-only, i386 and PPC universal, capable on running on all machines
+    supported by Mac OS X 10.3.9 through (at least) 10.6::
 
-* XCode 2.2 (or later), with the universal SDK
+        python build-installer.py \
+            --sdk-path=/Developer/SDKs/MacOSX10.4u.sdk \
+            --universal-archs=32-bit \
+            --dep-target=10.3 
+            # These are the current default options
 
-* No Fink (in ``/sw``) or DarwinPorts (in ``/opt/local``), those could
+    - builds the following third-party libraries
+
+        * Bzip2
+        * Zlib 1.2.3
+        * GNU Readline (GPL)
+        * SQLite 3
+        * NCurses
+        * Oracle Sleepycat DB 4.8 (Python 2.x only)
+
+    - requires ActiveState ``Tcl/Tk 8.4`` (currently 8.4.19) to be installed for building
+
+    - current target build environment:
+        
+        * Mac OS X 10.5.8 PPC or Intel
+        * Xcode 3.1.4 (or later)
+        * ``MacOSX10.4u`` SDK (later SDKs do not support PPC G3 processors)
+        * ``MACOSX_DEPLOYMENT_TARGET=10.3``
+        * Apple ``gcc-4.0``
+        * Python 2.n (n >= 4) for documentation build with Sphinx
+
+    - alternate build environments:
+
+        * Mac OS X 10.4.11 with Xcode 2.5
+        * Mac OS X 10.6.6 with Xcode 3.2.5
+            - need to change ``/System/Library/Frameworks/{Tcl,Tk}.framework/Version/Current`` to ``8.4``
+
+2.  64-bit / 32-bit, x86_64 and i386 universal, for OS X 10.6 (and later)::
+
+        python build-installer.py \
+            --sdk-path=/Developer/SDKs/MacOSX10.6.sdk \
+            --universal-archs=intel \
+            --dep-target=10.6
+
+    - uses system-supplied versions of third-party libraries
+    
+        * readline module links with Apple BSD editline (libedit)
+        * builds Oracle Sleepycat DB 4.8 (Python 2.x only)
+
+    - requires ActiveState Tcl/Tk 8.5.9 (or later) to be installed for building
+
+    - current target build environment:
+        
+        * Mac OS X 10.6.6 (or later)
+        * Xcode 3.2.5 (or later)
+        * ``MacOSX10.6`` SDK
+        * ``MACOSX_DEPLOYMENT_TARGET=10.6``
+        * Apple ``gcc-4.2``
+        * Python 2.n (n >= 4) for documentation build with Sphinx
+
+    - alternate build environments:
+
+        * none
+
+
+General Prerequisites
+---------------------
+
+* No Fink (in ``/sw``) or MacPorts (in ``/opt/local``) or other local
+  libraries or utilities (in ``/usr/local``) as they could
   interfere with the build.
 
-* The documentation for the release must be available on python.org
+* The documentation for the release is built using Sphinx
   because it is included in the installer.
 
+* It is safest to start each variant build with an empty source directory
+  populated with a fresh copy of the untarred source.
+
 
 The Recipe
 ----------
 
-Here are the steps you need to follow to build a MacPython installer:
+Here are the steps you need to follow to build a Python installer:
 
-*  Run ``./build-installer.py``. Optionally you can pass a number of arguments
-   to specify locations of various files. Please see the top of
+* Run ``build-installer.py``. Optionally you can pass a number of arguments
+  to specify locations of various files. Please see the top of
   ``build-installer.py`` for its usage.
 
-  Running this script takes some time, I will not only build Python itself
+  Running this script takes some time, it will not only build Python itself
   but also some 3th-party libraries that are needed for extensions.
 
 * When done the script will tell you where the DMG image is (by default
   somewhere in ``/tmp/_py``).
 
-Building a 4-way universal installer
-....................................
+Building other universal installers
+...................................
 
 It is also possible to build a 4-way universal installer that runs on 
-OSX Leopard or later::
+OS X Leopard or later::
 
-  $ ./build-installer.py --dep-target=10.5 --universal-archs=all --sdk=/
+  python 2.6 /build-installer.py \
+        --dep-target=10.5
+        --universal-archs=all
+        --sdk-path=/Developer/SDKs/MacOSX10.5.sdk
 
-This requires that the deployment target is 10.5 (or later), and hence
-also that your building on at least OSX 10.5.
-
-Testing
--------
-
-The resulting binaries should work on MacOSX 10.3.9 or later. I usually run
-the installer on a 10.3.9, a 10.4.x PPC and a 10.4.x Intel system and then
-run the testsuite to make sure.
+This requires that the deployment target is 10.5, and hence
+also that you are building on at least OS X 10.5.  4-way includes
+``i386``, ``x86_64``, ``ppc``, and ``ppc64`` (G5).  ``ppc64`` executable
+variants can only be run on G5 machines running 10.5.  Note that,
+while OS X 10.6 is only supported on Intel-based machines, it is possible
+to run ``ppc`` (32-bit) executables unmodified thanks to the Rosetta ppc
+emulation in OS X 10.5 and 10.6.
 
+Other ``--universal-archs`` options are ``64-bit`` (``x86_64``, ``ppc64``),
+and ``3-way`` (``ppc``, ``i386``, ``x86_64``).  None of these options
+are regularly exercised; use at your own risk.
 
-Announcements
--------------
 
-(This is mostly of historic interest)
+Testing
+-------
 
-When all is done, announcements can be posted to at least the following
-places:
--   pythonmac-sig@python.org
--   python-dev@python.org
--   python-announce@python.org
--   archivist@info-mac.org
--   adcnews@apple.com
--   news@macnn.com
--   http://www.macupdate.com
--   http://guide.apple.com/usindex.lasso
--   http://www.apple.com/downloads/macosx/submit
--   http://www.versiontracker.com/ (userid Jack.Jansen@oratrix.com)
--   http://www.macshareware.net (userid jackjansen)
+Ideally, the resulting binaries should be installed and the test suite run
+on all supported OS X releases and architectures.  As a practical matter,
+that is generally not possible.  At a minimum, variant 1 should be run on
+at least one Intel, one PPC G4, and one PPC G3 system and one each of 
+OS X 10.6, 10.5, 10.4, and 10.3.9.  Not all tests run on 10.3.9.
+Variant 2 should be run on 10.6 in both 32-bit and 64-bit modes.::
+
+    arch -i386 /usr/local/bin/pythonn.n -m test.regrtest -w -u all 
+    arch -X86_64 /usr/local/bin/pythonn.n -m test.regrtest -w -u all
+    
+Certain tests will be skipped and some cause the interpreter to fail
+which will likely generate ``Python quit unexpectedly`` alert messages
+to be generated at several points during a test run.  These can
+be ignored.
 
-Also, check out Stephan Deibels http://pythonology.org/market contact list
diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py
index b18465a..f0b3f3f 100755
--- a/Mac/BuildScript/build-installer.py
+++ b/Mac/BuildScript/build-installer.py
@@ -1,12 +1,14 @@
-#!/usr/bin/env python
+#!/usr/bin/python
 """
-This script is used to build the "official unofficial" universal build on
-Mac OS X. It requires Mac OS X 10.4, Xcode 2.2 and the 10.4u SDK to do its
-work.  64-bit or four-way universal builds require at least OS X 10.5 and
-the 10.5 SDK.
+This script is used to build "official" universal installers on Mac OS X.
+It requires at least Mac OS X 10.4, Xcode 2.2 and the 10.4u SDK for
+32-bit builds.  64-bit or four-way universal builds require at least
+OS X 10.5 and the 10.5 SDK.
 
-Please ensure that this script keeps working with Python 2.3, to avoid
-bootstrap issues (/usr/bin/python is Python 2.3 on OSX 10.4)
+Please ensure that this script keeps working with Python 2.5, to avoid
+bootstrap issues (/usr/bin/python is Python 2.5 on OSX 10.5).  Sphinx,
+which is used to build the documentation, currently requires at least
+Python 2.4.
 
 Usage: see USAGE variable in the script.
 """
@@ -40,16 +42,19 @@ def grepValue(fn, variable):
         if ln.startswith(variable):
             value = ln[len(variable):].strip()
             return value[1:-1]
+    raise RuntimeError, "Cannot find variable %s" % variable[:-1]
 
 def getVersion():
     return grepValue(os.path.join(SRCDIR, 'configure'), 'PACKAGE_VERSION')
 
+def getVersionTuple():
+    return tuple([int(n) for n in getVersion().split('.')])
+
 def getFullVersion():
     fn = os.path.join(SRCDIR, 'Include', 'patchlevel.h')
     for ln in open(fn):
         if 'PY_VERSION' in ln:
             return ln.split()[-1][1:-1]
-
     raise RuntimeError, "Cannot find full version??"
 
 # The directory we'll use to create the build (will be erased and recreated)
@@ -114,6 +119,8 @@ target_cc_map = {
 
 CC = target_cc_map[DEPTARGET]
 
+PYTHON_3 = getVersionTuple() >= (3, 0)
+
 USAGE = textwrap.dedent("""\
     Usage: build_python [options]
 
@@ -139,9 +146,9 @@ def library_recipes():
     if DEPTARGET < '10.5':
         result.extend([
           dict(
-              name="Bzip2 1.0.5",
-              url="http://www.bzip.org/1.0.5/bzip2-1.0.5.tar.gz",
-              checksum='3c15a0c8d1d3ee1c46a1634d00617b1a',
+              name="Bzip2 1.0.6",
+              url="http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz",
+              checksum='00b516f4704d4a7cb50a1d97e6e8e15b',
               configure=None,
               install='make install CC=%s PREFIX=%s/usr/local/ CFLAGS="-arch %s -isysroot %s"'%(
                   CC,
@@ -164,29 +171,33 @@ def library_recipes():
           ),
           dict(
               # Note that GNU readline is GPL'd software
-              name="GNU Readline 5.1.4",
-              url="http://ftp.gnu.org/pub/gnu/readline/readline-5.1.tar.gz" ,
-              checksum='7ee5a692db88b30ca48927a13fd60e46',
+              name="GNU Readline 6.1.2",
+              url="http://ftp.gnu.org/pub/gnu/readline/readline-6.1.tar.gz" ,
+              checksum='fc2f7e714fe792db1ce6ddc4c9fb4ef3',
               patchlevel='0',
               patches=[
                   # The readline maintainers don't do actual micro releases, but
                   # just ship a set of patches.
-                  'http://ftp.gnu.org/pub/gnu/readline/readline-5.1-patches/readline51-001',
-                  'http://ftp.gnu.org/pub/gnu/readline/readline-5.1-patches/readline51-002',
-                  'http://ftp.gnu.org/pub/gnu/readline/readline-5.1-patches/readline51-003',
-                  'http://ftp.gnu.org/pub/gnu/readline/readline-5.1-patches/readline51-004',
+                  'http://ftp.gnu.org/pub/gnu/readline/readline-6.1-patches/readline61-001',
+                  'http://ftp.gnu.org/pub/gnu/readline/readline-6.1-patches/readline61-002',
               ]
           ),
           dict(
-              name="SQLite 3.6.11",
-              url="http://www.sqlite.org/sqlite-3.6.11.tar.gz",
-              checksum='7ebb099696ab76cc6ff65dd496d17858',
+              name="SQLite 3.7.4",
+              url="http://www.sqlite.org/sqlite-autoconf-3070400.tar.gz",
+              checksum='8f0c690bfb33c3cbbc2471c3d9ba0158',
+              configure_env=('CFLAGS="-Os'
+                                  ' -DSQLITE_ENABLE_FTS3'
+                                  ' -DSQLITE_ENABLE_FTS3_PARENTHESIS'
+                                  ' -DSQLITE_ENABLE_RTREE'
+                                  ' -DSQLITE_TCL=0'
+                                  '"'),
               configure_pre=[
                   '--enable-threadsafe',
-                  '--enable-tempstore',
                   '--enable-shared=no',
                   '--enable-static=yes',
-                  '--disable-tcl',
+                  '--disable-readline',
+                  '--disable-dependency-tracking',
               ]
           ),
           dict(
@@ -194,6 +205,7 @@ def library_recipes():
               url="http://ftp.gnu.org/pub/gnu/ncurses/ncurses-5.5.tar.gz",
               checksum='e73c1ac10b4bfc46db43b2ddfd6244ef',
               configure_pre=[
+                  "--enable-widec",
                   "--without-cxx",
                   "--without-ada",
                   "--without-progs",
@@ -220,24 +232,26 @@ def library_recipes():
           ),
         ])
 
-    result.extend([
-      dict(
-          name="Sleepycat DB 4.7.25",
-          url="http://download.oracle.com/berkeley-db/db-4.7.25.tar.gz",
-          checksum='ec2b87e833779681a0c3a814aa71359e',
-          buildDir="build_unix",
-          configure="../dist/configure",
-          configure_pre=[
-              '--includedir=/usr/local/include/db4',
-          ]
-      ),
-    ])
+    if not PYTHON_3:
+        result.extend([
+          dict(
+              name="Sleepycat DB 4.7.25",
+              url="http://download.oracle.com/berkeley-db/db-4.7.25.tar.gz",
+              checksum='ec2b87e833779681a0c3a814aa71359e',
+              buildDir="build_unix",
+              configure="../dist/configure",
+              configure_pre=[
+                  '--includedir=/usr/local/include/db4',
+              ]
+          ),
+        ])
 
     return result
 
 
 # Instructions for building packages inside the .mpkg.
 def pkg_recipes():
+    unselected_for_python3 = ('selected', 'unselected')[PYTHON_3]
     result = [
         dict(
             name="PythonFramework",
@@ -308,7 +322,7 @@ def pkg_recipes():
             topdir="/Library/Frameworks/Python.framework",
             source="/empty-dir",
             required=False,
-            selected='selected',
+            selected=unselected_for_python3,
         ),
     ]
 
@@ -326,7 +340,7 @@ def pkg_recipes():
                 topdir="/Library/Frameworks/Python.framework",
                 source="/empty-dir",
                 required=False,
-                selected='selected',
+                selected=unselected_for_python3,
             )
         )
     return result
@@ -393,6 +407,9 @@ def checkEnvironment():
     Check that we're running on a supported system.
     """
 
+    if sys.version_info[0:2] < (2, 4):
+        fatal("This script must be run with Python 2.4 or later")
+
     if platform.system() != 'Darwin':
         fatal("This script should be run on a Mac OS X 10.4 (or later) system")
 
@@ -412,15 +429,16 @@ def checkEnvironment():
     #       to install a newer patch level.
 
     for framework in ['Tcl', 'Tk']:
-        fw = dict(lower=framework.lower(),
-                    upper=framework.upper(),
-                    cap=framework.capitalize())
-        fwpth = "Library/Frameworks/%(cap)s.framework/%(lower)sConfig.sh" % fw
-        sysfw = os.path.join('/System', fwpth)
+        #fw = dict(lower=framework.lower(),
+        #            upper=framework.upper(),
+        #            cap=framework.capitalize())
+        #fwpth = "Library/Frameworks/%(cap)s.framework/%(lower)sConfig.sh" % fw
+        fwpth = 'Library/Frameworks/Tcl.framework/Versions/Current'
+        sysfw = os.path.join(SDKPATH, 'System', fwpth)
         libfw = os.path.join('/', fwpth)
         usrfw = os.path.join(os.getenv('HOME'), fwpth)
-        version = "%(upper)s_VERSION" % fw
-        if getTclTkVersion(libfw, version) != getTclTkVersion(sysfw, version):
+        #version = "%(upper)s_VERSION" % fw
+        if os.readlink(libfw) != os.readlink(sysfw):
             fatal("Version of %s must match %s" % (libfw, sysfw) )
         if os.path.exists(usrfw):
             fatal("Please rename %s to avoid possible dynamic load issues."
@@ -690,6 +708,9 @@ def buildRecipe(recipe, basedir, archList):
         configure_args.insert(0, configure)
         configure_args = [ shellQuote(a) for a in configure_args ]
 
+        if 'configure_env' in recipe:
+            configure_args.insert(0, recipe['configure_env'])
+
         print "Running configure for %s"%(name,)
         runCommand(' '.join(configure_args) + ' 2>&1')
 
@@ -745,9 +766,9 @@ def buildPython():
         shutil.rmtree(buildDir)
     if os.path.exists(rootDir):
         shutil.rmtree(rootDir)
-    os.mkdir(buildDir)
-    os.mkdir(rootDir)
-    os.mkdir(os.path.join(rootDir, 'empty-dir'))
+    os.makedirs(buildDir)
+    os.makedirs(rootDir)
+    os.makedirs(os.path.join(rootDir, 'empty-dir'))
     curdir = os.getcwd()
     os.chdir(buildDir)
 
@@ -767,18 +788,20 @@ def buildPython():
     print "Running configure..."
     runCommand("%s -C --enable-framework --enable-universalsdk=%s "
                "--with-universal-archs=%s "
+               "%s "
                "LDFLAGS='-g -L%s/libraries/usr/local/lib' "
                "OPT='-g -O3 -I%s/libraries/usr/local/include' 2>&1"%(
         shellQuote(os.path.join(SRCDIR, 'configure')), shellQuote(SDKPATH),
         UNIVERSALARCHS,
+        (' ', '--with-computed-gotos ')[PYTHON_3],
         shellQuote(WORKDIR)[1:-1],
         shellQuote(WORKDIR)[1:-1]))
 
     print "Running make"
     runCommand("make")
 
-    print "Running make frameworkinstall"
-    runCommand("make frameworkinstall DESTDIR=%s"%(
+    print "Running make install"
+    runCommand("make install DESTDIR=%s"%(
         shellQuote(rootDir)))
 
     print "Running make frameworkinstallextras"
@@ -817,12 +840,33 @@ def buildPython():
             os.chmod(p, stat.S_IMODE(st.st_mode) | stat.S_IWGRP)
             os.chown(p, -1, gid)
 
+    if PYTHON_3:
+        LDVERSION=None
+        VERSION=None
+        ABIFLAGS=None
+
+        fp = open(os.path.join(buildDir, 'Makefile'), 'r')
+        for ln in fp:
+            if ln.startswith('VERSION='):
+                VERSION=ln.split()[1]
+            if ln.startswith('ABIFLAGS='):
+                ABIFLAGS=ln.split()[1]
+            if ln.startswith('LDVERSION='):
+                LDVERSION=ln.split()[1]
+        fp.close()
+
+        LDVERSION = LDVERSION.replace('$(VERSION)', VERSION)
+        LDVERSION = LDVERSION.replace('$(ABIFLAGS)', ABIFLAGS)
+        config_suffix = '-' + LDVERSION
+    else:
+        config_suffix = ''      # Python 2.x
+
     # We added some directories to the search path during the configure
     # phase. Remove those because those directories won't be there on
     # the end-users system.
     path =os.path.join(rootDir, 'Library', 'Frameworks', 'Python.framework',
                 'Versions', version, 'lib', 'python%s'%(version,),
-                'config', 'Makefile')
+                'config' + config_suffix, 'Makefile')
     fp = open(path, 'r')
     data = fp.read()
     fp.close()
@@ -847,7 +891,11 @@ def buildPython():
 
     os.chdir(curdir)
 
-
+    if PYTHON_3:
+        # Remove the 'Current' link, that way we don't accidently mess
+        # with an already installed version of python 2
+        os.unlink(os.path.join(rootDir, 'Library', 'Frameworks',
+                            'Python.framework', 'Versions', 'Current'))
 
 def patchFile(inPath, outPath):
     data = fileContents(inPath)