gh-114099: Add configure and Makefile targets to support iOS compilation. (GH-115390)

1 files changed, 321 insertions, 0 deletions

diff --git a/iOS/README.rst b/iOS/README.rst
new file mode 100644
index 0000000..1043b5e
--- /dev/null
+++ b/iOS/README.rst
@@ -0,0 +1,321 @@
+====================
+Python on iOS README
+====================
+
+:Authors:
+    Russell Keith-Magee (2023-11)
+
+This document provides a quick overview of some iOS specific features in the
+Python distribution.
+
+These instructions are only needed if you're planning to compile Python for iOS
+yourself. Most users should *not* need to do this. If you're looking to
+experiment with writing an iOS app in Python, tools such as `BeeWare's Briefcase
+<https://briefcase.readthedocs.io>`__ and `Kivy's Buildozer
+<https://buildozer.readthedocs.io>`__ will provide a much more approachable
+user experience.
+
+Compilers for building on iOS
+=============================
+
+Building for iOS requires the use of Apple's Xcode tooling. It is strongly
+recommended that you use the most recent stable release of Xcode. This will
+require the use of the most (or second-most) recently released macOS version,
+as Apple does not maintain Xcode for older macOS versions. The Xcode Command
+Line Tools are not sufficient for iOS development; you need a *full* Xcode
+install.
+
+If you want to run your code on the iOS simulator, you'll also need to install
+an iOS Simulator Platform. You should be prompted to select an iOS Simulator
+Platform when you first run Xcode. Alternatively, you can add an iOS Simulator
+Platform by selecting an open the Platforms tab of the Xcode Settings panel.
+
+iOS specific arguments to configure
+===================================
+
+* ``--enable-framework=DIR``
+
+  This argument specifies the location where the Python.framework will be
+  installed. This argument is required for all iOS builds; a directory *must*
+  be specified.
+
+* ``--with-framework-name=NAME``
+
+  Specify the name for the Python framework; defaults to ``Python``.
+
+Building Python on iOS
+======================
+
+ABIs and Architectures
+----------------------
+
+iOS apps can be deployed on physical devices, and on the iOS simulator. Although
+the API used on these devices is identical, the ABI is different - you need to
+link against different libraries for an iOS device build (``iphoneos``) or an
+iOS simulator build (``iphonesimulator``).
+
+Apple uses the ``XCframework`` format to allow specifying a single dependency
+that supports multiple ABIs. An ``XCframework`` is a wrapper around multiple
+ABI-specific frameworks that share a common API.
+
+iOS can also support different CPU architectures within each ABI. At present,
+there is only a single supported architecture on physical devices - ARM64.
+However, the *simulator* supports 2 architectures - ARM64 (for running on Apple
+Silicon machines), and x86_64 (for running on older Intel-based machines).
+
+To support multiple CPU architectures on a single platform, Apple uses a "fat
+binary" format - a single physical file that contains support for multiple
+architectures. It is possible to compile and use a "thin" single architecture
+version of a binary for testing purposes; however, the "thin" binary will not be
+portable to machines using other architectures.
+
+Building a single-architecture framework
+----------------------------------------
+
+The Python build system will create a ``Python.framework`` that supports a
+*single* ABI with a *single* architecture. Unlike macOS, iOS does not allow a
+framework to contain non-library content, so the iOS build will produce a
+``bin`` and ``lib`` folder in the same output folder as ``Python.framework``.
+The ``lib`` folder will be needed at runtime to support the Python library.
+
+If you want to use Python in a real iOS project, you need to produce multiple
+``Python.framework`` builds, one for each ABI and architecture. iOS builds of
+Python *must* be constructed as framework builds. To support this, you must
+provide the ``--enable-framework`` flag when configuring the build. The build
+also requires the use of cross-compilation. The minimal commands for building
+Python for the ARM64 iOS simulator will look something like::
+
+  $ export PATH="`pwd`/iOS/Resources/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin"
+  $ ./configure \
+        AR=arm64-apple-ios-simulator-ar \
+        CC=arm64-apple-ios-simulator-clang \
+        CPP=arm64-apple-ios-simulator-cpp \
+        CXX=arm64-apple-ios-simulator-clang \
+        --enable-framework=/path/to/install \
+        --host=arm64-apple-ios-simulator \
+        --build=arm64-apple-darwin \
+        --with-build-python=/path/to/python.exe
+  $ make
+  $ make install
+
+In this invocation:
+
+* ``iOS/Resources/bin`` has been added to the path, providing some shims for the
+  compilers and linkers needed by the build. Xcode requires the use of ``xcrun``
+  to invoke compiler tooling. However, if ``xcrun`` is pre-evaluated and the
+  result passed to ``configure``, these results can embed user- and
+  version-specific paths into the sysconfig data, which limits the portability
+  of the compiled Python. Alternatively, if ``xcrun`` is used *as* the compiler,
+  it requires that compiler variables like ``CC`` include spaces, which can
+  cause significant problems with many C configuration systems which assume that
+  ``CC`` will be a single executable.
+
+  To work around this problem, the ``iOS/Resources/bin`` folder contains some
+  wrapper scripts that present as simple compilers and linkers, but wrap
+  underlying calls to ``xcrun``. This allows configure to use a ``CC``
+  definition without spaces, and without user- or version-specific paths, while
+  retaining the ability to adapt to the local Xcode install. These scripts are
+  included in the ``bin`` directory of an iOS install.
+
+  These scripts will, by default, use the currently active Xcode installation.
+  If you want to use a different Xcode installation, you can use
+  ``xcode-select`` to set a new default Xcode globally, or you can use the
+  ``DEVELOPER_DIR`` environment variable to specify an Xcode install. The
+  scripts will use the default ``iphoneos``/``iphonesimulator`` SDK version for
+  the select Xcode install; if you want to use a different SDK, you can set the
+  ``IOS_SDK_VERSION`` environment variable. (e.g, setting
+  ``IOS_SDK_VERSION=17.1`` would cause the scripts to use the ``iphoneos17.1``
+  and ``iphonesimulator17.1`` SDKs, regardless of the Xcode default.)
+
+  The path has also been cleared of any user customizations. A common source of
+  bugs is for tools like Homebrew to accidentally leak macOS binaries into an iOS
+  build. Resetting the path to a known "bare bones" value is the easiest way to
+  avoid these problems.
+
+* ``/path/to/install`` is the location where the final ``Python.framework`` will
+  be output.
+
+* ``--host`` is the architecture and ABI that you want to build, in GNU compiler
+  triple format. This will be one of:
+
+  - ``arm64-apple-ios`` for ARM64 iOS devices.
+  - ``arm64-apple-ios-simulator`` for the iOS simulator running on Apple
+    Silicon devices.
+  - ``x86_64-apple-ios-simulator`` for the iOS simulator running on Intel
+    devices.
+
+* ``--build`` is the GNU compiler triple for the machine that will be running
+  the compiler. This is one of:
+
+  - ``arm64-apple-darwin`` for Apple Silicon devices.
+  - ``x86_64-apple-darwin`` for Intel devices.
+
+* ``/path/to/python.exe`` is the path to a Python binary on the machine that
+  will be running the compiler. This is needed because the Python compilation
+  process involves running some Python code. On a normal desktop build of
+  Python, you can compile a python interpreter and then use that interpreter to
+  run Python code. However, the binaries produced for iOS won't run on macOS, so
+  you need to provide an external Python interpreter. This interpreter must be
+  the same version as the Python that is being compiled. To be completely safe,
+  this should be the *exact* same commit hash. However, the longer a Python
+  release has been stable, the more likely it is that this constraint can be
+  relaxed - the same micro version will often be sufficient.
+
+For a full CPython build, you also need to specify the paths to iOS builds of
+the binary libraries that CPython depends on (XZ, BZip2, LibFFI and OpenSSL).
+This can be done by defining the ``LIBLZMA_CFLAGS``, ``LIBLZMA_LIBS``,
+``BZIP2_CFLAGS``, ``BZIP2_LIBS``, ``LIBFFI_CFLAGS``, and ``LIBFFI_LIBS``
+environment variables, and the ``--with-openssl`` configure option. Versions of
+these libraries pre-compiled for iOS can be found in `this repository
+<https://github.com/beeware/cpython-apple-source-deps/releases>`__.
+
+By default, Python will be compiled with an iOS deployment target (i.e., the
+minimum supported iOS version) of 12.0. To specify a different deployment
+target, provide the version number as part of the ``--host`` argument - for
+example, ``--host=arm64-apple-ios15.4-simulator`` would compile an ARM64
+simulator build with a deployment target of 15.4.
+
+Merge thin frameworks into fat frameworks
+-----------------------------------------
+
+Once you've built a ``Python.framework`` for each ABI and and architecture, you
+must produce a "fat" framework for each ABI that contains all the architectures
+for that ABI.
+
+The ``iphoneos`` build only needs to support a single architecture, so it can be
+used without modification.
+
+If you only want to support a single simulator architecture, (e.g., only support
+ARM64 simulators), you can use a single architecture ``Python.framework`` build.
+However, if you want to create ``Python.xcframework`` that supports *all*
+architectures, you'll need to merge the ``iphonesimulator`` builds for ARM64 and
+x86_64 into a single "fat" framework.
+
+The "fat" framework can be constructed by performing a directory merge of the
+content of the two "thin" ``Python.framework`` directories, plus the ``bin`` and
+``lib`` folders for each thin framework. When performing this merge:
+
+* The pure Python standard library content is identical for each architecture,
+  except for a handful of platform-specific files (such as the ``sysconfig``
+  module). Ensure that the "fat" framework has the union of all standard library
+  files.
+
+* Any binary files in the standard library, plus the main
+  ``libPython3.X.dylib``, can be merged using the ``lipo`` tool, provide by
+  Xcode::
+
+    $ lipo -create -output module.dylib path/to/x86_64/module.dylib path/to/arm64/module.dylib
+
+* The header files will be indentical on both architectures, except for
+  ``pyconfig.h``. Copy all the headers from one platform (say, arm64), rename
+  ``pyconfig.h`` to ``pyconfig-arm64.h``, and copy the ``pyconfig.h`` for the
+  other architecture into the merged header folder as ``pyconfig-x86_64.h``.
+  Then copy the ``iOS/Resources/pyconfig.h`` file from the CPython sources into
+  the merged headers folder. This will allow the two Python architectures to
+  share a common ``pyconfig.h`` header file.
+
+At this point, you should have 2 Python.framework folders - one for ``iphoneos``,
+and one for ``iphonesimulator`` that is a merge of x86+64 and ARM64 content.
+
+Merge frameworks into an XCframework
+------------------------------------
+
+Now that we have 2 (potentially fat) ABI-specific frameworks, we can merge those
+frameworks into a single ``XCframework``.
+
+The initial skeleton of an ``XCframework`` is built using::
+
+    xcodebuild -create-xcframework -output Python.xcframework -framework path/to/iphoneos/Python.framework -framework path/to/iphonesimulator/Python.framework
+
+Then, copy the ``bin`` and ``lib`` folders into the architecture-specific slices of
+the XCframework::
+
+    cp path/to/iphoneos/bin Python.xcframework/ios-arm64
+    cp path/to/iphoneos/lib Python.xcframework/ios-arm64
+
+    cp path/to/iphonesimulator/bin Python.xcframework/ios-arm64_x86-64-simulator
+    cp path/to/iphonesimulator/lib Python.xcframework/ios-arm64_x86-64-simulator
+
+Note that the name of the architecture-specific slice for the simulator will
+depend on the CPU architecture that you build.
+
+Then, add symbolic links to "common" platform names for each slice::
+
+    ln -si ios-arm64 Python.xcframework/iphoneos
+    ln -si ios-arm64_x86-64-simulator Python.xcframework/iphonesimulator
+
+You now have a Python.xcframework that can be used in a project.
+
+Testing Python on iOS
+=====================
+
+The ``iOS/testbed`` folder that contains an Xcode project that is able to run
+the iOS test suite. This project converts the Python test suite into a single
+test case in Xcode's XCTest framework. The single XCTest passes if the test
+suite passes.
+
+To run the test suite, configure a Python build for an iOS simulator (i.e.,
+``--host=arm64-apple-ios-simulator`` or ``--host=x86_64-apple-ios-simulator``
+), setting the framework location to the testbed project::
+
+    --enable-framework="./iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator"
+
+Then run ``make all install testiOS``. This will build an iOS framework for your
+chosen architecture, install the Python iOS framework into the testbed project,
+and run the test suite on an "iPhone SE (3rd generation)" simulator.
+
+While the test suite is running, Xcode does not display any console output.
+After showing some Xcode build commands, the console output will print ``Testing
+started``, and then appear to stop. It will remain in this state until the test
+suite completes. On a 2022 M1 MacBook Pro, the test suite takes approximately 12
+minutes to run; a couple of extra minutes is required to boot and prepare the
+iOS simulator.
+
+On success, the test suite will exit and report successful completion of the
+test suite. No output of the Python test suite will be displayed.
+
+On failure, the output of the Python test suite *will* be displayed. This will
+show the details of the tests that failed.
+
+Debugging test failures
+-----------------------
+
+The easiest way to diagnose a single test failure is to open the testbed project
+in Xcode and run the tests from there using the "Product > Test" menu item.
+
+Running specific tests
+^^^^^^^^^^^^^^^^^^^^^^
+
+As the test suite is being executed on an iOS simulator, it is not possible to
+pass in command line arguments to configure test suite operation. To work around
+this limitation, the arguments that would normally be passed as command line
+arguments are configured as a static string at the start of the XCTest method
+``- (void)testPython`` in ``iOSTestbedTests.m``. To pass an argument to the test
+suite, add a a string to the ``argv`` defintion. These arguments will be passed
+to the test suite as if they had been passed to ``python -m test`` at the
+command line.
+
+Disabling automated breakpoints
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+By default, Xcode will inserts an automatic breakpoint whenever a signal is
+raised. The Python test suite raises many of these signals as part of normal
+operation; unless you are trying to diagnose an issue with signals, the
+automatic breakpoints can be inconvenient. However, they can be disabled by
+creating a symbolic breakpoint that is triggered at the start of the test run.
+
+Select "Debug > Breakpoints > Create Symbolic Breakpoint" from the Xcode menu, and
+populate the new brewpoint with the following details:
+
+* **Name**: IgnoreSignals
+* **Symbol**: UIApplicationMain
+* **Action**: Add debugger commands for:
+  - ``process handle SIGINT -n true -p true -s false``
+  - ``process handle SIGUSR1 -n true -p true -s false``
+  - ``process handle SIGUSR2 -n true -p true -s false``
+  - ``process handle SIGXFSZ -n true -p true -s false``
+* Check the "Automatically continue after evaluating" box.
+
+All other details can be left blank. When the process executes the
+``UIApplicationMain`` entry point, the breakpoint will trigger, run the debugger
+commands to disable the automatic breakpoints, and automatically resume.