Third attempt at documenting an installation scheme; this one feels like

it could work--reasonably short and straightforward to document, but
flexible enough that it will be able to handle more than just modules
and extensions in the future.

1 files changed, 301 insertions, 192 deletions

diff --git a/Doc/inst/inst.tex b/Doc/inst/inst.tex
index bfc1bcf..c32fb06 100644
--- a/Doc/inst/inst.tex
+++ b/Doc/inst/inst.tex
@@ -30,12 +30,12 @@
 \newcommand{\option}[1]{\textsf{\small{#1}}}
 \newcommand{\filevar}[1]{{\textsl{\filenq{#1}}}}
 \newcommand{\homefile}[1]{\file{\tilde/#1}}
-\newcommand{\comingsoon}{\emph{Coming soon$\ \ldots$}}
+\newcommand{\comingsoon}{\emph{Coming soon...}}
 
 % And how about these?  Very handy for writing pathnames (tilde for
 % Unix, backslash for DOS/Windows).
 \renewcommand{\tilde}{\raisebox{-0.5ex}{\symbol{126}}}
-\newcommand{\bslash}{\symbol{92}}
+\newcommand{\bslash}{\symbol{92}}      % XXX only works in tt fonts!
 
 
 \begin{document}
@@ -52,8 +52,6 @@
 \section{Introduction}
 \label{sec:intro}
 
-\comingsoon
-
 \subsection{The new way: Distutils}
 \label{sec:new-way}
 
@@ -69,9 +67,10 @@
 
 
 
-\section{Normal Build and Install}
+\section{Standard Build and Install}
 \label{sec:normal-install}
 
+
 % This will cover:
 %   * setup.py install        (the usual thing)
 %   * setup.py build          (if you like doing things one-at-a-time)
@@ -82,229 +81,339 @@
 \comingsoon
 
 
-\section{Custom Extension Building}
-\label{sec:custom-ext}
-
-% This will cover:
-%   * normal extension build -- stress that it doesn't matter, you
-%     do the same thing whether there are extensions or not
-%   * what you might want to customize: compiler and compiler
-%     flags (warn of the dangers); per-file compiler flags
-%     (not handled yet!)
-%   * when things go wrong: I don't know! (and I don't know what
-%     to do, either!)
-\comingsoon
-
 
-\section{Custom Installation (\UNIX)}
-\label{sec:custom-install-unix}
-
-% XXX probably should banish mentions of Windows here to the
-% separate "Non-standard installation (Windows)" section.
-
-A \dfn{custom installation} is where you install modules to a location
-that's not in Python's default module search path.  There are a couple
-of reasons you might want to do this; the most typical is simply that
-you don't have permission to write to the standard Python library
-directory.  Or, even if you do have write permission to the standard
-library, you might wish to install a module distribution into a
-non-standard place for testing or experimentation.  (This is especially
-useful when upgrading an existing module distribution: you might want to
-make sure that your existing scripts continue to work as before, and
-only then install the upgrade ``for real.'')
-
-(XXX terminology: I keep saying ``standard Python library directory''
-when I really mean ``the site-packages directory under the standard
-Python library directory''.  Is there a better way?)
-
-In any event, you can easily install to non-standard locations with a
-couple of options to the \command{install} command:
-
-\begin{tableii}{ll}{option}{Option}{Description}
-  \lineii {install-lib}
-          {install directory for modules from pure Python distributions}
-  \lineii {install-platlib}
-          {install directory for modules from distributions with extensions}
-  \lineii {prefix}
-          {override \code{sys.prefix}:
-           point to an alternate Python installation}
-  \lineii {exec-prefix}
-          {override \code{sys.exec_prefix}:
-           point to an alternate Python installation}
-  \lineii {install-path}
-          {extra sub-path to append to \option{install-lib} (for
-           non-package-ized distributions)}
-\end{tableii}
-
-Of these, the most commonly used will probably be \option{install-lib}
-and \option{install-platlib}: you use them to point module installation
-right at a particular directory.  (You'll only need
-\option{install-platlib} if you maintain a multi-platform installation,
-which is often done on \UNIX{} networks with different architectures and
-operating systems.)  The two prefix options are intended for the
-somewhat arcane purpose of installing modules into a different Python
-installation than the Python binary used to perform the installation.
-The last, \option{install-path}, is mainly used for module developers to
-ensure that their module will go into a directory of their own, but it
-may occasionally be useful to you as a module installer.
-
-
-\subsection{Directly specifying installation directories}
-\label{sec:install-dirs}
-
-The most common type of custom module installation is where you maintain
-a personal stash of Python modules under your home directory, say in
-\homefile{lib/python}.  If you only care about a single platform
-there, then you only need to specify the \option{install-lib} option and 
-can forget about \option{install-platlib}:
-\begin{verbatim}
-python setup.py install --install-lib=~/lib/python
-\end{verbatim}
-You can, of course, supply whatever directory you like in place of
-\homefile{lib/python}.  More importantly, you can specify this
-directory permanently in your personal configuration file (XXX
-filename?):
+% takes eight args (four pairs):
+%   pure module distribution base + directory
+%   non-pure module distribution base + directory
+%   script base + directory
+%   data base + directory
+% ...and will no doubt take more args in future!
+\newcommand{\installscheme}[8]
+  {\begin{tableiii}{lll}{textrm}
+          {Type of file}
+          {Installation Directory}
+          {Override option}
+     \lineiii{pure module distribution}
+             {\filevar{#1}\filenq{#2}}
+             {\option{install-purelib}}
+     \lineiii{non-pure module distribution}
+             {\filevar{#3}\filenq{#4}}
+             {\option{install-platlib}}
+     \lineiii{scripts}
+             {\filevar{#5}\filenq{#6}}
+             {\option{install-scripts}}
+     \lineiii{data}
+             {\filevar{#7}\filenq{#8}}
+             {\option{install-data}}
+   \end{tableiii}}
+
+
+
+
+\section{Alternate Installation}
+\label{sec:alt-install}
+
+Often, it is necessary or desirable to install modules to a location
+other than the standard location for third-party Python modules.  For
+example, on a Unix system you might not have permission to write to the
+standard third-party module directory.  Or you might wish to try out a
+module before making it a standard part of your local Python
+installation; this is especially true when upgrading a distribution
+already present: you want to make sure your existing base of scripts
+still works with the new version before actually upgrading.
+
+The Distutils \command{install} command is designed to make installing
+module distributions to an alternate location simple and painless.  The
+basic idea is that you supply a base directory for the installation, and
+the \command{install} command picks a set of directories (called an
+\emph{installation scheme}) under this base directory in which to
+install files.  The details differ across platforms, so read whichever
+of the following section applies to you.
+
+
+\subsection{Alternate installation: Unix (the home scheme)}
+\label{sec:alt-unix-prefix}
+
+Under Unix, there are two ways to perform an alternate installation.
+The ``prefix scheme'' is similar to how alternate installation works
+under Windows and Mac OS, but is not necessarily the most useful way to
+maintain a personal Python library.  Hence, we document the more
+convenient and commonly useful ``home scheme'' first.
+
+The idea behind the ``home scheme'' is that you are building and
+maintaining a personal stash of Python modules, probably under your home 
+directory.  Installing a new module distribution is as simple as
 \begin{verbatim}
-[install]
-install-lib=~/lib/python
+python setup.py install --home
 \end{verbatim}
-Note that use of shell-style tilde and environment variable expansion is
-supported both on the command line and in configuration files.  (See
-section~\ref{sec:config-files} for more information on configuration
-files.)
-
-Of course, in order for this personal Python library scheme to work, you
-have to ensure that \homefile{lib/python} is in \code{sys.path} when you
-run Python.  The easiest way to do this under \UNIX{} is to add it to
-your \code{PYTHONPATH} environment variable when you login.  For
-example, if you use a Bourne shell derivative such as bash, zsh, or ksh,
-add the following to your \homefile{.profile} (or \homefile{.bashrc}, or
-\homefile{.zshenv}, depending on your shell and personal preferences):
+or
 \begin{verbatim}
-export PYTHONPATH=$HOME/lib/python
+python setup.py install --home=<dir>
 \end{verbatim}
-If you use a csh-derivative such as tcsh, add the following to your
-\homefile{.cshrc}:
+where you can supply any directory you like for the \option{home}
+option.  If you don't supply a directory (as in the first example
+above), the \command{install} command uses the \code{HOME} environment
+variable (or your official home directory as supplied by the password
+file, if \code{HOME} is not defined).
+
+The \option{home} option defines the installation base directory.  Files
+are installed to the following directories under the installation base
+as follows:
+\installscheme{home}{/lib/python}
+              {home}{/lib/python}
+              {home}{/bin}
+              {home}{/share}
+
+\subsection{Alternate installation: Unix (the prefix scheme)}
+\label{sec:alt-unix-home}
+
+The ``prefix scheme'' is useful when you wish to use one Python
+installation to perform the build/install (i.e., to run the setup
+script), but install modules into the third-party module directory of a
+different Python installation (or something that looks like a different
+Python installation).  If this sounds a trifle unusual, it is---that's
+why the ``home scheme'' comes first.  However, there are at least two
+known cases where the prefix scheme will be useful.
+
+First, consider that many Linux distribution put Python in \file{/usr},
+rather than the more traditional \file{/usr/local}.  This is entirely
+appropriate, since in those cases Python is part of ``the system''
+rather than a local add-on.  However, if you are installing Python
+modules from source, you probably want them to go in
+\file{/usr/local/lib/python1.\filevar{X}} rather than
+\file{/usr/lib/python1.\filevar{X}}.  This can be done with
 \begin{verbatim}
-setenv PYTHONPATH $HOME/lib/python
+/usr/bin/python setup.py install --prefix=/usr/local
 \end{verbatim}
 
-If you use multiple platforms (architectures and/or operating systems)
-from the same home directory, then you probably want to maintain a
-multi-platform personal Python library.  One possible scheme is to put
-platform-neutral (pure Python) distributions in \homefile{lib/python}
-and platform-specific distributions (any that containe extension
-modules) in \homefile{lib/python.\filevar{plat}}:
+Another possibility is a network filesystem where the name used to write
+to a remote directory is different from the name used to read it: for
+example, the Python interpreter accessed as \file{/usr/local/bin/python}
+might search for modules in \file{/usr/local/lib/python1.\filevar{X}},
+but those modules would have to be installed to, say,
+\file{/mnt/\filevar{@server}/export/lib/python1.\filevar{X}}.  This
+could be done with
 \begin{verbatim}
-python setup.py install --install-lib=~/lib/python \
-                        --install-lib-plat=~/lib/python.plat \
+/usr/local/bin/python setup.py install --prefix=/mnt/@server/export
 \end{verbatim}
-On the command line, of course, you can just type in the current
-platform in place of \filevar{plat}: \file{linux-x86},
-\file{solaris-sparc}, \file{linux-alpha}, whatever.  That's not an
-option in a configuration file, though---the same file has to cover all
-platforms for which you maintain a personal Python library.  So the
-Distutils provide a \code{PLAT} environment variable which will expand
-to the current platform name:
+
+In either case, the \option{prefix} option defines the installation
+base, and the \option{exec-prefix} option defines the platform-specific
+installation base, which is used for platform-specific files.
+(Currently, this just means non-pure module distributions, but could be
+expanded to C libraries, binary executables, etc.)  If
+\option{exec-prefix} is not supplied, it defaults to \option{prefix}.
+Files are installed as follows:
+
+\installscheme{prefix}{/lib/python1.\filevar{X}/site-packages}
+              {exec-prefix}{/lib/python1.\filevar{X}/site-packages}
+              {prefix}{/bin}
+              {prefix}{/share}
+
+There is no requirement that \option{prefix} or \option{exec-prefix}
+actually point to an alternate Python installation; if the directories
+listed above do not already exist, they are created at installation
+time.
+
+Incidentally, the real reason the prefix scheme is important is simply
+that a standard Unix installation uses the prefix scheme, but with
+\option{prefix} and \option{exec-prefix} supplied by Python itself (as
+\code{sys.prefix} and \code{sys.exec\_prefix}).  Thus, you might think
+you'll never use the prefix scheme, but every time you run \code{python
+  setup.py install} without any other options, you're using it.
+
+Note that installing extensions to an alternate Python installation has
+no effect on how those extensions are built: in particular, the Python
+header files (\file{Python.h} and friends) installed with the Python
+interpreter used to run the setup script will be used in compiling
+extensions.  It is your responsibility to ensure that the interpreter
+used to run extensions installed in this way is compatibile with the
+interpreter used to build them.  The best way to ensure this is that the
+two interpreters are the same version of Python (possibly different
+builds, or possibly copies of the same build).  (Of course, if your
+\option{prefix} and \option{exec-prefix} don't even point to an
+alternate Python installation, this is immaterial.)
+
+
+\subsection{Alternate installation: Windows}
+\label{sec:alt-windows}
+
+Since Windows has no conception of a user's home directory, and since
+the standard Python installation under Windows is simpler than that
+under Unix, there's no point in having separate \option{prefix} and
+\option{home} options.  Just use the \option{prefix} option to specify
+a base directory, e.g.
 \begin{verbatim}
-[install]
-install-lib=~/lib/python
-install-platlib=~/lib/python.$PLAT
+python setup.py install --base="\Temp\Python"
 \end{verbatim}
-(If \code{PLAT} is already defined in your environment, the Distutils
-won't override it: that way you can maintain consistency with other
-applications that look for a \code{PLAT} variable; this is especially
-useful when you refer to \code{PLAT} in your login scripts, as explained
-below.)
-
-(XXX danger danger! this environment-variable-in-config-file thing is
-frighteningly make-like: is there any way to avoid it?)
-
-Again, you have to make sure that your personal Python library appears
-in \code{sys.path}, and again the easiest way to do this is to set
-\code{PYTHONPATH} in your login scripts.  This time, though, you have to 
-be sure to set \emph{both} directories (platform-neutral and the current 
-platform-specific directory).  For Bourne-shell derivatives:
+to install modules to the \file{\bslash{}Temp} directory on the current
+drive.
+
+The installation base is defined by the \option{prefix} option; the
+\option{exec-prefix} option is not supported under Windows.  Files are
+installed as follows:
+\installscheme{prefix}{}
+              {prefix}{}
+              {prefix}{\bslash{}Scripts}
+              {prefix}{\bslash{}Data}
+
+
+\subsection{Alternate installation: Mac OS}
+\label{sec:alt-macos}
+
+Like Windows, Mac OS has no notion of home directories (or even of
+users), and a fairly simple standard Python installation.  Thus, only a
+\option{prefix} option is needed.  It defines the installation base, and 
+files are installed under it as follows:
+
+XXX how do MacPython users run the interpreter with command-line args?
+
+\installscheme{prefix}{:Lib}
+              {prefix}{:Mac:PlugIns}
+              {prefix}{}
+              {prefix}{}
+
+XXX Corran Webster says: ``Modules are found in either \file{:Lib} or
+\file{:Mac:Lib}, while extensions usually go in
+\file{:Mac:PlugIns}''---does this mean that non-pure distributions should
+be divided between \file{:Mac:PlugIns} and \file{:Mac:Lib}?  If so, that
+changes the granularity at which we care about modules: instead of
+``modules from pure distributions'' and ``modules from non-pure
+distributions'', it becomes ``modules from pure distributions'',
+``Python modules from non-pure distributions'', and ``extensions from
+non-pure distributions''.  Is this necessary?!?
+
+
+\section{Custom Installation}
+\label{sec:custom-install}
+
+Sometimes, the alternate installation schemes described in
+section~\ref{sec:alt-install} just don't do what you want.  You might
+want to tweak just one or two directories while keeping everything under
+the same base directory, or you might want to completely redefine the
+installation scheme.  In either case, you're creating a \emph{custom
+  installation scheme}.
+
+You probably noticed the column of ``override options'' in the tables
+describing the alternate installation schemes above.  Those options are
+how you define a custom installation scheme.  These override options can
+be relative, absolute, or explicitly defined in terms of one of the
+installation base directories.  (There are two installation base
+directories, and they are normally the same---they only differ when you
+use the Unix ``prefix scheme'' and supply different \option{prefix} and
+\option{exec-prefix} options.)
+
+For example, say you're installing a module distribution to your home
+directory under Unix---but you want scripts to go in
+\file{\tilde/scripts} rather than \file{\tilde/bin}.  As you might
+expect, you can override this directory with the
+\option{install-scripts} option; in this case, it makes most sense to
+supply a relative path, which will be interpreted relative to the
+installation base directory (your home directory, in this case):
 \begin{verbatim}
-export PYTHONPATH=$HOME/lib/python:$HOME/lib/python.$PLAT
+python setup.py install --home --install-scripts=scripts
 \end{verbatim}
-and for csh-derivatives:
+
+Another Unix example: suppose your Python installation was built and
+installed with a prefix of \file{/usr/local/python}, so under a standard 
+installation scripts will wind up in \file{/usr/local/python/bin}.  If
+you want them in \file{/usr/local/bin} instead, you would supply this
+absolute directory for the \option{install-scripts} option:
 \begin{verbatim}
-setenv PYTHONPATH $HOME/lib/python:$HOME/lib/python.$PLAT
+python setup.py install --install-scripts=/usr/local/bin
 \end{verbatim}
-Note that it is your responsibility to set the \code{PATH} environment
-variable (unless your system administrator has kindly taken care of it
-in the system-wide login scripts, which is a wise thing to do on
-multi-platform networks).  One way to do this is with the \code{uname}
-command:
+(This performs an installation using the ``prefix scheme,'' where the
+prefix is whatever your Python interpreter was installed with---
+\file{/usr/local/python} in this case.)
+
+If you maintain Python on Windows, you might want third-party modules to
+live in a subdirectory of \filevar{prefix}, rather than right in
+\filevar{prefix} itself.  This is almost as easy as customizing the
+script installation directory---you just have to remember that there are
+two types of modules to worry about, pure modules and non-pure modules
+(i.e., modules from a non-pure distribution).  For example:
 \begin{verbatim}
-export PLAT=`uname -sm | tr 'A-Z ' 'a-z-'`
+python setup.py install --install-purelib=Site --install-platlib=Site
 \end{verbatim}
-(XXX check that this works well on other Unices: on Linux, \code{-m}
-becomes eg. \code{i586}, which is not the \emph{machine} but the
-\emph{processor}.  Arggh!)
-
-Of course, there are more reasons to do custom installation than
-maintaining a personal Python library.  Even if you have write access to
-the system-wide directories for third-party modules
-(\file{\filevar{prefix}/lib/python1.\filevar{x}/site-packages} and
-\file{\filevar{exec-prefix}/lib/python1.\filevar{x}/site-packages}), you
-might want to install new module distributions---especially upgrades of
-modules that are crucial to your local infrastructure---to a temporary
-location, in order to test them before installing them ``for real.''
-This is fundamentally no different from installing to your home
-directory, except that you probably won't bother to set
-\code{PYTHONPATH} permanently.  For example, to install a module
-distribution to \file{/tmp/pylib}:
+The specified installation directories are relative to \filevar{prefix}.
+Of course, you also have to ensure that these directories are in
+Python's module search path, e.g. by putting a \file{.pth} file in
+\filevar{prefix} (XXX should have a section describing .pth files and
+cross-ref it here).
+
+If you want to define an entire installation scheme, you just have to
+supply all of the installation directory options.  The recommended way
+to do this is to supply relative paths; for example, if want to maintain
+all Python module-related files under \file{python} in your home
+directory, and you want a separate directory for each platform that you
+use your home directory from, you might define the following
+installation scheme:
 \begin{verbatim}
-python setup.py install --install-lib=/tmp/pylib
+python setup.py install --home \
+                        --install-purelib=python/lib \
+                        --install-platlib=python/lib.$PLAT \
+                        --install-scripts=python/scripts
+                        --install-data=python/data
 \end{verbatim}
-Then, of course, you'll want to run some script that depends on these
-modules to make sure that they still work with your installed base of
-code:
+or, equivalently,
 \begin{verbatim}
-env PYTHONPATH=/tmp/pylib python /usr/local/bin/crucial_script ...
+python setup.py install --home=~/python \
+                        --install-purelib=lib \
+                        --install-platlib=lib.$PLAT \
+                        --install-scripts=scripts
+                        --install-data=data
 \end{verbatim}
-
-Of course, you can do this temporary installation with separate
-\option{install-lib} and \option{install-platlib} options.  If you're
-doing this to a network-wide directory, not \file{/tmp}, this might be
-essential.  As you might have guessed, it's not too hard:
+\code{\$PLAT} is not (necessarily) an environment variable---it will be
+expanded by the Distutils as it parses your command line options (just
+as it does when parsing your configuration file(s)).
+
+Obviously, specifying the entire installation scheme every time you
+install a new module distribution would be very tedious.  Thus, you can
+put these options into your Distutils config file (see
+section~\ref{sec:config-files}):
 \begin{verbatim}
-python setup.py install --install-lib=/scratch/pylib \
-                        --install-platlib=/scratch/pylib.plat
+[install]
+install-base=$HOME
+install-purelib=python/lib
+install-platlib=python/lib.$PLAT
+install-scripts=python/scripts
+install-data=python/data
 \end{verbatim}
-and then, testing your crucial scripts on multiple platforms:
+or, equivalently,
 \begin{verbatim}
-env PYTHONPATH=/scratch/pylib:/scratch/pylib.plat \
-    python /usr/local/bin/crucial_script ...
+[install]
+install-base=$HOME/python
+install-purelib=lib
+install-platlib=lib.$PLAT
+install-scripts=scripts
+install-data=data
 \end{verbatim}
-
-However you do the testing, once you're satisfied that the new version
-doesn't break anything, you can install it to the system-wide
-third-party module directory as usual:
+Note that these two are \emph{not} equivalent if you supply a different
+installation base directory when you run the setup script.  For example,
 \begin{verbatim}
-python setup.py install
+python setup.py --install-base=/tmp
 \end{verbatim}
+would install pure modules to \filevar{/tmp/python/lib} in the first
+case, and to \filevar{/tmp/lib} in the second case.  (For the second
+case, you probably want to supply an installation base of
+\file{/tmp/python}.)
 
+You probably noticed the use of \code{\$HOME} and \code{\$PLAT} in the
+sample configuration file input.  These are Distutils configuration
+variables, which bear a strong resemblance to environment variables.  In
+fact, you can use environment variables in config files, but the
+Distutils additionally define a few extra variables that may not be in
+your environment, such as \code{\$PATH}.  See
+section~\ref{sec:config-files} for details.
 
-\subsection{Indirect specification: prefix directories}
-\label{sec:prefix-dirs}
-
-Occasionally, you may want to install a module distribution
+XXX need some Windows and Mac OS examples---when would custom
+installation schemes be needed on those platforms?
 
 
-\section{Custom Installation (Windows)}
-\label{sec:custom-install-windows}
-
-\comingsoon
-
 
 \section{Configuration Files}
 \label{sec:config-files}
 
 \comingsoon
 
-
-
 \end{document}