From 7c1e5f65e32f22734b88e1b2461d8e1e3fb2d292 Mon Sep 17 00:00:00 2001 From: Greg Ward Date: Fri, 10 Mar 2000 01:56:58 +0000 Subject: [from 2000-02-23] Description of the baroque and already-obsolete installation scheme that Fred Drake and I cooked up. Only saved for posterity. --- Doc/inst/inst.tex | 534 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 534 insertions(+) create mode 100644 Doc/inst/inst.tex diff --git a/Doc/inst/inst.tex b/Doc/inst/inst.tex new file mode 100644 index 0000000..e7442df --- /dev/null +++ b/Doc/inst/inst.tex @@ -0,0 +1,534 @@ +\documentclass{howto} +\usepackage{ltxmarkup} +\usepackage{times} + +\title{Installing Python Modules} + +% The audience for this document includes people who don't know anything +% about Python and aren't about to learn the language just in order to +% install and maintain it for their users, i.e. system administrators. +% Thus, I have to be sure to explain the basics at some point: +% sys.path and PYTHONPATH at least. Should probably give pointers to +% other docs on "import site", PYTHONSTARTUP, PYTHONHOME, etc. +% +% Also, I need to take into account that most modules out there don't +% (yet) use Distutils: briefly explain the old Makefile.pre.in +% convention (maybe move material from the E&E manual to here?), and +% explain where to copy .py and .so files manually if the distribution +% doesn't provide a mechanism for doing so. +% +% Finally, it might be useful to include all the material from my "Care +% and Feeding of a Python Installation" talk in here somewhere. Yow! + +% Hey wow, Guido didn't write this one either! +\author{Greg Ward} +\authoraddress{E-mail: \email{gward@python.net}} + +% Should these be added to the standard Python doc tools? (They'll be +% needed for my "Distributing Python Modules" guide, too.) +\newcommand{\command}[1]{\code{#1}} +\newcommand{\option}[1]{\code{#1}} +\newcommand{\filevar}[1]{{\textsl{\filenq{#1}}}} +\newcommand{\comingsoon}{\emph{Coming soon$\ \ldots$}} + +% And how about these? Very handy for writing pathnames (tilde for +% Unix, backslash for DOS/Windows). +\renewcommand{\tilde}{\raisebox{-0.5ex}{\symbol{126}}} +\newcommand{\bslash}{\symbol{92}} + + +\begin{document} + +\maketitle + +%\begin{abstract} +%\noindent +%Abstract this! +%\end{abstract} + +\tableofcontents + +\section{Introduction} +\label{sec:intro} + +\comingsoon + +\subsection{The new way: Distutils} +\label{sec:new-way} + + +\subsection{The old way (pure Python): whatever you feel like} +\label{sec:old-way-pure} + + +\subsection{The old way (extensions, \UNIX{} only): Makefile.pre.in} +\label{sec:old-way-ext} + + + + + +\section{Normal Build and Install} +\label{sec:normal-install} + +% This will cover: +% * setup.py install (the usual thing) +% * setup.py build (if you like doing things one-at-a-time) +% * setup.py build install (not necessary unless you need to supply +% build options--ref. next section) +% * where things are installed, on Unix and Windows (Mac...?) +% * simple custom install: "install --prefix=$HOME" +\comingsoon + + +\section{Custom Extension Building} +\label{sec:custom-ext} + +% This will cover: +% * normal extension build -- stress that it doesn't matter, you +% do the same thing whether there are extensions or not +% * what you might want to customize: compiler and compiler +% flags (warn of the dangers); per-file compiler flags +% (not handled yet!) +% * when things go wrong: I don't know! (and I don't know what +% to do, either!) +\comingsoon + + +\section{Custom Installation (\UNIX)} +\label{sec:custom-install-unix} + +% XXX probably should banish mentions of Windows here to the +% separate "Non-standard installation (Windows)" section. + +A \dfn{custom installation} is where you install modules to a location +that's not in Python's default module search path. There are a couple +of reasons you might want to do this; the most typical is simply that +you don't have permission to write to the standard Python library +directory. Or, even if you do have write permission to the standard +library, you might wish to install a module distribution into a +non-standard place for testing or experimentation. (This is especially +useful when upgrading an existing module distribution: you might want to +make sure that your existing scripts continue to work as before, and +only then install the upgrade ``for real.'') + +(XXX terminology: I keep saying ``standard Python library directory'' +when I really mean ``the site-packages directory under the standard +Python library directory''. Is there a better way?) + +In any event, you can easily install to non-standard locations with a +couple of options to the \command{install} command: + +\begin{tableii}{ll}{option}{Option}{Description} + \lineii{prefix}{base dir for pure Python distributions + (overrides \code{sys.prefix})} + \lineii{exec-prefix}{base dir for distributions with extensions + (overrides \code{sys.exec_prefix})} + \lineii{install-lib}{install dir for top-level modules from pure + Python distributions} + \lineii{install-platlib}{install dir for top-level modules from + distributions with extensions} + \lineii{install-path}{extra path under \option{install-lib} or + \option{install-platlib} to install to} +\end{tableii} + + +\subsection{Prefix options} +\label{sec:prefix-options} + +There are a lot of picky little rules that govern the interactions of +these five options. As usual, it's easier to explain things with +examples, so we'll save all the picky rules for later, after you've seen +a bunch of examples. However, we really have to establish some ground +rules before we can dive into the examples: +\begin{itemize} +\item in a normal \UNIX{} installation, \code{sys.prefix} and + \code{sys.exec\_prefix} are both \file{/usr/local}. +\item in a multi-platform \UNIX{} installation, \code{sys.prefix} and + \code{sys.exec\_prefix} are different, and are selected when you + configure and build Python itself. Our canonical example of a + multi-platform installation will have a \code{sys.prefix} of + \file{/usr/local} and a \code{sys.exec\_prefix} of + \file{/usr/local.\filevar{plat}} (for whatever value of \filevar{plat} + is appropriate). +\item the canonical place to install third-party modules is + either \file{\filevar{prefix}/lib/python1.\filevar{X}/site-packages} + or \file{\filevar{exec\_prefix}/lib/python1.\filevar{X}/site-packages}. + These will be referred to as ``the site-packages directories.'' +\end{itemize} + + +\subsubsection{Pure Python module distribution} + +To demonstrate, consider a hypothetical module distribution that +contains one top-level module and a package with two modules: +\begin{tableii}{ll}{module}{Module}{Filename} + \lineii{mymod}{\filenq{mymod.py}} + \lineii{mypkg.mod1}{\filenq{mypkg/mod1.py}} + \lineii{mypkg.mod2}{\filenq{mypkg/mod2.py}} +\end{tableii} +where the filenames are relative to \file{build/lib} after building, or +to some directory in \code{sys.path} after installation. + +The goal of installation is to copy these files into a directory in +\code{sys.path} without interfering with the standard Python library. +The canonical, preferred, and most obvious thing to do is to put them in +the ``site-packages'' directory, which is exactly what the +\command{install} comand does by default: under a normal \UNIX{} Python +installation, +\begin{verbatim} + python setup.py install +\end{verbatim} +installs \file{/usr/local/lib/python1.\filevar{X}/lib/site-packages/mymod.py}, +with the \module{mypkg} package in a \file{mypkg} directory under +\file{site-packages}. + +However, if you were interested in a standard installation, you wouldn't +be reading this section. The next-most-standard thing to do is to +specify a custom prefix to override \code{sys.prefix}. For example: +\begin{verbatim} + python setup.py install --prefix=/home/greg +\end{verbatim} +is a sensible way to install Python modules to your home directory: this +results in the installation of \file{/home/greg/lib/python/mymod.py}, +with the \module{mypkg} modules in \file{/home/greg/lib/python/mypkg/}. +An important point here is that in both this example and the ``plain +vanilla'' example above, the actual installation directory is derived +from the \option{prefix} option. However, when \option{prefix} differs +from \code{sys.prefix}, the installation directory is derived +differently: the Python version and \file{site-packages} are omitted. +(The version number is part of the standard library directory name to +describe the version of the standard library, so it doesn't make sense +to include it in the name of a non-standard-library directory; likewise, +\file{site-packages} is meant to denote non-standard modules living in +the same area as the standard library, so it doesn't make sense to +include it when installing to a non-standard library. [XXX check with +Guido that this reasoning is valid and correct; Fred disagrees!]) + + +\subsubsection{Module distribution with extensions} + +Now let's consider a different hypothetical module distribution, which +consists of a single package, \module{foo}, containing one pure Python +module and one extension module: +\begin{tableii}{ll}{module}{Module}{Filename} + \lineii{foo.pure}{\filenq{foo/pure.py}} + \lineii{foo.ext}{\filenq{foo/ext.so} (or \file{foo/extmodule.so})} +\end{tableii} +In this case, the two modules will be in different locations in the +build tree: \file{build/lib/foo/pure.py} and +\file{build/platlib/foo/ext.so}. (The \file{.so} (``shared object'') +extension isn't universal, but it's the norm on \UNIX-like systems; +under Windows, the extension module will be in \file{foo/ext.pyd} or +\file{foo/extmodule.pyd}.) + +Consider again a standard, plain-vanilla installation: +\begin{verbatim} + python setup.py install +\end{verbatim} +In this case, \emph{both} modules will be installed to the site-packages +directory under \code{sys.exec\_prefix}, e.g. to +\file{/usr/local.\filevar{plat}/lib/python1.\filevar{X}/site-packages} +on a \UNIX{} system where Python was configured with +\samp{--exec-prefix=/usr/local.plat}. (On Windows, again, there is no +site-packages directory and \code{sys.prefix} and +\code{sys.exec\_prefix} are the same---so both modules will just be +installed to \code{sys.prefix}.) + +Of course, we've already established that you're not interested in +standard installations. If you just want to install these modules to +your home directory, and you don't maintain a multi-platform home +directory, no problem---just set the prefix as before: +\begin{verbatim} +python setup.py install --prefix=/home/greg +\end{verbatim} +and both modules will be installed to \file{/home/greg/lib/python}. + +Now let's say your Python installation is in \file{/usr}---as is the +case in many Linux distributions---but your local policy is to install +third-party software to a network-wide \file{/usr/local} and +\file{/usr/local.\filevar{plat}}. That is, \code{sys.prefix} and +\code{sys.exec\_prefix} are both \file{/usr}, and you want Python +modules to be installed to either \file{/usr/local/lib/python} or +\file{/usr/local.\filevar{plat}/lib/python}. This is one case where you +want to specify both \option{prefix} and \option{exec-prefix}: +\begin{verbatim} +python setup.py install --prefix=/usr/local \ + --exec-prefix=/usr/local.plat +\end{verbatim} +An oddity of this situation is that for any given module distribution, +you only have to supply \emph{one} of \option{prefix} and +\option{exec-prefix}, because pure Python distributions are always +installed under \option{prefix}, and extension-containing distributions +are always installed under \option{exec-prefix}. For consistency's +sake, though, it's best always to supply both---and the best way to do +that is by using a system-wide configuration file (see +Section~\ref{sec:config-files}). + +You could use a similar scheme to maintain a multi-platform personal +Python library. For example, if you install lots of stuff to your home +directory (not just Python modules), you might have a complete +\file{\tilde/usr} with \file{include}, \file{man}, \file{lib}, and so +forth. (The advantage of this scheme is that it keeps those mock system +directories out of your home directory and makes it easier to support a +multi-platform personal \file{usr} tree.) If you don't care about a +multi-platform installation, you can just install with +\begin{verbatim} +python setup.py install --prefix=$HOME/usr +\end{verbatim} +But if you want to keep separate \file{usr} trees for each architecture +that you use, you could say +\begin{verbatim} +python setup.py install --prefix=$HOME/usr \ + --exec-prefix=$HOME/usr.plat +\end{verbatim} +for various values of \file{plat}. + +% this paragraph is for Michel Sanner ;-) +(Perceptive readers will note that on a multi-platform Python +installation, multiple identical copies of \file{foo/pure.py} will be +installed, one for each platform. This is deliberate. First, it makes +Python's module search algorithm simpler (XXX check this): when you say +\samp{import foo.pure}, Python searches \code{sys.path} until it finds a +directory containing \file{foo/__init__.py}. When it finds one, that +directory is deemed to be the directory containing the \module{foo} +package for this import. Even if the search algorithm were changed +(necessitating a trip back in time to ``fix'' Python 1.5), the only way +to make multiple candidate \module{foo} directories (one for pure +Python, one for extension modules) would be to make copies of +\file{__init__.py}---in which case, why not make copies of all the pure +Python modules? Second, if you kept pure Python modules related to +extension modules in a platform-shared directory, what happens while you +are upgrading your favourite extension from version 1.0 to 1.1 on +platforms X and Y? After you install 1.1 for platform X, the 1.1 +\file{.py} files will be in the platform-shared directory---but the 1.0 +extensions will still be in the platform Y directory. If the interval +between installing 1.1 for platform X and for platform Y is long---e.g., +there are portability problems with platform Y---then there's a good +probability of a version mismatch between the 1.1 Python modules and the +1.0 extensions on platform Y. The solution to both problems is to +install separate copies of the pure Python modules for every platform. +In this day and age, unnecessary disk use is no argument.) + +Other ways to support a multi-platform personal Python library are +discussed below, when we cover the \option{install-lib} and +\option{install-platlib} options. + + +% Gory details on the prefix options (still need to work these into the +% surrounding text): +XXX need to finish these rules and give them some context! +\begin{itemize} +\item \code{sys.exec\_prefix} (and the \option{exec-prefix} option) + only matters on a multi-platform installation. If you don't have a + multi-platform installation (or even know what that is), then you + don't care about \option{exec-prefix}. +\item in a normal Windows installation, \code{sys.prefix} and + \code{sys.exec\_prefix} are both \file{C:\bslash Program Files\bslash + Python}; they are never different under Windows (XXX check!). +\item you may supply \emph{both} of \option{prefix} and + \option{exec-prefix}, or \emph{neither} of them, or \emph{just} + \option{prefix}---but you may not supply just \option{exec-prefix}. +\end{itemize} + + +\subsection{Installation directory options} +\label{sec:install-dirs} + +Most of the time, it's enough to specify just \option{prefix} (and +possibly \option{exec-prefix})---your modules are installed to +\file{lib/python} under one or the other, you add the appropriate +directory to \code{sys.path}, and that's it. + +However, there will inevitably be times when you want finer control over +the installation directories, and that is when the \option{install-lib}, +\option{install-platlib}, and \option{install-path} options are +essential. Normally, \option{install-lib} and \option{install-platlib} +are simply the directories where pure Python modules and extension +modules, respectively, are installed. That is, top-level modules +(modules not in a package) are installed straight to +\option{install-lib} (or \option{install-platlib} if there are any +extensions in the module distribution). (If \option{install-path} is +supplied, then things are a bit more complex; we'll deal with that +below.) + +Normally, \option{install-lib} and \option{install-platlib} are derived +from \option{prefix} and/or \option{exec-prefix}. For example, if you +don't supply anything, then \option{prefix} defaults to +\code{sys.prefix}, and \option{install-lib} defaults to +\file{\filevar{prefix}/lib/python1.\filevar{X}/site-packages}. If you +supply \option{prefix} but not \option{install-lib}, then +\option{install-lib} defaults to \file{\filevar{prefix}/lib/python} +(unless you just happen to supply a prefix which equals +\code{sys.prefix}, which is treated the same as if you don't supply +\option{prefix} at all). (The rules for \option{exec-prefix} and +\option{install-platlib} are a bit more complex; the following examples +should clarify. Consult the Distutils source for the gory details.) + +To illustrate, let's go back to our hypothetical pure-Python module +distribution containing \module{mymod}, \module{mypkg.mod1}, and +\module{mypkg.mod2}. If you maintain a personal stash of Python modules +in your home directory, but don't like the \file{\tilde/lib/python} +convention, no problem---you can put the modules right in a +\file{\tilde/python} directory with +\begin{verbatim} +python setup.py install --install-lib=$HOME/python +\end{verbatim} +which will install \file{\$HOME/python/mymod.py}, +\file{\$HOME/python/mypkg/mod1.py}, and +\file{\$HOME/python/mypkg/mod2.py}. + +If you happen to install a module distribution that contains extensions, +again that's no problem---in the absence of \option{exec-prefix}, +\option{install-platlib} defaults to \option{install-lib}, so the above +example will also put extension modules in \file{\$HOME/python}. +(XXX is this correct? is this the best way to describe it? should it be +implemented this way or some other way? how should it be described?) + +This may not be what you want, though, if you maintain a multi-platform +stash of Python modules in your home directory. In that case, you need +to specify \option{install-platlib}---this is the directory where module +distributions with extensions will be installed. For example, if you +keep pure Python module distributions in \file{\tilde/python} and +extension distributions in \file{\tilde/python.plat}: +\begin{verbatim} +python setup.py install --install-lib=$HOME/python \ + --install-platlib=$HOME/python.plat +\end{verbatim} +(Just as with \option{prefix} and \option{exec-prefix}, it's only +necessary to supply one of \option{install-lib} and +\option{install-platlib} for any given module distribution, but to +ensure consistency you should always supply them both using a +configuration file (section~\ref{sec:config-files}).) + +An alternate way to maintain a multi-platform personal Python library is +in \file{\tilde/lib/python} and \file{\tilde/lib/python.plat}. In that +case, you can get away with supplying \option{prefix} and +\option{install-platlib}: +\begin{verbatim} +python setup.py install --prefix=$HOME \ + --install-platlib=$HOME/lib/python.plat +\end{verbatim} + +Finally, the \option{install-path} option, which exists mainly to gum up +the whole works---but in a productive (and important) way. +Specifically, \option{install-path} exists to give a directory of their +own to module distributions that wouldn't otherwise have one, i.e.\ that +are not distributed as a (Python) package. + +Consider a module distribution, Foo, that consists of (pure Python) +modules \module{foobar}, \module{foobaz}, and \module{fooqux}. +Obviously these are related, and if the project had started in the +Python 1.5 era (and doesn't worry about backwards compatibility with +Python 1.4), they probably would be packaged up and called +\module{foo.bar}, \module{foo.baz}, and \module{foo.qux}. +Unfortunately, they aren't, but we still want the Foo modules to go into +a directory of their own. + +Normally, this will be taken care of by the module developer: he adds a +line \samp{install_path = 'Foo'} to his setup script, which has the +following consequences: +\begin{enumerate} +\item instead of \option{install-lib} the modules would be installed in + \file{\filevar{install-lib}/Foo} +\item if \option{install-lib} is the same as the default + \option{install-lib}---e.g., you supplied neither \option{prefix} or + \option{install-lib}---then a \file{Foo.pth} will be created in + \option{install-lib}, so that Python adds + \file{\filevar{install-lib}/Foo} to \code{sys.path} +\item if \option{install-lib} is not the default, then a warning will be + printed, reminding you to add \file{\filevar{install-lib}/Foo} to + \code{sys.path} yourself, such as with the \code{PYTHONPATH} + environment variable +\end{enumerate} + +Thus, you as a module installer have to be aware of the +\option{install-path} option---especially if you maintain a personal +stash of Python modules and don't have write permission to the standard +library, so Distutils can't create \file{.pth} files for you---but you +don't often have to supply it yourself. There are situations in which +you might want to supply it, though: +\begin{itemize} +\item a module developer forgot to include it (the distribution really + should go in a directory of its own, but it won't unless you make it) +\item you want to override the \option{install-path} supplied by the + developer (e.g., you'd rather have a huge jumble of files in + \file{site-packages} than make Python wade through a bunch of + \file{.pth} files at startup) +\end{itemize} + +The first case is easy: say we're dealing with the Foo distribution +again, but the developer forgot to include \option{install-path}. No +problem, you can supply it on the command line: +\begin{verbatim} +python setup.py install --install-path=Foo +\end{verbatim} +Note that this will work just fine if you supply \option{prefix} or +\option{install-lib}---but of course, you'll probably have to ensure +that the \file{Foo} directory is in \code{sys.path} yourself. + +If you're really fanatical about keeping track of what you have +installed, you might want to supply your own \option{install-path} that +records the version as well as the name of the module distribution; this +overrides any \option{install-path} included by the module developer in +the setup script: +\begin{verbatim} +python setup.py install --install-path=Foo-1.3 +\end{verbatim} + +Finally, you can disable \option{install-path} entirely: +\begin{verbatim} +python setup.py install --install-path='' +\end{verbatim} +...but the mess that will result (modules from many different +distributions in the same \option{install-lib} and +\option{install-platlib} directories) is your own problem. + +% Points to make +% * only one of prefix or exec_prefix matters +% * don't have to specify exec_prefix unless != prefix +% * thus, usually enough to supply prefix +% * only have to supply install_lib if you don't like +% "prefix/lib/python" +% * likewise for install_platlib and exec_prefix +% * don't have to supply install_platlib unless != install_lib (??) +% * in the absence of install_path, top-level modules wind up in +% install_lib or install_platlib +In case you're interested, here are the exact rules for how +\option{install-lib} and \option{install-platlib} are initialized, and +how they and \option{install-path} affect where modules (pure Python and +extensions) are installed to: +\begin{itemize} +\item If you don't supply \option{prefix} (and possibly + \option{exec-prefix}), then \option{install-lib} and + \option{install-platlib} will be, respectively, + \file{\filevar{\$prefix}/lib/python1.\filevar{X}/site-packages} and + \file{\filevar{\$exec\_prefix}/lib/python1.\filevar{X}/site-packages}. In a + normal \UNIX{} installation, both of these resolve to + \file{/usr/local/lib/python1.\filevar{X}/site-packages}. +\item in the absence of an \option{install-path} option, top-level + modules and packages from a pure Python distribution are installed to + \option{install-lib} +\item in the absence of an \option{install-path} option, top-level + modules and packages from a distribution that contains \emph{any} + extension modules are installed to \option{install-platlib}. +\item \emph{there're more, but I don't remember everything offhand} +%\item \option{install-lib} is initialized from \option{prefix} (which +% in turn is initialized from \code{sys.prefix})---so you should +\end{itemize} + + +\section{Custom Installation (Windows)} +\label{sec:custom-install-windows} + +\comingsoon + + +\section{Configuration Files} +\label{sec:config-files} + +\comingsoon + + + +\end{document} -- cgit v0.12