diff options
Diffstat (limited to 'Doc/ext/extending.tex')
| -rw-r--r-- | Doc/ext/extending.tex | 1695 |
1 files changed, 1695 insertions, 0 deletions
diff --git a/Doc/ext/extending.tex b/Doc/ext/extending.tex new file mode 100644 index 0000000..ee1b678 --- /dev/null +++ b/Doc/ext/extending.tex @@ -0,0 +1,1695 @@ +\chapter{Extending Python with C or \Cpp{} \label{intro}} + + +It is quite easy to add new built-in modules to Python, if you know +how to program in C. Such \dfn{extension modules} can do two things +that can't be done directly in Python: they can implement new built-in +object types, and they can call C library functions and system calls. + +To support extensions, the Python API (Application Programmers +Interface) defines a set of functions, macros and variables that +provide access to most aspects of the Python run-time system. The +Python API is incorporated in a C source file by including the header +\code{"Python.h"}. + +The compilation of an extension module depends on its intended use as +well as on your system setup; details are given in later chapters. + + +\section{A Simple Example + \label{simpleExample}} + +Let's create an extension module called \samp{spam} (the favorite food +of Monty Python fans...) and let's say we want to create a Python +interface to the C library function \cfunction{system()}.\footnote{An +interface for this function already exists in the standard module +\module{os} --- it was chosen as a simple and straightfoward example.} +This function takes a null-terminated character string as argument and +returns an integer. We want this function to be callable from Python +as follows: + +\begin{verbatim} +>>> import spam +>>> status = spam.system("ls -l") +\end{verbatim} + +Begin by creating a file \file{spammodule.c}. (Historically, if a +module is called \samp{spam}, the C file containing its implementation +is called \file{spammodule.c}; if the module name is very long, like +\samp{spammify}, the module name can be just \file{spammify.c}.) + +The first line of our file can be: + +\begin{verbatim} +#include <Python.h> +\end{verbatim} + +which pulls in the Python API (you can add a comment describing the +purpose of the module and a copyright notice if you like). + +All user-visible symbols defined by \code{"Python.h"} have a prefix of +\samp{Py} or \samp{PY}, except those defined in standard header files. +For convenience, and since they are used extensively by the Python +interpreter, \code{"Python.h"} includes a few standard header files: +\code{<stdio.h>}, \code{<string.h>}, \code{<errno.h>}, and +\code{<stdlib.h>}. If the latter header file does not exist on your +system, it declares the functions \cfunction{malloc()}, +\cfunction{free()} and \cfunction{realloc()} directly. + +The next thing we add to our module file is the C function that will +be called when the Python expression \samp{spam.system(\var{string})} +is evaluated (we'll see shortly how it ends up being called): + +\begin{verbatim} +static PyObject * +spam_system(self, args) + PyObject *self; + PyObject *args; +{ + char *command; + int sts; + + if (!PyArg_ParseTuple(args, "s", &command)) + return NULL; + sts = system(command); + return Py_BuildValue("i", sts); +} +\end{verbatim} + +There is a straightforward translation from the argument list in +Python (for example, the single expression \code{"ls -l"}) to the +arguments passed to the C function. The C function always has two +arguments, conventionally named \var{self} and \var{args}. + +The \var{self} argument is only used when the C function implements a +built-in method, not a function. In the example, \var{self} will +always be a \NULL{} pointer, since we are defining a function, not a +method. (This is done so that the interpreter doesn't have to +understand two different types of C functions.) + +The \var{args} argument will be a pointer to a Python tuple object +containing the arguments. Each item of the tuple corresponds to an +argument in the call's argument list. The arguments are Python +objects --- in order to do anything with them in our C function we have +to convert them to C values. The function \cfunction{PyArg_ParseTuple()} +in the Python API checks the argument types and converts them to C +values. It uses a template string to determine the required types of +the arguments as well as the types of the C variables into which to +store the converted values. More about this later. + +\cfunction{PyArg_ParseTuple()} returns true (nonzero) if all arguments have +the right type and its components have been stored in the variables +whose addresses are passed. It returns false (zero) if an invalid +argument list was passed. In the latter case it also raises an +appropriate exception so the calling function can return +\NULL{} immediately (as we saw in the example). + + +\section{Intermezzo: Errors and Exceptions + \label{errors}} + +An important convention throughout the Python interpreter is the +following: when a function fails, it should set an exception condition +and return an error value (usually a \NULL{} pointer). Exceptions +are stored in a static global variable inside the interpreter; if this +variable is \NULL{} no exception has occurred. A second global +variable stores the ``associated value'' of the exception (the second +argument to \keyword{raise}). A third variable contains the stack +traceback in case the error originated in Python code. These three +variables are the C equivalents of the Python variables +\code{sys.exc_type}, \code{sys.exc_value} and \code{sys.exc_traceback} (see +the section on module \module{sys} in the +\citetitle[../lib/lib.html]{Python Library Reference}). It is +important to know about them to understand how errors are passed +around. + +The Python API defines a number of functions to set various types of +exceptions. + +The most common one is \cfunction{PyErr_SetString()}. Its arguments +are an exception object and a C string. The exception object is +usually a predefined object like \cdata{PyExc_ZeroDivisionError}. The +C string indicates the cause of the error and is converted to a +Python string object and stored as the ``associated value'' of the +exception. + +Another useful function is \cfunction{PyErr_SetFromErrno()}, which only +takes an exception argument and constructs the associated value by +inspection of the global variable \cdata{errno}. The most +general function is \cfunction{PyErr_SetObject()}, which takes two object +arguments, the exception and its associated value. You don't need to +\cfunction{Py_INCREF()} the objects passed to any of these functions. + +You can test non-destructively whether an exception has been set with +\cfunction{PyErr_Occurred()}. This returns the current exception object, +or \NULL{} if no exception has occurred. You normally don't need +to call \cfunction{PyErr_Occurred()} to see whether an error occurred in a +function call, since you should be able to tell from the return value. + +When a function \var{f} that calls another function \var{g} detects +that the latter fails, \var{f} should itself return an error value +(usually \NULL{} or \code{-1}). It should \emph{not} call one of the +\cfunction{PyErr_*()} functions --- one has already been called by \var{g}. +\var{f}'s caller is then supposed to also return an error indication +to \emph{its} caller, again \emph{without} calling \cfunction{PyErr_*()}, +and so on --- the most detailed cause of the error was already +reported by the function that first detected it. Once the error +reaches the Python interpreter's main loop, this aborts the currently +executing Python code and tries to find an exception handler specified +by the Python programmer. + +(There are situations where a module can actually give a more detailed +error message by calling another \cfunction{PyErr_*()} function, and in +such cases it is fine to do so. As a general rule, however, this is +not necessary, and can cause information about the cause of the error +to be lost: most operations can fail for a variety of reasons.) + +To ignore an exception set by a function call that failed, the exception +condition must be cleared explicitly by calling \cfunction{PyErr_Clear()}. +The only time C code should call \cfunction{PyErr_Clear()} is if it doesn't +want to pass the error on to the interpreter but wants to handle it +completely by itself (possibly by trying something else, or pretending +nothing went wrong). + +Every failing \cfunction{malloc()} call must be turned into an +exception --- the direct caller of \cfunction{malloc()} (or +\cfunction{realloc()}) must call \cfunction{PyErr_NoMemory()} and +return a failure indicator itself. All the object-creating functions +(for example, \cfunction{PyInt_FromLong()}) already do this, so this +note is only relevant to those who call \cfunction{malloc()} directly. + +Also note that, with the important exception of +\cfunction{PyArg_ParseTuple()} and friends, functions that return an +integer status usually return a positive value or zero for success and +\code{-1} for failure, like \UNIX{} system calls. + +Finally, be careful to clean up garbage (by making +\cfunction{Py_XDECREF()} or \cfunction{Py_DECREF()} calls for objects +you have already created) when you return an error indicator! + +The choice of which exception to raise is entirely yours. There are +predeclared C objects corresponding to all built-in Python exceptions, +such as \cdata{PyExc_ZeroDivisionError}, which you can use directly. +Of course, you should choose exceptions wisely --- don't use +\cdata{PyExc_TypeError} to mean that a file couldn't be opened (that +should probably be \cdata{PyExc_IOError}). If something's wrong with +the argument list, the \cfunction{PyArg_ParseTuple()} function usually +raises \cdata{PyExc_TypeError}. If you have an argument whose value +must be in a particular range or must satisfy other conditions, +\cdata{PyExc_ValueError} is appropriate. + +You can also define a new exception that is unique to your module. +For this, you usually declare a static object variable at the +beginning of your file: + +\begin{verbatim} +static PyObject *SpamError; +\end{verbatim} + +and initialize it in your module's initialization function +(\cfunction{initspam()}) with an exception object (leaving out +the error checking for now): + +\begin{verbatim} +void +initspam() +{ + PyObject *m, *d; + + m = Py_InitModule("spam", SpamMethods); + d = PyModule_GetDict(m); + SpamError = PyErr_NewException("spam.error", NULL, NULL); + PyDict_SetItemString(d, "error", SpamError); +} +\end{verbatim} + +Note that the Python name for the exception object is +\exception{spam.error}. The \cfunction{PyErr_NewException()} function +may create a class with the base class being \exception{Exception} +(unless another class is passed in instead of \NULL), described in the +\citetitle[../lib/lib.html]{Python Library Reference} under ``Built-in +Exceptions.'' + +Note also that the \cdata{SpamError} variable retains a reference to +the newly created exception class; this is intentional! Since the +exception could be removed from the module by external code, an owned +reference to the class is needed to ensure that it will not be +discarded, causing \cdata{SpamError} to become a dangling pointer. +Should it become a dangling pointer, C code which raises the exception +could cause a core dump or other unintended side effects. + + +\section{Back to the Example + \label{backToExample}} + +Going back to our example function, you should now be able to +understand this statement: + +\begin{verbatim} + if (!PyArg_ParseTuple(args, "s", &command)) + return NULL; +\end{verbatim} + +It returns \NULL{} (the error indicator for functions returning +object pointers) if an error is detected in the argument list, relying +on the exception set by \cfunction{PyArg_ParseTuple()}. Otherwise the +string value of the argument has been copied to the local variable +\cdata{command}. This is a pointer assignment and you are not supposed +to modify the string to which it points (so in Standard C, the variable +\cdata{command} should properly be declared as \samp{const char +*command}). + +The next statement is a call to the \UNIX{} function +\cfunction{system()}, passing it the string we just got from +\cfunction{PyArg_ParseTuple()}: + +\begin{verbatim} + sts = system(command); +\end{verbatim} + +Our \function{spam.system()} function must return the value of +\cdata{sts} as a Python object. This is done using the function +\cfunction{Py_BuildValue()}, which is something like the inverse of +\cfunction{PyArg_ParseTuple()}: it takes a format string and an +arbitrary number of C values, and returns a new Python object. +More info on \cfunction{Py_BuildValue()} is given later. + +\begin{verbatim} + return Py_BuildValue("i", sts); +\end{verbatim} + +In this case, it will return an integer object. (Yes, even integers +are objects on the heap in Python!) + +If you have a C function that returns no useful argument (a function +returning \ctype{void}), the corresponding Python function must return +\code{None}. You need this idiom to do so: + +\begin{verbatim} + Py_INCREF(Py_None); + return Py_None; +\end{verbatim} + +\cdata{Py_None} is the C name for the special Python object +\code{None}. It is a genuine Python object rather than a \NULL{} +pointer, which means ``error'' in most contexts, as we have seen. + + +\section{The Module's Method Table and Initialization Function + \label{methodTable}} + +I promised to show how \cfunction{spam_system()} is called from Python +programs. First, we need to list its name and address in a ``method +table'': + +\begin{verbatim} +static PyMethodDef SpamMethods[] = { + ... + {"system", spam_system, METH_VARARGS}, + ... + {NULL, NULL} /* Sentinel */ +}; +\end{verbatim} + +Note the third entry (\samp{METH_VARARGS}). This is a flag telling +the interpreter the calling convention to be used for the C +function. It should normally always be \samp{METH_VARARGS} or +\samp{METH_VARARGS | METH_KEYWORDS}; a value of \code{0} means that an +obsolete variant of \cfunction{PyArg_ParseTuple()} is used. + +When using only \samp{METH_VARARGS}, the function should expect +the Python-level parameters to be passed in as a tuple acceptable for +parsing via \cfunction{PyArg_ParseTuple()}; more information on this +function is provided below. + +The \constant{METH_KEYWORDS} bit may be set in the third field if +keyword arguments should be passed to the function. In this case, the +C function should accept a third \samp{PyObject *} parameter which +will be a dictionary of keywords. Use +\cfunction{PyArg_ParseTupleAndKeywords()} to parse the arguments to +such a function. + +The method table must be passed to the interpreter in the module's +initialization function. The initialization function must be named +\cfunction{init\var{name}()}, where \var{name} is the name of the +module, and should be the only non-\keyword{static} item defined in +the module file: + +\begin{verbatim} +void +initspam() +{ + (void) Py_InitModule("spam", SpamMethods); +} +\end{verbatim} + +Note that for \Cpp, this method must be declared \code{extern "C"}. + +When the Python program imports module \module{spam} for the first +time, \cfunction{initspam()} is called. (See below for comments about +embedding Python.) It calls +\cfunction{Py_InitModule()}, which creates a ``module object'' (which +is inserted in the dictionary \code{sys.modules} under the key +\code{"spam"}), and inserts built-in function objects into the newly +created module based upon the table (an array of \ctype{PyMethodDef} +structures) that was passed as its second argument. +\cfunction{Py_InitModule()} returns a pointer to the module object +that it creates (which is unused here). It aborts with a fatal error +if the module could not be initialized satisfactorily, so the caller +doesn't need to check for errors. + +When embedding Python, the \cfunction{initspam()} function is not +called automatically unless there's an entry in the +\cdata{_PyImport_Inittab} table. The easiest way to handle this is to +statically initialize your statically-linked modules by directly +calling \cfunction{initspam()} after the call to +\cfunction{Py_Initialize()} or \cfunction{PyMac_Initialize()}: + +\begin{verbatim} +int main(int argc, char **argv) +{ + /* Pass argv[0] to the Python interpreter */ + Py_SetProgramName(argv[0]); + + /* Initialize the Python interpreter. Required. */ + Py_Initialize(); + + /* Add a static module */ + initspam(); +\end{verbatim} + +An example may be found in the file \file{Demo/embed/demo.c} in the +Python source distribution. + +\strong{Note:} Removing entries from \code{sys.modules} or importing +compiled modules into multiple interpreters within a process (or +following a \cfunction{fork()} without an intervening +\cfunction{exec()}) can create problems for some extension modules. +Extension module authors should exercise caution when initializing +internal data structures. +Note also that the \function{reload()} function can be used with +extension modules, and will call the module initialization function +(\cfunction{initspam()} in the example), but will not load the module +again if it was loaded from a dynamically loadable object file +(\file{.so} on \UNIX, \file{.dll} on Windows). + +A more substantial example module is included in the Python source +distribution as \file{Modules/xxmodule.c}. This file may be used as a +template or simply read as an example. The \program{modulator.py} +script included in the source distribution or Windows install provides +a simple graphical user interface for declaring the functions and +objects which a module should implement, and can generate a template +which can be filled in. The script lives in the +\file{Tools/modulator/} directory; see the \file{README} file there +for more information. + + +\section{Compilation and Linkage + \label{compilation}} + +There are two more things to do before you can use your new extension: +compiling and linking it with the Python system. If you use dynamic +loading, the details depend on the style of dynamic loading your +system uses; see the chapters about building extension modules on +\UNIX{} (chapter \ref{building-on-unix}) and Windows (chapter +\ref{building-on-windows}) for more information about this. +% XXX Add information about MacOS + +If you can't use dynamic loading, or if you want to make your module a +permanent part of the Python interpreter, you will have to change the +configuration setup and rebuild the interpreter. Luckily, this is +very simple: just place your file (\file{spammodule.c} for example) in +the \file{Modules/} directory of an unpacked source distribution, add +a line to the file \file{Modules/Setup.local} describing your file: + +\begin{verbatim} +spam spammodule.o +\end{verbatim} + +and rebuild the interpreter by running \program{make} in the toplevel +directory. You can also run \program{make} in the \file{Modules/} +subdirectory, but then you must first rebuild \file{Makefile} +there by running `\program{make} Makefile'. (This is necessary each +time you change the \file{Setup} file.) + +If your module requires additional libraries to link with, these can +be listed on the line in the configuration file as well, for instance: + +\begin{verbatim} +spam spammodule.o -lX11 +\end{verbatim} + +\section{Calling Python Functions from C + \label{callingPython}} + +So far we have concentrated on making C functions callable from +Python. The reverse is also useful: calling Python functions from C. +This is especially the case for libraries that support so-called +``callback'' functions. If a C interface makes use of callbacks, the +equivalent Python often needs to provide a callback mechanism to the +Python programmer; the implementation will require calling the Python +callback functions from a C callback. Other uses are also imaginable. + +Fortunately, the Python interpreter is easily called recursively, and +there is a standard interface to call a Python function. (I won't +dwell on how to call the Python parser with a particular string as +input --- if you're interested, have a look at the implementation of +the \programopt{-c} command line option in \file{Python/pythonmain.c} +from the Python source code.) + +Calling a Python function is easy. First, the Python program must +somehow pass you the Python function object. You should provide a +function (or some other interface) to do this. When this function is +called, save a pointer to the Python function object (be careful to +\cfunction{Py_INCREF()} it!) in a global variable --- or wherever you +see fit. For example, the following function might be part of a module +definition: + +\begin{verbatim} +static PyObject *my_callback = NULL; + +static PyObject * +my_set_callback(dummy, args) + PyObject *dummy, *args; +{ + PyObject *result = NULL; + PyObject *temp; + + if (PyArg_ParseTuple(args, "O:set_callback", &temp)) { + if (!PyCallable_Check(temp)) { + PyErr_SetString(PyExc_TypeError, "parameter must be callable"); + return NULL; + } + Py_XINCREF(temp); /* Add a reference to new callback */ + Py_XDECREF(my_callback); /* Dispose of previous callback */ + my_callback = temp; /* Remember new callback */ + /* Boilerplate to return "None" */ + Py_INCREF(Py_None); + result = Py_None; + } + return result; +} +\end{verbatim} + +This function must be registered with the interpreter using the +\constant{METH_VARARGS} flag; this is described in section +\ref{methodTable}, ``The Module's Method Table and Initialization +Function.'' The \cfunction{PyArg_ParseTuple()} function and its +arguments are documented in section \ref{parseTuple}, ``Extracting +Parameters in Extension Functions.'' + +The macros \cfunction{Py_XINCREF()} and \cfunction{Py_XDECREF()} +increment/decrement the reference count of an object and are safe in +the presence of \NULL{} pointers (but note that \var{temp} will not be +\NULL{} in this context). More info on them in section +\ref{refcounts}, ``Reference Counts.'' + +Later, when it is time to call the function, you call the C function +\cfunction{PyEval_CallObject()}. This function has two arguments, both +pointers to arbitrary Python objects: the Python function, and the +argument list. The argument list must always be a tuple object, whose +length is the number of arguments. To call the Python function with +no arguments, pass an empty tuple; to call it with one argument, pass +a singleton tuple. \cfunction{Py_BuildValue()} returns a tuple when its +format string consists of zero or more format codes between +parentheses. For example: + +\begin{verbatim} + int arg; + PyObject *arglist; + PyObject *result; + ... + arg = 123; + ... + /* Time to call the callback */ + arglist = Py_BuildValue("(i)", arg); + result = PyEval_CallObject(my_callback, arglist); + Py_DECREF(arglist); +\end{verbatim} + +\cfunction{PyEval_CallObject()} returns a Python object pointer: this is +the return value of the Python function. \cfunction{PyEval_CallObject()} is +``reference-count-neutral'' with respect to its arguments. In the +example a new tuple was created to serve as the argument list, which +is \cfunction{Py_DECREF()}-ed immediately after the call. + +The return value of \cfunction{PyEval_CallObject()} is ``new'': either it +is a brand new object, or it is an existing object whose reference +count has been incremented. So, unless you want to save it in a +global variable, you should somehow \cfunction{Py_DECREF()} the result, +even (especially!) if you are not interested in its value. + +Before you do this, however, it is important to check that the return +value isn't \NULL{}. If it is, the Python function terminated by +raising an exception. If the C code that called +\cfunction{PyEval_CallObject()} is called from Python, it should now +return an error indication to its Python caller, so the interpreter +can print a stack trace, or the calling Python code can handle the +exception. If this is not possible or desirable, the exception should +be cleared by calling \cfunction{PyErr_Clear()}. For example: + +\begin{verbatim} + if (result == NULL) + return NULL; /* Pass error back */ + ...use result... + Py_DECREF(result); +\end{verbatim} + +Depending on the desired interface to the Python callback function, +you may also have to provide an argument list to +\cfunction{PyEval_CallObject()}. In some cases the argument list is +also provided by the Python program, through the same interface that +specified the callback function. It can then be saved and used in the +same manner as the function object. In other cases, you may have to +construct a new tuple to pass as the argument list. The simplest way +to do this is to call \cfunction{Py_BuildValue()}. For example, if +you want to pass an integral event code, you might use the following +code: + +\begin{verbatim} + PyObject *arglist; + ... + arglist = Py_BuildValue("(l)", eventcode); + result = PyEval_CallObject(my_callback, arglist); + Py_DECREF(arglist); + if (result == NULL) + return NULL; /* Pass error back */ + /* Here maybe use the result */ + Py_DECREF(result); +\end{verbatim} + +Note the placement of \samp{Py_DECREF(arglist)} immediately after the +call, before the error check! Also note that strictly spoken this +code is not complete: \cfunction{Py_BuildValue()} may run out of +memory, and this should be checked. + + +\section{Extracting Parameters in Extension Functions + \label{parseTuple}} + +The \cfunction{PyArg_ParseTuple()} function is declared as follows: + +\begin{verbatim} +int PyArg_ParseTuple(PyObject *arg, char *format, ...); +\end{verbatim} + +The \var{arg} argument must be a tuple object containing an argument +list passed from Python to a C function. The \var{format} argument +must be a format string, whose syntax is explained below. The +remaining arguments must be addresses of variables whose type is +determined by the format string. For the conversion to succeed, the +\var{arg} object must match the format and the format must be +exhausted. On success, \cfunction{PyArg_ParseTuple()} returns true, +otherwise it returns false and raises an appropriate exception. + +Note that while \cfunction{PyArg_ParseTuple()} checks that the Python +arguments have the required types, it cannot check the validity of the +addresses of C variables passed to the call: if you make mistakes +there, your code will probably crash or at least overwrite random bits +in memory. So be careful! + +A format string consists of zero or more ``format units''. A format +unit describes one Python object; it is usually a single character or +a parenthesized sequence of format units. With a few exceptions, a +format unit that is not a parenthesized sequence normally corresponds +to a single address argument to \cfunction{PyArg_ParseTuple()}. In the +following description, the quoted form is the format unit; the entry +in (round) parentheses is the Python object type that matches the +format unit; and the entry in [square] brackets is the type of the C +variable(s) whose address should be passed. (Use the \samp{\&} +operator to pass a variable's address.) + +Note that any Python object references which are provided to the +caller are \emph{borrowed} references; do not decrement their +reference count! + +\begin{description} + +\item[\samp{s} (string or Unicode object) {[char *]}] +Convert a Python string or Unicode object to a C pointer to a +character string. You must not provide storage for the string +itself; a pointer to an existing string is stored into the character +pointer variable whose address you pass. The C string is +null-terminated. The Python string must not contain embedded null +bytes; if it does, a \exception{TypeError} exception is raised. +Unicode objects are converted to C strings using the default +encoding. If this conversion fails, an \exception{UnicodeError} is +raised. + +\item[\samp{s\#} (string, Unicode or any read buffer compatible object) +{[char *, int]}] +This variant on \samp{s} stores into two C variables, the first one a +pointer to a character string, the second one its length. In this +case the Python string may contain embedded null bytes. Unicode +objects pass back a pointer to the default encoded string version of the +object if such a conversion is possible. All other read buffer +compatible objects pass back a reference to the raw internal data +representation. + +\item[\samp{z} (string or \code{None}) {[char *]}] +Like \samp{s}, but the Python object may also be \code{None}, in which +case the C pointer is set to \NULL{}. + +\item[\samp{z\#} (string or \code{None} or any read buffer compatible object) +{[char *, int]}] +This is to \samp{s\#} as \samp{z} is to \samp{s}. + +\item[\samp{u} (Unicode object) {[Py_UNICODE *]}] +Convert a Python Unicode object to a C pointer to a null-terminated +buffer of 16-bit Unicode (UTF-16) data. As with \samp{s}, there is no need +to provide storage for the Unicode data buffer; a pointer to the +existing Unicode data is stored into the Py_UNICODE pointer variable whose +address you pass. + +\item[\samp{u\#} (Unicode object) {[Py_UNICODE *, int]}] +This variant on \samp{u} stores into two C variables, the first one +a pointer to a Unicode data buffer, the second one its length. + +\item[\samp{es} (string, Unicode object or character buffer compatible +object) {[const char *encoding, char **buffer]}] +This variant on \samp{s} is used for encoding Unicode and objects +convertible to Unicode into a character buffer. It only works for +encoded data without embedded \NULL{} bytes. + +The variant reads one C variable and stores into two C variables, the +first one a pointer to an encoding name string (\var{encoding}), and the +second a pointer to a pointer to a character buffer (\var{**buffer}, +the buffer used for storing the encoded data). + +The encoding name must map to a registered codec. If set to \NULL{}, +the default encoding is used. + +\cfunction{PyArg_ParseTuple()} will allocate a buffer of the needed +size using \cfunction{PyMem_NEW()}, copy the encoded data into this +buffer and adjust \var{*buffer} to reference the newly allocated +storage. The caller is responsible for calling +\cfunction{PyMem_Free()} to free the allocated buffer after usage. + +\item[\samp{et} (string, Unicode object or character buffer compatible +object) {[const char *encoding, char **buffer]}] +Same as \samp{es} except that string objects are passed through without +recoding them. Instead, the implementation assumes that the string +object uses the encoding passed in as parameter. + +\item[\samp{es\#} (string, Unicode object or character buffer compatible +object) {[const char *encoding, char **buffer, int *buffer_length]}] +This variant on \samp{s\#} is used for encoding Unicode and objects +convertible to Unicode into a character buffer. It reads one C +variable and stores into three C variables, the first one a pointer to +an encoding name string (\var{encoding}), the second a pointer to a +pointer to a character buffer (\var{**buffer}, the buffer used for +storing the encoded data) and the third one a pointer to an integer +(\var{*buffer_length}, the buffer length). + +The encoding name must map to a registered codec. If set to \NULL{}, +the default encoding is used. + +There are two modes of operation: + +If \var{*buffer} points a \NULL{} pointer, +\cfunction{PyArg_ParseTuple()} will allocate a buffer of the needed +size using \cfunction{PyMem_NEW()}, copy the encoded data into this +buffer and adjust \var{*buffer} to reference the newly allocated +storage. The caller is responsible for calling +\cfunction{PyMem_Free()} to free the allocated buffer after usage. + +If \var{*buffer} points to a non-\NULL{} pointer (an already allocated +buffer), \cfunction{PyArg_ParseTuple()} will use this location as +buffer and interpret \var{*buffer_length} as buffer size. It will then +copy the encoded data into the buffer and 0-terminate it. Buffer +overflow is signalled with an exception. + +In both cases, \var{*buffer_length} is set to the length of the +encoded data without the trailing 0-byte. + +\item[\samp{et\#} (string, Unicode object or character buffer compatible +object) {[const char *encoding, char **buffer]}] +Same as \samp{es\#} except that string objects are passed through without +recoding them. Instead, the implementation assumes that the string +object uses the encoding passed in as parameter. + +\item[\samp{b} (integer) {[char]}] +Convert a Python integer to a tiny int, stored in a C \ctype{char}. + +\item[\samp{h} (integer) {[short int]}] +Convert a Python integer to a C \ctype{short int}. + +\item[\samp{i} (integer) {[int]}] +Convert a Python integer to a plain C \ctype{int}. + +\item[\samp{l} (integer) {[long int]}] +Convert a Python integer to a C \ctype{long int}. + +\item[\samp{c} (string of length 1) {[char]}] +Convert a Python character, represented as a string of length 1, to a +C \ctype{char}. + +\item[\samp{f} (float) {[float]}] +Convert a Python floating point number to a C \ctype{float}. + +\item[\samp{d} (float) {[double]}] +Convert a Python floating point number to a C \ctype{double}. + +\item[\samp{D} (complex) {[Py_complex]}] +Convert a Python complex number to a C \ctype{Py_complex} structure. + +\item[\samp{O} (object) {[PyObject *]}] +Store a Python object (without any conversion) in a C object pointer. +The C program thus receives the actual object that was passed. The +object's reference count is not increased. The pointer stored is not +\NULL{}. + +\item[\samp{O!} (object) {[\var{typeobject}, PyObject *]}] +Store a Python object in a C object pointer. This is similar to +\samp{O}, but takes two C arguments: the first is the address of a +Python type object, the second is the address of the C variable (of +type \ctype{PyObject *}) into which the object pointer is stored. +If the Python object does not have the required type, +\exception{TypeError} is raised. + +\item[\samp{O\&} (object) {[\var{converter}, \var{anything}]}] +Convert a Python object to a C variable through a \var{converter} +function. This takes two arguments: the first is a function, the +second is the address of a C variable (of arbitrary type), converted +to \ctype{void *}. The \var{converter} function in turn is called as +follows: + +\var{status}\code{ = }\var{converter}\code{(}\var{object}, \var{address}\code{);} + +where \var{object} is the Python object to be converted and +\var{address} is the \ctype{void *} argument that was passed to +\cfunction{PyArg_ConvertTuple()}. The returned \var{status} should be +\code{1} for a successful conversion and \code{0} if the conversion +has failed. When the conversion fails, the \var{converter} function +should raise an exception. + +\item[\samp{S} (string) {[PyStringObject *]}] +Like \samp{O} but requires that the Python object is a string object. +Raises \exception{TypeError} if the object is not a string object. +The C variable may also be declared as \ctype{PyObject *}. + +\item[\samp{U} (Unicode string) {[PyUnicodeObject *]}] +Like \samp{O} but requires that the Python object is a Unicode object. +Raises \exception{TypeError} if the object is not a Unicode object. +The C variable may also be declared as \ctype{PyObject *}. + +\item[\samp{t\#} (read-only character buffer) {[char *, int]}] +Like \samp{s\#}, but accepts any object which implements the read-only +buffer interface. The \ctype{char *} variable is set to point to the +first byte of the buffer, and the \ctype{int} is set to the length of +the buffer. Only single-segment buffer objects are accepted; +\exception{TypeError} is raised for all others. + +\item[\samp{w} (read-write character buffer) {[char *]}] +Similar to \samp{s}, but accepts any object which implements the +read-write buffer interface. The caller must determine the length of +the buffer by other means, or use \samp{w\#} instead. Only +single-segment buffer objects are accepted; \exception{TypeError} is +raised for all others. + +\item[\samp{w\#} (read-write character buffer) {[char *, int]}] +Like \samp{s\#}, but accepts any object which implements the +read-write buffer interface. The \ctype{char *} variable is set to +point to the first byte of the buffer, and the \ctype{int} is set to +the length of the buffer. Only single-segment buffer objects are +accepted; \exception{TypeError} is raised for all others. + +\item[\samp{(\var{items})} (tuple) {[\var{matching-items}]}] +The object must be a Python sequence whose length is the number of +format units in \var{items}. The C arguments must correspond to the +individual format units in \var{items}. Format units for sequences +may be nested. + +\strong{Note:} Prior to Python version 1.5.2, this format specifier +only accepted a tuple containing the individual parameters, not an +arbitrary sequence. Code which previously caused +\exception{TypeError} to be raised here may now proceed without an +exception. This is not expected to be a problem for existing code. + +\end{description} + +It is possible to pass Python long integers where integers are +requested; however no proper range checking is done --- the most +significant bits are silently truncated when the receiving field is +too small to receive the value (actually, the semantics are inherited +from downcasts in C --- your mileage may vary). + +A few other characters have a meaning in a format string. These may +not occur inside nested parentheses. They are: + +\begin{description} + +\item[\samp{|}] +Indicates that the remaining arguments in the Python argument list are +optional. The C variables corresponding to optional arguments should +be initialized to their default value --- when an optional argument is +not specified, \cfunction{PyArg_ParseTuple()} does not touch the contents +of the corresponding C variable(s). + +\item[\samp{:}] +The list of format units ends here; the string after the colon is used +as the function name in error messages (the ``associated value'' of +the exception that \cfunction{PyArg_ParseTuple()} raises). + +\item[\samp{;}] +The list of format units ends here; the string after the semicolon is +used as the error message \emph{instead} of the default error message. +Clearly, \samp{:} and \samp{;} mutually exclude each other. + +\end{description} + +Some example calls: + +\begin{verbatim} + int ok; + int i, j; + long k, l; + char *s; + int size; + + ok = PyArg_ParseTuple(args, ""); /* No arguments */ + /* Python call: f() */ +\end{verbatim} + +\begin{verbatim} + ok = PyArg_ParseTuple(args, "s", &s); /* A string */ + /* Possible Python call: f('whoops!') */ +\end{verbatim} + +\begin{verbatim} + ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Two longs and a string */ + /* Possible Python call: f(1, 2, 'three') */ +\end{verbatim} + +\begin{verbatim} + ok = PyArg_ParseTuple(args, "(ii)s#", &i, &j, &s, &size); + /* A pair of ints and a string, whose size is also returned */ + /* Possible Python call: f((1, 2), 'three') */ +\end{verbatim} + +\begin{verbatim} + { + char *file; + char *mode = "r"; + int bufsize = 0; + ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize); + /* A string, and optionally another string and an integer */ + /* Possible Python calls: + f('spam') + f('spam', 'w') + f('spam', 'wb', 100000) */ + } +\end{verbatim} + +\begin{verbatim} + { + int left, top, right, bottom, h, v; + ok = PyArg_ParseTuple(args, "((ii)(ii))(ii)", + &left, &top, &right, &bottom, &h, &v); + /* A rectangle and a point */ + /* Possible Python call: + f(((0, 0), (400, 300)), (10, 10)) */ + } +\end{verbatim} + +\begin{verbatim} + { + Py_complex c; + ok = PyArg_ParseTuple(args, "D:myfunction", &c); + /* a complex, also providing a function name for errors */ + /* Possible Python call: myfunction(1+2j) */ + } +\end{verbatim} + + +\section{Keyword Parameters for Extension Functions + \label{parseTupleAndKeywords}} + +The \cfunction{PyArg_ParseTupleAndKeywords()} function is declared as +follows: + +\begin{verbatim} +int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict, + char *format, char **kwlist, ...); +\end{verbatim} + +The \var{arg} and \var{format} parameters are identical to those of the +\cfunction{PyArg_ParseTuple()} function. The \var{kwdict} parameter +is the dictionary of keywords received as the third parameter from the +Python runtime. The \var{kwlist} parameter is a \NULL{}-terminated +list of strings which identify the parameters; the names are matched +with the type information from \var{format} from left to right. On +success, \cfunction{PyArg_ParseTupleAndKeywords()} returns true, +otherwise it returns false and raises an appropriate exception. + +\strong{Note:} Nested tuples cannot be parsed when using keyword +arguments! Keyword parameters passed in which are not present in the +\var{kwlist} will cause \exception{TypeError} to be raised. + +Here is an example module which uses keywords, based on an example by +Geoff Philbrick (\email{philbrick@hks.com}):% +\index{Philbrick, Geoff} + +\begin{verbatim} +#include <stdio.h> +#include "Python.h" + +static PyObject * +keywdarg_parrot(self, args, keywds) + PyObject *self; + PyObject *args; + PyObject *keywds; +{ + int voltage; + char *state = "a stiff"; + char *action = "voom"; + char *type = "Norwegian Blue"; + + static char *kwlist[] = {"voltage", "state", "action", "type", NULL}; + + if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist, + &voltage, &state, &action, &type)) + return NULL; + + printf("-- This parrot wouldn't %s if you put %i Volts through it.\n", + action, voltage); + printf("-- Lovely plumage, the %s -- It's %s!\n", type, state); + + Py_INCREF(Py_None); + + return Py_None; +} + +static PyMethodDef keywdarg_methods[] = { + /* The cast of the function is necessary since PyCFunction values + * only take two PyObject* parameters, and keywdarg_parrot() takes + * three. + */ + {"parrot", (PyCFunction)keywdarg_parrot, METH_VARARGS|METH_KEYWORDS}, + {NULL, NULL} /* sentinel */ +}; + +void +initkeywdarg() +{ + /* Create the module and add the functions */ + Py_InitModule("keywdarg", keywdarg_methods); +} +\end{verbatim} + + +\section{Building Arbitrary Values + \label{buildValue}} + +This function is the counterpart to \cfunction{PyArg_ParseTuple()}. It is +declared as follows: + +\begin{verbatim} +PyObject *Py_BuildValue(char *format, ...); +\end{verbatim} + +It recognizes a set of format units similar to the ones recognized by +\cfunction{PyArg_ParseTuple()}, but the arguments (which are input to the +function, not output) must not be pointers, just values. It returns a +new Python object, suitable for returning from a C function called +from Python. + +One difference with \cfunction{PyArg_ParseTuple()}: while the latter +requires its first argument to be a tuple (since Python argument lists +are always represented as tuples internally), +\cfunction{Py_BuildValue()} does not always build a tuple. It builds +a tuple only if its format string contains two or more format units. +If the format string is empty, it returns \code{None}; if it contains +exactly one format unit, it returns whatever object is described by +that format unit. To force it to return a tuple of size 0 or one, +parenthesize the format string. + +When memory buffers are passed as parameters to supply data to build +objects, as for the \samp{s} and \samp{s\#} formats, the required data +is copied. Buffers provided by the caller are never referenced by the +objects created by \cfunction{Py_BuildValue()}. In other words, if +your code invokes \cfunction{malloc()} and passes the allocated memory +to \cfunction{Py_BuildValue()}, your code is responsible for +calling \cfunction{free()} for that memory once +\cfunction{Py_BuildValue()} returns. + +In the following description, the quoted form is the format unit; the +entry in (round) parentheses is the Python object type that the format +unit will return; and the entry in [square] brackets is the type of +the C value(s) to be passed. + +The characters space, tab, colon and comma are ignored in format +strings (but not within format units such as \samp{s\#}). This can be +used to make long format strings a tad more readable. + +\begin{description} + +\item[\samp{s} (string) {[char *]}] +Convert a null-terminated C string to a Python object. If the C +string pointer is \NULL{}, \code{None} is used. + +\item[\samp{s\#} (string) {[char *, int]}] +Convert a C string and its length to a Python object. If the C string +pointer is \NULL{}, the length is ignored and \code{None} is +returned. + +\item[\samp{z} (string or \code{None}) {[char *]}] +Same as \samp{s}. + +\item[\samp{z\#} (string or \code{None}) {[char *, int]}] +Same as \samp{s\#}. + +\item[\samp{u} (Unicode string) {[Py_UNICODE *]}] +Convert a null-terminated buffer of Unicode (UCS-2) data to a Python +Unicode object. If the Unicode buffer pointer is \NULL, +\code{None} is returned. + +\item[\samp{u\#} (Unicode string) {[Py_UNICODE *, int]}] +Convert a Unicode (UCS-2) data buffer and its length to a Python +Unicode object. If the Unicode buffer pointer is \NULL, the length +is ignored and \code{None} is returned. + +\item[\samp{i} (integer) {[int]}] +Convert a plain C \ctype{int} to a Python integer object. + +\item[\samp{b} (integer) {[char]}] +Same as \samp{i}. + +\item[\samp{h} (integer) {[short int]}] +Same as \samp{i}. + +\item[\samp{l} (integer) {[long int]}] +Convert a C \ctype{long int} to a Python integer object. + +\item[\samp{c} (string of length 1) {[char]}] +Convert a C \ctype{int} representing a character to a Python string of +length 1. + +\item[\samp{d} (float) {[double]}] +Convert a C \ctype{double} to a Python floating point number. + +\item[\samp{f} (float) {[float]}] +Same as \samp{d}. + +\item[\samp{D} (complex) {[Py_complex *]}] +Convert a C \ctype{Py_complex} structure to a Python complex number. + +\item[\samp{O} (object) {[PyObject *]}] +Pass a Python object untouched (except for its reference count, which +is incremented by one). If the object passed in is a \NULL{} +pointer, it is assumed that this was caused because the call producing +the argument found an error and set an exception. Therefore, +\cfunction{Py_BuildValue()} will return \NULL{} but won't raise an +exception. If no exception has been raised yet, +\cdata{PyExc_SystemError} is set. + +\item[\samp{S} (object) {[PyObject *]}] +Same as \samp{O}. + +\item[\samp{U} (object) {[PyObject *]}] +Same as \samp{O}. + +\item[\samp{N} (object) {[PyObject *]}] +Same as \samp{O}, except it doesn't increment the reference count on +the object. Useful when the object is created by a call to an object +constructor in the argument list. + +\item[\samp{O\&} (object) {[\var{converter}, \var{anything}]}] +Convert \var{anything} to a Python object through a \var{converter} +function. The function is called with \var{anything} (which should be +compatible with \ctype{void *}) as its argument and should return a +``new'' Python object, or \NULL{} if an error occurred. + +\item[\samp{(\var{items})} (tuple) {[\var{matching-items}]}] +Convert a sequence of C values to a Python tuple with the same number +of items. + +\item[\samp{[\var{items}]} (list) {[\var{matching-items}]}] +Convert a sequence of C values to a Python list with the same number +of items. + +\item[\samp{\{\var{items}\}} (dictionary) {[\var{matching-items}]}] +Convert a sequence of C values to a Python dictionary. Each pair of +consecutive C values adds one item to the dictionary, serving as key +and value, respectively. + +\end{description} + +If there is an error in the format string, the +\cdata{PyExc_SystemError} exception is raised and \NULL{} returned. + +Examples (to the left the call, to the right the resulting Python value): + +\begin{verbatim} + Py_BuildValue("") None + Py_BuildValue("i", 123) 123 + Py_BuildValue("iii", 123, 456, 789) (123, 456, 789) + Py_BuildValue("s", "hello") 'hello' + Py_BuildValue("ss", "hello", "world") ('hello', 'world') + Py_BuildValue("s#", "hello", 4) 'hell' + Py_BuildValue("()") () + Py_BuildValue("(i)", 123) (123,) + Py_BuildValue("(ii)", 123, 456) (123, 456) + Py_BuildValue("(i,i)", 123, 456) (123, 456) + Py_BuildValue("[i,i]", 123, 456) [123, 456] + Py_BuildValue("{s:i,s:i}", + "abc", 123, "def", 456) {'abc': 123, 'def': 456} + Py_BuildValue("((ii)(ii)) (ii)", + 1, 2, 3, 4, 5, 6) (((1, 2), (3, 4)), (5, 6)) +\end{verbatim} + + +\section{Reference Counts + \label{refcounts}} + +In languages like C or \Cpp{}, the programmer is responsible for +dynamic allocation and deallocation of memory on the heap. In C, +this is done using the functions \cfunction{malloc()} and +\cfunction{free()}. In \Cpp{}, the operators \keyword{new} and +\keyword{delete} are used with essentially the same meaning; they are +actually implemented using \cfunction{malloc()} and +\cfunction{free()}, so we'll restrict the following discussion to the +latter. + +Every block of memory allocated with \cfunction{malloc()} should +eventually be returned to the pool of available memory by exactly one +call to \cfunction{free()}. It is important to call +\cfunction{free()} at the right time. If a block's address is +forgotten but \cfunction{free()} is not called for it, the memory it +occupies cannot be reused until the program terminates. This is +called a \dfn{memory leak}. On the other hand, if a program calls +\cfunction{free()} for a block and then continues to use the block, it +creates a conflict with re-use of the block through another +\cfunction{malloc()} call. This is called \dfn{using freed memory}. +It has the same bad consequences as referencing uninitialized data --- +core dumps, wrong results, mysterious crashes. + +Common causes of memory leaks are unusual paths through the code. For +instance, a function may allocate a block of memory, do some +calculation, and then free the block again. Now a change in the +requirements for the function may add a test to the calculation that +detects an error condition and can return prematurely from the +function. It's easy to forget to free the allocated memory block when +taking this premature exit, especially when it is added later to the +code. Such leaks, once introduced, often go undetected for a long +time: the error exit is taken only in a small fraction of all calls, +and most modern machines have plenty of virtual memory, so the leak +only becomes apparent in a long-running process that uses the leaking +function frequently. Therefore, it's important to prevent leaks from +happening by having a coding convention or strategy that minimizes +this kind of errors. + +Since Python makes heavy use of \cfunction{malloc()} and +\cfunction{free()}, it needs a strategy to avoid memory leaks as well +as the use of freed memory. The chosen method is called +\dfn{reference counting}. The principle is simple: every object +contains a counter, which is incremented when a reference to the +object is stored somewhere, and which is decremented when a reference +to it is deleted. When the counter reaches zero, the last reference +to the object has been deleted and the object is freed. + +An alternative strategy is called \dfn{automatic garbage collection}. +(Sometimes, reference counting is also referred to as a garbage +collection strategy, hence my use of ``automatic'' to distinguish the +two.) The big advantage of automatic garbage collection is that the +user doesn't need to call \cfunction{free()} explicitly. (Another claimed +advantage is an improvement in speed or memory usage --- this is no +hard fact however.) The disadvantage is that for C, there is no +truly portable automatic garbage collector, while reference counting +can be implemented portably (as long as the functions \cfunction{malloc()} +and \cfunction{free()} are available --- which the C Standard guarantees). +Maybe some day a sufficiently portable automatic garbage collector +will be available for C. Until then, we'll have to live with +reference counts. + +\subsection{Reference Counting in Python + \label{refcountsInPython}} + +There are two macros, \code{Py_INCREF(x)} and \code{Py_DECREF(x)}, +which handle the incrementing and decrementing of the reference count. +\cfunction{Py_DECREF()} also frees the object when the count reaches zero. +For flexibility, it doesn't call \cfunction{free()} directly --- rather, it +makes a call through a function pointer in the object's \dfn{type +object}. For this purpose (and others), every object also contains a +pointer to its type object. + +The big question now remains: when to use \code{Py_INCREF(x)} and +\code{Py_DECREF(x)}? Let's first introduce some terms. Nobody +``owns'' an object; however, you can \dfn{own a reference} to an +object. An object's reference count is now defined as the number of +owned references to it. The owner of a reference is responsible for +calling \cfunction{Py_DECREF()} when the reference is no longer +needed. Ownership of a reference can be transferred. There are three +ways to dispose of an owned reference: pass it on, store it, or call +\cfunction{Py_DECREF()}. Forgetting to dispose of an owned reference +creates a memory leak. + +It is also possible to \dfn{borrow}\footnote{The metaphor of +``borrowing'' a reference is not completely correct: the owner still +has a copy of the reference.} a reference to an object. The borrower +of a reference should not call \cfunction{Py_DECREF()}. The borrower must +not hold on to the object longer than the owner from which it was +borrowed. Using a borrowed reference after the owner has disposed of +it risks using freed memory and should be avoided +completely.\footnote{Checking that the reference count is at least 1 +\strong{does not work} --- the reference count itself could be in +freed memory and may thus be reused for another object!} + +The advantage of borrowing over owning a reference is that you don't +need to take care of disposing of the reference on all possible paths +through the code --- in other words, with a borrowed reference you +don't run the risk of leaking when a premature exit is taken. The +disadvantage of borrowing over leaking is that there are some subtle +situations where in seemingly correct code a borrowed reference can be +used after the owner from which it was borrowed has in fact disposed +of it. + +A borrowed reference can be changed into an owned reference by calling +\cfunction{Py_INCREF()}. This does not affect the status of the owner from +which the reference was borrowed --- it creates a new owned reference, +and gives full owner responsibilities (the new owner must +dispose of the reference properly, as well as the previous owner). + + +\subsection{Ownership Rules + \label{ownershipRules}} + +Whenever an object reference is passed into or out of a function, it +is part of the function's interface specification whether ownership is +transferred with the reference or not. + +Most functions that return a reference to an object pass on ownership +with the reference. In particular, all functions whose function it is +to create a new object, such as \cfunction{PyInt_FromLong()} and +\cfunction{Py_BuildValue()}, pass ownership to the receiver. Even if in +fact, in some cases, you don't receive a reference to a brand new +object, you still receive ownership of the reference. For instance, +\cfunction{PyInt_FromLong()} maintains a cache of popular values and can +return a reference to a cached item. + +Many functions that extract objects from other objects also transfer +ownership with the reference, for instance +\cfunction{PyObject_GetAttrString()}. The picture is less clear, here, +however, since a few common routines are exceptions: +\cfunction{PyTuple_GetItem()}, \cfunction{PyList_GetItem()}, +\cfunction{PyDict_GetItem()}, and \cfunction{PyDict_GetItemString()} +all return references that you borrow from the tuple, list or +dictionary. + +The function \cfunction{PyImport_AddModule()} also returns a borrowed +reference, even though it may actually create the object it returns: +this is possible because an owned reference to the object is stored in +\code{sys.modules}. + +When you pass an object reference into another function, in general, +the function borrows the reference from you --- if it needs to store +it, it will use \cfunction{Py_INCREF()} to become an independent +owner. There are exactly two important exceptions to this rule: +\cfunction{PyTuple_SetItem()} and \cfunction{PyList_SetItem()}. These +functions take over ownership of the item passed to them --- even if +they fail! (Note that \cfunction{PyDict_SetItem()} and friends don't +take over ownership --- they are ``normal.'') + +When a C function is called from Python, it borrows references to its +arguments from the caller. The caller owns a reference to the object, +so the borrowed reference's lifetime is guaranteed until the function +returns. Only when such a borrowed reference must be stored or passed +on, it must be turned into an owned reference by calling +\cfunction{Py_INCREF()}. + +The object reference returned from a C function that is called from +Python must be an owned reference --- ownership is tranferred from the +function to its caller. + + +\subsection{Thin Ice + \label{thinIce}} + +There are a few situations where seemingly harmless use of a borrowed +reference can lead to problems. These all have to do with implicit +invocations of the interpreter, which can cause the owner of a +reference to dispose of it. + +The first and most important case to know about is using +\cfunction{Py_DECREF()} on an unrelated object while borrowing a +reference to a list item. For instance: + +\begin{verbatim} +bug(PyObject *list) { + PyObject *item = PyList_GetItem(list, 0); + + PyList_SetItem(list, 1, PyInt_FromLong(0L)); + PyObject_Print(item, stdout, 0); /* BUG! */ +} +\end{verbatim} + +This function first borrows a reference to \code{list[0]}, then +replaces \code{list[1]} with the value \code{0}, and finally prints +the borrowed reference. Looks harmless, right? But it's not! + +Let's follow the control flow into \cfunction{PyList_SetItem()}. The list +owns references to all its items, so when item 1 is replaced, it has +to dispose of the original item 1. Now let's suppose the original +item 1 was an instance of a user-defined class, and let's further +suppose that the class defined a \method{__del__()} method. If this +class instance has a reference count of 1, disposing of it will call +its \method{__del__()} method. + +Since it is written in Python, the \method{__del__()} method can execute +arbitrary Python code. Could it perhaps do something to invalidate +the reference to \code{item} in \cfunction{bug()}? You bet! Assuming +that the list passed into \cfunction{bug()} is accessible to the +\method{__del__()} method, it could execute a statement to the effect of +\samp{del list[0]}, and assuming this was the last reference to that +object, it would free the memory associated with it, thereby +invalidating \code{item}. + +The solution, once you know the source of the problem, is easy: +temporarily increment the reference count. The correct version of the +function reads: + +\begin{verbatim} +no_bug(PyObject *list) { + PyObject *item = PyList_GetItem(list, 0); + + Py_INCREF(item); + PyList_SetItem(list, 1, PyInt_FromLong(0L)); + PyObject_Print(item, stdout, 0); + Py_DECREF(item); +} +\end{verbatim} + +This is a true story. An older version of Python contained variants +of this bug and someone spent a considerable amount of time in a C +debugger to figure out why his \method{__del__()} methods would fail... + +The second case of problems with a borrowed reference is a variant +involving threads. Normally, multiple threads in the Python +interpreter can't get in each other's way, because there is a global +lock protecting Python's entire object space. However, it is possible +to temporarily release this lock using the macro +\code{Py_BEGIN_ALLOW_THREADS}, and to re-acquire it using +\code{Py_END_ALLOW_THREADS}. This is common around blocking I/O +calls, to let other threads use the processor while waiting for the I/O to +complete. Obviously, the following function has the same problem as +the previous one: + +\begin{verbatim} +bug(PyObject *list) { + PyObject *item = PyList_GetItem(list, 0); + Py_BEGIN_ALLOW_THREADS + ...some blocking I/O call... + Py_END_ALLOW_THREADS + PyObject_Print(item, stdout, 0); /* BUG! */ +} +\end{verbatim} + + +\subsection{NULL Pointers + \label{nullPointers}} + +In general, functions that take object references as arguments do not +expect you to pass them \NULL{} pointers, and will dump core (or +cause later core dumps) if you do so. Functions that return object +references generally return \NULL{} only to indicate that an +exception occurred. The reason for not testing for \NULL{} +arguments is that functions often pass the objects they receive on to +other function --- if each function were to test for \NULL{}, +there would be a lot of redundant tests and the code would run more +slowly. + +It is better to test for \NULL{} only at the ``source:'' when a +pointer that may be \NULL{} is received, for example, from +\cfunction{malloc()} or from a function that may raise an exception. + +The macros \cfunction{Py_INCREF()} and \cfunction{Py_DECREF()} +do not check for \NULL{} pointers --- however, their variants +\cfunction{Py_XINCREF()} and \cfunction{Py_XDECREF()} do. + +The macros for checking for a particular object type +(\code{Py\var{type}_Check()}) don't check for \NULL{} pointers --- +again, there is much code that calls several of these in a row to test +an object against various different expected types, and this would +generate redundant tests. There are no variants with \NULL{} +checking. + +The C function calling mechanism guarantees that the argument list +passed to C functions (\code{args} in the examples) is never +\NULL{} --- in fact it guarantees that it is always a tuple.\footnote{ +These guarantees don't hold when you use the ``old'' style +calling convention --- this is still found in much existing code.} + +It is a severe error to ever let a \NULL{} pointer ``escape'' to +the Python user. + +% Frank Stajano: +% A pedagogically buggy example, along the lines of the previous listing, +% would be helpful here -- showing in more concrete terms what sort of +% actions could cause the problem. I can't very well imagine it from the +% description. + + +\section{Writing Extensions in \Cpp{} + \label{cplusplus}} + +It is possible to write extension modules in \Cpp{}. Some restrictions +apply. If the main program (the Python interpreter) is compiled and +linked by the C compiler, global or static objects with constructors +cannot be used. This is not a problem if the main program is linked +by the \Cpp{} compiler. Functions that will be called by the +Python interpreter (in particular, module initalization functions) +have to be declared using \code{extern "C"}. +It is unnecessary to enclose the Python header files in +\code{extern "C" \{...\}} --- they use this form already if the symbol +\samp{__cplusplus} is defined (all recent \Cpp{} compilers define this +symbol). + + +\section{Providing a C API for an Extension Module + \label{using-cobjects}} +\sectionauthor{Konrad Hinsen}{hinsen@cnrs-orleans.fr} + +Many extension modules just provide new functions and types to be +used from Python, but sometimes the code in an extension module can +be useful for other extension modules. For example, an extension +module could implement a type ``collection'' which works like lists +without order. Just like the standard Python list type has a C API +which permits extension modules to create and manipulate lists, this +new collection type should have a set of C functions for direct +manipulation from other extension modules. + +At first sight this seems easy: just write the functions (without +declaring them \keyword{static}, of course), provide an appropriate +header file, and document the C API. And in fact this would work if +all extension modules were always linked statically with the Python +interpreter. When modules are used as shared libraries, however, the +symbols defined in one module may not be visible to another module. +The details of visibility depend on the operating system; some systems +use one global namespace for the Python interpreter and all extension +modules (Windows, for example), whereas others require an explicit +list of imported symbols at module link time (AIX is one example), or +offer a choice of different strategies (most Unices). And even if +symbols are globally visible, the module whose functions one wishes to +call might not have been loaded yet! + +Portability therefore requires not to make any assumptions about +symbol visibility. This means that all symbols in extension modules +should be declared \keyword{static}, except for the module's +initialization function, in order to avoid name clashes with other +extension modules (as discussed in section~\ref{methodTable}). And it +means that symbols that \emph{should} be accessible from other +extension modules must be exported in a different way. + +Python provides a special mechanism to pass C-level information +(pointers) from one extension module to another one: CObjects. +A CObject is a Python data type which stores a pointer (\ctype{void +*}). CObjects can only be created and accessed via their C API, but +they can be passed around like any other Python object. In particular, +they can be assigned to a name in an extension module's namespace. +Other extension modules can then import this module, retrieve the +value of this name, and then retrieve the pointer from the CObject. + +There are many ways in which CObjects can be used to export the C API +of an extension module. Each name could get its own CObject, or all C +API pointers could be stored in an array whose address is published in +a CObject. And the various tasks of storing and retrieving the pointers +can be distributed in different ways between the module providing the +code and the client modules. + +The following example demonstrates an approach that puts most of the +burden on the writer of the exporting module, which is appropriate +for commonly used library modules. It stores all C API pointers +(just one in the example!) in an array of \ctype{void} pointers which +becomes the value of a CObject. The header file corresponding to +the module provides a macro that takes care of importing the module +and retrieving its C API pointers; client modules only have to call +this macro before accessing the C API. + +The exporting module is a modification of the \module{spam} module from +section~\ref{simpleExample}. The function \function{spam.system()} +does not call the C library function \cfunction{system()} directly, +but a function \cfunction{PySpam_System()}, which would of course do +something more complicated in reality (such as adding ``spam'' to +every command). This function \cfunction{PySpam_System()} is also +exported to other extension modules. + +The function \cfunction{PySpam_System()} is a plain C function, +declared \keyword{static} like everything else: + +\begin{verbatim} +static int +PySpam_System(command) + char *command; +{ + return system(command); +} +\end{verbatim} + +The function \cfunction{spam_system()} is modified in a trivial way: + +\begin{verbatim} +static PyObject * +spam_system(self, args) + PyObject *self; + PyObject *args; +{ + char *command; + int sts; + + if (!PyArg_ParseTuple(args, "s", &command)) + return NULL; + sts = PySpam_System(command); + return Py_BuildValue("i", sts); +} +\end{verbatim} + +In the beginning of the module, right after the line + +\begin{verbatim} +#include "Python.h" +\end{verbatim} + +two more lines must be added: + +\begin{verbatim} +#define SPAM_MODULE +#include "spammodule.h" +\end{verbatim} + +The \code{\#define} is used to tell the header file that it is being +included in the exporting module, not a client module. Finally, +the module's initialization function must take care of initializing +the C API pointer array: + +\begin{verbatim} +void +initspam() +{ + PyObject *m; + static void *PySpam_API[PySpam_API_pointers]; + PyObject *c_api_object; + + m = Py_InitModule("spam", SpamMethods); + + /* Initialize the C API pointer array */ + PySpam_API[PySpam_System_NUM] = (void *)PySpam_System; + + /* Create a CObject containing the API pointer array's address */ + c_api_object = PyCObject_FromVoidPtr((void *)PySpam_API, NULL); + + if (c_api_object != NULL) { + /* Create a name for this object in the module's namespace */ + PyObject *d = PyModule_GetDict(m); + + PyDict_SetItemString(d, "_C_API", c_api_object); + Py_DECREF(c_api_object); + } +} +\end{verbatim} + +Note that \code{PySpam_API} is declared \code{static}; otherwise +the pointer array would disappear when \code{initspam} terminates! + +The bulk of the work is in the header file \file{spammodule.h}, +which looks like this: + +\begin{verbatim} +#ifndef Py_SPAMMODULE_H +#define Py_SPAMMODULE_H +#ifdef __cplusplus +extern "C" { +#endif + +/* Header file for spammodule */ + +/* C API functions */ +#define PySpam_System_NUM 0 +#define PySpam_System_RETURN int +#define PySpam_System_PROTO (char *command) + +/* Total number of C API pointers */ +#define PySpam_API_pointers 1 + + +#ifdef SPAM_MODULE +/* This section is used when compiling spammodule.c */ + +static PySpam_System_RETURN PySpam_System PySpam_System_PROTO; + +#else +/* This section is used in modules that use spammodule's API */ + +static void **PySpam_API; + +#define PySpam_System \ + (*(PySpam_System_RETURN (*)PySpam_System_PROTO) PySpam_API[PySpam_System_NUM]) + +#define import_spam() \ +{ \ + PyObject *module = PyImport_ImportModule("spam"); \ + if (module != NULL) { \ + PyObject *module_dict = PyModule_GetDict(module); \ + PyObject *c_api_object = PyDict_GetItemString(module_dict, "_C_API"); \ + if (PyCObject_Check(c_api_object)) { \ + PySpam_API = (void **)PyCObject_AsVoidPtr(c_api_object); \ + } \ + } \ +} + +#endif + +#ifdef __cplusplus +} +#endif + +#endif /* !defined(Py_SPAMMODULE_H */ +\end{verbatim} + +All that a client module must do in order to have access to the +function \cfunction{PySpam_System()} is to call the function (or +rather macro) \cfunction{import_spam()} in its initialization +function: + +\begin{verbatim} +void +initclient() +{ + PyObject *m; + + Py_InitModule("client", ClientMethods); + import_spam(); +} +\end{verbatim} + +The main disadvantage of this approach is that the file +\file{spammodule.h} is rather complicated. However, the +basic structure is the same for each function that is +exported, so it has to be learned only once. + +Finally it should be mentioned that CObjects offer additional +functionality, which is especially useful for memory allocation and +deallocation of the pointer stored in a CObject. The details +are described in the \citetitle[../api/api.html]{Python/C API +Reference Manual} in the section ``CObjects'' and in the +implementation of CObjects (files \file{Include/cobject.h} and +\file{Objects/cobject.c} in the Python source code distribution). |