diff options
Diffstat (limited to 'Doc/lib/libpickle.tex')
| -rw-r--r-- | Doc/lib/libpickle.tex | 888 |
1 files changed, 0 insertions, 888 deletions
diff --git a/Doc/lib/libpickle.tex b/Doc/lib/libpickle.tex deleted file mode 100644 index 0c4cd98..0000000 --- a/Doc/lib/libpickle.tex +++ /dev/null @@ -1,888 +0,0 @@ -\section{\module{pickle} --- Python object serialization} - -\declaremodule{standard}{pickle} -\modulesynopsis{Convert Python objects to streams of bytes and back.} -% Substantial improvements by Jim Kerr <jbkerr@sr.hp.com>. -% Rewritten by Barry Warsaw <barry@zope.com> - -\index{persistence} -\indexii{persistent}{objects} -\indexii{serializing}{objects} -\indexii{marshalling}{objects} -\indexii{flattening}{objects} -\indexii{pickling}{objects} - -The \module{pickle} module implements a fundamental, but powerful -algorithm for serializing and de-serializing a Python object -structure. ``Pickling'' is the process whereby a Python object -hierarchy is converted into a byte stream, and ``unpickling'' is the -inverse operation, whereby a byte stream is converted back into an -object hierarchy. Pickling (and unpickling) is alternatively known as -``serialization'', ``marshalling,''\footnote{Don't confuse this with -the \refmodule{marshal} module} or ``flattening'', -however, to avoid confusion, the terms used here are ``pickling'' and -``unpickling''. - -This documentation describes both the \module{pickle} module and the -\refmodule{cPickle} module. - -\subsection{Relationship to other Python modules} - -The \module{pickle} module has an optimized cousin called the -\module{cPickle} module. As its name implies, \module{cPickle} is -written in C, so it can be up to 1000 times faster than -\module{pickle}. However it does not support subclassing of the -\function{Pickler()} and \function{Unpickler()} classes, because in -\module{cPickle} these are functions, not classes. Most applications -have no need for this functionality, and can benefit from the improved -performance of \module{cPickle}. Other than that, the interfaces of -the two modules are nearly identical; the common interface is -described in this manual and differences are pointed out where -necessary. In the following discussions, we use the term ``pickle'' -to collectively describe the \module{pickle} and -\module{cPickle} modules. - -The data streams the two modules produce are guaranteed to be -interchangeable. - -Python has a more primitive serialization module called -\refmodule{marshal}, but in general -\module{pickle} should always be the preferred way to serialize Python -objects. \module{marshal} exists primarily to support Python's -\file{.pyc} files. - -The \module{pickle} module differs from \refmodule{marshal} several -significant ways: - -\begin{itemize} - -\item The \module{pickle} module keeps track of the objects it has - already serialized, so that later references to the same object - won't be serialized again. \module{marshal} doesn't do this. - - This has implications both for recursive objects and object - sharing. Recursive objects are objects that contain references - to themselves. These are not handled by marshal, and in fact, - attempting to marshal recursive objects will crash your Python - interpreter. Object sharing happens when there are multiple - references to the same object in different places in the object - hierarchy being serialized. \module{pickle} stores such objects - only once, and ensures that all other references point to the - master copy. Shared objects remain shared, which can be very - important for mutable objects. - -\item \module{marshal} cannot be used to serialize user-defined - classes and their instances. \module{pickle} can save and - restore class instances transparently, however the class - definition must be importable and live in the same module as - when the object was stored. - -\item The \module{marshal} serialization format is not guaranteed to - be portable across Python versions. Because its primary job in - life is to support \file{.pyc} files, the Python implementers - reserve the right to change the serialization format in - non-backwards compatible ways should the need arise. The - \module{pickle} serialization format is guaranteed to be - backwards compatible across Python releases. - -\end{itemize} - -\begin{notice}[warning] -The \module{pickle} module is not intended to be secure against -erroneous or maliciously constructed data. Never unpickle data -received from an untrusted or unauthenticated source. -\end{notice} - -Note that serialization is a more primitive notion than persistence; -although -\module{pickle} reads and writes file objects, it does not handle the -issue of naming persistent objects, nor the (even more complicated) -issue of concurrent access to persistent objects. The \module{pickle} -module can transform a complex object into a byte stream and it can -transform the byte stream into an object with the same internal -structure. Perhaps the most obvious thing to do with these byte -streams is to write them onto a file, but it is also conceivable to -send them across a network or store them in a database. The module -\refmodule{shelve} provides a simple interface -to pickle and unpickle objects on DBM-style database files. - -\subsection{Data stream format} - -The data format used by \module{pickle} is Python-specific. This has -the advantage that there are no restrictions imposed by external -standards such as XDR\index{XDR}\index{External Data Representation} -(which can't represent pointer sharing); however it means that -non-Python programs may not be able to reconstruct pickled Python -objects. - -By default, the \module{pickle} data format uses a printable \ASCII{} -representation. This is slightly more voluminous than a binary -representation. The big advantage of using printable \ASCII{} (and of -some other characteristics of \module{pickle}'s representation) is that -for debugging or recovery purposes it is possible for a human to read -the pickled file with a standard text editor. - -There are currently 3 different protocols which can be used for pickling. - -\begin{itemize} - -\item Protocol version 0 is the original ASCII protocol and is backwards -compatible with earlier versions of Python. - -\item Protocol version 1 is the old binary format which is also compatible -with earlier versions of Python. - -\item Protocol version 2 was introduced in Python 2.3. It provides -much more efficient pickling of new-style classes. - -\end{itemize} - -Refer to PEP 307 for more information. - -If a \var{protocol} is not specified, protocol 0 is used. -If \var{protocol} is specified as a negative value -or \constant{HIGHEST_PROTOCOL}, -the highest protocol version available will be used. - -\versionchanged[Introduced the \var{protocol} parameter]{2.3} - -A binary format, which is slightly more efficient, can be chosen by -specifying a \var{protocol} version >= 1. - -\subsection{Usage} - -To serialize an object hierarchy, you first create a pickler, then you -call the pickler's \method{dump()} method. To de-serialize a data -stream, you first create an unpickler, then you call the unpickler's -\method{load()} method. The \module{pickle} module provides the -following constant: - -\begin{datadesc}{HIGHEST_PROTOCOL} -The highest protocol version available. This value can be passed -as a \var{protocol} value. -\versionadded{2.3} -\end{datadesc} - -\note{Be sure to always open pickle files created with protocols >= 1 in - binary mode. For the old ASCII-based pickle protocol 0 you can use - either text mode or binary mode as long as you stay consistent. - - A pickle file written with protocol 0 in binary mode will contain - lone linefeeds as line terminators and therefore will look ``funny'' - when viewed in Notepad or other editors which do not support this - format.} - -The \module{pickle} module provides the -following functions to make the pickling process more convenient: - -\begin{funcdesc}{dump}{obj, file\optional{, protocol}} -Write a pickled representation of \var{obj} to the open file object -\var{file}. This is equivalent to -\code{Pickler(\var{file}, \var{protocol}).dump(\var{obj})}. - -If the \var{protocol} parameter is omitted, protocol 0 is used. -If \var{protocol} is specified as a negative value -or \constant{HIGHEST_PROTOCOL}, -the highest protocol version will be used. - -\versionchanged[Introduced the \var{protocol} parameter]{2.3} - -\var{file} must have a \method{write()} method that accepts a single -string argument. It can thus be a file object opened for writing, a -\refmodule{StringIO} object, or any other custom -object that meets this interface. -\end{funcdesc} - -\begin{funcdesc}{load}{file} -Read a string from the open file object \var{file} and interpret it as -a pickle data stream, reconstructing and returning the original object -hierarchy. This is equivalent to \code{Unpickler(\var{file}).load()}. - -\var{file} must have two methods, a \method{read()} method that takes -an integer argument, and a \method{readline()} method that requires no -arguments. Both methods should return a string. Thus \var{file} can -be a file object opened for reading, a -\module{StringIO} object, or any other custom -object that meets this interface. - -This function automatically determines whether the data stream was -written in binary mode or not. -\end{funcdesc} - -\begin{funcdesc}{dumps}{obj\optional{, protocol}} -Return the pickled representation of the object as a string, instead -of writing it to a file. - -If the \var{protocol} parameter is omitted, protocol 0 is used. -If \var{protocol} is specified as a negative value -or \constant{HIGHEST_PROTOCOL}, -the highest protocol version will be used. - -\versionchanged[The \var{protocol} parameter was added]{2.3} - -\end{funcdesc} - -\begin{funcdesc}{loads}{string} -Read a pickled object hierarchy from a string. Characters in the -string past the pickled object's representation are ignored. -\end{funcdesc} - -The \module{pickle} module also defines three exceptions: - -\begin{excdesc}{PickleError} -A common base class for the other exceptions defined below. This -inherits from \exception{Exception}. -\end{excdesc} - -\begin{excdesc}{PicklingError} -This exception is raised when an unpicklable object is passed to -the \method{dump()} method. -\end{excdesc} - -\begin{excdesc}{UnpicklingError} -This exception is raised when there is a problem unpickling an object. -Note that other exceptions may also be raised during unpickling, -including (but not necessarily limited to) \exception{AttributeError}, -\exception{EOFError}, \exception{ImportError}, and \exception{IndexError}. -\end{excdesc} - -The \module{pickle} module also exports two callables\footnote{In the -\module{pickle} module these callables are classes, which you could -subclass to customize the behavior. However, in the \refmodule{cPickle} -module these callables are factory functions and so cannot be -subclassed. One common reason to subclass is to control what -objects can actually be unpickled. See section~\ref{pickle-sub} for -more details.}, \class{Pickler} and \class{Unpickler}: - -\begin{classdesc}{Pickler}{file\optional{, protocol}} -This takes a file-like object to which it will write a pickle data -stream. - -If the \var{protocol} parameter is omitted, protocol 0 is used. -If \var{protocol} is specified as a negative value, -the highest protocol version will be used. - -\versionchanged[Introduced the \var{protocol} parameter]{2.3} - -\var{file} must have a \method{write()} method that accepts a single -string argument. It can thus be an open file object, a -\module{StringIO} object, or any other custom -object that meets this interface. -\end{classdesc} - -\class{Pickler} objects define one (or two) public methods: - -\begin{methoddesc}[Pickler]{dump}{obj} -Write a pickled representation of \var{obj} to the open file object -given in the constructor. Either the binary or \ASCII{} format will -be used, depending on the value of the \var{protocol} argument passed to the -constructor. -\end{methoddesc} - -\begin{methoddesc}[Pickler]{clear_memo}{} -Clears the pickler's ``memo''. The memo is the data structure that -remembers which objects the pickler has already seen, so that shared -or recursive objects pickled by reference and not by value. This -method is useful when re-using picklers. - -\begin{notice} -Prior to Python 2.3, \method{clear_memo()} was only available on the -picklers created by \refmodule{cPickle}. In the \module{pickle} module, -picklers have an instance variable called \member{memo} which is a -Python dictionary. So to clear the memo for a \module{pickle} module -pickler, you could do the following: - -\begin{verbatim} -mypickler.memo.clear() -\end{verbatim} - -Code that does not need to support older versions of Python should -simply use \method{clear_memo()}. -\end{notice} -\end{methoddesc} - -It is possible to make multiple calls to the \method{dump()} method of -the same \class{Pickler} instance. These must then be matched to the -same number of calls to the \method{load()} method of the -corresponding \class{Unpickler} instance. If the same object is -pickled by multiple \method{dump()} calls, the \method{load()} will -all yield references to the same object.\footnote{\emph{Warning}: this -is intended for pickling multiple objects without intervening -modifications to the objects or their parts. If you modify an object -and then pickle it again using the same \class{Pickler} instance, the -object is not pickled again --- a reference to it is pickled and the -\class{Unpickler} will return the old value, not the modified one. -There are two problems here: (1) detecting changes, and (2) -marshalling a minimal set of changes. Garbage Collection may also -become a problem here.} - -\class{Unpickler} objects are defined as: - -\begin{classdesc}{Unpickler}{file} -This takes a file-like object from which it will read a pickle data -stream. This class automatically determines whether the data stream -was written in binary mode or not, so it does not need a flag as in -the \class{Pickler} factory. - -\var{file} must have two methods, a \method{read()} method that takes -an integer argument, and a \method{readline()} method that requires no -arguments. Both methods should return a string. Thus \var{file} can -be a file object opened for reading, a -\module{StringIO} object, or any other custom -object that meets this interface. -\end{classdesc} - -\class{Unpickler} objects have one (or two) public methods: - -\begin{methoddesc}[Unpickler]{load}{} -Read a pickled object representation from the open file object given -in the constructor, and return the reconstituted object hierarchy -specified therein. -\end{methoddesc} - -\begin{methoddesc}[Unpickler]{noload}{} -This is just like \method{load()} except that it doesn't actually -create any objects. This is useful primarily for finding what's -called ``persistent ids'' that may be referenced in a pickle data -stream. See section~\ref{pickle-protocol} below for more details. - -\strong{Note:} the \method{noload()} method is currently only -available on \class{Unpickler} objects created with the -\module{cPickle} module. \module{pickle} module \class{Unpickler}s do -not have the \method{noload()} method. -\end{methoddesc} - -\subsection{What can be pickled and unpickled?} - -The following types can be pickled: - -\begin{itemize} - -\item \code{None}, \code{True}, and \code{False} - -\item integers, long integers, floating point numbers, complex numbers - -\item normal and Unicode strings - -\item tuples, lists, sets, and dictionaries containing only picklable objects - -\item functions defined at the top level of a module - -\item built-in functions defined at the top level of a module - -\item classes that are defined at the top level of a module - -\item instances of such classes whose \member{__dict__} or -\method{__setstate__()} is picklable (see -section~\ref{pickle-protocol} for details) - -\end{itemize} - -Attempts to pickle unpicklable objects will raise the -\exception{PicklingError} exception; when this happens, an unspecified -number of bytes may have already been written to the underlying file. -Trying to pickle a highly recursive data structure may exceed the -maximum recursion depth, a \exception{RuntimeError} will be raised -in this case. You can carefully raise this limit with -\function{sys.setrecursionlimit()}. - -Note that functions (built-in and user-defined) are pickled by ``fully -qualified'' name reference, not by value. This means that only the -function name is pickled, along with the name of module the function -is defined in. Neither the function's code, nor any of its function -attributes are pickled. Thus the defining module must be importable -in the unpickling environment, and the module must contain the named -object, otherwise an exception will be raised.\footnote{The exception -raised will likely be an \exception{ImportError} or an -\exception{AttributeError} but it could be something else.} - -Similarly, classes are pickled by named reference, so the same -restrictions in the unpickling environment apply. Note that none of -the class's code or data is pickled, so in the following example the -class attribute \code{attr} is not restored in the unpickling -environment: - -\begin{verbatim} -class Foo: - attr = 'a class attr' - -picklestring = pickle.dumps(Foo) -\end{verbatim} - -These restrictions are why picklable functions and classes must be -defined in the top level of a module. - -Similarly, when class instances are pickled, their class's code and -data are not pickled along with them. Only the instance data are -pickled. This is done on purpose, so you can fix bugs in a class or -add methods to the class and still load objects that were created with -an earlier version of the class. If you plan to have long-lived -objects that will see many versions of a class, it may be worthwhile -to put a version number in the objects so that suitable conversions -can be made by the class's \method{__setstate__()} method. - -\subsection{The pickle protocol -\label{pickle-protocol}}\setindexsubitem{(pickle protocol)} - -This section describes the ``pickling protocol'' that defines the -interface between the pickler/unpickler and the objects that are being -serialized. This protocol provides a standard way for you to define, -customize, and control how your objects are serialized and -de-serialized. The description in this section doesn't cover specific -customizations that you can employ to make the unpickling environment -slightly safer from untrusted pickle data streams; see section~\ref{pickle-sub} -for more details. - -\subsubsection{Pickling and unpickling normal class - instances\label{pickle-inst}} - -When a pickled class instance is unpickled, its \method{__init__()} -method is normally \emph{not} invoked. If it is desirable that the -\method{__init__()} method be called on unpickling, an old-style class -can define a method \method{__getinitargs__()}, which should return a -\emph{tuple} containing the arguments to be passed to the class -constructor (\method{__init__()} for example). The -\method{__getinitargs__()} method is called at -pickle time; the tuple it returns is incorporated in the pickle for -the instance. -\withsubitem{(copy protocol)}{\ttindex{__getinitargs__()}} -\withsubitem{(instance constructor)}{\ttindex{__init__()}} - -\withsubitem{(copy protocol)}{\ttindex{__getnewargs__()}} - -New-style types can provide a \method{__getnewargs__()} method that is -used for protocol 2. Implementing this method is needed if the type -establishes some internal invariants when the instance is created, or -if the memory allocation is affected by the values passed to the -\method{__new__()} method for the type (as it is for tuples and -strings). Instances of a new-style type \class{C} are created using - -\begin{alltt} -obj = C.__new__(C, *\var{args}) -\end{alltt} - -where \var{args} is the result of calling \method{__getnewargs__()} on -the original object; if there is no \method{__getnewargs__()}, an -empty tuple is assumed. - -\withsubitem{(copy protocol)}{ - \ttindex{__getstate__()}\ttindex{__setstate__()}} -\withsubitem{(instance attribute)}{ - \ttindex{__dict__}} - -Classes can further influence how their instances are pickled; if the -class defines the method \method{__getstate__()}, it is called and the -return state is pickled as the contents for the instance, instead of -the contents of the instance's dictionary. If there is no -\method{__getstate__()} method, the instance's \member{__dict__} is -pickled. - -Upon unpickling, if the class also defines the method -\method{__setstate__()}, it is called with the unpickled -state.\footnote{These methods can also be used to implement copying -class instances.} If there is no \method{__setstate__()} method, the -pickled state must be a dictionary and its items are assigned to the -new instance's dictionary. If a class defines both -\method{__getstate__()} and \method{__setstate__()}, the state object -needn't be a dictionary and these methods can do what they -want.\footnote{This protocol is also used by the shallow and deep -copying operations defined in the -\refmodule{copy} module.} - -\begin{notice}[warning] - For new-style classes, if \method{__getstate__()} returns a false - value, the \method{__setstate__()} method will not be called. -\end{notice} - - -\subsubsection{Pickling and unpickling extension types} - -When the \class{Pickler} encounters an object of a type it knows -nothing about --- such as an extension type --- it looks in two places -for a hint of how to pickle it. One alternative is for the object to -implement a \method{__reduce__()} method. If provided, at pickling -time \method{__reduce__()} will be called with no arguments, and it -must return either a string or a tuple. - -If a string is returned, it names a global variable whose contents are -pickled as normal. The string returned by \method{__reduce__} should -be the object's local name relative to its module; the pickle module -searches the module namespace to determine the object's module. - -When a tuple is returned, it must be between two and five elements -long. Optional elements can either be omitted, or \code{None} can be provided -as their value. The semantics of each element are: - -\begin{itemize} - -\item A callable object that will be called to create the initial -version of the object. The next element of the tuple will provide -arguments for this callable, and later elements provide additional -state information that will subsequently be used to fully reconstruct -the pickled data. - -In the unpickling environment this object must be either a class, a -callable registered as a ``safe constructor'' (see below), or it must -have an attribute \member{__safe_for_unpickling__} with a true value. -Otherwise, an \exception{UnpicklingError} will be raised in the -unpickling environment. Note that as usual, the callable itself is -pickled by name. - -\item A tuple of arguments for the callable object. -\versionchanged[Formerly, this argument could also be \code{None}]{2.5} - -\item Optionally, the object's state, which will be passed to - the object's \method{__setstate__()} method as described in - section~\ref{pickle-inst}. If the object has no - \method{__setstate__()} method, then, as above, the value must - be a dictionary and it will be added to the object's - \member{__dict__}. - -\item Optionally, an iterator (and not a sequence) yielding successive -list items. These list items will be pickled, and appended to the -object using either \code{obj.append(\var{item})} or -\code{obj.extend(\var{list_of_items})}. This is primarily used for -list subclasses, but may be used by other classes as long as they have -\method{append()} and \method{extend()} methods with the appropriate -signature. (Whether \method{append()} or \method{extend()} is used -depends on which pickle protocol version is used as well as the number -of items to append, so both must be supported.) - -\item Optionally, an iterator (not a sequence) -yielding successive dictionary items, which should be tuples of the -form \code{(\var{key}, \var{value})}. These items will be pickled -and stored to the object using \code{obj[\var{key}] = \var{value}}. -This is primarily used for dictionary subclasses, but may be used by -other classes as long as they implement \method{__setitem__}. - -\end{itemize} - -It is sometimes useful to know the protocol version when implementing -\method{__reduce__}. This can be done by implementing a method named -\method{__reduce_ex__} instead of \method{__reduce__}. -\method{__reduce_ex__}, when it exists, is called in preference over -\method{__reduce__} (you may still provide \method{__reduce__} for -backwards compatibility). The \method{__reduce_ex__} method will be -called with a single integer argument, the protocol version. - -The \class{object} class implements both \method{__reduce__} and -\method{__reduce_ex__}; however, if a subclass overrides -\method{__reduce__} but not \method{__reduce_ex__}, the -\method{__reduce_ex__} implementation detects this and calls -\method{__reduce__}. - -An alternative to implementing a \method{__reduce__()} method on the -object to be pickled, is to register the callable with the -\refmodule[copyreg]{copy_reg} module. This module provides a way -for programs to register ``reduction functions'' and constructors for -user-defined types. Reduction functions have the same semantics and -interface as the \method{__reduce__()} method described above, except -that they are called with a single argument, the object to be pickled. - -The registered constructor is deemed a ``safe constructor'' for purposes -of unpickling as described above. - - -\subsubsection{Pickling and unpickling external objects} - -For the benefit of object persistence, the \module{pickle} module -supports the notion of a reference to an object outside the pickled -data stream. Such objects are referenced by a ``persistent id'', -which is just an arbitrary string of printable \ASCII{} characters. -The resolution of such names is not defined by the \module{pickle} -module; it will delegate this resolution to user defined functions on -the pickler and unpickler.\footnote{The actual mechanism for -associating these user defined functions is slightly different for -\module{pickle} and \module{cPickle}. The description given here -works the same for both implementations. Users of the \module{pickle} -module could also use subclassing to effect the same results, -overriding the \method{persistent_id()} and \method{persistent_load()} -methods in the derived classes.} - -To define external persistent id resolution, you need to set the -\member{persistent_id} attribute of the pickler object and the -\member{persistent_load} attribute of the unpickler object. - -To pickle objects that have an external persistent id, the pickler -must have a custom \function{persistent_id()} method that takes an -object as an argument and returns either \code{None} or the persistent -id for that object. When \code{None} is returned, the pickler simply -pickles the object as normal. When a persistent id string is -returned, the pickler will pickle that string, along with a marker -so that the unpickler will recognize the string as a persistent id. - -To unpickle external objects, the unpickler must have a custom -\function{persistent_load()} function that takes a persistent id -string and returns the referenced object. - -Here's a silly example that \emph{might} shed more light: - -\begin{verbatim} -import pickle -from cStringIO import StringIO - -src = StringIO() -p = pickle.Pickler(src) - -def persistent_id(obj): - if hasattr(obj, 'x'): - return 'the value %d' % obj.x - else: - return None - -p.persistent_id = persistent_id - -class Integer: - def __init__(self, x): - self.x = x - def __str__(self): - return 'My name is integer %d' % self.x - -i = Integer(7) -print i -p.dump(i) - -datastream = src.getvalue() -print repr(datastream) -dst = StringIO(datastream) - -up = pickle.Unpickler(dst) - -class FancyInteger(Integer): - def __str__(self): - return 'I am the integer %d' % self.x - -def persistent_load(persid): - if persid.startswith('the value '): - value = int(persid.split()[2]) - return FancyInteger(value) - else: - raise pickle.UnpicklingError, 'Invalid persistent id' - -up.persistent_load = persistent_load - -j = up.load() -print j -\end{verbatim} - -In the \module{cPickle} module, the unpickler's -\member{persistent_load} attribute can also be set to a Python -list, in which case, when the unpickler reaches a persistent id, the -persistent id string will simply be appended to this list. This -functionality exists so that a pickle data stream can be ``sniffed'' -for object references without actually instantiating all the objects -in a pickle.\footnote{We'll leave you with the image of Guido and Jim -sitting around sniffing pickles in their living rooms.} Setting -\member{persistent_load} to a list is usually used in conjunction with -the \method{noload()} method on the Unpickler. - -% BAW: Both pickle and cPickle support something called -% inst_persistent_id() which appears to give unknown types a second -% shot at producing a persistent id. Since Jim Fulton can't remember -% why it was added or what it's for, I'm leaving it undocumented. - -\subsection{Subclassing Unpicklers \label{pickle-sub}} - -By default, unpickling will import any class that it finds in the -pickle data. You can control exactly what gets unpickled and what -gets called by customizing your unpickler. Unfortunately, exactly how -you do this is different depending on whether you're using -\module{pickle} or \module{cPickle}.\footnote{A word of caution: the -mechanisms described here use internal attributes and methods, which -are subject to change in future versions of Python. We intend to -someday provide a common interface for controlling this behavior, -which will work in either \module{pickle} or \module{cPickle}.} - -In the \module{pickle} module, you need to derive a subclass from -\class{Unpickler}, overriding the \method{load_global()} -method. \method{load_global()} should read two lines from the pickle -data stream where the first line will the name of the module -containing the class and the second line will be the name of the -instance's class. It then looks up the class, possibly importing the -module and digging out the attribute, then it appends what it finds to -the unpickler's stack. Later on, this class will be assigned to the -\member{__class__} attribute of an empty class, as a way of magically -creating an instance without calling its class's \method{__init__()}. -Your job (should you choose to accept it), would be to have -\method{load_global()} push onto the unpickler's stack, a known safe -version of any class you deem safe to unpickle. It is up to you to -produce such a class. Or you could raise an error if you want to -disallow all unpickling of instances. If this sounds like a hack, -you're right. Refer to the source code to make this work. - -Things are a little cleaner with \module{cPickle}, but not by much. -To control what gets unpickled, you can set the unpickler's -\member{find_global} attribute to a function or \code{None}. If it is -\code{None} then any attempts to unpickle instances will raise an -\exception{UnpicklingError}. If it is a function, -then it should accept a module name and a class name, and return the -corresponding class object. It is responsible for looking up the -class and performing any necessary imports, and it may raise an -error to prevent instances of the class from being unpickled. - -The moral of the story is that you should be really careful about the -source of the strings your application unpickles. - -\subsection{Example \label{pickle-example}} - -For the simplest code, use the \function{dump()} and \function{load()} -functions. Note that a self-referencing list is pickled and restored -correctly. - -\begin{verbatim} -import pickle - -data1 = {'a': [1, 2.0, 3, 4+6j], - 'b': ('string', u'Unicode string'), - 'c': None} - -selfref_list = [1, 2, 3] -selfref_list.append(selfref_list) - -output = open('data.pkl', 'wb') - -# Pickle dictionary using protocol 0. -pickle.dump(data1, output) - -# Pickle the list using the highest protocol available. -pickle.dump(selfref_list, output, -1) - -output.close() -\end{verbatim} - -The following example reads the resulting pickled data. When reading -a pickle-containing file, you should open the file in binary mode -because you can't be sure if the ASCII or binary format was used. - -\begin{verbatim} -import pprint, pickle - -pkl_file = open('data.pkl', 'rb') - -data1 = pickle.load(pkl_file) -pprint.pprint(data1) - -data2 = pickle.load(pkl_file) -pprint.pprint(data2) - -pkl_file.close() -\end{verbatim} - -Here's a larger example that shows how to modify pickling behavior for a -class. The \class{TextReader} class opens a text file, and returns -the line number and line contents each time its \method{readline()} -method is called. If a \class{TextReader} instance is pickled, all -attributes \emph{except} the file object member are saved. When the -instance is unpickled, the file is reopened, and reading resumes from -the last location. The \method{__setstate__()} and -\method{__getstate__()} methods are used to implement this behavior. - -\begin{verbatim} -class TextReader: - """Print and number lines in a text file.""" - def __init__(self, file): - self.file = file - self.fh = open(file) - self.lineno = 0 - - def readline(self): - self.lineno = self.lineno + 1 - line = self.fh.readline() - if not line: - return None - if line.endswith("\n"): - line = line[:-1] - return "%d: %s" % (self.lineno, line) - - def __getstate__(self): - odict = self.__dict__.copy() # copy the dict since we change it - del odict['fh'] # remove filehandle entry - return odict - - def __setstate__(self, dict): - fh = open(dict['file']) # reopen file - count = dict['lineno'] # read from file... - while count: # until line count is restored - fh.readline() - count = count - 1 - self.__dict__.update(dict) # update attributes - self.fh = fh # save the file object -\end{verbatim} - -A sample usage might be something like this: - -\begin{verbatim} ->>> import TextReader ->>> obj = TextReader.TextReader("TextReader.py") ->>> obj.readline() -'1: #!/usr/local/bin/python' ->>> # (more invocations of obj.readline() here) -... obj.readline() -'7: class TextReader:' ->>> import pickle ->>> pickle.dump(obj,open('save.p', 'wb')) -\end{verbatim} - -If you want to see that \refmodule{pickle} works across Python -processes, start another Python session, before continuing. What -follows can happen from either the same process or a new process. - -\begin{verbatim} ->>> import pickle ->>> reader = pickle.load(open('save.p', 'rb')) ->>> reader.readline() -'8: "Print and number lines in a text file."' -\end{verbatim} - - -\begin{seealso} - \seemodule[copyreg]{copy_reg}{Pickle interface constructor - registration for extension types.} - - \seemodule{shelve}{Indexed databases of objects; uses \module{pickle}.} - - \seemodule{copy}{Shallow and deep object copying.} - - \seemodule{marshal}{High-performance serialization of built-in types.} -\end{seealso} - - -\section{\module{cPickle} --- A faster \module{pickle}} - -\declaremodule{builtin}{cPickle} -\modulesynopsis{Faster version of \refmodule{pickle}, but not subclassable.} -\moduleauthor{Jim Fulton}{jim@zope.com} -\sectionauthor{Fred L. Drake, Jr.}{fdrake@acm.org} - -The \module{cPickle} module supports serialization and -de-serialization of Python objects, providing an interface and -functionality nearly identical to the -\refmodule{pickle}\refstmodindex{pickle} module. There are several -differences, the most important being performance and subclassability. - -First, \module{cPickle} can be up to 1000 times faster than -\module{pickle} because the former is implemented in C. Second, in -the \module{cPickle} module the callables \function{Pickler()} and -\function{Unpickler()} are functions, not classes. This means that -you cannot use them to derive custom pickling and unpickling -subclasses. Most applications have no need for this functionality and -should benefit from the greatly improved performance of the -\module{cPickle} module. - -The pickle data stream produced by \module{pickle} and -\module{cPickle} are identical, so it is possible to use -\module{pickle} and \module{cPickle} interchangeably with existing -pickles.\footnote{Since the pickle data format is actually a tiny -stack-oriented programming language, and some freedom is taken in the -encodings of certain objects, it is possible that the two modules -produce different data streams for the same input objects. However it -is guaranteed that they will always be able to read each other's -data streams.} - -There are additional minor differences in API between \module{cPickle} -and \module{pickle}, however for most applications, they are -interchangeable. More documentation is provided in the -\module{pickle} module documentation, which -includes a list of the documented differences. - - |