Moving pickletools.py from the sandbox into the std library. I started

this over the weekend, and it made faster & better progress than I
expected -- it's already useful <wink>.

1 files changed, 1794 insertions, 0 deletions

diff --git a/Lib/pickletools.py b/Lib/pickletools.py
new file mode 100644
index 0000000..2f03082
--- /dev/null
+++ b/Lib/pickletools.py
@@ -0,0 +1,1794 @@
+""""Executable documentation" for the pickle module.
+
+Extensive comments about the pickle protocols and pickle-machine opcodes
+can be found here.  Some functions meant for external use:
+
+genops(pickle)
+   Generate all the opcodes in a pickle, as (opcode, arg, position) triples.
+
+dis(pickle, out=None, indentlevel=4)
+   Print a symbolic disassembly of a pickle.
+"""
+
+# Other ideas:
+#
+# - A pickle verifier:  read a pickle and check it exhaustively for
+#   well-formedness.
+#
+# - A protocol identifier:  examine a pickle and return its protocol number
+#   (== the highest .proto attr value among all the opcodes in the pickle).
+#
+# - A pickle optimizer:  for example, tuple-building code is sometimes more
+#   elaborate than necessary, catering for the possibility that the tuple
+#   is recursive.  Or lots of times a PUT is generated that's never accessed
+#   by a later GET.
+
+
+"""
+"A pickle" is a program for a virtual pickle machine (PM, but more accurately
+called an unpickling machine).  It's a sequence of opcodes, interpreted by the
+PM, building an arbitrarily complex Python object.
+
+For the most part, the PM is very simple:  there are no looping, testing, or
+conditional instructions, no arithmetic and no function calls.  Opcodes are
+executed once each, from first to last, until a STOP opcode is reached.
+
+The PM has two data areas, "the stack" and "the memo".
+
+Many opcodes push Python objects onto the stack; e.g., INT pushes a Python
+integer object on the stack, whose value is gotten from a decimal string
+literal immediately following the INT opcode in the pickle bytestream.  Other
+opcodes take Python objects off the stack.  The result of unpickling is
+whatever object is left on the stack when the final STOP opcode is executed.
+
+The memo is simply an array of objects, or it can be implemented as a dict
+mapping little integers to objects.  The memo serves as the PM's "long term
+memory", and the little integers indexing the memo are akin to variable
+names.  Some opcodes pop a stack object into the memo at a given index,
+and others push a memo object at a given index onto the stack again.
+
+At heart, that's all the PM has.  Subtleties arise for these reasons:
+
++ Object identity.  Objects can be arbitrarily complex, and subobjects
+  may be shared (for example, the list [a, a] refers to the same object a
+  twice).  It can be vital that unpickling recreate an isomorphic object
+  graph, faithfully reproducing sharing.
+
++ Recursive objects.  For example, after "L = []; L.append(L)", L is a
+  list, and L[0] is the same list.  This is related to the object identity
+  point, and some sequences of pickle opcodes are subtle in order to
+  get the right result in all cases.
+
++ Things pickle doesn't know everything about.  Examples of things pickle
+  does know everything about are Python's builtin scalar and container
+  types, like ints and tuples.  They generally have opcodes dedicated to
+  them.  For things like module references and instances of user-defined
+  classes, pickle's knowledge is limited.  Historically, many enhancements
+  have been made to the pickle protocol in order to do a better (faster,
+  and/or more compact) job on those.
+
++ Backward compatibility and micro-optimization.  As explained below,
+  pickle opcodes never go away, not even when better ways to do a thing
+  get invented.  The repertoire of the PM just keeps growing over time.
+  So, e.g., there are now six distinct opcodes for building a Python integer,
+  five of them devoted to "short" integers.  Even so, the only way to pickle
+  a Python long int takes time quadratic in the number of digits, for both
+  pickling and unpickling.  This isn't so much a subtlety as a source of
+  wearying complication.
+
+
+Pickle protocols:
+
+For compatibility, the meaning of a pickle opcode never changes.  Instead new
+pickle opcodes get added, and each version's unpickler can handle all the
+pickle opcodes in all protocol versions to date.  So old pickles continue to
+be readable forever.  The pickler can generally be told to restrict itself to
+the subset of opcodes available under previous protocol versions too, so that
+users can create pickles under the current version readable by older
+versions.  However, a pickle does not contain its version number embedded
+within it.  If an older unpickler tries to read a pickle using a later
+protocol, the result is most likely an exception due to seeing an unknown (in
+the older unpickler) opcode.
+
+The original pickle used what's now called "protocol 0", and what was called
+"text mode" before Python 2.3.  The entire pickle bytestream is made up of
+printable 7-bit ASCII characters, plus the newline character, in protocol 0.
+That's why it was called text mode.
+
+The second major set of additions is now called "protocol 1", and was called
+"binary mode" before Python 2.3.  This added many opcodes with arguments
+consisting of arbitrary bytes, including NUL bytes and unprintable "high bit"
+bytes.  Binary mode pickles can be substantially smaller than equivalent
+text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte
+int as 4 bytes following the opcode, which is cheaper to unpickle than the
+(perhaps) 11-character decimal string attached to INT.
+
+The third major set of additions came in Python 2.3, and is called "protocol
+2".  XXX Write a short blurb when Guido figures out what they are <wink>. XXX
+"""
+
+# Meta-rule:  Descriptions are stored in instances of descriptor objects,
+# with plain constructors.  No meta-language is defined from which
+# descriptors could be constructed.  If you want, e.g., XML, write a little
+# program to generate XML from the objects.
+
+##############################################################################
+# Some pickle opcodes have an argument, following the opcode in the
+# bytestream.  An argument is of a specific type, described by an instance
+# of ArgumentDescriptor.  These are not to be confused with arguments taken
+# off the stack -- ArgumentDescriptor applies only to arguments embedded in
+# the opcode stream, immediately following an opcode.
+
+# Represents the number of bytes consumed by an argument delimited by the
+# next newline character.
+UP_TO_NEWLINE = -1
+
+# Represents the number of bytes consumed by a two-argument opcode where
+# the first argument gives the number of bytes in the second argument.
+TAKEN_FROM_ARGUMENT = -2
+
+class ArgumentDescriptor(object):
+    __slots__ = (
+        # name of descriptor record, also a module global name; a string
+        'name',
+
+        # length of argument, in bytes; an int; UP_TO_NEWLINE and
+        # TAKEN_FROM_ARGUMENT are negative values for variable-length cases
+        'n',
+
+        # a function taking a file-like object, reading this kind of argument
+        # from the object at the current position, advancing the current
+        # position by n bytes, and returning the value of the argument
+        'reader',
+
+        # human-readable docs for this arg descriptor; a string
+        'doc',
+    )
+
+    def __init__(self, name, n, reader, doc):
+        assert isinstance(name, str)
+        self.name = name
+
+        assert isinstance(n, int) and (n >= 0 or
+                                       n is UP_TO_NEWLINE or
+                                       n is TAKEN_FROM_ARGUMENT)
+        self.n = n
+
+        self.reader = reader
+
+        assert isinstance(doc, str)
+        self.doc = doc
+
+from struct import unpack as _unpack
+
+def read_uint1(f):
+    """
+    >>> import StringIO
+    >>> read_uint1(StringIO.StringIO('\\xff'))
+    255
+    """
+
+    data = f.read(1)
+    if data:
+        return ord(data)
+    raise ValueError("not enough data in stream to read uint1")
+
+uint1 = ArgumentDescriptor(
+            name='uint1',
+            n=1,
+            reader=read_uint1,
+            doc="One-byte unsigned integer.")
+
+
+def read_uint2(f):
+    """
+    >>> import StringIO
+    >>> read_uint2(StringIO.StringIO('\\xff\\x00'))
+    255
+    >>> read_uint2(StringIO.StringIO('\\xff\\xff'))
+    65535
+    """
+
+    data = f.read(2)
+    if len(data) == 2:
+        return _unpack("<H", data)[0]
+    raise ValueError("not enough data in stream to read uint2")
+
+uint2 = ArgumentDescriptor(
+            name='uint2',
+            n=2,
+            reader=read_uint2,
+            doc="Two-byte unsigned integer, little-endian.")
+
+
+def read_int4(f):
+    """
+    >>> import StringIO
+    >>> read_int4(StringIO.StringIO('\\xff\\x00\\x00\\x00'))
+    255
+    >>> read_int4(StringIO.StringIO('\\x00\\x00\\x00\\x80')) == -(2**31)
+    True
+    """
+
+    data = f.read(4)
+    if len(data) == 4:
+        return _unpack("<i", data)[0]
+    raise ValueError("not enough data in stream to read int4")
+
+int4 = ArgumentDescriptor(
+           name='int4',
+           n=4,
+           reader=read_int4,
+           doc="Four-byte signed integer, little-endian, 2's complement.")
+
+
+def read_stringnl(f, decode=True, stripquotes=True):
+    """
+    >>> import StringIO
+    >>> read_stringnl(StringIO.StringIO("'abcd'\\nefg\\n"))
+    'abcd'
+
+    >>> read_stringnl(StringIO.StringIO("\\n"))
+    Traceback (most recent call last):
+    ...
+    ValueError: no string quotes around ''
+
+    >>> read_stringnl(StringIO.StringIO("\\n"), stripquotes=False)
+    ''
+
+    >>> read_stringnl(StringIO.StringIO("''\\n"))
+    ''
+
+    >>> read_stringnl(StringIO.StringIO('"abcd"'))
+    Traceback (most recent call last):
+    ...
+    ValueError: no newline found when trying to read stringnl
+
+    Embedded escapes are undone in the result.
+    >>> read_stringnl(StringIO.StringIO("'a\\\\nb\\x00c\\td'\\n'e'"))
+    'a\\nb\\x00c\\td'
+    """
+
+    data = f.readline()
+    if not data.endswith('\n'):
+        raise ValueError("no newline found when trying to read stringnl")
+    data = data[:-1]    # lose the newline
+
+    if stripquotes:
+        for q in "'\"":
+            if data.startswith(q):
+                if not data.endswith(q):
+                    raise ValueError("strinq quote %r not found at both "
+                                     "ends of %r" % (q, data))
+                data = data[1:-1]
+                break
+        else:
+            raise ValueError("no string quotes around %r" % data)
+
+    # I'm not sure when 'string_escape' was added to the std codecs; it's
+    # crazy not to use it if it's there.
+    if decode:
+        data = data.decode('string_escape')
+    return data
+
+stringnl = ArgumentDescriptor(
+               name='stringnl',
+               n=UP_TO_NEWLINE,
+               reader=read_stringnl,
+               doc="""A newline-terminated string.
+
+                   This is a repr-style string, with embedded escapes, and
+                   bracketing quotes.
+                   """)
+
+def read_stringnl_noescape(f):
+    return read_stringnl(f, decode=False, stripquotes=False)
+
+stringnl_noescape = ArgumentDescriptor(
+                        name='stringnl_noescape',
+                        n=UP_TO_NEWLINE,
+                        reader=read_stringnl_noescape,
+                        doc="""A newline-terminated string.
+
+                        This is a str-style string, without embedded escapes,
+                        or bracketing quotes.  It should consist solely of
+                        printable ASCII characters.
+                        """)
+
+def read_stringnl_noescape_pair(f):
+    """
+    >>> import StringIO
+    >>> read_stringnl_noescape_pair(StringIO.StringIO("Queue\\nEmpty\\njunk"))
+    'Queue.Empty'
+    """
+
+    return "%s.%s" % (read_stringnl_noescape(f), read_stringnl_noescape(f))
+
+stringnl_noescape_pair = ArgumentDescriptor(
+                             name='stringnl_noescape_pair',
+                             n=UP_TO_NEWLINE,
+                             reader=read_stringnl_noescape_pair,
+                             doc="""A pair of newline-terminated strings.
+
+                             These are str-style strings, without embedded
+                             escapes, or bracketing quotes.  They should
+                             consist solely of printable ASCII characters.
+                             The pair is returned as a single string, with
+                             a single '.' separating the two strings.
+                             """)
+
+def read_string4(f):
+    """
+    >>> import StringIO
+    >>> read_string4(StringIO.StringIO("\\x00\\x00\\x00\\x00abc"))
+    ''
+    >>> read_string4(StringIO.StringIO("\\x03\\x00\\x00\\x00abcdef"))
+    'abc'
+    >>> read_string4(StringIO.StringIO("\\x00\\x00\\x00\\x03abcdef"))
+    Traceback (most recent call last):
+    ...
+    ValueError: expected 50331648 bytes in a string4, but only 6 remain
+    """
+
+    n = read_int4(f)
+    if n < 0:
+        raise ValueError("string4 byte count < 0: %d" % n)
+    data = f.read(n)
+    if len(data) == n:
+        return data
+    raise ValueError("expected %d bytes in a string4, but only %d remain" %
+                     (n, len(data)))
+
+string4 = ArgumentDescriptor(
+              name="string4",
+              n=TAKEN_FROM_ARGUMENT,
+              reader=read_string4,
+              doc="""A counted string.
+
+              The first argument is a 4-byte little-endian signed int giving
+              the number of bytes in the string, and the second argument is
+              that many bytes.
+              """)
+
+
+def read_string1(f):
+    """
+    >>> import StringIO
+    >>> read_string1(StringIO.StringIO("\\x00"))
+    ''
+    >>> read_string1(StringIO.StringIO("\\x03abcdef"))
+    'abc'
+    """
+
+    n = read_uint1(f)
+    assert n >= 0
+    data = f.read(n)
+    if len(data) == n:
+        return data
+    raise ValueError("expected %d bytes in a string1, but only %d remain" %
+                     (n, len(data)))
+
+string1 = ArgumentDescriptor(
+              name="string1",
+              n=TAKEN_FROM_ARGUMENT,
+              reader=read_string1,
+              doc="""A counted string.
+
+              The first argument is a 1-byte unsigned int giving the number
+              of bytes in the string, and the second argument is that many
+              bytes.
+              """)
+
+
+def read_unicodestringnl(f):
+    """
+    >>> import StringIO
+    >>> read_unicodestringnl(StringIO.StringIO("abc\\uabcd\\njunk"))
+    u'abc\\uabcd'
+    """
+
+    data = f.readline()
+    if not data.endswith('\n'):
+        raise ValueError("no newline found when trying to read "
+                         "unicodestringnl")
+    data = data[:-1]    # lose the newline
+    return unicode(data, 'raw-unicode-escape')
+
+unicodestringnl = ArgumentDescriptor(
+                      name='unicodestringnl',
+                      n=UP_TO_NEWLINE,
+                      reader=read_unicodestringnl,
+                      doc="""A newline-terminated Unicode string.
+
+                      This is raw-unicode-escape encoded, so consists of
+                      printable ASCII characters, and may contain embedded
+                      escape sequences.
+                      """)
+
+def read_unicodestring4(f):
+    """
+    >>> import StringIO
+    >>> s = u'abcd\\uabcd'
+    >>> enc = s.encode('utf-8')
+    >>> enc
+    'abcd\\xea\\xaf\\x8d'
+    >>> n = chr(len(enc)) + chr(0) * 3  # little-endian 4-byte length
+    >>> t = read_unicodestring4(StringIO.StringIO(n + enc + 'junk'))
+    >>> s == t
+    True
+
+    >>> read_unicodestring4(StringIO.StringIO(n + enc[:-1]))
+    Traceback (most recent call last):
+    ...
+    ValueError: expected 7 bytes in a unicodestring4, but only 6 remain
+    """
+
+    n = read_int4(f)
+    if n < 0:
+        raise ValueError("unicodestring4 byte count < 0: %d" % n)
+    data = f.read(n)
+    if len(data) == n:
+        return unicode(data, 'utf-8')
+    raise ValueError("expected %d bytes in a unicodestring4, but only %d "
+                     "remain" % (n, len(data)))
+
+unicodestring4 = ArgumentDescriptor(
+                    name="unicodestring4",
+                    n=TAKEN_FROM_ARGUMENT,
+                    reader=read_unicodestring4,
+                    doc="""A counted Unicode string.
+
+                    The first argument is a 4-byte little-endian signed int
+                    giving the number of bytes in the string, and the second
+                    argument-- the UTF-8 encoding of the Unicode string --
+                    contains that many bytes.
+                    """)
+
+
+def read_decimalnl_short(f):
+    """
+    >>> import StringIO
+    >>> read_decimalnl_short(StringIO.StringIO("1234\\n56"))
+    1234
+
+    >>> read_decimalnl_short(StringIO.StringIO("1234L\\n56"))
+    Traceback (most recent call last):
+    ...
+    ValueError: trailing 'L' not allowed in '1234L'
+    """
+
+    s = read_stringnl(f, decode=False, stripquotes=False)
+    if s.endswith("L"):
+        raise ValueError("trailing 'L' not allowed in %r" % s)
+
+    # It's not necessarily true that the result fits in a Python short int:
+    # the pickle may have been written on a 64-bit box.  There's also a hack
+    # for True and False here.
+    if s == "00":
+        return False
+    elif s == "01":
+        return True
+
+    try:
+        return int(s)
+    except OverflowError:
+        return long(s)
+
+def read_decimalnl_long(f):
+    """
+    >>> import StringIO
+
+    >>> read_decimalnl_long(StringIO.StringIO("1234\\n56"))
+    Traceback (most recent call last):
+    ...
+    ValueError: trailing 'L' required in '1234'
+
+    Someday the trailing 'L' will probably go away from this output.
+
+    >>> read_decimalnl_long(StringIO.StringIO("1234L\\n56"))
+    1234L
+
+    >>> read_decimalnl_long(StringIO.StringIO("123456789012345678901234L\\n6"))
+    123456789012345678901234L
+    """
+
+    s = read_stringnl(f, decode=False, stripquotes=False)
+    if not s.endswith("L"):
+        raise ValueError("trailing 'L' required in %r" % s)
+    return long(s)
+
+
+decimalnl_short = ArgumentDescriptor(
+                      name='decimalnl_short',
+                      n=UP_TO_NEWLINE,
+                      reader=read_decimalnl_short,
+                      doc="""A newline-terminated decimal integer literal.
+
+                          This never has a trailing 'L', and the integer fit
+                          in a short Python int on the box where the pickle
+                          was written -- but there's no guarantee it will fit
+                          in a short Python int on the box where the pickle
+                          is read.
+                          """)
+
+decimalnl_long = ArgumentDescriptor(
+                     name='decimalnl_long',
+                     n=UP_TO_NEWLINE,
+                     reader=read_decimalnl_long,
+                     doc="""A newline-terminated decimal integer literal.
+
+                         This has a trailing 'L', and can represent integers
+                         of any size.
+                         """)
+
+
+def read_floatnl(f):
+    """
+    >>> import StringIO
+    >>> read_floatnl(StringIO.StringIO("-1.25\\n6"))
+    -1.25
+    """
+    s = read_stringnl(f, decode=False, stripquotes=False)
+    return float(s)
+
+floatnl = ArgumentDescriptor(
+              name='floatnl',
+              n=UP_TO_NEWLINE,
+              reader=read_floatnl,
+              doc="""A newline-terminated decimal floating literal.
+
+              In general this requires 17 significant digits for roundtrip
+              identity, and pickling then unpickling infinities, NaNs, and
+              minus zero doesn't work across boxes, or on some boxes even
+              on itself (e.g., Windows can't read the strings it produces
+              for infinities or NaNs).
+              """)
+
+def read_float8(f):
+    """
+    >>> import StringIO, struct
+    >>> raw = struct.pack(">d", -1.25)
+    >>> raw
+    '\\xbf\\xf4\\x00\\x00\\x00\\x00\\x00\\x00'
+    >>> read_float8(StringIO.StringIO(raw + "\\n"))
+    -1.25
+    """
+
+    data = f.read(8)
+    if len(data) == 8:
+        return _unpack(">d", data)[0]
+    raise ValueError("not enough data in stream to read float8")
+
+
+float8 = ArgumentDescriptor(
+             name='float8',
+             n=8,
+             reader=read_float8,
+             doc="""An 8-byte binary representation of a float, big-endian.
+
+             The format is unique to Python, and shared with the struct
+             module (format string '>d') "in theory" (the struct and cPickle
+             implementations don't share the code -- they should).  It's
+             strongly related to the IEEE-754 double format, and, in normal
+             cases, is in fact identical to the big-endian 754 double format.
+             On other boxes the dynamic range is limited to that of a 754
+             double, and "add a half and chop" rounding is used to reduce
+             the precision to 53 bits.  However, even on a 754 box,
+             infinities, NaNs, and minus zero may not be handled correctly
+             (may not survive roundtrip pickling intact).
+             """)
+
+##############################################################################
+# Object descriptors.  The stack used by the pickle machine holds objects,
+# and in the stack_before and stack_after attributes of OpcodeInfo
+# descriptors we need names to describe the various types of objects that can
+# appear on the stack.
+
+class StackObject(object):
+    __slots__ = (
+        # name of descriptor record, for info only
+        'name',
+
+        # type of object, or tuple of type objects (meaning the object can
+        # be of any type in the tuple)
+        'obtype',
+
+        # human-readable docs for this kind of stack object; a string
+        'doc',
+    )
+
+    def __init__(self, name, obtype, doc):
+        assert isinstance(name, str)
+        self.name = name
+
+        assert isinstance(obtype, type) or isinstance(obtype, tuple)
+        if isinstance(obtype, tuple):
+            for contained in obtype:
+                assert isinstance(contained, type)
+        self.obtype = obtype
+
+        assert isinstance(doc, str)
+        self.doc = doc
+
+
+pyint = StackObject(
+            name='int',
+            obtype=int,
+            doc="A short (as opposed to long) Python integer object.")
+
+pylong = StackObject(
+             name='long',
+             obtype=long,
+             doc="A long (as opposed to short) Python integer object.")
+
+pyinteger_or_bool = StackObject(
+                        name='int_or_bool',
+                        obtype=(int, long, bool),
+                        doc="A Python integer object (short or long), or "
+                            "a Python bool.")
+
+pyfloat = StackObject(
+              name='float',
+              obtype=float,
+              doc="A Python float object.")
+
+pystring = StackObject(
+               name='str',
+               obtype=str,
+               doc="A Python string object.")
+
+pyunicode = StackObject(
+                name='unicode',
+                obtype=unicode,
+                doc="A Python Unicode string object.")
+
+pynone = StackObject(
+             name="None",
+             obtype=type(None),
+             doc="The Python None object.")
+
+pytuple = StackObject(
+              name="tuple",
+              obtype=tuple,
+              doc="A Python tuple object.")
+
+pylist = StackObject(
+             name="list",
+             obtype=list,
+             doc="A Python list object.")
+
+pydict = StackObject(
+             name="dict",
+             obtype=dict,
+             doc="A Python dict object.")
+
+anyobject = StackObject(
+                name='any',
+                obtype=object,
+                doc="Any kind of object whatsoever.")
+
+markobject = StackObject(
+                 name="mark",
+                 obtype=StackObject,
+                 doc="""'The mark' is a unique object.
+
+                 Opcodes that operate on a variable number of objects
+                 generally don't embed the count of objects in the opcode,
+                 or pull it off the stack.  Instead the MARK opcode is used
+                 to push a special marker object on the stack, and then
+                 some other opcodes grab all the objects from the top of
+                 the stack down to (but not including) the topmost marker
+                 object.
+                 """)
+
+stackslice = StackObject(
+                 name="stackslice",
+                 obtype=StackObject,
+                 doc="""An object representing a contiguous slice of the stack.
+
+                 This is used in conjuction with markobject, to represent all
+                 of the stack following the topmost markobject.  For example,
+                 the POP_MARK opcode changes the stack from
+
+                     [..., markobject, stackslice]
+                 to
+                     [...]
+
+                 No matter how many object are on the stack after the topmost
+                 markobject, POP_MARK gets rid of all of them (including the
+                 topmost markobject too).
+                 """)
+
+##############################################################################
+# Descriptors for pickle opcodes.
+
+class OpcodeInfo(object):
+
+    __slots__ = (
+        # symbolic name of opcode; a string
+        'name',
+
+        # the code used in a bytestream to represent the opcode; a
+        # one-character string
+        'code',
+
+        # If the opcode has an argument embedded in the byte string, an
+        # instance of ArgumentDescriptor specifying its type.  Note that
+        # arg.reader(s) can be used to read and decode the argument from
+        # the bytestream s, and arg.doc documents the format of the raw
+        # argument bytes.  If the opcode doesn't have an argument embedded
+        # in the bytestream, arg should be None.
+        'arg',
+
+        # what the stack looks like before this opcode runs; a list
+        'stack_before',
+
+        # what the stack looks like after this opcode runs; a list
+        'stack_after',
+
+        # the protocol number in which this opcode was introduced; an int
+        'proto',
+
+        # human-readable docs for this opcode; a string
+        'doc',
+    )
+
+    def __init__(self, name, code, arg,
+                 stack_before, stack_after, proto, doc):
+        assert isinstance(name, str)
+        self.name = name
+
+        assert isinstance(code, str)
+        assert len(code) == 1
+        self.code = code
+
+        assert arg is None or isinstance(arg, ArgumentDescriptor)
+        self.arg = arg
+
+        assert isinstance(stack_before, list)
+        for x in stack_before:
+            assert isinstance(x, StackObject)
+        self.stack_before = stack_before
+
+        assert isinstance(stack_after, list)
+        for x in stack_after:
+            assert isinstance(x, StackObject)
+        self.stack_after = stack_after
+
+        assert isinstance(proto, int) and 0 <= proto <= 2
+        self.proto = proto
+
+        assert isinstance(doc, str)
+        self.doc = doc
+
+I = OpcodeInfo
+opcodes = [
+
+    # Ways to spell integers.
+
+    I(name='INT',
+      code='I',
+      arg=decimalnl_short,
+      stack_before=[],
+      stack_after=[pyinteger_or_bool],
+      proto=0,
+      doc="""Push an integer or bool.
+
+      The argument is a newline-terminated decimal literal string.
+
+      The intent may have been that this always fit in a short Python int,
+      but INT can be generated in pickles written on a 64-bit box that
+      require a Python long on a 32-bit box.  The difference between this
+      and LONG then is that INT skips a trailing 'L', and produces a short
+      int whenever possible.
+
+      Another difference is due to that, when bool was introduced as a
+      distinct type in 2.3, builtin names True and False were also added to
+      2.2.2, mapping to ints 1 and 0.  For compatibility in both directions,
+      True gets pickled as INT + "I01\\n", and False as INT + "I00\\n".
+      Leading zeroes are never produced for a genuine integer.  The 2.3
+      (and later) unpicklers special-case these and return bool instead;
+      earlier unpicklers ignore the leading "0" and return the int.
+      """),
+
+    I(name='LONG',
+      code='L',
+      arg=decimalnl_long,
+      stack_before=[],
+      stack_after=[pylong],
+      proto=0,
+      doc="""Push a long integer.
+
+      The same as INT, except that the literal ends with 'L', and always
+      unpickles to a Python long.  There doesn't seem a real purpose to the
+      trailing 'L'.
+      """),
+
+    I(name='BININT',
+      code='J',
+      arg=int4,
+      stack_before=[],
+      stack_after=[pyint],
+      proto=1,
+      doc="""Push a four-byte signed integer.
+
+      This handles the full range of Python (short) integers on a 32-bit
+      box, directly as binary bytes (1 for the opcode and 4 for the integer).
+      If the integer is non-negative and fits in 1 or 2 bytes, pickling via
+      BININT1 or BININT2 saves space.
+      """),
+
+    I(name='BININT1',
+      code='K',
+      arg=uint1,
+      stack_before=[],
+      stack_after=[pyint],
+      proto=1,
+      doc="""Push a one-byte unsigned integer.
+
+      This is a space optimization for pickling very small non-negative ints,
+      in range(256).
+      """),
+
+    I(name='BININT2',
+      code='M',
+      arg=uint2,
+      stack_before=[],
+      stack_after=[pyint],
+      proto=1,
+      doc="""Push a two-byte unsigned integer.
+
+      This is a space optimization for pickling small positive ints, in
+      range(256, 2**16).  Integers in range(256) can also be pickled via
+      BININT2, but BININT1 instead saves a byte.
+      """),
+
+    # Ways to spell strings (8-bit, not Unicode).
+
+    I(name='STRING',
+      code='S',
+      arg=stringnl,
+      stack_before=[],
+      stack_after=[pystring],
+      proto=0,
+      doc="""Push a Python string object.
+
+      The argument is a repr-style string, with bracketing quote characters,
+      and perhaps embedded escapes.  The argument extends until the next
+      newline character.
+      """),
+
+    I(name='BINSTRING',
+      code='T',
+      arg=string4,
+      stack_before=[],
+      stack_after=[pystring],
+      proto=1,
+      doc="""Push a Python string object.
+
+      There are two arguments:  the first is a 4-byte little-endian signed int
+      giving the number of bytes in the string, and the second is that many
+      bytes, which are taken literally as the string content.
+      """),
+
+    I(name='SHORT_BINSTRING',
+      code='U',
+      arg=string1,
+      stack_before=[],
+      stack_after=[pystring],
+      proto=1,
+      doc="""Push a Python string object.
+
+      There are two arguments:  the first is a 1-byte unsigned int giving
+      the number of bytes in the string, and the second is that many bytes,
+      which are taken literally as the string content.
+      """),
+
+    # Ways to spell None.
+
+    I(name='NONE',
+      code='N',
+      arg=None,
+      stack_before=[],
+      stack_after=[pynone],
+      proto=0,
+      doc="Push None on the stack."),
+
+    # Ways to spell Unicode strings.
+
+    I(name='UNICODE',
+      code='V',
+      arg=unicodestringnl,
+      stack_before=[],
+      stack_after=[pyunicode],
+      proto=0,  # this may be pure-text, but it's a later addition
+      doc="""Push a Python Unicode string object.
+
+      The argument is a raw-unicode-escape encoding of a Unicode string,
+      and so may contain embedded escape sequences.  The argument extends
+      until the next newline character.
+      """),
+
+    I(name='BINUNICODE',
+      code='X',
+      arg=unicodestring4,
+      stack_before=[],
+      stack_after=[pyunicode],
+      proto=1,
+      doc="""Push a Python Unicode string object.
+
+      There are two arguments:  the first is a 4-byte little-endian signed int
+      giving the number of bytes in the string.  The second is that many
+      bytes, and is the UTF-8 encoding of the Unicode string.
+      """),
+
+    # Ways to spell floats.
+
+    I(name='FLOAT',
+      code='F',
+      arg=floatnl,
+      stack_before=[],
+      stack_after=[pyfloat],
+      proto=0,
+      doc="""Newline-terminated decimal float literal.
+
+      The argument is repr(a_float), and in general requires 17 significant
+      digits for roundtrip conversion to be an identity (this is so for
+      IEEE-754 double precision values, which is what Python float maps to
+      on most boxes).
+
+      In general, FLOAT cannot be used to transport infinities, NaNs, or
+      minus zero across boxes (or even on a single box, if the platform C
+      library can't read the strings it produces for such things -- Windows
+      is like that), but may do less damage than BINFLOAT on boxes with
+      greater precision or dynamic range than IEEE-754 double.
+      """),
+
+    I(name='BINFLOAT',
+      code='G',
+      arg=float8,
+      stack_before=[],
+      stack_after=[pyfloat],
+      proto=1,
+      doc="""Float stored in binary form, with 8 bytes of data.
+
+      This generally requires less than half the space of FLOAT encoding.
+      In general, BINFLOAT cannot be used to transport infinities, NaNs, or
+      minus zero, raises an exception if the exponent exceeds the range of
+      an IEEE-754 double, and retains no more than 53 bits of precision (if
+      there are more than that, "add a half and chop" rounding is used to
+      cut it back to 53 significant bits).
+      """),
+
+    # Ways to build lists.
+
+    I(name='EMPTY_LIST',
+      code=']',
+      arg=None,
+      stack_before=[],
+      stack_after=[pylist],
+      proto=1,
+      doc="Push an empty list."),
+
+    I(name='APPEND',
+      code='a',
+      arg=None,
+      stack_before=[pylist, anyobject],
+      stack_after=[pylist],
+      proto=0,
+      doc="""Append an object to a list.
+
+      Stack before:  ... pylist anyobject
+      Stack after:   ... pylist+[anyobject]
+      """),
+
+    I(name='APPENDS',
+      code='e',
+      arg=None,
+      stack_before=[pylist, markobject, stackslice],
+      stack_after=[pylist],
+      proto=1,
+      doc="""Extend a list by a slice of stack objects.
+
+      Stack before:  ... pylist markobject stackslice
+      Stack after:   ... pylist+stackslice
+      """),
+
+    I(name='LIST',
+      code='l',
+      arg=None,
+      stack_before=[markobject, stackslice],
+      stack_after=[pylist],
+      proto=0,
+      doc="""Build a list out of the topmost stack slice, after markobject.
+
+      All the stack entries following the topmost markobject are placed into
+      a single Python list, which single list object replaces all of the
+      stack from the topmost markobject onward.  For example,
+
+      Stack before: ... markobject 1 2 3 'abc'
+      Stack after:  ... [1, 2, 3, 'abc']
+      """),
+
+    # Ways to build tuples.
+
+    I(name='EMPTY_TUPLE',
+      code=')',
+      arg=None,
+      stack_before=[],
+      stack_after=[pytuple],
+      proto=1,
+      doc="Push an empty tuple."),
+
+    I(name='TUPLE',
+      code='t',
+      arg=None,
+      stack_before=[markobject, stackslice],
+      stack_after=[pytuple],
+      proto=0,
+      doc="""Build a tuple out of the topmost stack slice, after markobject.
+
+      All the stack entries following the topmost markobject are placed into
+      a single Python tuple, which single tuple object replaces all of the
+      stack from the topmost markobject onward.  For example,
+
+      Stack before: ... markobject 1 2 3 'abc'
+      Stack after:  ... (1, 2, 3, 'abc')
+      """),
+
+    # Ways to build dicts.
+
+    I(name='EMPTY_DICT',
+      code='}',
+      arg=None,
+      stack_before=[],
+      stack_after=[pydict],
+      proto=1,
+      doc="Push an empty dict."),
+
+    I(name='DICT',
+      code='d',
+      arg=None,
+      stack_before=[markobject, stackslice],
+      stack_after=[pydict],
+      proto=0,
+      doc="""Build a dict out of the topmost stack slice, after markobject.
+
+      All the stack entries following the topmost markobject are placed into
+      a single Python dict, which single dict object replaces all of the
+      stack from the topmost markobject onward.  The stack slice alternates
+      key, value, key, value, ....  For example,
+
+      Stack before: ... markobject 1 2 3 'abc'
+      Stack after:  ... {1: 2, 3: 'abc'}
+      """),
+
+    I(name='SETITEM',
+      code='s',
+      arg=None,
+      stack_before=[pydict, anyobject, anyobject],
+      stack_after=[pydict],
+      proto=0,
+      doc="""Add a key+value pair to an existing dict.
+
+      Stack before:  ... pydict key value
+      Stack after:   ... pydict
+
+      where pydict has been modified via pydict[key] = value.
+      """),
+
+    I(name='SETITEMS',
+      code='u',
+      arg=None,
+      stack_before=[pydict, markobject, stackslice],
+      stack_after=[pydict],
+      proto=1,
+      doc="""Add an arbitrary number of key+value pairs to an existing dict.
+
+      The slice of the stack following the topmost markobject is taken as
+      an alternating sequence of keys and values, added to the dict
+      immediately under the topmost markobject.  Everything at and after the
+      topmost markobject is popped, leaving the mutated dict at the top
+      of the stack.
+
+      Stack before:  ... pydict markobject key_1 value_1 ... key_n value_n
+      Stack after:   ... pydict
+
+      where pydict has been modified via pydict[key_i] = value_i for i in
+      1, 2, ..., n, and in that order.
+      """),
+
+    # Stack manipulation.
+
+    I(name='POP',
+      code='0',
+      arg=None,
+      stack_before=[anyobject],
+      stack_after=[],
+      proto=0,
+      doc="Discard the top stack item, shrinking the stack by one item."),
+
+    I(name='DUP',
+      code='2',
+      arg=None,
+      stack_before=[anyobject],
+      stack_after=[anyobject, anyobject],
+      proto=0,
+      doc="Push the top stack item onto the stack again, duplicating it."),
+
+    I(name='MARK',
+      code='(',
+      arg=None,
+      stack_before=[],
+      stack_after=[markobject],
+      proto=0,
+      doc="""Push markobject onto the stack.
+
+      markobject is a unique object, used by other opcodes to identify a
+      region of the stack containing a variable number of objects for them
+      to work on.  See markobject.doc for more detail.
+      """),
+
+    I(name='POP_MARK',
+      code='1',
+      arg=None,
+      stack_before=[markobject, stackslice],
+      stack_after=[],
+      proto=0,
+      doc="""Pop all the stack objects at and above the topmost markobject.
+
+      When an opcode using a variable number of stack objects is done,
+      POP_MARK is used to remove those objects, and to remove the markobject
+      that delimited their starting position on the stack.
+      """),
+
+    # Memo manipulation.  There are really only two operations (get and put),
+    # each in all-text, "short binary", and "long binary" flavors.
+
+    I(name='GET',
+      code='g',
+      arg=decimalnl_short,
+      stack_before=[],
+      stack_after=[anyobject],
+      proto=0,
+      doc="""Read an object from the memo and push it on the stack.
+
+      The index of the memo object to push is given by the newline-teriminated
+      decimal string following.  BINGET and LONG_BINGET are space-optimized
+      versions.
+      """),
+
+    I(name='BINGET',
+      code='h',
+      arg=uint1,
+      stack_before=[],
+      stack_after=[anyobject],
+      proto=1,
+      doc="""Read an object from the memo and push it on the stack.
+
+      The index of the memo object to push is given by the 1-byte unsigned
+      integer following.
+      """),
+
+    I(name='LONG_BINGET',
+      code='j',
+      arg=int4,
+      stack_before=[],
+      stack_after=[anyobject],
+      proto=1,
+      doc="""Read an object from the memo and push it on the stack.
+
+      The index of the memo object to push is given by the 4-byte signed
+      little-endian integer following.
+      """),
+
+    I(name='PUT',
+      code='p',
+      arg=decimalnl_short,
+      stack_before=[],
+      stack_after=[],
+      proto=0,
+      doc="""Store the stack top into the memo.  The stack is not popped.
+
+      The index of the memo location to write into is given by the newline-
+      terminated decimal string following.  BINPUT and LONG_BINPUT are
+      space-optimized versions.
+      """),
+
+    I(name='BINPUT',
+      code='q',
+      arg=uint1,
+      stack_before=[],
+      stack_after=[],
+      proto=1,
+      doc="""Store the stack top into the memo.  The stack is not popped.
+
+      The index of the memo location to write into is given by the 1-byte
+      unsigned integer following.
+      """),
+
+    I(name='LONG_BINPUT',
+      code='r',
+      arg=int4,
+      stack_before=[],
+      stack_after=[],
+      proto=1,
+      doc="""Store the stack top into the memo.  The stack is not popped.
+
+      The index of the memo location to write into is given by the 4-byte
+      signed little-endian integer following.
+      """),
+
+    # Push a class object, or module function, on the stack, via its module
+    # and name.
+
+    I(name='GLOBAL',
+      code='c',
+      arg=stringnl_noescape_pair,
+      stack_before=[],
+      stack_after=[anyobject],
+      proto=0,
+      doc="""Push a global object (module.attr) on the stack.
+
+      Two newline-terminated strings follow the GLOBAL opcode.  The first is
+      taken as a module name, and the second as a class name.  The class
+      object module.class is pushed on the stack.  More accurately, the
+      object returned by self.find_class(module, class) is pushed on the
+      stack, so unpickling subclasses can override this form of lookup.
+      """),
+
+    # Ways to build objects of classes pickle doesn't know about directly
+    # (user-defined classes).  I despair of documenting this accurately
+    # and comprehensibly -- you really have to read the pickle code to
+    # find all the special cases.
+
+    I(name='REDUCE',
+      code='R',
+      arg=None,
+      stack_before=[anyobject, anyobject],
+      stack_after=[anyobject],
+      proto=0,
+      doc="""Push an object built from a callable and an argument tuple.
+
+      The opcode is named to remind of the __reduce__() method.
+
+      Stack before: ... callable pytuple
+      Stack after:  ... callable(*pytuple)
+
+      The callable and the argument tuple are the first two items returned
+      by a __reduce__ method.  Applying the callable to the argtuple is
+      supposed to reproduce the original object, or at least get it started.
+      If the __reduce__ method returns a 3-tuple, the last component is an
+      argument to be passed to the object's __setstate__, and then the REDUCE
+      opcode is followed by code to create setstate's argument, and then a
+      BUILD opcode to apply  __setstate__ to that argument.
+
+      There are lots of special cases here.  The argtuple can be None, in
+      which case callable.__basicnew__() is called instead to produce the
+      object to be pushed on the stack.  This appears to be a trick unique
+      to ExtensionClasses, and is deprecated regardless.
+
+      If type(callable) is not ClassType, REDUCE complains unless the
+      callable has been registered with the copy_reg module's
+      safe_constructors dict, or the callable has a magic
+      '__safe_for_unpickling__' attribute with a true value.  I'm not sure
+      why it does this, but I've sure seen this complaint often enough when
+      I didn't want to <wink>.
+      """),
+
+    I(name='BUILD',
+      code='b',
+      arg=None,
+      stack_before=[anyobject, anyobject],
+      stack_after=[anyobject],
+      proto=0,
+      doc="""Finish building an object, via __setstate__ or dict update.
+
+      Stack before: ... anyobject argument
+      Stack after:  ... anyobject
+
+      where anyobject may have been mutated, as follows:
+
+      If the object has a __setstate__ method,
+
+          anyobject.__setstate__(argument)
+
+      is called.
+
+      Else the argument must be a dict, the object must have a __dict__, and
+      the object is updated via
+
+          anyobject.__dict__.update(argument)
+
+      This may raise RuntimeError in restricted execution mode (which
+      disallows access to __dict__ directly); in that case, the object
+      is updated instead via
+
+          for k, v in argument.items():
+              anyobject[k] = v
+      """),
+
+    I(name='INST',
+      code='i',
+      arg=stringnl_noescape_pair,
+      stack_before=[markobject, stackslice],
+      stack_after=[anyobject],
+      proto=0,
+      doc="""Build a class instance.
+
+      This is the protocol 0 version of protocol 1's OBJ opcode.
+      INST is followed by two newline-terminated strings, giving a
+      module and class name, just as for the GLOBAL opcode (and see
+      GLOBAL for more details about that).  self.find_class(module, name)
+      is used to get a class object.
+
+      In addition, all the objects on the stack following the topmost
+      markobject are gathered into a tuple and popped (along with the
+      topmost markobject), just as for the TUPLE opcode.
+
+      Now it gets complicated.  If all of these are true:
+
+        + The argtuple is empty (markobject was at the top of the stack
+          at the start).
+
+        + It's an old-style class object (the type of the class object is
+          ClassType).
+
+        + The class object does not have a __getinitargs__ attribute.
+
+      then we want to create an old-style class instance without invoking
+      its __init__() method (pickle has waffled on this over the years; not
+      calling __init__() is current wisdom).  In this case, an instance of
+      an old-style dummy class is created, and then we try to rebind its
+      __class__ attribute to the desired class object.  If this succeeds,
+      the new instance object is pushed on the stack, and we're done.  In
+      restricted execution mode it can fail (assignment to __class__ is
+      disallowed), and I'm not really sure what happens then -- it looks
+      like the code ends up calling the class object's __init__ anyway,
+      via falling into the next case.
+
+      Else (the argtuple is not empty, it's not an old-style class object,
+      or the class object does have a __getinitargs__ attribute), the code
+      first insists that the class object have a __safe_for_unpickling__
+      attribute.  Unlike as for the __safe_for_unpickling__ check in REDUCE,
+      it doesn't matter whether this attribute has a true or false value, it
+      only matters whether it exists (XXX this smells like a bug).  If
+      __safe_for_unpickling__ dosn't exist, UnpicklingError is raised.
+
+      Else (the class object does have a __safe_for_unpickling__ attr),
+      the class object obtained from INST's arguments is applied to the
+      argtuple obtained from the stack, and the resulting instance object
+      is pushed on the stack.
+      """),
+
+    I(name='OBJ',
+      code='o',
+      arg=None,
+      stack_before=[markobject, anyobject, stackslice],
+      stack_after=[anyobject],
+      proto=1,
+      doc="""Build a class instance.
+
+      This is the protocol 1 version of protocol 0's INST opcode, and is
+      very much like it.  The major difference is that the class object
+      is taken off the stack, allowing it to be retrieved from the memo
+      repeatedly if several instances of the same class are created.  This
+      can be much more efficient (in both time and space) than repeatedly
+      embedding the module and class names in INST opcodes.
+
+      Unlike INST, OBJ takes no arguments from the opcode stream.  Instead
+      the class object is taken off the stack, immediately above the
+      topmost markobject:
+
+      Stack before: ... markobject classobject stackslice
+      Stack after:  ... new_instance_object
+
+      As for INST, the remainder of the stack above the markobject is
+      gathered into an argument tuple, and then the logic seems identical,
+      except that no __safe_for_unpickling__ check is done (XXX this smells
+      like a bug).  See INST for the gory details.
+      """),
+
+    # Machine control.
+
+    I(name='STOP',
+      code='.',
+      arg=None,
+      stack_before=[anyobject],
+      stack_after=[],
+      proto=0,
+      doc="""Stop the unpickling machine.
+
+      Every pickle ends with this opcode.  The object at the top of the stack
+      is popped, and that's the result of unpickling.  The stack should be
+      empty then.
+      """),
+
+    # Ways to deal with persistent IDs.
+
+    I(name='PERSID',
+      code='P',
+      arg=stringnl_noescape,
+      stack_before=[],
+      stack_after=[anyobject],
+      proto=0,
+      doc="""Push an object identified by a persistent ID.
+
+      The pickle module doesn't define what a persistent ID means.  PERSID's
+      argument is a newline-terminated str-style (no embedded escapes, no
+      bracketing quote characters) string, which *is* "the persistent ID".
+      The unpickler passes this string to self.persistent_load().  Whatever
+      object that returns is pushed on the stack.  There is no implementation
+      of persistent_load() in Python's unpickler:  it must be supplied by an
+      unpickler subclass.
+      """),
+
+    I(name='BINPERSID',
+      code='Q',
+      arg=None,
+      stack_before=[anyobject],
+      stack_after=[anyobject],
+      proto=1,
+      doc="""Push an object identified by a persistent ID.
+
+      Like PERSID, except the persistent ID is popped off the stack (instead
+      of being a string embedded in the opcode bytestream).  The persistent
+      ID is passed to self.persistent_load(), and whatever object that
+      returns is pushed on the stack.  See PERSID for more detail.
+      """),
+]
+del I
+
+# Verify uniqueness of .name and .code members.
+name2i = {}
+code2i = {}
+
+for i, d in enumerate(opcodes):
+    if d.name in name2i:
+        raise ValueError("repeated name %r at indices %d and %d" %
+                         (d.name, name2i[d.name], i))
+    if d.code in code2i:
+        raise ValueError("repeated code %r at indices %d and %d" %
+                         (d.code, code2i[d.code], i))
+
+    name2i[d.name] = i
+    code2i[d.code] = i
+
+del name2i, code2i, i, d
+
+##############################################################################
+# Build a code2op dict, mapping opcode characters to OpcodeInfo records.
+# Also ensure we've got the same stuff as pickle.py, although the
+# introspection here is dicey.
+
+code2op = {}
+for d in opcodes:
+    code2op[d.code] = d
+del d
+
+def assure_pickle_consistency(verbose=False):
+    import pickle, re
+
+    copy = code2op.copy()
+    for name in pickle.__all__:
+        if not re.match("[A-Z][A-Z0-9_]+$", name):
+            if verbose:
+                print "skipping %r: it doesn't look like an opcode name" % name
+            continue
+        picklecode = getattr(pickle, name)
+        if not isinstance(picklecode, str) or len(picklecode) != 1:
+            if verbose:
+                print ("skipping %r: value %r doesn't look like a pickle "
+                       "code" % (name, picklecode))
+            continue
+        if picklecode in copy:
+            if verbose:
+                print "checking name %r w/ code %r for consistency" % (
+                      name, picklecode)
+            d = copy[picklecode]
+            if d.name != name:
+                raise ValueError("for pickle code %r, pickle.py uses name %r "
+                                 "but we're using name %r" % (picklecode,
+                                                              name,
+                                                              d.name))
+            # Forget this one.  Any left over in copy at the end are a problem
+            # of a different kind.
+            del copy[picklecode]
+        else:
+            raise ValueError("pickle.py appears to have a pickle opcode with "
+                             "name %r and code %r, but we don't" %
+                             (name, picklecode))
+    if copy:
+        msg = ["we appear to have pickle opcodes that pickle.py doesn't have:"]
+        for code, d in copy.items():
+            msg.append("    name %r with code %r" % (d.name, code))
+        raise ValueError("\n".join(msg))
+
+assure_pickle_consistency()
+
+##############################################################################
+# A pickle opcode generator.
+
+def genops(pickle):
+    """"Generate all the opcodes in a pickle.
+
+    'pickle' is a file-like object, or string, containing the pickle.
+
+    Each opcode in the pickle is generated, from the current pickle position,
+    stopping after a STOP opcode is delivered.  A triple is generated for
+    each opcode:
+
+        opcode, arg, pos
+
+    opcode is an OpcodeInfo record, describing the current opcode.
+
+    If the opcode has an argument embedded in the pickle, arg is its decoded
+    value, as a Python object.  If the opcode doesn't have an argument, arg
+    is None.
+
+    If the pickle has a tell() method, pos was the value of pickle.tell()
+    before reading the current opcode.  If the pickle is a string object,
+    it's wrapped in a StringIO object, and the latter's tell() result is
+    used.  Else (the pickle doesn't have a tell(), and it's not obvious how
+    to query its current position) pos is None.
+    """
+
+    import cStringIO as StringIO
+
+    if isinstance(pickle, str):
+        pickle = StringIO.StringIO(pickle)
+
+    if hasattr(pickle, "tell"):
+        getpos = pickle.tell
+    else:
+        getpos = lambda: None
+
+    while True:
+        pos = getpos()
+        code = pickle.read(1)
+        opcode = code2op.get(code)
+        if opcode is None:
+            if code == "":
+                raise ValueError("pickle exhausted before seeing STOP")
+            else:
+                raise ValueError("at position %s, opcode %r unknown" % (
+                                 pos is None and "<unknown>" or pos,
+                                 code))
+        if opcode.arg is None:
+            arg = None
+        else:
+            arg = opcode.arg.reader(pickle)
+        yield opcode, arg, pos
+        if code == '.':
+            assert opcode.name == 'STOP'
+            break
+
+##############################################################################
+# A symbolic pickle disassembler.
+
+def dis(pickle, out=None, indentlevel=4):
+    """Produce a symbolic disassembly of a pickle.
+
+    'pickle' is a file-like object, or string, containing a (at least one)
+    pickle.  The pickle is disassembled from the current position, through
+    the first STOP opcode encountered.
+
+    Optional arg 'out' is a file-like object to which the disassembly is
+    printed.  It defaults to sys.stdout.
+
+    Optional arg indentlevel is the number of blanks by which to indent
+    a new MARK level.  It defaults to 4.
+    """
+
+    markstack = []
+    indentchunk = ' ' * indentlevel
+    for opcode, arg, pos in genops(pickle):
+        if pos is not None:
+            print >> out, "%5d:" % pos,
+
+        line = "%s %s%s" % (opcode.code,
+                            indentchunk * len(markstack),
+                            opcode.name)
+
+        markmsg = None
+        if markstack and markobject in opcode.stack_before:
+                assert markobject not in opcode.stack_after
+                markpos = markstack.pop()
+                if markpos is not None:
+                    markmsg = "(MARK at %d)" % markpos
+
+        if arg is not None or markmsg:
+            # make a mild effort to align arguments
+            line += ' ' * (10 - len(opcode.name))
+            if arg is not None:
+                line += ' ' + repr(arg)
+            if markmsg:
+                line += ' ' + markmsg
+        print >> out, line
+
+        if markobject in opcode.stack_after:
+            assert markobject not in opcode.stack_before
+            markstack.append(pos)
+
+
+_dis_test = """
+>>> import pickle
+>>> x = [1, 2, (3, 4), {'abc': u"def"}]
+>>> pik = pickle.dumps(x)
+>>> dis(pik)
+    0: ( MARK
+    1: l     LIST       (MARK at 0)
+    2: p PUT        0
+    5: I INT        1
+    8: a APPEND
+    9: I INT        2
+   12: a APPEND
+   13: ( MARK
+   14: I     INT        3
+   17: I     INT        4
+   20: t     TUPLE      (MARK at 13)
+   21: p PUT        1
+   24: a APPEND
+   25: ( MARK
+   26: d     DICT       (MARK at 25)
+   27: p PUT        2
+   30: S STRING     'abc'
+   37: p PUT        3
+   40: V UNICODE    u'def'
+   45: p PUT        4
+   48: s SETITEM
+   49: a APPEND
+   50: . STOP
+
+Try again with a "binary" pickle.
+
+>>> pik = pickle.dumps(x, 1)
+>>> dis(pik)
+    0: ] EMPTY_LIST
+    1: q BINPUT     0
+    3: ( MARK
+    4: K     BININT1    1
+    6: K     BININT1    2
+    8: (     MARK
+    9: K         BININT1    3
+   11: K         BININT1    4
+   13: t         TUPLE      (MARK at 8)
+   14: q     BINPUT     1
+   16: }     EMPTY_DICT
+   17: q     BINPUT     2
+   19: U     SHORT_BINSTRING 'abc'
+   24: q     BINPUT     3
+   26: X     BINUNICODE u'def'
+   34: q     BINPUT     4
+   36: s     SETITEM
+   37: e     APPENDS    (MARK at 3)
+   38: . STOP
+
+Exercise the INST/OBJ/BUILD family.
+
+>>> import random
+>>> dis(pickle.dumps(random.random))
+    0: c GLOBAL     'random.random'
+   15: p PUT        0
+   18: . STOP
+
+>>> x = [pickle.PicklingError()] * 2
+>>> dis(pickle.dumps(x))
+    0: ( MARK
+    1: l     LIST       (MARK at 0)
+    2: p PUT        0
+    5: ( MARK
+    6: i     INST       'pickle.PicklingError' (MARK at 5)
+   28: p PUT        1
+   31: ( MARK
+   32: d     DICT       (MARK at 31)
+   33: p PUT        2
+   36: S STRING     'args'
+   44: p PUT        3
+   47: ( MARK
+   48: t     TUPLE      (MARK at 47)
+   49: p PUT        4
+   52: s SETITEM
+   53: b BUILD
+   54: a APPEND
+   55: g GET        1
+   58: a APPEND
+   59: . STOP
+
+>>> dis(pickle.dumps(x, 1))
+    0: ] EMPTY_LIST
+    1: q BINPUT     0
+    3: ( MARK
+    4: (     MARK
+    5: c         GLOBAL     'pickle.PicklingError'
+   27: q         BINPUT     1
+   29: o         OBJ        (MARK at 4)
+   30: q     BINPUT     2
+   32: }     EMPTY_DICT
+   33: q     BINPUT     3
+   35: U     SHORT_BINSTRING 'args'
+   41: q     BINPUT     4
+   43: )     EMPTY_TUPLE
+   44: s     SETITEM
+   45: b     BUILD
+   46: h     BINGET     2
+   48: e     APPENDS    (MARK at 3)
+   49: . STOP
+
+Try "the canonical" recursive-object test.
+
+>>> L = []
+>>> T = L,
+>>> L.append(T)
+>>> L[0] is T
+True
+>>> T[0] is L
+True
+>>> L[0][0] is L
+True
+>>> T[0][0] is T
+True
+>>> dis(pickle.dumps(L))
+    0: ( MARK
+    1: l     LIST       (MARK at 0)
+    2: p PUT        0
+    5: ( MARK
+    6: g     GET        0
+    9: t     TUPLE      (MARK at 5)
+   10: p PUT        1
+   13: a APPEND
+   14: . STOP
+>>> dis(pickle.dumps(L, 1))
+    0: ] EMPTY_LIST
+    1: q BINPUT     0
+    3: ( MARK
+    4: h     BINGET     0
+    6: t     TUPLE      (MARK at 3)
+    7: q BINPUT     1
+    9: a APPEND
+   10: . STOP
+
+The protocol 0 pickle of the tuple causes the disassembly to get confused,
+as it doesn't realize that the POP opcode at 16 gets rid of the MARK at 0
+(so the output remains indented until the end).  The protocol 1 pickle
+doesn't trigger this glitch, because the disassembler realizes that
+POP_MARK gets rid of the MARK.  Doing a better job on the protocol 0
+pickle would require the disassembler to emulate the stack.
+
+>>> dis(pickle.dumps(T))
+    0: ( MARK
+    1: (     MARK
+    2: l         LIST       (MARK at 1)
+    3: p     PUT        0
+    6: (     MARK
+    7: g         GET        0
+   10: t         TUPLE      (MARK at 6)
+   11: p     PUT        1
+   14: a     APPEND
+   15: 0     POP
+   16: 0     POP
+   17: g     GET        1
+   20: .     STOP
+>>> dis(pickle.dumps(T, 1))
+    0: ( MARK
+    1: ]     EMPTY_LIST
+    2: q     BINPUT     0
+    4: (     MARK
+    5: h         BINGET     0
+    7: t         TUPLE      (MARK at 4)
+    8: q     BINPUT     1
+   10: a     APPEND
+   11: 1     POP_MARK   (MARK at 0)
+   12: h BINGET     1
+   14: . STOP
+"""
+
+__test__ = {'dissassembler_test': _dis_test,
+           }
+
+def _test():
+    import doctest
+    return doctest.testmod()
+
+if __name__ == "__main__":
+    _test()