diff options
| -rw-r--r-- | Lib/pickletools.py | 1794 | ||||
| -rw-r--r-- | Lib/test/test_pickletools.py | 3 | ||||
| -rw-r--r-- | Misc/NEWS | 8 |
3 files changed, 1803 insertions, 2 deletions
diff --git a/Lib/pickletools.py b/Lib/pickletools.py new file mode 100644 index 0000000..2f03082 --- /dev/null +++ b/Lib/pickletools.py @@ -0,0 +1,1794 @@ +""""Executable documentation" for the pickle module. + +Extensive comments about the pickle protocols and pickle-machine opcodes +can be found here. Some functions meant for external use: + +genops(pickle) + Generate all the opcodes in a pickle, as (opcode, arg, position) triples. + +dis(pickle, out=None, indentlevel=4) + Print a symbolic disassembly of a pickle. +""" + +# Other ideas: +# +# - A pickle verifier: read a pickle and check it exhaustively for +# well-formedness. +# +# - A protocol identifier: examine a pickle and return its protocol number +# (== the highest .proto attr value among all the opcodes in the pickle). +# +# - A pickle optimizer: for example, tuple-building code is sometimes more +# elaborate than necessary, catering for the possibility that the tuple +# is recursive. Or lots of times a PUT is generated that's never accessed +# by a later GET. + + +""" +"A pickle" is a program for a virtual pickle machine (PM, but more accurately +called an unpickling machine). It's a sequence of opcodes, interpreted by the +PM, building an arbitrarily complex Python object. + +For the most part, the PM is very simple: there are no looping, testing, or +conditional instructions, no arithmetic and no function calls. Opcodes are +executed once each, from first to last, until a STOP opcode is reached. + +The PM has two data areas, "the stack" and "the memo". + +Many opcodes push Python objects onto the stack; e.g., INT pushes a Python +integer object on the stack, whose value is gotten from a decimal string +literal immediately following the INT opcode in the pickle bytestream. Other +opcodes take Python objects off the stack. The result of unpickling is +whatever object is left on the stack when the final STOP opcode is executed. + +The memo is simply an array of objects, or it can be implemented as a dict +mapping little integers to objects. The memo serves as the PM's "long term +memory", and the little integers indexing the memo are akin to variable +names. Some opcodes pop a stack object into the memo at a given index, +and others push a memo object at a given index onto the stack again. + +At heart, that's all the PM has. Subtleties arise for these reasons: + ++ Object identity. Objects can be arbitrarily complex, and subobjects + may be shared (for example, the list [a, a] refers to the same object a + twice). It can be vital that unpickling recreate an isomorphic object + graph, faithfully reproducing sharing. + ++ Recursive objects. For example, after "L = []; L.append(L)", L is a + list, and L[0] is the same list. This is related to the object identity + point, and some sequences of pickle opcodes are subtle in order to + get the right result in all cases. + ++ Things pickle doesn't know everything about. Examples of things pickle + does know everything about are Python's builtin scalar and container + types, like ints and tuples. They generally have opcodes dedicated to + them. For things like module references and instances of user-defined + classes, pickle's knowledge is limited. Historically, many enhancements + have been made to the pickle protocol in order to do a better (faster, + and/or more compact) job on those. + ++ Backward compatibility and micro-optimization. As explained below, + pickle opcodes never go away, not even when better ways to do a thing + get invented. The repertoire of the PM just keeps growing over time. + So, e.g., there are now six distinct opcodes for building a Python integer, + five of them devoted to "short" integers. Even so, the only way to pickle + a Python long int takes time quadratic in the number of digits, for both + pickling and unpickling. This isn't so much a subtlety as a source of + wearying complication. + + +Pickle protocols: + +For compatibility, the meaning of a pickle opcode never changes. Instead new +pickle opcodes get added, and each version's unpickler can handle all the +pickle opcodes in all protocol versions to date. So old pickles continue to +be readable forever. The pickler can generally be told to restrict itself to +the subset of opcodes available under previous protocol versions too, so that +users can create pickles under the current version readable by older +versions. However, a pickle does not contain its version number embedded +within it. If an older unpickler tries to read a pickle using a later +protocol, the result is most likely an exception due to seeing an unknown (in +the older unpickler) opcode. + +The original pickle used what's now called "protocol 0", and what was called +"text mode" before Python 2.3. The entire pickle bytestream is made up of +printable 7-bit ASCII characters, plus the newline character, in protocol 0. +That's why it was called text mode. + +The second major set of additions is now called "protocol 1", and was called +"binary mode" before Python 2.3. This added many opcodes with arguments +consisting of arbitrary bytes, including NUL bytes and unprintable "high bit" +bytes. Binary mode pickles can be substantially smaller than equivalent +text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte +int as 4 bytes following the opcode, which is cheaper to unpickle than the +(perhaps) 11-character decimal string attached to INT. + +The third major set of additions came in Python 2.3, and is called "protocol +2". XXX Write a short blurb when Guido figures out what they are <wink>. XXX +""" + +# Meta-rule: Descriptions are stored in instances of descriptor objects, +# with plain constructors. No meta-language is defined from which +# descriptors could be constructed. If you want, e.g., XML, write a little +# program to generate XML from the objects. + +############################################################################## +# Some pickle opcodes have an argument, following the opcode in the +# bytestream. An argument is of a specific type, described by an instance +# of ArgumentDescriptor. These are not to be confused with arguments taken +# off the stack -- ArgumentDescriptor applies only to arguments embedded in +# the opcode stream, immediately following an opcode. + +# Represents the number of bytes consumed by an argument delimited by the +# next newline character. +UP_TO_NEWLINE = -1 + +# Represents the number of bytes consumed by a two-argument opcode where +# the first argument gives the number of bytes in the second argument. +TAKEN_FROM_ARGUMENT = -2 + +class ArgumentDescriptor(object): + __slots__ = ( + # name of descriptor record, also a module global name; a string + 'name', + + # length of argument, in bytes; an int; UP_TO_NEWLINE and + # TAKEN_FROM_ARGUMENT are negative values for variable-length cases + 'n', + + # a function taking a file-like object, reading this kind of argument + # from the object at the current position, advancing the current + # position by n bytes, and returning the value of the argument + 'reader', + + # human-readable docs for this arg descriptor; a string + 'doc', + ) + + def __init__(self, name, n, reader, doc): + assert isinstance(name, str) + self.name = name + + assert isinstance(n, int) and (n >= 0 or + n is UP_TO_NEWLINE or + n is TAKEN_FROM_ARGUMENT) + self.n = n + + self.reader = reader + + assert isinstance(doc, str) + self.doc = doc + +from struct import unpack as _unpack + +def read_uint1(f): + """ + >>> import StringIO + >>> read_uint1(StringIO.StringIO('\\xff')) + 255 + """ + + data = f.read(1) + if data: + return ord(data) + raise ValueError("not enough data in stream to read uint1") + +uint1 = ArgumentDescriptor( + name='uint1', + n=1, + reader=read_uint1, + doc="One-byte unsigned integer.") + + +def read_uint2(f): + """ + >>> import StringIO + >>> read_uint2(StringIO.StringIO('\\xff\\x00')) + 255 + >>> read_uint2(StringIO.StringIO('\\xff\\xff')) + 65535 + """ + + data = f.read(2) + if len(data) == 2: + return _unpack("<H", data)[0] + raise ValueError("not enough data in stream to read uint2") + +uint2 = ArgumentDescriptor( + name='uint2', + n=2, + reader=read_uint2, + doc="Two-byte unsigned integer, little-endian.") + + +def read_int4(f): + """ + >>> import StringIO + >>> read_int4(StringIO.StringIO('\\xff\\x00\\x00\\x00')) + 255 + >>> read_int4(StringIO.StringIO('\\x00\\x00\\x00\\x80')) == -(2**31) + True + """ + + data = f.read(4) + if len(data) == 4: + return _unpack("<i", data)[0] + raise ValueError("not enough data in stream to read int4") + +int4 = ArgumentDescriptor( + name='int4', + n=4, + reader=read_int4, + doc="Four-byte signed integer, little-endian, 2's complement.") + + +def read_stringnl(f, decode=True, stripquotes=True): + """ + >>> import StringIO + >>> read_stringnl(StringIO.StringIO("'abcd'\\nefg\\n")) + 'abcd' + + >>> read_stringnl(StringIO.StringIO("\\n")) + Traceback (most recent call last): + ... + ValueError: no string quotes around '' + + >>> read_stringnl(StringIO.StringIO("\\n"), stripquotes=False) + '' + + >>> read_stringnl(StringIO.StringIO("''\\n")) + '' + + >>> read_stringnl(StringIO.StringIO('"abcd"')) + Traceback (most recent call last): + ... + ValueError: no newline found when trying to read stringnl + + Embedded escapes are undone in the result. + >>> read_stringnl(StringIO.StringIO("'a\\\\nb\\x00c\\td'\\n'e'")) + 'a\\nb\\x00c\\td' + """ + + data = f.readline() + if not data.endswith('\n'): + raise ValueError("no newline found when trying to read stringnl") + data = data[:-1] # lose the newline + + if stripquotes: + for q in "'\"": + if data.startswith(q): + if not data.endswith(q): + raise ValueError("strinq quote %r not found at both " + "ends of %r" % (q, data)) + data = data[1:-1] + break + else: + raise ValueError("no string quotes around %r" % data) + + # I'm not sure when 'string_escape' was added to the std codecs; it's + # crazy not to use it if it's there. + if decode: + data = data.decode('string_escape') + return data + +stringnl = ArgumentDescriptor( + name='stringnl', + n=UP_TO_NEWLINE, + reader=read_stringnl, + doc="""A newline-terminated string. + + This is a repr-style string, with embedded escapes, and + bracketing quotes. + """) + +def read_stringnl_noescape(f): + return read_stringnl(f, decode=False, stripquotes=False) + +stringnl_noescape = ArgumentDescriptor( + name='stringnl_noescape', + n=UP_TO_NEWLINE, + reader=read_stringnl_noescape, + doc="""A newline-terminated string. + + This is a str-style string, without embedded escapes, + or bracketing quotes. It should consist solely of + printable ASCII characters. + """) + +def read_stringnl_noescape_pair(f): + """ + >>> import StringIO + >>> read_stringnl_noescape_pair(StringIO.StringIO("Queue\\nEmpty\\njunk")) + 'Queue.Empty' + """ + + return "%s.%s" % (read_stringnl_noescape(f), read_stringnl_noescape(f)) + +stringnl_noescape_pair = ArgumentDescriptor( + name='stringnl_noescape_pair', + n=UP_TO_NEWLINE, + reader=read_stringnl_noescape_pair, + doc="""A pair of newline-terminated strings. + + These are str-style strings, without embedded + escapes, or bracketing quotes. They should + consist solely of printable ASCII characters. + The pair is returned as a single string, with + a single '.' separating the two strings. + """) + +def read_string4(f): + """ + >>> import StringIO + >>> read_string4(StringIO.StringIO("\\x00\\x00\\x00\\x00abc")) + '' + >>> read_string4(StringIO.StringIO("\\x03\\x00\\x00\\x00abcdef")) + 'abc' + >>> read_string4(StringIO.StringIO("\\x00\\x00\\x00\\x03abcdef")) + Traceback (most recent call last): + ... + ValueError: expected 50331648 bytes in a string4, but only 6 remain + """ + + n = read_int4(f) + if n < 0: + raise ValueError("string4 byte count < 0: %d" % n) + data = f.read(n) + if len(data) == n: + return data + raise ValueError("expected %d bytes in a string4, but only %d remain" % + (n, len(data))) + +string4 = ArgumentDescriptor( + name="string4", + n=TAKEN_FROM_ARGUMENT, + reader=read_string4, + doc="""A counted string. + + The first argument is a 4-byte little-endian signed int giving + the number of bytes in the string, and the second argument is + that many bytes. + """) + + +def read_string1(f): + """ + >>> import StringIO + >>> read_string1(StringIO.StringIO("\\x00")) + '' + >>> read_string1(StringIO.StringIO("\\x03abcdef")) + 'abc' + """ + + n = read_uint1(f) + assert n >= 0 + data = f.read(n) + if len(data) == n: + return data + raise ValueError("expected %d bytes in a string1, but only %d remain" % + (n, len(data))) + +string1 = ArgumentDescriptor( + name="string1", + n=TAKEN_FROM_ARGUMENT, + reader=read_string1, + doc="""A counted string. + + The first argument is a 1-byte unsigned int giving the number + of bytes in the string, and the second argument is that many + bytes. + """) + + +def read_unicodestringnl(f): + """ + >>> import StringIO + >>> read_unicodestringnl(StringIO.StringIO("abc\\uabcd\\njunk")) + u'abc\\uabcd' + """ + + data = f.readline() + if not data.endswith('\n'): + raise ValueError("no newline found when trying to read " + "unicodestringnl") + data = data[:-1] # lose the newline + return unicode(data, 'raw-unicode-escape') + +unicodestringnl = ArgumentDescriptor( + name='unicodestringnl', + n=UP_TO_NEWLINE, + reader=read_unicodestringnl, + doc="""A newline-terminated Unicode string. + + This is raw-unicode-escape encoded, so consists of + printable ASCII characters, and may contain embedded + escape sequences. + """) + +def read_unicodestring4(f): + """ + >>> import StringIO + >>> s = u'abcd\\uabcd' + >>> enc = s.encode('utf-8') + >>> enc + 'abcd\\xea\\xaf\\x8d' + >>> n = chr(len(enc)) + chr(0) * 3 # little-endian 4-byte length + >>> t = read_unicodestring4(StringIO.StringIO(n + enc + 'junk')) + >>> s == t + True + + >>> read_unicodestring4(StringIO.StringIO(n + enc[:-1])) + Traceback (most recent call last): + ... + ValueError: expected 7 bytes in a unicodestring4, but only 6 remain + """ + + n = read_int4(f) + if n < 0: + raise ValueError("unicodestring4 byte count < 0: %d" % n) + data = f.read(n) + if len(data) == n: + return unicode(data, 'utf-8') + raise ValueError("expected %d bytes in a unicodestring4, but only %d " + "remain" % (n, len(data))) + +unicodestring4 = ArgumentDescriptor( + name="unicodestring4", + n=TAKEN_FROM_ARGUMENT, + reader=read_unicodestring4, + doc="""A counted Unicode string. + + The first argument is a 4-byte little-endian signed int + giving the number of bytes in the string, and the second + argument-- the UTF-8 encoding of the Unicode string -- + contains that many bytes. + """) + + +def read_decimalnl_short(f): + """ + >>> import StringIO + >>> read_decimalnl_short(StringIO.StringIO("1234\\n56")) + 1234 + + >>> read_decimalnl_short(StringIO.StringIO("1234L\\n56")) + Traceback (most recent call last): + ... + ValueError: trailing 'L' not allowed in '1234L' + """ + + s = read_stringnl(f, decode=False, stripquotes=False) + if s.endswith("L"): + raise ValueError("trailing 'L' not allowed in %r" % s) + + # It's not necessarily true that the result fits in a Python short int: + # the pickle may have been written on a 64-bit box. There's also a hack + # for True and False here. + if s == "00": + return False + elif s == "01": + return True + + try: + return int(s) + except OverflowError: + return long(s) + +def read_decimalnl_long(f): + """ + >>> import StringIO + + >>> read_decimalnl_long(StringIO.StringIO("1234\\n56")) + Traceback (most recent call last): + ... + ValueError: trailing 'L' required in '1234' + + Someday the trailing 'L' will probably go away from this output. + + >>> read_decimalnl_long(StringIO.StringIO("1234L\\n56")) + 1234L + + >>> read_decimalnl_long(StringIO.StringIO("123456789012345678901234L\\n6")) + 123456789012345678901234L + """ + + s = read_stringnl(f, decode=False, stripquotes=False) + if not s.endswith("L"): + raise ValueError("trailing 'L' required in %r" % s) + return long(s) + + +decimalnl_short = ArgumentDescriptor( + name='decimalnl_short', + n=UP_TO_NEWLINE, + reader=read_decimalnl_short, + doc="""A newline-terminated decimal integer literal. + + This never has a trailing 'L', and the integer fit + in a short Python int on the box where the pickle + was written -- but there's no guarantee it will fit + in a short Python int on the box where the pickle + is read. + """) + +decimalnl_long = ArgumentDescriptor( + name='decimalnl_long', + n=UP_TO_NEWLINE, + reader=read_decimalnl_long, + doc="""A newline-terminated decimal integer literal. + + This has a trailing 'L', and can represent integers + of any size. + """) + + +def read_floatnl(f): + """ + >>> import StringIO + >>> read_floatnl(StringIO.StringIO("-1.25\\n6")) + -1.25 + """ + s = read_stringnl(f, decode=False, stripquotes=False) + return float(s) + +floatnl = ArgumentDescriptor( + name='floatnl', + n=UP_TO_NEWLINE, + reader=read_floatnl, + doc="""A newline-terminated decimal floating literal. + + In general this requires 17 significant digits for roundtrip + identity, and pickling then unpickling infinities, NaNs, and + minus zero doesn't work across boxes, or on some boxes even + on itself (e.g., Windows can't read the strings it produces + for infinities or NaNs). + """) + +def read_float8(f): + """ + >>> import StringIO, struct + >>> raw = struct.pack(">d", -1.25) + >>> raw + '\\xbf\\xf4\\x00\\x00\\x00\\x00\\x00\\x00' + >>> read_float8(StringIO.StringIO(raw + "\\n")) + -1.25 + """ + + data = f.read(8) + if len(data) == 8: + return _unpack(">d", data)[0] + raise ValueError("not enough data in stream to read float8") + + +float8 = ArgumentDescriptor( + name='float8', + n=8, + reader=read_float8, + doc="""An 8-byte binary representation of a float, big-endian. + + The format is unique to Python, and shared with the struct + module (format string '>d') "in theory" (the struct and cPickle + implementations don't share the code -- they should). It's + strongly related to the IEEE-754 double format, and, in normal + cases, is in fact identical to the big-endian 754 double format. + On other boxes the dynamic range is limited to that of a 754 + double, and "add a half and chop" rounding is used to reduce + the precision to 53 bits. However, even on a 754 box, + infinities, NaNs, and minus zero may not be handled correctly + (may not survive roundtrip pickling intact). + """) + +############################################################################## +# Object descriptors. The stack used by the pickle machine holds objects, +# and in the stack_before and stack_after attributes of OpcodeInfo +# descriptors we need names to describe the various types of objects that can +# appear on the stack. + +class StackObject(object): + __slots__ = ( + # name of descriptor record, for info only + 'name', + + # type of object, or tuple of type objects (meaning the object can + # be of any type in the tuple) + 'obtype', + + # human-readable docs for this kind of stack object; a string + 'doc', + ) + + def __init__(self, name, obtype, doc): + assert isinstance(name, str) + self.name = name + + assert isinstance(obtype, type) or isinstance(obtype, tuple) + if isinstance(obtype, tuple): + for contained in obtype: + assert isinstance(contained, type) + self.obtype = obtype + + assert isinstance(doc, str) + self.doc = doc + + +pyint = StackObject( + name='int', + obtype=int, + doc="A short (as opposed to long) Python integer object.") + +pylong = StackObject( + name='long', + obtype=long, + doc="A long (as opposed to short) Python integer object.") + +pyinteger_or_bool = StackObject( + name='int_or_bool', + obtype=(int, long, bool), + doc="A Python integer object (short or long), or " + "a Python bool.") + +pyfloat = StackObject( + name='float', + obtype=float, + doc="A Python float object.") + +pystring = StackObject( + name='str', + obtype=str, + doc="A Python string object.") + +pyunicode = StackObject( + name='unicode', + obtype=unicode, + doc="A Python Unicode string object.") + +pynone = StackObject( + name="None", + obtype=type(None), + doc="The Python None object.") + +pytuple = StackObject( + name="tuple", + obtype=tuple, + doc="A Python tuple object.") + +pylist = StackObject( + name="list", + obtype=list, + doc="A Python list object.") + +pydict = StackObject( + name="dict", + obtype=dict, + doc="A Python dict object.") + +anyobject = StackObject( + name='any', + obtype=object, + doc="Any kind of object whatsoever.") + +markobject = StackObject( + name="mark", + obtype=StackObject, + doc="""'The mark' is a unique object. + + Opcodes that operate on a variable number of objects + generally don't embed the count of objects in the opcode, + or pull it off the stack. Instead the MARK opcode is used + to push a special marker object on the stack, and then + some other opcodes grab all the objects from the top of + the stack down to (but not including) the topmost marker + object. + """) + +stackslice = StackObject( + name="stackslice", + obtype=StackObject, + doc="""An object representing a contiguous slice of the stack. + + This is used in conjuction with markobject, to represent all + of the stack following the topmost markobject. For example, + the POP_MARK opcode changes the stack from + + [..., markobject, stackslice] + to + [...] + + No matter how many object are on the stack after the topmost + markobject, POP_MARK gets rid of all of them (including the + topmost markobject too). + """) + +############################################################################## +# Descriptors for pickle opcodes. + +class OpcodeInfo(object): + + __slots__ = ( + # symbolic name of opcode; a string + 'name', + + # the code used in a bytestream to represent the opcode; a + # one-character string + 'code', + + # If the opcode has an argument embedded in the byte string, an + # instance of ArgumentDescriptor specifying its type. Note that + # arg.reader(s) can be used to read and decode the argument from + # the bytestream s, and arg.doc documents the format of the raw + # argument bytes. If the opcode doesn't have an argument embedded + # in the bytestream, arg should be None. + 'arg', + + # what the stack looks like before this opcode runs; a list + 'stack_before', + + # what the stack looks like after this opcode runs; a list + 'stack_after', + + # the protocol number in which this opcode was introduced; an int + 'proto', + + # human-readable docs for this opcode; a string + 'doc', + ) + + def __init__(self, name, code, arg, + stack_before, stack_after, proto, doc): + assert isinstance(name, str) + self.name = name + + assert isinstance(code, str) + assert len(code) == 1 + self.code = code + + assert arg is None or isinstance(arg, ArgumentDescriptor) + self.arg = arg + + assert isinstance(stack_before, list) + for x in stack_before: + assert isinstance(x, StackObject) + self.stack_before = stack_before + + assert isinstance(stack_after, list) + for x in stack_after: + assert isinstance(x, StackObject) + self.stack_after = stack_after + + assert isinstance(proto, int) and 0 <= proto <= 2 + self.proto = proto + + assert isinstance(doc, str) + self.doc = doc + +I = OpcodeInfo +opcodes = [ + + # Ways to spell integers. + + I(name='INT', + code='I', + arg=decimalnl_short, + stack_before=[], + stack_after=[pyinteger_or_bool], + proto=0, + doc="""Push an integer or bool. + + The argument is a newline-terminated decimal literal string. + + The intent may have been that this always fit in a short Python int, + but INT can be generated in pickles written on a 64-bit box that + require a Python long on a 32-bit box. The difference between this + and LONG then is that INT skips a trailing 'L', and produces a short + int whenever possible. + + Another difference is due to that, when bool was introduced as a + distinct type in 2.3, builtin names True and False were also added to + 2.2.2, mapping to ints 1 and 0. For compatibility in both directions, + True gets pickled as INT + "I01\\n", and False as INT + "I00\\n". + Leading zeroes are never produced for a genuine integer. The 2.3 + (and later) unpicklers special-case these and return bool instead; + earlier unpicklers ignore the leading "0" and return the int. + """), + + I(name='LONG', + code='L', + arg=decimalnl_long, + stack_before=[], + stack_after=[pylong], + proto=0, + doc="""Push a long integer. + + The same as INT, except that the literal ends with 'L', and always + unpickles to a Python long. There doesn't seem a real purpose to the + trailing 'L'. + """), + + I(name='BININT', + code='J', + arg=int4, + stack_before=[], + stack_after=[pyint], + proto=1, + doc="""Push a four-byte signed integer. + + This handles the full range of Python (short) integers on a 32-bit + box, directly as binary bytes (1 for the opcode and 4 for the integer). + If the integer is non-negative and fits in 1 or 2 bytes, pickling via + BININT1 or BININT2 saves space. + """), + + I(name='BININT1', + code='K', + arg=uint1, + stack_before=[], + stack_after=[pyint], + proto=1, + doc="""Push a one-byte unsigned integer. + + This is a space optimization for pickling very small non-negative ints, + in range(256). + """), + + I(name='BININT2', + code='M', + arg=uint2, + stack_before=[], + stack_after=[pyint], + proto=1, + doc="""Push a two-byte unsigned integer. + + This is a space optimization for pickling small positive ints, in + range(256, 2**16). Integers in range(256) can also be pickled via + BININT2, but BININT1 instead saves a byte. + """), + + # Ways to spell strings (8-bit, not Unicode). + + I(name='STRING', + code='S', + arg=stringnl, + stack_before=[], + stack_after=[pystring], + proto=0, + doc="""Push a Python string object. + + The argument is a repr-style string, with bracketing quote characters, + and perhaps embedded escapes. The argument extends until the next + newline character. + """), + + I(name='BINSTRING', + code='T', + arg=string4, + stack_before=[], + stack_after=[pystring], + proto=1, + doc="""Push a Python string object. + + There are two arguments: the first is a 4-byte little-endian signed int + giving the number of bytes in the string, and the second is that many + bytes, which are taken literally as the string content. + """), + + I(name='SHORT_BINSTRING', + code='U', + arg=string1, + stack_before=[], + stack_after=[pystring], + proto=1, + doc="""Push a Python string object. + + There are two arguments: the first is a 1-byte unsigned int giving + the number of bytes in the string, and the second is that many bytes, + which are taken literally as the string content. + """), + + # Ways to spell None. + + I(name='NONE', + code='N', + arg=None, + stack_before=[], + stack_after=[pynone], + proto=0, + doc="Push None on the stack."), + + # Ways to spell Unicode strings. + + I(name='UNICODE', + code='V', + arg=unicodestringnl, + stack_before=[], + stack_after=[pyunicode], + proto=0, # this may be pure-text, but it's a later addition + doc="""Push a Python Unicode string object. + + The argument is a raw-unicode-escape encoding of a Unicode string, + and so may contain embedded escape sequences. The argument extends + until the next newline character. + """), + + I(name='BINUNICODE', + code='X', + arg=unicodestring4, + stack_before=[], + stack_after=[pyunicode], + proto=1, + doc="""Push a Python Unicode string object. + + There are two arguments: the first is a 4-byte little-endian signed int + giving the number of bytes in the string. The second is that many + bytes, and is the UTF-8 encoding of the Unicode string. + """), + + # Ways to spell floats. + + I(name='FLOAT', + code='F', + arg=floatnl, + stack_before=[], + stack_after=[pyfloat], + proto=0, + doc="""Newline-terminated decimal float literal. + + The argument is repr(a_float), and in general requires 17 significant + digits for roundtrip conversion to be an identity (this is so for + IEEE-754 double precision values, which is what Python float maps to + on most boxes). + + In general, FLOAT cannot be used to transport infinities, NaNs, or + minus zero across boxes (or even on a single box, if the platform C + library can't read the strings it produces for such things -- Windows + is like that), but may do less damage than BINFLOAT on boxes with + greater precision or dynamic range than IEEE-754 double. + """), + + I(name='BINFLOAT', + code='G', + arg=float8, + stack_before=[], + stack_after=[pyfloat], + proto=1, + doc="""Float stored in binary form, with 8 bytes of data. + + This generally requires less than half the space of FLOAT encoding. + In general, BINFLOAT cannot be used to transport infinities, NaNs, or + minus zero, raises an exception if the exponent exceeds the range of + an IEEE-754 double, and retains no more than 53 bits of precision (if + there are more than that, "add a half and chop" rounding is used to + cut it back to 53 significant bits). + """), + + # Ways to build lists. + + I(name='EMPTY_LIST', + code=']', + arg=None, + stack_before=[], + stack_after=[pylist], + proto=1, + doc="Push an empty list."), + + I(name='APPEND', + code='a', + arg=None, + stack_before=[pylist, anyobject], + stack_after=[pylist], + proto=0, + doc="""Append an object to a list. + + Stack before: ... pylist anyobject + Stack after: ... pylist+[anyobject] + """), + + I(name='APPENDS', + code='e', + arg=None, + stack_before=[pylist, markobject, stackslice], + stack_after=[pylist], + proto=1, + doc="""Extend a list by a slice of stack objects. + + Stack before: ... pylist markobject stackslice + Stack after: ... pylist+stackslice + """), + + I(name='LIST', + code='l', + arg=None, + stack_before=[markobject, stackslice], + stack_after=[pylist], + proto=0, + doc="""Build a list out of the topmost stack slice, after markobject. + + All the stack entries following the topmost markobject are placed into + a single Python list, which single list object replaces all of the + stack from the topmost markobject onward. For example, + + Stack before: ... markobject 1 2 3 'abc' + Stack after: ... [1, 2, 3, 'abc'] + """), + + # Ways to build tuples. + + I(name='EMPTY_TUPLE', + code=')', + arg=None, + stack_before=[], + stack_after=[pytuple], + proto=1, + doc="Push an empty tuple."), + + I(name='TUPLE', + code='t', + arg=None, + stack_before=[markobject, stackslice], + stack_after=[pytuple], + proto=0, + doc="""Build a tuple out of the topmost stack slice, after markobject. + + All the stack entries following the topmost markobject are placed into + a single Python tuple, which single tuple object replaces all of the + stack from the topmost markobject onward. For example, + + Stack before: ... markobject 1 2 3 'abc' + Stack after: ... (1, 2, 3, 'abc') + """), + + # Ways to build dicts. + + I(name='EMPTY_DICT', + code='}', + arg=None, + stack_before=[], + stack_after=[pydict], + proto=1, + doc="Push an empty dict."), + + I(name='DICT', + code='d', + arg=None, + stack_before=[markobject, stackslice], + stack_after=[pydict], + proto=0, + doc="""Build a dict out of the topmost stack slice, after markobject. + + All the stack entries following the topmost markobject are placed into + a single Python dict, which single dict object replaces all of the + stack from the topmost markobject onward. The stack slice alternates + key, value, key, value, .... For example, + + Stack before: ... markobject 1 2 3 'abc' + Stack after: ... {1: 2, 3: 'abc'} + """), + + I(name='SETITEM', + code='s', + arg=None, + stack_before=[pydict, anyobject, anyobject], + stack_after=[pydict], + proto=0, + doc="""Add a key+value pair to an existing dict. + + Stack before: ... pydict key value + Stack after: ... pydict + + where pydict has been modified via pydict[key] = value. + """), + + I(name='SETITEMS', + code='u', + arg=None, + stack_before=[pydict, markobject, stackslice], + stack_after=[pydict], + proto=1, + doc="""Add an arbitrary number of key+value pairs to an existing dict. + + The slice of the stack following the topmost markobject is taken as + an alternating sequence of keys and values, added to the dict + immediately under the topmost markobject. Everything at and after the + topmost markobject is popped, leaving the mutated dict at the top + of the stack. + + Stack before: ... pydict markobject key_1 value_1 ... key_n value_n + Stack after: ... pydict + + where pydict has been modified via pydict[key_i] = value_i for i in + 1, 2, ..., n, and in that order. + """), + + # Stack manipulation. + + I(name='POP', + code='0', + arg=None, + stack_before=[anyobject], + stack_after=[], + proto=0, + doc="Discard the top stack item, shrinking the stack by one item."), + + I(name='DUP', + code='2', + arg=None, + stack_before=[anyobject], + stack_after=[anyobject, anyobject], + proto=0, + doc="Push the top stack item onto the stack again, duplicating it."), + + I(name='MARK', + code='(', + arg=None, + stack_before=[], + stack_after=[markobject], + proto=0, + doc="""Push markobject onto the stack. + + markobject is a unique object, used by other opcodes to identify a + region of the stack containing a variable number of objects for them + to work on. See markobject.doc for more detail. + """), + + I(name='POP_MARK', + code='1', + arg=None, + stack_before=[markobject, stackslice], + stack_after=[], + proto=0, + doc="""Pop all the stack objects at and above the topmost markobject. + + When an opcode using a variable number of stack objects is done, + POP_MARK is used to remove those objects, and to remove the markobject + that delimited their starting position on the stack. + """), + + # Memo manipulation. There are really only two operations (get and put), + # each in all-text, "short binary", and "long binary" flavors. + + I(name='GET', + code='g', + arg=decimalnl_short, + stack_before=[], + stack_after=[anyobject], + proto=0, + doc="""Read an object from the memo and push it on the stack. + + The index of the memo object to push is given by the newline-teriminated + decimal string following. BINGET and LONG_BINGET are space-optimized + versions. + """), + + I(name='BINGET', + code='h', + arg=uint1, + stack_before=[], + stack_after=[anyobject], + proto=1, + doc="""Read an object from the memo and push it on the stack. + + The index of the memo object to push is given by the 1-byte unsigned + integer following. + """), + + I(name='LONG_BINGET', + code='j', + arg=int4, + stack_before=[], + stack_after=[anyobject], + proto=1, + doc="""Read an object from the memo and push it on the stack. + + The index of the memo object to push is given by the 4-byte signed + little-endian integer following. + """), + + I(name='PUT', + code='p', + arg=decimalnl_short, + stack_before=[], + stack_after=[], + proto=0, + doc="""Store the stack top into the memo. The stack is not popped. + + The index of the memo location to write into is given by the newline- + terminated decimal string following. BINPUT and LONG_BINPUT are + space-optimized versions. + """), + + I(name='BINPUT', + code='q', + arg=uint1, + stack_before=[], + stack_after=[], + proto=1, + doc="""Store the stack top into the memo. The stack is not popped. + + The index of the memo location to write into is given by the 1-byte + unsigned integer following. + """), + + I(name='LONG_BINPUT', + code='r', + arg=int4, + stack_before=[], + stack_after=[], + proto=1, + doc="""Store the stack top into the memo. The stack is not popped. + + The index of the memo location to write into is given by the 4-byte + signed little-endian integer following. + """), + + # Push a class object, or module function, on the stack, via its module + # and name. + + I(name='GLOBAL', + code='c', + arg=stringnl_noescape_pair, + stack_before=[], + stack_after=[anyobject], + proto=0, + doc="""Push a global object (module.attr) on the stack. + + Two newline-terminated strings follow the GLOBAL opcode. The first is + taken as a module name, and the second as a class name. The class + object module.class is pushed on the stack. More accurately, the + object returned by self.find_class(module, class) is pushed on the + stack, so unpickling subclasses can override this form of lookup. + """), + + # Ways to build objects of classes pickle doesn't know about directly + # (user-defined classes). I despair of documenting this accurately + # and comprehensibly -- you really have to read the pickle code to + # find all the special cases. + + I(name='REDUCE', + code='R', + arg=None, + stack_before=[anyobject, anyobject], + stack_after=[anyobject], + proto=0, + doc="""Push an object built from a callable and an argument tuple. + + The opcode is named to remind of the __reduce__() method. + + Stack before: ... callable pytuple + Stack after: ... callable(*pytuple) + + The callable and the argument tuple are the first two items returned + by a __reduce__ method. Applying the callable to the argtuple is + supposed to reproduce the original object, or at least get it started. + If the __reduce__ method returns a 3-tuple, the last component is an + argument to be passed to the object's __setstate__, and then the REDUCE + opcode is followed by code to create setstate's argument, and then a + BUILD opcode to apply __setstate__ to that argument. + + There are lots of special cases here. The argtuple can be None, in + which case callable.__basicnew__() is called instead to produce the + object to be pushed on the stack. This appears to be a trick unique + to ExtensionClasses, and is deprecated regardless. + + If type(callable) is not ClassType, REDUCE complains unless the + callable has been registered with the copy_reg module's + safe_constructors dict, or the callable has a magic + '__safe_for_unpickling__' attribute with a true value. I'm not sure + why it does this, but I've sure seen this complaint often enough when + I didn't want to <wink>. + """), + + I(name='BUILD', + code='b', + arg=None, + stack_before=[anyobject, anyobject], + stack_after=[anyobject], + proto=0, + doc="""Finish building an object, via __setstate__ or dict update. + + Stack before: ... anyobject argument + Stack after: ... anyobject + + where anyobject may have been mutated, as follows: + + If the object has a __setstate__ method, + + anyobject.__setstate__(argument) + + is called. + + Else the argument must be a dict, the object must have a __dict__, and + the object is updated via + + anyobject.__dict__.update(argument) + + This may raise RuntimeError in restricted execution mode (which + disallows access to __dict__ directly); in that case, the object + is updated instead via + + for k, v in argument.items(): + anyobject[k] = v + """), + + I(name='INST', + code='i', + arg=stringnl_noescape_pair, + stack_before=[markobject, stackslice], + stack_after=[anyobject], + proto=0, + doc="""Build a class instance. + + This is the protocol 0 version of protocol 1's OBJ opcode. + INST is followed by two newline-terminated strings, giving a + module and class name, just as for the GLOBAL opcode (and see + GLOBAL for more details about that). self.find_class(module, name) + is used to get a class object. + + In addition, all the objects on the stack following the topmost + markobject are gathered into a tuple and popped (along with the + topmost markobject), just as for the TUPLE opcode. + + Now it gets complicated. If all of these are true: + + + The argtuple is empty (markobject was at the top of the stack + at the start). + + + It's an old-style class object (the type of the class object is + ClassType). + + + The class object does not have a __getinitargs__ attribute. + + then we want to create an old-style class instance without invoking + its __init__() method (pickle has waffled on this over the years; not + calling __init__() is current wisdom). In this case, an instance of + an old-style dummy class is created, and then we try to rebind its + __class__ attribute to the desired class object. If this succeeds, + the new instance object is pushed on the stack, and we're done. In + restricted execution mode it can fail (assignment to __class__ is + disallowed), and I'm not really sure what happens then -- it looks + like the code ends up calling the class object's __init__ anyway, + via falling into the next case. + + Else (the argtuple is not empty, it's not an old-style class object, + or the class object does have a __getinitargs__ attribute), the code + first insists that the class object have a __safe_for_unpickling__ + attribute. Unlike as for the __safe_for_unpickling__ check in REDUCE, + it doesn't matter whether this attribute has a true or false value, it + only matters whether it exists (XXX this smells like a bug). If + __safe_for_unpickling__ dosn't exist, UnpicklingError is raised. + + Else (the class object does have a __safe_for_unpickling__ attr), + the class object obtained from INST's arguments is applied to the + argtuple obtained from the stack, and the resulting instance object + is pushed on the stack. + """), + + I(name='OBJ', + code='o', + arg=None, + stack_before=[markobject, anyobject, stackslice], + stack_after=[anyobject], + proto=1, + doc="""Build a class instance. + + This is the protocol 1 version of protocol 0's INST opcode, and is + very much like it. The major difference is that the class object + is taken off the stack, allowing it to be retrieved from the memo + repeatedly if several instances of the same class are created. This + can be much more efficient (in both time and space) than repeatedly + embedding the module and class names in INST opcodes. + + Unlike INST, OBJ takes no arguments from the opcode stream. Instead + the class object is taken off the stack, immediately above the + topmost markobject: + + Stack before: ... markobject classobject stackslice + Stack after: ... new_instance_object + + As for INST, the remainder of the stack above the markobject is + gathered into an argument tuple, and then the logic seems identical, + except that no __safe_for_unpickling__ check is done (XXX this smells + like a bug). See INST for the gory details. + """), + + # Machine control. + + I(name='STOP', + code='.', + arg=None, + stack_before=[anyobject], + stack_after=[], + proto=0, + doc="""Stop the unpickling machine. + + Every pickle ends with this opcode. The object at the top of the stack + is popped, and that's the result of unpickling. The stack should be + empty then. + """), + + # Ways to deal with persistent IDs. + + I(name='PERSID', + code='P', + arg=stringnl_noescape, + stack_before=[], + stack_after=[anyobject], + proto=0, + doc="""Push an object identified by a persistent ID. + + The pickle module doesn't define what a persistent ID means. PERSID's + argument is a newline-terminated str-style (no embedded escapes, no + bracketing quote characters) string, which *is* "the persistent ID". + The unpickler passes this string to self.persistent_load(). Whatever + object that returns is pushed on the stack. There is no implementation + of persistent_load() in Python's unpickler: it must be supplied by an + unpickler subclass. + """), + + I(name='BINPERSID', + code='Q', + arg=None, + stack_before=[anyobject], + stack_after=[anyobject], + proto=1, + doc="""Push an object identified by a persistent ID. + + Like PERSID, except the persistent ID is popped off the stack (instead + of being a string embedded in the opcode bytestream). The persistent + ID is passed to self.persistent_load(), and whatever object that + returns is pushed on the stack. See PERSID for more detail. + """), +] +del I + +# Verify uniqueness of .name and .code members. +name2i = {} +code2i = {} + +for i, d in enumerate(opcodes): + if d.name in name2i: + raise ValueError("repeated name %r at indices %d and %d" % + (d.name, name2i[d.name], i)) + if d.code in code2i: + raise ValueError("repeated code %r at indices %d and %d" % + (d.code, code2i[d.code], i)) + + name2i[d.name] = i + code2i[d.code] = i + +del name2i, code2i, i, d + +############################################################################## +# Build a code2op dict, mapping opcode characters to OpcodeInfo records. +# Also ensure we've got the same stuff as pickle.py, although the +# introspection here is dicey. + +code2op = {} +for d in opcodes: + code2op[d.code] = d +del d + +def assure_pickle_consistency(verbose=False): + import pickle, re + + copy = code2op.copy() + for name in pickle.__all__: + if not re.match("[A-Z][A-Z0-9_]+$", name): + if verbose: + print "skipping %r: it doesn't look like an opcode name" % name + continue + picklecode = getattr(pickle, name) + if not isinstance(picklecode, str) or len(picklecode) != 1: + if verbose: + print ("skipping %r: value %r doesn't look like a pickle " + "code" % (name, picklecode)) + continue + if picklecode in copy: + if verbose: + print "checking name %r w/ code %r for consistency" % ( + name, picklecode) + d = copy[picklecode] + if d.name != name: + raise ValueError("for pickle code %r, pickle.py uses name %r " + "but we're using name %r" % (picklecode, + name, + d.name)) + # Forget this one. Any left over in copy at the end are a problem + # of a different kind. + del copy[picklecode] + else: + raise ValueError("pickle.py appears to have a pickle opcode with " + "name %r and code %r, but we don't" % + (name, picklecode)) + if copy: + msg = ["we appear to have pickle opcodes that pickle.py doesn't have:"] + for code, d in copy.items(): + msg.append(" name %r with code %r" % (d.name, code)) + raise ValueError("\n".join(msg)) + +assure_pickle_consistency() + +############################################################################## +# A pickle opcode generator. + +def genops(pickle): + """"Generate all the opcodes in a pickle. + + 'pickle' is a file-like object, or string, containing the pickle. + + Each opcode in the pickle is generated, from the current pickle position, + stopping after a STOP opcode is delivered. A triple is generated for + each opcode: + + opcode, arg, pos + + opcode is an OpcodeInfo record, describing the current opcode. + + If the opcode has an argument embedded in the pickle, arg is its decoded + value, as a Python object. If the opcode doesn't have an argument, arg + is None. + + If the pickle has a tell() method, pos was the value of pickle.tell() + before reading the current opcode. If the pickle is a string object, + it's wrapped in a StringIO object, and the latter's tell() result is + used. Else (the pickle doesn't have a tell(), and it's not obvious how + to query its current position) pos is None. + """ + + import cStringIO as StringIO + + if isinstance(pickle, str): + pickle = StringIO.StringIO(pickle) + + if hasattr(pickle, "tell"): + getpos = pickle.tell + else: + getpos = lambda: None + + while True: + pos = getpos() + code = pickle.read(1) + opcode = code2op.get(code) + if opcode is None: + if code == "": + raise ValueError("pickle exhausted before seeing STOP") + else: + raise ValueError("at position %s, opcode %r unknown" % ( + pos is None and "<unknown>" or pos, + code)) + if opcode.arg is None: + arg = None + else: + arg = opcode.arg.reader(pickle) + yield opcode, arg, pos + if code == '.': + assert opcode.name == 'STOP' + break + +############################################################################## +# A symbolic pickle disassembler. + +def dis(pickle, out=None, indentlevel=4): + """Produce a symbolic disassembly of a pickle. + + 'pickle' is a file-like object, or string, containing a (at least one) + pickle. The pickle is disassembled from the current position, through + the first STOP opcode encountered. + + Optional arg 'out' is a file-like object to which the disassembly is + printed. It defaults to sys.stdout. + + Optional arg indentlevel is the number of blanks by which to indent + a new MARK level. It defaults to 4. + """ + + markstack = [] + indentchunk = ' ' * indentlevel + for opcode, arg, pos in genops(pickle): + if pos is not None: + print >> out, "%5d:" % pos, + + line = "%s %s%s" % (opcode.code, + indentchunk * len(markstack), + opcode.name) + + markmsg = None + if markstack and markobject in opcode.stack_before: + assert markobject not in opcode.stack_after + markpos = markstack.pop() + if markpos is not None: + markmsg = "(MARK at %d)" % markpos + + if arg is not None or markmsg: + # make a mild effort to align arguments + line += ' ' * (10 - len(opcode.name)) + if arg is not None: + line += ' ' + repr(arg) + if markmsg: + line += ' ' + markmsg + print >> out, line + + if markobject in opcode.stack_after: + assert markobject not in opcode.stack_before + markstack.append(pos) + + +_dis_test = """ +>>> import pickle +>>> x = [1, 2, (3, 4), {'abc': u"def"}] +>>> pik = pickle.dumps(x) +>>> dis(pik) + 0: ( MARK + 1: l LIST (MARK at 0) + 2: p PUT 0 + 5: I INT 1 + 8: a APPEND + 9: I INT 2 + 12: a APPEND + 13: ( MARK + 14: I INT 3 + 17: I INT 4 + 20: t TUPLE (MARK at 13) + 21: p PUT 1 + 24: a APPEND + 25: ( MARK + 26: d DICT (MARK at 25) + 27: p PUT 2 + 30: S STRING 'abc' + 37: p PUT 3 + 40: V UNICODE u'def' + 45: p PUT 4 + 48: s SETITEM + 49: a APPEND + 50: . STOP + +Try again with a "binary" pickle. + +>>> pik = pickle.dumps(x, 1) +>>> dis(pik) + 0: ] EMPTY_LIST + 1: q BINPUT 0 + 3: ( MARK + 4: K BININT1 1 + 6: K BININT1 2 + 8: ( MARK + 9: K BININT1 3 + 11: K BININT1 4 + 13: t TUPLE (MARK at 8) + 14: q BINPUT 1 + 16: } EMPTY_DICT + 17: q BINPUT 2 + 19: U SHORT_BINSTRING 'abc' + 24: q BINPUT 3 + 26: X BINUNICODE u'def' + 34: q BINPUT 4 + 36: s SETITEM + 37: e APPENDS (MARK at 3) + 38: . STOP + +Exercise the INST/OBJ/BUILD family. + +>>> import random +>>> dis(pickle.dumps(random.random)) + 0: c GLOBAL 'random.random' + 15: p PUT 0 + 18: . STOP + +>>> x = [pickle.PicklingError()] * 2 +>>> dis(pickle.dumps(x)) + 0: ( MARK + 1: l LIST (MARK at 0) + 2: p PUT 0 + 5: ( MARK + 6: i INST 'pickle.PicklingError' (MARK at 5) + 28: p PUT 1 + 31: ( MARK + 32: d DICT (MARK at 31) + 33: p PUT 2 + 36: S STRING 'args' + 44: p PUT 3 + 47: ( MARK + 48: t TUPLE (MARK at 47) + 49: p PUT 4 + 52: s SETITEM + 53: b BUILD + 54: a APPEND + 55: g GET 1 + 58: a APPEND + 59: . STOP + +>>> dis(pickle.dumps(x, 1)) + 0: ] EMPTY_LIST + 1: q BINPUT 0 + 3: ( MARK + 4: ( MARK + 5: c GLOBAL 'pickle.PicklingError' + 27: q BINPUT 1 + 29: o OBJ (MARK at 4) + 30: q BINPUT 2 + 32: } EMPTY_DICT + 33: q BINPUT 3 + 35: U SHORT_BINSTRING 'args' + 41: q BINPUT 4 + 43: ) EMPTY_TUPLE + 44: s SETITEM + 45: b BUILD + 46: h BINGET 2 + 48: e APPENDS (MARK at 3) + 49: . STOP + +Try "the canonical" recursive-object test. + +>>> L = [] +>>> T = L, +>>> L.append(T) +>>> L[0] is T +True +>>> T[0] is L +True +>>> L[0][0] is L +True +>>> T[0][0] is T +True +>>> dis(pickle.dumps(L)) + 0: ( MARK + 1: l LIST (MARK at 0) + 2: p PUT 0 + 5: ( MARK + 6: g GET 0 + 9: t TUPLE (MARK at 5) + 10: p PUT 1 + 13: a APPEND + 14: . STOP +>>> dis(pickle.dumps(L, 1)) + 0: ] EMPTY_LIST + 1: q BINPUT 0 + 3: ( MARK + 4: h BINGET 0 + 6: t TUPLE (MARK at 3) + 7: q BINPUT 1 + 9: a APPEND + 10: . STOP + +The protocol 0 pickle of the tuple causes the disassembly to get confused, +as it doesn't realize that the POP opcode at 16 gets rid of the MARK at 0 +(so the output remains indented until the end). The protocol 1 pickle +doesn't trigger this glitch, because the disassembler realizes that +POP_MARK gets rid of the MARK. Doing a better job on the protocol 0 +pickle would require the disassembler to emulate the stack. + +>>> dis(pickle.dumps(T)) + 0: ( MARK + 1: ( MARK + 2: l LIST (MARK at 1) + 3: p PUT 0 + 6: ( MARK + 7: g GET 0 + 10: t TUPLE (MARK at 6) + 11: p PUT 1 + 14: a APPEND + 15: 0 POP + 16: 0 POP + 17: g GET 1 + 20: . STOP +>>> dis(pickle.dumps(T, 1)) + 0: ( MARK + 1: ] EMPTY_LIST + 2: q BINPUT 0 + 4: ( MARK + 5: h BINGET 0 + 7: t TUPLE (MARK at 4) + 8: q BINPUT 1 + 10: a APPEND + 11: 1 POP_MARK (MARK at 0) + 12: h BINGET 1 + 14: . STOP +""" + +__test__ = {'dissassembler_test': _dis_test, + } + +def _test(): + import doctest + return doctest.testmod() + +if __name__ == "__main__": + _test() diff --git a/Lib/test/test_pickletools.py b/Lib/test/test_pickletools.py new file mode 100644 index 0000000..2e2fe50 --- /dev/null +++ b/Lib/test/test_pickletools.py @@ -0,0 +1,3 @@ +import pickletools +from test import test_support +test_support.run_doctest(pickletools) @@ -122,6 +122,10 @@ Extension modules Library ------- +- The new pickletools.py contains lots of documentation about pickle + internals, and supplies some helpers for working with pickles, such as + a symbolic pickle disassembler. + - Xmlrpclib.py now supports the builtin boolean type. - py_compile has a new 'doraise' flag and a new PyCompileError @@ -151,7 +155,7 @@ Library the gain value which is passed to Tk. SF bug# 602259. - Fix logging.handlers.SysLogHandler protocol when using UNIX domain sockets. - See SF patch #642974. + See SF patch #642974. Tools/Demos ----------- @@ -214,7 +218,7 @@ Mac - There are new dialogs EasyDialogs.AskFileForOpen, AskFileForSave and AskFolder. The old macfs.StandardGetFile and friends are deprecated. - + - Type Carbon.File.FSCatalogInfo and supporting methods have been implemented. This also makes macfs.FSSpec.SetDates() work again. |