\documentclass{howto} \usepackage{distutils} % $Id$ \title{What's New in Python 2.4} \release{0.0} \author{A.M.\ Kuchling} \authoraddress{ \strong{Python Software Foundation}\\ Email: \email{amk@amk.ca} } \begin{document} \maketitle \tableofcontents This article explains the new features in Python 2.4. The release date is expected to be around September 2004. While Python 2.3 was primarily a library development release, Python 2.4 may extend the core language and interpreter in as-yet-undetermined ways. This article doesn't attempt to provide a complete specification of the new features, but instead provides a convenient overview. For full details, you should refer to the documentation for Python 2.4, such as the \citetitle[../lib/lib.html]{Python Library Reference} and the \citetitle[../ref/ref.html]{Python Reference Manual}. If you want to understand the complete implementation and design rationale, refer to the PEP for a particular new feature. %====================================================================== \section{PEP 218: Built-In Set Objects} Two new built-in types, \function{set(\var{iterable})} and \function{frozenset(\var{iterable})} provide high speed data types for membership testing, for eliminating duplicates from sequences, and for mathematical operations like unions, intersections, differences, and symmetric differences. \begin{verbatim} >>> a = set('abracadabra') # form a set from a string >>> 'z' in a # fast membership testing False >>> a # unique letters in a set(['a', 'r', 'b', 'c', 'd']) >>> ''.join(a) # convert back into a string 'arbcd' >>> b = set('alacazam') # form a second set >>> a - b # letters in a but not in b set(['r', 'd', 'b']) >>> a | b # letters in either a or b set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l']) >>> a & b # letters in both a and b set(['a', 'c']) >>> a ^ b # letters in a or b but not both set(['r', 'd', 'b', 'm', 'z', 'l']) >>> a.add('z') # add a new element >>> a.update('wxy') # add multiple new elements >>> a set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z']) >>> a.remove('x') # take one element out >>> a set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z']) \end{verbatim} The type \function{frozenset()} is an immutable version of \function{set()}. Since it is immutable and hashable, it may be used as a dictionary key or as a member of another set. Accordingly, it does not have methods like \method{add()} and \method{remove()} which could alter its contents. % XXX what happens to the sets module? % The current thinking is that the sets module will be left alone. % That way, existing code will continue to run without alteration. % Also, the module provides an autoconversion feature not supported by set() % and frozenset(). \begin{seealso} \seepep{218}{Adding a Built-In Set Object Type}{Originally proposed by Greg Wilson and ultimately implemented by Raymond Hettinger.} \end{seealso} %====================================================================== \section{PEP 237: Unifying Long Integers and Integers} XXX write this. %====================================================================== \section{PEP 229: Generator Expressions} Now, simple generators can be coded succinctly as expressions using a syntax like list comprehensions but with parentheses instead of brackets. These expressions are designed for situations where the generator is used right away by an enclosing function. Generator expressions are more compact but less versatile than full generator definitions and they tend to be more memory friendly than equivalent list comprehensions. \begin{verbatim} g = (tgtexp for var1 in exp1 for var2 in exp2 if exp3) \end{verbatim} is equivalent to: \begin{verbatim} def __gen(exp): for var1 in exp: for var2 in exp2: if exp3: yield tgtexp g = __gen(iter(exp1)) del __gen \end{verbatim} The advantage over full generator definitions is in economy of expression. Their advantage over list comprehensions is in saving memory by creating data only when it is needed rather than forming a whole list is memory all at once. Applications using memory friendly generator expressions may scale-up to high volumes of data more readily than with list comprehensions. Generator expressions are best used in functions that consume their data all at once and would not benefit from having a full list instead of a generator as an input: \begin{verbatim} >>> sum(i*i for i in range(10)) 285 >>> sorted(set(i*i for i in xrange(-20, 20) if i%2==1)) # odd squares [1, 9, 25, 49, 81, 121, 169, 225, 289, 361] >>> from itertools import izip >>> xvec = [10, 20, 30] >>> yvec = [7, 5, 3] >>> sum(x*y for x,y in izip(xvec, yvec)) # dot product 260 >>> from math import pi, sin >>> sine_table = dict((x, sin(x*pi/180)) for x in xrange(0, 91)) >>> unique_words = set(word for line in page for word in line.split()) >>> valedictorian = max((student.gpa, student.name) for student in graduates) \end{verbatim} For more complex uses of generators, it is strongly recommended that the traditional full generator definitions be used instead. In a generator expression, the first for-loop expression is evaluated as soon as the expression is defined while the other expressions do not get evaluated until the generator is run. This nuance is never an issue when the generator is used immediately; however, if it is not used right away, a full generator definition would be much more clear about when the sub-expressions are evaluated and would be more obvious about the visibility and lifetime of the variables. \begin{seealso} \seepep{289}{Generator Expressions}{Proposed by Raymond Hettinger and implemented by Jiwon Seo with early efforts steered by Hye-Shik Chang.} \end{seealso} %====================================================================== \section{PEP 322: Reverse Iteration} A new built-in function, \function{reversed(\var{seq})}, takes a sequence and returns an iterator that returns the elements of the sequence in reverse order. \begin{verbatim} >>> for i in reversed(xrange(1,4)): ... print i ... 3 2 1 \end{verbatim} Compared to extended slicing, \code{range(1,4)[::-1]}, \function{reversed()} is easier to read, runs faster, and uses substantially less memory. Note that \function{reversed()} only accepts sequences, not arbitrary iterators. If you want to reverse an iterator, first convert it to a list with \function{list()}. \begin{verbatim} >>> input= open('/etc/passwd', 'r') >>> for line in reversed(list(input)): ... print line ... root:*:0:0:System Administrator:/var/root:/bin/tcsh ... \end{verbatim} \begin{seealso} \seepep{322}{Reverse Iteration}{Written and implemented by Raymond Hettinger.} \end{seealso} %====================================================================== \section{Other Language Changes} Here are all of the changes that Python 2.4 makes to the core Python language. \begin{itemize} \item The \method{dict.update()} method now accepts the same argument forms as the \class{dict} constructor. This includes any mapping, any iterable of key/value pairs, and/or keyword arguments. \item The string methods, \method{ljust()}, \method{rjust()}, and \method{center()} now take an optional argument for specifying a fill character other than a space. \item Strings also gained an \method{rsplit()} method that works like the \method{split()} method but splits from the end of the string. \begin{verbatim} >>> 'www.python.org'.split('.', 1) ['www', 'python.org'] 'www.python.org'.rsplit('.', 1) ['www.python', 'org'] \end{verbatim} \item The \method{sort()} method of lists gained three keyword arguments, \var{cmp}, \var{key}, and \var{reverse}. These arguments make some common usages of \method{sort()} simpler. All are optional. \var{cmp} is the same as the previous single argument to \method{sort()}; if provided, the value should be a comparison function that takes two arguments and returns -1, 0, or +1 depending on how the arguments compare. \var{key} should be a single-argument function that takes a list element and returns a comparison key for the element. The list is then sorted using the comparison keys. The following example sorts a list case-insensitively: \begin{verbatim} >>> L = ['A', 'b', 'c', 'D'] >>> L.sort() # Case-sensitive sort >>> L ['A', 'D', 'b', 'c'] >>> L.sort(key=lambda x: x.lower()) >>> L ['A', 'b', 'c', 'D'] >>> L.sort(cmp=lambda x,y: cmp(x.lower(), y.lower())) >>> L ['A', 'b', 'c', 'D'] \end{verbatim} The last example, which uses the \var{cmp} parameter, is the old way to perform a case-insensitive sort. It works but is slower than using a \var{key} parameter. Using \var{key} results in calling the \method{lower()} method once for each element in the list while using \var{cmp} will call the method twice for each comparison. For simple key functions and comparison functions, it is often possible to avoid a \keyword{lambda} expression by using an unbound method instead. For example, the above case-insensitive sort is best coded as: \begin{verbatim} >>> L.sort(key=str.lower) >>> L ['A', 'b', 'c', 'D'] \end{verbatim} The \var{reverse} parameter should have a Boolean value. If the value is \constant{True}, the list will be sorted into reverse order. Instead of \code{L.sort(lambda x,y: cmp(y.score, x.score))}, you can now write: \code{L.sort(key = lambda x: x.score, reverse=True)}. The results of sorting are now guaranteed to be stable. This means that two entries with equal keys will be returned in the same order as they were input. For example, you can sort a list of people by name, and then sort the list by age, resulting in a list sorted by age where people with the same age are in name-sorted order. \item There is a new built-in function \function{sorted(\var{iterable})} that works like the in-place \method{list.sort()} method but has been made suitable for use in expressions. The differences are: \begin{itemize} \item the input may be any iterable; \item a newly formed copy is sorted, leaving the original intact; and \item the expression returns the new sorted copy \end{itemize} \begin{verbatim} >>> L = [9,7,8,3,2,4,1,6,5] >>> [10+i for i in sorted(L)] # usable in a list comprehension [11, 12, 13, 14, 15, 16, 17, 18, 19] >>> L = [9,7,8,3,2,4,1,6,5] # original is left unchanged [9,7,8,3,2,4,1,6,5] >>> sorted('Monte Python') # any iterable may be an input [' ', 'M', 'P', 'e', 'h', 'n', 'n', 'o', 'o', 't', 't', 'y'] >>> # List the contents of a dict sorted by key values >>> colormap = dict(red=1, blue=2, green=3, black=4, yellow=5) >>> for k, v in sorted(colormap.iteritems()): ... print k, v ... black 4 blue 2 green 3 red 1 yellow 5 \end{verbatim} \item The \function{zip()} built-in function and \function{itertools.izip()} now return an empty list instead of raising a \exception{TypeError} exception if called with no arguments. This makes them more suitable for use with variable length argument lists: \begin{verbatim} >>> def transpose(array): ... return zip(*array) ... >>> transpose([(1,2,3), (4,5,6)]) [(1, 4), (2, 5), (3, 6)] >>> transpose([]) [] \end{verbatim} \end{itemize} %====================================================================== \subsection{Optimizations} \begin{itemize} \item The inner loops for \class{list} and \class{tuple} slicing were optimized and now run about one-third faster. The inner loops were also optimized for \class{dict} with performance boosts to \method{keys()}, \method{values()}, \method{items()}, \method{iterkeys()}, \method{itervalues()}, and \method{iteritems()}. \item The machinery for growing and shrinking lists was optimized for speed and for space efficiency. Small lists (under eight elements) never over-allocate by more than three elements. Large lists do not over-allocate by more than 1/8th. Appending and popping from lists now runs faster due to more efficient code paths and less frequent use of the underlying system realloc(). List comprehensions also benefit. The amount of improvement varies between systems and shows the greatest improvement on systems with poor realloc() implementations. \method{list.extend()} was also optimized and no longer converts its argument into a temporary list prior to extending the base list. \item \function{list()}, \function{tuple()}, \function{map()}, \function{filter()}, and \function{zip()} now run several times faster with non-sequence arguments that supply a \method{__len__()} method. Previously, the pre-sizing optimization only applied to sequence arguments. \item The methods \method{list.__getitem__()}, \method{dict.__getitem__()}, and \method{dict.__contains__()} are are now implemented as \class{method_descriptor} objects rather than \class{wrapper_descriptor} objects. This form of optimized access doubles their performance and makes them more suitable for use as arguments to functionals: \samp{map(mydict.__getitem__, keylist)}. \item Added a new opcode, \code{LIST_APPEND}, that simplifies the generated bytecode for list comprehensions and speeds them up by about a third. \end{itemize} The net result of the 2.4 optimizations is that Python 2.4 runs the pystone benchmark around XX\% faster than Python 2.3 and YY\% faster than Python 2.2. %====================================================================== \section{New, Improved, and Deprecated Modules} As usual, Python's standard library received a number of enhancements and bug fixes. Here's a partial list of the most notable changes, sorted alphabetically by module name. Consult the \file{Misc/NEWS} file in the source tree for a more complete list of changes, or look through the CVS logs for all the details. \begin{itemize} \item The \module{curses} modules now supports the ncurses extension \function{use_default_colors()}. On platforms where the terminal supports transparency, this makes it possible to use a transparent background. (Contributed by J\"org Lehmann.) \item The \module{bisect} module now has an underlying C implementation for improved performance. (Contributed by Dmitry Vasiliev.) \item The CJKCodecs collections of East Asian codecs, maintained by Hye-Shik Chang, was integrated into 2.4. The new encodings are: \begin{itemize} \item Chinese (PRC): gb2312, gbk, gb18030, hz \item Chinese (ROC): big5, cp950 \item Japanese: cp932, shift-jis, shift-jisx0213, euc-jp, euc-jisx0213, iso-2022-jp, iso-2022-jp-1, iso-2022-jp-2, iso-2022-jp-3, iso-2022-jp-ext \item Korean: cp949, euc-kr, johab, iso-2022-kr \end{itemize} \item There is a new \module{collections} module for various specialized collection datatypes. Currently it contains just one type, \class{deque}, a double-ended queue that supports efficiently adding and removing elements from either end. \begin{verbatim} >>> from collections import deque >>> d = deque('ghi') # make a new deque with three items >>> d.append('j') # add a new entry to the right side >>> d.appendleft('f') # add a new entry to the left side >>> d # show the representation of the deque deque(['f', 'g', 'h', 'i', 'j']) >>> d.pop() # return and remove the rightmost item 'j' >>> d.popleft() # return and remove the leftmost item 'f' >>> list(d) # list the contents of the deque ['g', 'h', 'i'] >>> 'h' in d # search the deque True \end{verbatim} Several modules now take advantage of \class{collections.deque} for improved performance: \module{Queue}, \module{mutex}, \module{shlex} \module{threading}, and \module{pydoc}. \item The \module{ConfigParser} classes have been enhanced slightly. The \method{read()} method now returns a list of the files that were successfully parsed, and the \method{set()} method raises \exception{TypeError} if passed a \var{value} argument that isn't a string. \item The \module{heapq} module has been converted to C. The resulting tenfold improvement in speed makes the module suitable for handling high volumes of data. \item The \module{imaplib} module now supports IMAP's THREAD command. (Contributed by Yves Dionne.) \item The \module{itertools} module gained a \function{groupby(\var{iterable}\optional{, \var{func}})} function, inspired by the GROUP BY clause from SQL. \var{iterable} returns a succession of elements, and the optional \var{func} is a function that takes an element and returns a key value; if omitted, the key is simply the element itself. \function{groupby()} then groups the elements into subsequences which have matching values of the key, and returns a series of 2-tuples containing the key value and an iterator over the subsequence. Here's an example. The \var{key} function simply returns whether a number is even or odd, so the result of \function{groupby()} is to return consecutive runs of odd or even numbers. \begin{verbatim} >>> import itertools >>> L = [2,4,6, 7,8,9,11, 12, 14] >>> for key_val, it in itertools.groupby(L, lambda x: x % 2): ... print key_val, list(it) ... 0 [2, 4, 6] 1 [7] 0 [8] 1 [9, 11] 0 [12, 14] >>> \end{verbatim} Like its SQL counterpart, \function{groupby()} is typically used with sorted input. The logic for \function{groupby()} is similar to the \UNIX{} \code{uniq} filter which makes it handy for eliminating, counting, or identifying duplicate elements: \begin{verbatim} >>> word = 'abracadabra' >>> letters = sorted(word) # Turn string into a sorted list of letters >>> letters ['a', 'a', 'a', 'a', 'a', 'b', 'b', 'c', 'd', 'r', 'r'] >>> [k for k, g in groupby(letters)] # List unique letters ['a', 'b', 'c', 'd', 'r'] >>> [(k, len(list(g))) for k, g in groupby(letters)] # Count letter occurences [('a', 5), ('b', 2), ('c', 1), ('d', 1), ('r', 2)] >>> [k for k, g in groupby(letters) if len(list(g)) > 1] # List duplicated letters ['a', 'b', 'r'] \end{verbatim} \item \module{itertools} also gained a function named \function{tee(\var{iterator}, \var{N})} that returns \var{N} independent iterators that replicate \var{iterator}. If \var{N} is omitted, the default is 2. \begin{verbatim} >>> L = [1,2,3] >>> i1, i2 = itertools.tee(L) >>> i1,i2 (, ) >>> list(i1) # Run the first iterator to exhaustion [1, 2, 3] >>> list(i2) # Run the second iterator to exhaustion [1, 2, 3] >\end{verbatim} Note that \function{tee()} has to keep copies of the values returned by the iterator; in the worst case, it may need to keep all of them. This should therefore be used carefully if the leading iterator can run far ahead of the trailing iterator in a long stream of inputs. If the separation is large, then it becomes preferable to use \function{list()} instead. When the iterators track closely with one another, \function{tee()} is ideal. Possible applications include bookmarking, windowing, or lookahead iterators. \item A new \function{getsid()} function was added to the \module{posix} module that underlies the \module{os} module. (Contributed by J. Raynor.) \item The \module{operator} module gained two new functions, \function{attrgetter(\var{attr})} and \function{itemgetter(\var{index})}. Both functions return callables that take a single argument and return the corresponding attribute or item; these callables make excellent data extractors when used with \function{map()} or \function{sorted()}. For example: \begin{verbatim} >>> L = [('c', 2), ('d', 1), ('a', 4), ('b', 3)] >>> map(operator.itemgetter(0), L) ['c', 'd', 'a', 'b'] >>> map(operator.itemgetter(1), L) [2, 1, 4, 3] >>> sorted(L, key=operator.itemgetter(1)) # Sort list by second tuple item [('d', 1), ('c', 2), ('b', 3), ('a', 4)] \end{verbatim} \item The \module{random} module has a new method called \method{getrandbits(N)} which returns an N-bit long integer. This method supports the existing \method{randrange()} method, making it possible to efficiently generate arbitrarily large random numbers. \item The regular expression language accepted by the \module{re} module was extended with simple conditional expressions, written as \code{(?(\var{group})\var{A}|\var{B})}. \var{group} is either a numeric group ID or a group name defined with \code{(?P...)} earlier in the expression. If the specified group matched, the regular expression pattern \var{A} will be tested against the string; if the group didn't match, the pattern \var{B} will be used instead. \item The \module{weakref} module now supports a wider variety of objects including Python functions, class instances, sets, frozensets, deques, arrays, files, sockets, and regular expression pattern objects. \end{itemize} %====================================================================== % whole new modules get described in \subsections here \subsection{cookielib} The \module{cookielib} library supports client-side handling for HTTP cookies, just as the \module{Cookie} provides server-side cookie support in CGI scripts. This library manages cookies in a way similar to web browsers. Cookies are stored in cookie jars; the library transparently stores cookies offered by the web server in the cookie jar, and fetches the cookie from the jar when connecting to the server. Similar to web browsers, policy objects control whether cookies are accepted or not. In order to store cookies across sessions, two implementations of cookie jars are provided: one that stores cookies in the Netscape format, so applications can use the Mozilla or Lynx cookie jars, and one that stores cookies in the same format as the Perl libwww libary. \module{urllib2} has been changed to interact with \module{cookielib}: \class{HTTPCookieProcessor} manages a cookie jar that is used when accessing URLs. % ====================================================================== \section{Build and C API Changes} Changes to Python's build process and to the C API include: \begin{itemize} \item Three new convenience macros were added for common return values from extension functions: \csimplemacro{Py_RETURN_NONE}, \csimplemacro{Py_RETURN_TRUE}, and \csimplemacro{Py_RETURN_FALSE}. \item A new function, \cfunction{PyTuple_Pack(\var{N}, \var{obj1}, \var{obj2}, ..., \var{objN})}, constructs tuples from a variable length argument list of Python objects. \item A new function, \cfunction{PyDict_Contains(\var{d}, \var{k})}, implements fast dictionary lookups without masking exceptions raised during the look-up process. \item A new method flag, \constant{METH_COEXISTS}, allows a function defined in slots to co-exist with a PyCFunction having the same name. This can halve the access to time to a method such as \method{set.__contains__()} \end{itemize} %====================================================================== \subsection{Port-Specific Changes} \begin{itemize} \item The Windows port now builds under MSVC++ 7.1 as well as version 6. \end{itemize} %====================================================================== \section{Other Changes and Fixes \label{section-other}} As usual, there were a bunch of other improvements and bugfixes scattered throughout the source tree. A search through the CVS change logs finds there were XXX patches applied and YYY bugs fixed between Python 2.3 and 2.4. Both figures are likely to be underestimates. Some of the more notable changes are: \begin{itemize} \item The \module{timeit} module now automatically disables periodic garbarge collection during the timing loop. This change makes consecutive timings more comparable. \item The \module{base64} module now has more complete RFC 3548 support for Base64, Base32, and Base16 encoding and decoding, including optional case folding and optional alternative alphabets. (Contributed by Barry Warsaw.) \end{itemize} %====================================================================== \section{Porting to Python 2.4} This section lists previously described changes that may require changes to your code: \begin{itemize} \item The \function{zip()} built-in function and \function{itertools.izip()} now return an empty list instead of raising a \exception{TypeError} exception if called with no arguments. \item \function{dircache.listdir()} now passes exceptions to the caller instead of returning empty lists. \item \function{LexicalHandler.startDTD()} used to receive public and system ID in the wrong order. This has been corrected; applications relying on the wrong order need to be fixed. \item \function{fcntl.ioctl} now warns if the mutate arg is omitted and relevant. \end{itemize} %====================================================================== \section{Acknowledgements \label{acks}} The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this article: Raymond Hettinger. \end{document}