summaryrefslogtreecommitdiffstats
path: root/Doc/whatsnew/2.4.rst
blob: 9e438ac7615037d282262b2d366ffb1a65146078 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
****************************
  What's New in Python 2.4
****************************

:Author: A.M. Kuchling

.. |release| replace:: 1.02

.. $Id: whatsnew24.tex 54632 2007-03-31 11:59:54Z georg.brandl $
.. Don't write extensive text for new sections; I'll do that.
.. Feel free to add commented-out reminders of things that need
.. to be covered.  --amk

This article explains the new features in Python 2.4.1, released on March 30,
2005.

Python 2.4 is a medium-sized release.  It doesn't introduce as many changes as
the radical Python 2.2, but introduces more features than the conservative 2.3
release.  The most significant new language features are function decorators and
generator expressions; most other changes are to the standard library.

According to the CVS change logs, there were 481 patches applied and 502 bugs
fixed between Python 2.3 and 2.4.  Both figures are likely to be underestimates.

This article doesn't attempt to provide a complete specification of every single
new feature, but instead provides a brief introduction to each feature.  For
full details, you should refer to the documentation for Python 2.4, such as the
Python Library Reference and the Python Reference Manual.  Often you will be
referred to the PEP for a particular new feature for explanations of the
implementation and design rationale.

.. ======================================================================


PEP 218: Built-In Set Objects
=============================

Python 2.3 introduced the :mod:`sets` module.  C implementations of set data
types have now been added to the Python core as two new built-in types,
:func:`set(iterable)` and :func:`frozenset(iterable)`.  They provide high speed
operations for membership testing, for eliminating duplicates from sequences,
and for mathematical operations like unions, intersections, differences, and
symmetric differences. ::

   >>> a = set('abracadabra')              # form a set from a string
   >>> 'z' in a                            # fast membership testing
   False
   >>> a                                   # unique letters in a
   set(['a', 'r', 'b', 'c', 'd'])
   >>> ''.join(a)                          # convert back into a string
   'arbcd'

   >>> b = set('alacazam')                 # form a second set
   >>> a - b                               # letters in a but not in b
   set(['r', 'd', 'b'])
   >>> a | b                               # letters in either a or b
   set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])
   >>> a & b                               # letters in both a and b
   set(['a', 'c'])
   >>> a ^ b                               # letters in a or b but not both
   set(['r', 'd', 'b', 'm', 'z', 'l'])

   >>> a.add('z')                          # add a new element
   >>> a.update('wxy')                     # add multiple new elements
   >>> a
   set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z'])
   >>> a.remove('x')                       # take one element out
   >>> a
   set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z'])

The :func:`frozenset` type is an immutable version of :func:`set`. Since it is
immutable and hashable, it may be used as a dictionary key or as a member of
another set.

The :mod:`sets` module remains in the standard library, and may be useful if you
wish to subclass the :class:`Set` or :class:`ImmutableSet` classes.  There are
currently no plans to deprecate the module.


.. seealso::

   :pep:`218` - Adding a Built-In Set Object Type
      Originally proposed by Greg Wilson and ultimately implemented by Raymond
      Hettinger.

.. ======================================================================


PEP 237: Unifying Long Integers and Integers
============================================

The lengthy transition process for this PEP, begun in Python 2.2, takes another
step forward in Python 2.4.  In 2.3, certain integer operations that would
behave differently after int/long unification triggered :exc:`FutureWarning`
warnings and returned values limited to 32 or 64 bits (depending on your
platform).  In 2.4, these expressions no longer produce a warning and instead
produce a different result that's usually a long integer.

The problematic expressions are primarily left shifts and lengthy hexadecimal
and octal constants.  For example, ``2 << 32`` results in a warning in 2.3,
evaluating to 0 on 32-bit platforms.  In Python 2.4, this expression now returns
the correct answer, 8589934592.


.. seealso::

   :pep:`237` - Unifying Long Integers and Integers
      Original PEP written by Moshe Zadka and GvR.  The changes for 2.4 were
      implemented by  Kalle Svensson.

.. ======================================================================


PEP 289: Generator Expressions
==============================

The iterator feature introduced in Python 2.2 and the :mod:`itertools` module
make it easier to write programs that loop through large data sets without
having the entire data set in memory at one time.  List comprehensions don't fit
into this picture very well because they produce a Python list object containing
all of the items.  This unavoidably pulls all of the objects into memory, which
can be a problem if your data set is very large.  When trying to write a
functionally-styled program, it would be natural to write something like::

   links = [link for link in get_all_links() if not link.followed]
   for link in links:
       ...

instead of  ::

   for link in get_all_links():
       if link.followed:
           continue
       ...

The first form is more concise and perhaps more readable, but if you're dealing
with a large number of link objects you'd have to write the second form to avoid
having all link objects in memory at the same time.

Generator expressions work similarly to list comprehensions but don't
materialize the entire list; instead they create a generator that will return
elements one by one.  The above example could be written as::

   links = (link for link in get_all_links() if not link.followed)
   for link in links:
       ...

Generator expressions always have to be written inside parentheses, as in the
above example.  The parentheses signalling a function call also count, so if you
want to create an iterator that will be immediately passed to a function you
could write::

   print sum(obj.count for obj in list_all_objects())

Generator expressions differ from list comprehensions in various small ways.
Most notably, the loop variable (*obj* in the above example) is not accessible
outside of the generator expression.  List comprehensions leave the variable
assigned to its last value; future versions of Python will change this, making
list comprehensions match generator expressions in this respect.


.. seealso::

   :pep:`289` - Generator Expressions
      Proposed by Raymond Hettinger and implemented by Jiwon Seo with early efforts
      steered by Hye-Shik Chang.

.. ======================================================================


PEP 292: Simpler String Substitutions
=====================================

Some new classes in the standard library provide an alternative mechanism for
substituting variables into strings; this style of substitution may be better
for applications where untrained users need to edit templates.

The usual way of substituting variables by name is the ``%`` operator::

   >>> '%(page)i: %(title)s' % {'page':2, 'title': 'The Best of Times'}
   '2: The Best of Times'

When writing the template string, it can be easy to forget the ``i`` or ``s``
after the closing parenthesis.  This isn't a big problem if the template is in a
Python module, because you run the code, get an "Unsupported format character"
:exc:`ValueError`, and fix the problem.  However, consider an application such
as Mailman where template strings or translations are being edited by users who
aren't aware of the Python language.  The format string's syntax is complicated
to explain to such users, and if they make a mistake, it's difficult to provide
helpful feedback to them.

PEP 292 adds a :class:`Template` class to the :mod:`string` module that uses
``$`` to indicate a substitution::

   >>> import string
   >>> t = string.Template('$page: $title')
   >>> t.substitute({'page':2, 'title': 'The Best of Times'})
   '2: The Best of Times'

If a key is missing from the dictionary, the :meth:`substitute` method will
raise a :exc:`KeyError`.  There's also a :meth:`safe_substitute` method that
ignores missing keys::

   >>> t = string.Template('$page: $title')
   >>> t.safe_substitute({'page':3})
   '3: $title'


.. seealso::

   :pep:`292` - Simpler String Substitutions
      Written and implemented  by Barry Warsaw.

.. ======================================================================


PEP 318: Decorators for Functions and Methods
=============================================

Python 2.2 extended Python's object model by adding static methods and class
methods, but it didn't extend Python's syntax to provide any new way of defining
static or class methods.  Instead, you had to write a :keyword:`def` statement
in the usual way, and pass the resulting method to a :func:`staticmethod` or
:func:`classmethod` function that would wrap up the function as a method of the
new type. Your code would look like this::

   class C:
      def meth (cls):
          ...

      meth = classmethod(meth)   # Rebind name to wrapped-up class method

If the method was very long, it would be easy to miss or forget the
:func:`classmethod` invocation after the function body.

The intention was always to add some syntax to make such definitions more
readable, but at the time of 2.2's release a good syntax was not obvious.  Today
a good syntax *still* isn't obvious but users are asking for easier access to
the feature; a new syntactic feature has been added to meet this need.

The new feature is called "function decorators".  The name comes from the idea
that :func:`classmethod`, :func:`staticmethod`, and friends are storing
additional information on a function object; they're *decorating* functions with
more details.

The notation borrows from Java and uses the ``'@'`` character as an indicator.
Using the new syntax, the example above would be written::

   class C:

      @classmethod
      def meth (cls):
          ...


The ``@classmethod`` is shorthand for the ``meth=classmethod(meth)`` assignment.
More generally, if you have the following::

   @A
   @B
   @C
   def f ():
       ...

It's equivalent to the following pre-decorator code::

   def f(): ...
   f = A(B(C(f)))

Decorators must come on the line before a function definition, one decorator per
line, and can't be on the same line as the def statement, meaning that ``@A def
f(): ...`` is illegal.  You can only decorate function definitions, either at
the module level or inside a class; you can't decorate class definitions.

A decorator is just a function that takes the function to be decorated as an
argument and returns either the same function or some new object.  The return
value of the decorator need not be callable (though it typically is), unless
further decorators will be applied to the result.  It's easy to write your own
decorators.  The following simple example just sets an attribute on the function
object::

   >>> def deco(func):
   ...    func.attr = 'decorated'
   ...    return func
   ...
   >>> @deco
   ... def f(): pass
   ...
   >>> f
   <function f at 0x402ef0d4>
   >>> f.attr
   'decorated'
   >>>

As a slightly more realistic example, the following decorator checks that the
supplied argument is an integer::

   def require_int (func):
       def wrapper (arg):
           assert isinstance(arg, int)
           return func(arg)

       return wrapper

   @require_int
   def p1 (arg):
       print arg

   @require_int
   def p2(arg):
       print arg*2

An example in :pep:`318` contains a fancier version of this idea that lets you
both specify the required type and check the returned type.

Decorator functions can take arguments.  If arguments are supplied, your
decorator function is called with only those arguments and must return a new
decorator function; this function must take a single function and return a
function, as previously described.  In other words, ``@A @B @C(args)`` becomes::

   def f(): ...
   _deco = C(args)
   f = A(B(_deco(f)))

Getting this right can be slightly brain-bending, but it's not too difficult.

A small related change makes the :attr:`func_name` attribute of functions
writable.  This attribute is used to display function names in tracebacks, so
decorators should change the name of any new function that's constructed and
returned.


.. seealso::

   :pep:`318` - Decorators for Functions, Methods and Classes
      Written  by Kevin D. Smith, Jim Jewett, and Skip Montanaro.  Several people
      wrote patches implementing function decorators, but the one that was actually
      checked in was patch #979728, written by Mark Russell.

   http://www.python.org/moin/PythonDecoratorLibrary
      This Wiki page contains several examples of decorators.

.. ======================================================================


PEP 322: Reverse Iteration
==========================

A new built-in function, :func:`reversed(seq)`, takes a sequence and returns an
iterator that loops over the elements of the sequence  in reverse order.   ::

   >>> for i in reversed(xrange(1,4)):
   ...    print i
   ...
   3
   2
   1

Compared to extended slicing, such as ``range(1,4)[::-1]``, :func:`reversed` is
easier to read, runs faster, and uses substantially less memory.

Note that :func:`reversed` only accepts sequences, not arbitrary iterators.  If
you want to reverse an iterator, first convert it to  a list with :func:`list`.
::

   >>> input = open('/etc/passwd', 'r')
   >>> for line in reversed(list(input)):
   ...   print line
   ...
   root:*:0:0:System Administrator:/var/root:/bin/tcsh
     ...


.. seealso::

   :pep:`322` - Reverse Iteration
      Written and implemented by Raymond Hettinger.

.. ======================================================================


PEP 324: New subprocess Module
==============================

The standard library provides a number of ways to execute a subprocess, offering
different features and different levels of complexity.
:func:`os.system(command)` is easy to use, but slow (it runs a shell process
which executes the command) and dangerous (you have to be careful about escaping
the shell's metacharacters).  The :mod:`popen2` module offers classes that can
capture standard output and standard error from the subprocess, but the naming
is confusing.  The :mod:`subprocess` module cleans  this up, providing a unified
interface that offers all the features you might need.

Instead of :mod:`popen2`'s collection of classes, :mod:`subprocess` contains a
single class called :class:`Popen`  whose constructor supports a number of
different keyword arguments. ::

   class Popen(args, bufsize=0, executable=None,
               stdin=None, stdout=None, stderr=None,
               preexec_fn=None, close_fds=False, shell=False,
               cwd=None, env=None, universal_newlines=False,
               startupinfo=None, creationflags=0):

*args* is commonly a sequence of strings that will be the arguments to the
program executed as the subprocess.  (If the *shell* argument is true, *args*
can be a string which will then be passed on to the shell for interpretation,
just as :func:`os.system` does.)

*stdin*, *stdout*, and *stderr* specify what the subprocess's input, output, and
error streams will be.  You can provide a file object or a file descriptor, or
you can use the constant ``subprocess.PIPE`` to create a pipe between the
subprocess and the parent.

The constructor has a number of handy options:

* *close_fds* requests that all file descriptors be closed before running the
  subprocess.

* *cwd* specifies the working directory in which the subprocess will be executed
  (defaulting to whatever the parent's working directory is).

* *env* is a dictionary specifying environment variables.

* *preexec_fn* is a function that gets called before the child is started.

* *universal_newlines* opens the child's input and output using Python's
  universal newline feature.

Once you've created the :class:`Popen` instance,  you can call its :meth:`wait`
method to pause until the subprocess has exited, :meth:`poll` to check if it's
exited without pausing,  or :meth:`communicate(data)` to send the string *data*
to the subprocess's standard input.   :meth:`communicate(data)`  then reads any
data that the subprocess has sent to its standard output  or standard error,
returning a tuple ``(stdout_data, stderr_data)``.

:func:`call` is a shortcut that passes its arguments along to the :class:`Popen`
constructor, waits for the command to complete, and returns the status code of
the subprocess.  It can serve as a safer analog to :func:`os.system`::

   sts = subprocess.call(['dpkg', '-i', '/tmp/new-package.deb'])
   if sts == 0:
       # Success
       ...
   else:
       # dpkg returned an error
       ...

The command is invoked without use of the shell.  If you really do want to  use
the shell, you can add ``shell=True`` as a keyword argument and provide a string
instead of a sequence::

   sts = subprocess.call('dpkg -i /tmp/new-package.deb', shell=True)

The PEP takes various examples of shell and Python code and shows how they'd be
translated into Python code that uses :mod:`subprocess`.  Reading this section
of the PEP is highly recommended.


.. seealso::

   :pep:`324` - subprocess - New process module
      Written and implemented by Peter Åstrand, with assistance from Fredrik Lundh and
      others.

.. ======================================================================


PEP 327: Decimal Data Type
==========================

Python has always supported floating-point (FP) numbers, based on the underlying
C :ctype:`double` type, as a data type.  However, while most programming
languages provide a floating-point type, many people (even programmers) are
unaware that floating-point numbers don't represent certain decimal fractions
accurately.  The new :class:`Decimal` type can represent these fractions
accurately, up to a user-specified precision limit.


Why is Decimal needed?
----------------------

The limitations arise from the representation used for floating-point numbers.
FP numbers are made up of three components:

* The sign, which is positive or negative.

* The mantissa, which is a single-digit binary number   followed by a fractional
  part.  For example, ``1.01`` in base-2 notation is ``1 + 0/2 + 1/4``, or 1.25 in
  decimal notation.

* The exponent, which tells where the decimal point is located in the number
  represented.

For example, the number 1.25 has positive sign, a mantissa value of 1.01 (in
binary), and an exponent of 0 (the decimal point doesn't need to be shifted).
The number 5 has the same sign and mantissa, but the exponent is 2 because the
mantissa is multiplied by 4 (2 to the power of the exponent 2); 1.25 \* 4 equals
5.

Modern systems usually provide floating-point support that conforms to a
standard called IEEE 754.  C's :ctype:`double` type is usually implemented as a
64-bit IEEE 754 number, which uses 52 bits of space for the mantissa.  This
means that numbers can only be specified to 52 bits of precision.  If you're
trying to represent numbers whose expansion repeats endlessly, the expansion is
cut off after 52 bits. Unfortunately, most software needs to produce output in
base 10, and common fractions in base 10 are often repeating decimals in binary.
For example, 1.1 decimal is binary ``1.0001100110011 ...``; .1 = 1/16 + 1/32 +
1/256 plus an infinite number of additional terms.  IEEE 754 has to chop off
that infinitely repeated decimal after 52 digits, so the representation is
slightly inaccurate.

Sometimes you can see this inaccuracy when the number is printed::

   >>> 1.1
   1.1000000000000001

The inaccuracy isn't always visible when you print the number because the FP-to-
decimal-string conversion is provided by the C library, and most C libraries try
to produce sensible output.  Even if it's not displayed, however, the inaccuracy
is still there and subsequent operations can magnify the error.

For many applications this doesn't matter.  If I'm plotting points and
displaying them on my monitor, the difference between 1.1 and 1.1000000000000001
is too small to be visible.  Reports often limit output to a certain number of
decimal places, and if you round the number to two or three or even eight
decimal places, the error is never apparent.  However, for applications where it
does matter,  it's a lot of work to implement your own custom arithmetic
routines.

Hence, the :class:`Decimal` type was created.


The :class:`Decimal` type
-------------------------

A new module, :mod:`decimal`, was added to Python's standard library.  It
contains two classes, :class:`Decimal` and :class:`Context`.  :class:`Decimal`
instances represent numbers, and :class:`Context` instances are used to wrap up
various settings such as the precision and default rounding mode.

:class:`Decimal` instances are immutable, like regular Python integers and FP
numbers; once it's been created, you can't change the value an instance
represents.  :class:`Decimal` instances can be created from integers or
strings::

   >>> import decimal
   >>> decimal.Decimal(1972)
   Decimal("1972")
   >>> decimal.Decimal("1.1")
   Decimal("1.1")

You can also provide tuples containing the sign, the mantissa represented  as a
tuple of decimal digits, and the exponent::

   >>> decimal.Decimal((1, (1, 4, 7, 5), -2))
   Decimal("-14.75")

Cautionary note: the sign bit is a Boolean value, so 0 is positive and 1 is
negative.

Converting from floating-point numbers poses a bit of a problem: should the FP
number representing 1.1 turn into the decimal number for exactly 1.1, or for 1.1
plus whatever inaccuracies are introduced? The decision was to dodge the issue
and leave such a conversion out of the API.  Instead, you should convert the
floating-point number into a string using the desired precision and pass the
string to the :class:`Decimal` constructor::

   >>> f = 1.1
   >>> decimal.Decimal(str(f))
   Decimal("1.1")
   >>> decimal.Decimal('%.12f' % f)
   Decimal("1.100000000000")

Once you have :class:`Decimal` instances, you can perform the usual mathematical
operations on them.  One limitation: exponentiation requires an integer
exponent::

   >>> a = decimal.Decimal('35.72')
   >>> b = decimal.Decimal('1.73')
   >>> a+b
   Decimal("37.45")
   >>> a-b
   Decimal("33.99")
   >>> a*b
   Decimal("61.7956")
   >>> a/b
   Decimal("20.64739884393063583815028902")
   >>> a ** 2
   Decimal("1275.9184")
   >>> a**b
   Traceback (most recent call last):
     ...
   decimal.InvalidOperation: x ** (non-integer)

You can combine :class:`Decimal` instances with integers, but not with floating-
point numbers::

   >>> a + 4
   Decimal("39.72")
   >>> a + 4.5
   Traceback (most recent call last):
     ...
   TypeError: You can interact Decimal only with int, long or Decimal data types.
   >>>

:class:`Decimal` numbers can be used with the :mod:`math` and :mod:`cmath`
modules, but note that they'll be immediately converted to  floating-point
numbers before the operation is performed, resulting in a possible loss of
precision and accuracy.  You'll also get back a regular floating-point number
and not a :class:`Decimal`.   ::

   >>> import math, cmath
   >>> d = decimal.Decimal('123456789012.345')
   >>> math.sqrt(d)
   351364.18288201344
   >>> cmath.sqrt(-d)
   351364.18288201344j

:class:`Decimal` instances have a :meth:`sqrt` method that returns a
:class:`Decimal`, but if you need other things such as trigonometric functions
you'll have to implement them. ::

   >>> d.sqrt()
   Decimal("351364.1828820134592177245001")


The :class:`Context` type
-------------------------

Instances of the :class:`Context` class encapsulate several settings for
decimal operations:

* :attr:`prec` is the precision, the number of decimal places.

* :attr:`rounding` specifies the rounding mode.  The :mod:`decimal` module has
  constants for the various possibilities: :const:`ROUND_DOWN`,
  :const:`ROUND_CEILING`,  :const:`ROUND_HALF_EVEN`, and various others.

* :attr:`traps` is a dictionary specifying what happens on encountering certain
  error conditions: either  an exception is raised or  a value is returned.  Some
  examples of error conditions are division by zero, loss of precision, and
  overflow.

There's a thread-local default context available by calling :func:`getcontext`;
you can change the properties of this context to alter the default precision,
rounding, or trap handling.  The following example shows the effect of changing
the precision of the default context::

   >>> decimal.getcontext().prec
   28
   >>> decimal.Decimal(1) / decimal.Decimal(7)
   Decimal("0.1428571428571428571428571429")
   >>> decimal.getcontext().prec = 9
   >>> decimal.Decimal(1) / decimal.Decimal(7)
   Decimal("0.142857143")

The default action for error conditions is selectable; the module can either
return a special value such as infinity or not-a-number, or exceptions can be
raised::

   >>> decimal.Decimal(1) / decimal.Decimal(0)
   Traceback (most recent call last):
     ...
   decimal.DivisionByZero: x / 0
   >>> decimal.getcontext().traps[decimal.DivisionByZero] = False
   >>> decimal.Decimal(1) / decimal.Decimal(0)
   Decimal("Infinity")
   >>>

The :class:`Context` instance also has various methods for formatting  numbers
such as :meth:`to_eng_string` and :meth:`to_sci_string`.

For more information, see the documentation for the :mod:`decimal` module, which
includes a quick-start tutorial and a reference.


.. seealso::

   :pep:`327` - Decimal Data Type
      Written by Facundo Batista and implemented by Facundo Batista, Eric Price,
      Raymond Hettinger, Aahz, and Tim Peters.

   http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html
      A more detailed overview of the IEEE-754 representation.

   http://www.lahey.com/float.htm
      The article uses Fortran code to illustrate many of the problems that floating-
      point inaccuracy can cause.

   http://www2.hursley.ibm.com/decimal/
      A description of a decimal-based representation.  This representation is being
      proposed as a standard, and underlies the new Python decimal type.  Much of this
      material was written by Mike Cowlishaw, designer of the Rexx language.

.. ======================================================================


PEP 328: Multi-line Imports
===========================

One language change is a small syntactic tweak aimed at making it easier to
import many names from a module.  In a ``from module import names`` statement,
*names* is a sequence of names separated by commas.  If the sequence is  very
long, you can either write multiple imports from the same module, or you can use
backslashes to escape the line endings like this::

   from SimpleXMLRPCServer import SimpleXMLRPCServer,\
               SimpleXMLRPCRequestHandler,\
               CGIXMLRPCRequestHandler,\
               resolve_dotted_attribute

The syntactic change in Python 2.4 simply allows putting the names within
parentheses.  Python ignores newlines within a parenthesized expression, so the
backslashes are no longer needed::

   from SimpleXMLRPCServer import (SimpleXMLRPCServer,
                                   SimpleXMLRPCRequestHandler,
                                   CGIXMLRPCRequestHandler,
                                   resolve_dotted_attribute)

The PEP also proposes that all :keyword:`import` statements be absolute imports,
with a leading ``.`` character to indicate a relative import.  This part of the
PEP was not implemented for Python 2.4, but was completed for Python 2.5.


.. seealso::

   :pep:`328` - Imports: Multi-Line and Absolute/Relative
      Written by Aahz.  Multi-line imports were implemented by Dima Dorfman.

.. ======================================================================


PEP 331: Locale-Independent Float/String Conversions
====================================================

The :mod:`locale` modules lets Python software select various conversions and
display conventions that are localized to a particular country or language.
However, the module was careful to not change the numeric locale because various
functions in Python's implementation required that the numeric locale remain set
to the ``'C'`` locale.  Often this was because the code was using the C
library's :cfunc:`atof` function.

Not setting the numeric locale caused trouble for extensions that used third-
party C libraries, however, because they wouldn't have the correct locale set.
The motivating example was GTK+, whose user interface widgets weren't displaying
numbers in the current locale.

The solution described in the PEP is to add three new functions to the Python
API that perform ASCII-only conversions, ignoring the locale setting:

* :cfunc:`PyOS_ascii_strtod(str, ptr)`  and :cfunc:`PyOS_ascii_atof(str, ptr)`
  both convert a string to a C :ctype:`double`.

* :cfunc:`PyOS_ascii_formatd(buffer, buf_len, format, d)` converts a
  :ctype:`double` to an ASCII string.

The code for these functions came from the GLib library
(http://developer.gnome.org/arch/gtk/glib.html), whose developers kindly
relicensed the relevant functions and donated them to the Python Software
Foundation.  The :mod:`locale` module  can now change the numeric locale,
letting extensions such as GTK+  produce the correct results.


.. seealso::

   :pep:`331` - Locale-Independent Float/String Conversions
      Written by Christian R. Reis, and implemented by Gustavo Carneiro.

.. ======================================================================


Other Language Changes
======================

Here are all of the changes that Python 2.4 makes to the core Python language.

* Decorators for functions and methods were added (:pep:`318`).

* Built-in :func:`set` and :func:`frozenset` types were  added (:pep:`218`).
  Other new built-ins include the :func:`reversed(seq)` function (:pep:`322`).

* Generator expressions were added (:pep:`289`).

* Certain numeric expressions no longer return values restricted to 32 or 64
  bits (:pep:`237`).

* You can now put parentheses around the list of names in a ``from module import
  names`` statement (:pep:`328`).

* The :meth:`dict.update` method now accepts the same argument forms as the
  :class:`dict` constructor.  This includes any mapping, any iterable of key/value
  pairs, and keyword arguments. (Contributed by Raymond Hettinger.)

* The string methods :meth:`ljust`, :meth:`rjust`, and :meth:`center` now take
  an optional argument for specifying a fill character other than a space.
  (Contributed by Raymond Hettinger.)

* Strings also gained an :meth:`rsplit` method that works like the :meth:`split`
  method but splits from the end of the string.   (Contributed by Sean
  Reifschneider.) ::

     >>> 'www.python.org'.split('.', 1)
     ['www', 'python.org']
     'www.python.org'.rsplit('.', 1)
     ['www.python', 'org']

* Three keyword parameters, *cmp*, *key*, and *reverse*, were added to the
  :meth:`sort` method of lists. These parameters make some common usages of
  :meth:`sort` simpler. All of these parameters are optional.

  For the *cmp* parameter, the value should be a comparison function that takes
  two parameters and returns -1, 0, or +1 depending on how the parameters compare.
  This function will then be used to sort the list.  Previously this was the only
  parameter that could be provided to :meth:`sort`.

  *key* should be a single-parameter function that takes a list element and
  returns a comparison key for the element.  The list is then sorted using the
  comparison keys.  The following example sorts a list case-insensitively::

     >>> L = ['A', 'b', 'c', 'D']
     >>> L.sort()                 # Case-sensitive sort
     >>> L
     ['A', 'D', 'b', 'c']
     >>> # Using 'key' parameter to sort list
     >>> L.sort(key=lambda x: x.lower())
     >>> L
     ['A', 'b', 'c', 'D']
     >>> # Old-fashioned way
     >>> L.sort(cmp=lambda x,y: cmp(x.lower(), y.lower()))
     >>> L
     ['A', 'b', 'c', 'D']

  The last example, which uses the *cmp* parameter, is the old way to perform a
  case-insensitive sort.  It works but is slower than using a *key* parameter.
  Using *key* calls :meth:`lower` method once for each element in the list while
  using *cmp* will call it twice for each comparison, so using *key* saves on
  invocations of the :meth:`lower` method.

  For simple key functions and comparison functions, it is often possible to avoid
  a :keyword:`lambda` expression by using an unbound method instead.  For example,
  the above case-insensitive sort is best written as::

     >>> L.sort(key=str.lower)
     >>> L
     ['A', 'b', 'c', 'D']

  Finally, the *reverse* parameter takes a Boolean value.  If the value is true,
  the list will be sorted into reverse order. Instead of ``L.sort() ;
  L.reverse()``, you can now write ``L.sort(reverse=True)``.

  The results of sorting are now guaranteed to be stable.  This means that two
  entries with equal keys will be returned in the same order as they were input.
  For example, you can sort a list of people by name, and then sort the list by
  age, resulting in a list sorted by age where people with the same age are in
  name-sorted order.

  (All changes to :meth:`sort` contributed by Raymond Hettinger.)

* There is a new built-in function :func:`sorted(iterable)` that works like the
  in-place :meth:`list.sort` method but can be used in expressions.  The
  differences are:

* the input may be any iterable;

* a newly formed copy is sorted, leaving the original intact; and

* the expression returns the new sorted copy

  ::

     >>> L = [9,7,8,3,2,4,1,6,5]
     >>> [10+i for i in sorted(L)]       # usable in a list comprehension
     [11, 12, 13, 14, 15, 16, 17, 18, 19]
     >>> L                               # original is left unchanged
     [9,7,8,3,2,4,1,6,5]
     >>> sorted('Monty Python')          # any iterable may be an input
     [' ', 'M', 'P', 'h', 'n', 'n', 'o', 'o', 't', 't', 'y', 'y']

     >>> # List the contents of a dict sorted by key values
     >>> colormap = dict(red=1, blue=2, green=3, black=4, yellow=5)
     >>> for k, v in sorted(colormap.iteritems()):
     ...     print k, v
     ...
     black 4
     blue 2
     green 3
     red 1
     yellow 5

  (Contributed by Raymond Hettinger.)

* Integer operations will no longer trigger an :exc:`OverflowWarning`. The
  :exc:`OverflowWarning` warning will disappear in Python 2.5.

* The interpreter gained a new switch, :option:`-m`, that takes a name, searches
  for the corresponding  module on ``sys.path``, and runs the module as a script.
  For example,  you can now run the Python profiler with ``python -m profile``.
  (Contributed by Nick Coghlan.)

* The :func:`eval(expr, globals, locals)` and :func:`execfile(filename, globals,
  locals)` functions and the :keyword:`exec` statement now accept any mapping type
  for the *locals* parameter.  Previously this had to be a regular Python
  dictionary.  (Contributed by Raymond Hettinger.)

* The :func:`zip` built-in function and :func:`itertools.izip` now return an
  empty list if called with no arguments. Previously they raised a
  :exc:`TypeError` exception.  This makes them more suitable for use with variable
  length argument lists::

     >>> def transpose(array):
     ...    return zip(*array)
     ...
     >>> transpose([(1,2,3), (4,5,6)])
     [(1, 4), (2, 5), (3, 6)]
     >>> transpose([])
     []

  (Contributed by Raymond Hettinger.)

* Encountering a failure while importing a module no longer leaves a partially-
  initialized module object in ``sys.modules``.  The incomplete module object left
  behind would fool further imports of the same module into succeeding, leading to
  confusing errors.   (Fixed by Tim Peters.)

* :const:`None` is now a constant; code that binds a new value to  the name
  ``None`` is now a syntax error. (Contributed by Raymond Hettinger.)

.. ======================================================================


Optimizations
-------------

* The inner loops for list and tuple slicing were optimized and now run about
  one-third faster.  The inner loops for dictionaries were also optimized,
  resulting in performance boosts for :meth:`keys`, :meth:`values`, :meth:`items`,
  :meth:`iterkeys`, :meth:`itervalues`, and :meth:`iteritems`. (Contributed by
  Raymond Hettinger.)

* The machinery for growing and shrinking lists was optimized for speed and for
  space efficiency.  Appending and popping from lists now runs faster due to more
  efficient code paths and less frequent use of the underlying system
  :cfunc:`realloc`.  List comprehensions also benefit.   :meth:`list.extend` was
  also optimized and no longer converts its argument into a temporary list before
  extending the base list.  (Contributed by Raymond Hettinger.)

* :func:`list`, :func:`tuple`, :func:`map`, :func:`filter`, and :func:`zip` now
  run several times faster with non-sequence arguments that supply a
  :meth:`__len__` method.  (Contributed by Raymond Hettinger.)

* The methods :meth:`list.__getitem__`, :meth:`dict.__getitem__`, and
  :meth:`dict.__contains__` are are now implemented as :class:`method_descriptor`
  objects rather than :class:`wrapper_descriptor` objects.  This form of  access
  doubles their performance and makes them more suitable for use as arguments to
  functionals: ``map(mydict.__getitem__, keylist)``. (Contributed by Raymond
  Hettinger.)

* Added a new opcode, ``LIST_APPEND``, that simplifies the generated bytecode
  for list comprehensions and speeds them up by about a third.  (Contributed by
  Raymond Hettinger.)

* The peephole bytecode optimizer has been improved to  produce shorter, faster
  bytecode; remarkably, the resulting bytecode is  more readable.  (Enhanced by
  Raymond Hettinger.)

* String concatenations in statements of the form ``s = s + "abc"`` and ``s +=
  "abc"`` are now performed more efficiently in certain circumstances.  This
  optimization won't be present in other Python implementations such as Jython, so
  you shouldn't rely on it; using the :meth:`join` method of strings is still
  recommended when you want to efficiently glue a large number of strings
  together. (Contributed by Armin Rigo.)

The net result of the 2.4 optimizations is that Python 2.4 runs the pystone
benchmark around 5% faster than Python 2.3 and 35% faster than Python 2.2.
(pystone is not a particularly good benchmark, but it's the most commonly used
measurement of Python's performance.  Your own applications may show greater or
smaller benefits from Python 2.4.)

.. pystone is almost useless for comparing different versions of Python;
   instead, it excels at predicting relative Python performance on different
   machines.  So, this section would be more informative if it used other tools
   such as pybench and parrotbench.  For a more application oriented benchmark,
   try comparing the timings of test_decimal.py under 2.3 and 2.4.

.. ======================================================================


New, Improved, and Deprecated Modules
=====================================

As usual, Python's standard library received a number of enhancements and bug
fixes.  Here's a partial list of the most notable changes, sorted alphabetically
by module name. Consult the :file:`Misc/NEWS` file in the source tree for a more
complete list of changes, or look through the CVS logs for all the details.

* The :mod:`asyncore` module's :func:`loop` function now has a *count* parameter
  that lets you perform a limited number of passes through the polling loop.  The
  default is still to loop forever.

* The :mod:`base64` module now has more complete RFC 3548 support for Base64,
  Base32, and Base16 encoding and decoding, including optional case folding and
  optional alternative alphabets. (Contributed by Barry Warsaw.)

* The :mod:`bisect` module now has an underlying C implementation for improved
  performance. (Contributed by Dmitry Vasiliev.)

* The CJKCodecs collections of East Asian codecs, maintained by Hye-Shik Chang,
  was integrated into 2.4.   The new encodings are:

* Chinese (PRC): gb2312, gbk, gb18030, big5hkscs, hz

* Chinese (ROC): big5, cp950

* Japanese: cp932, euc-jis-2004, euc-jp, euc-jisx0213, iso-2022-jp,
    iso-2022-jp-1, iso-2022-jp-2, iso-2022-jp-3, iso-2022-jp-ext, iso-2022-jp-2004,
    shift-jis, shift-jisx0213, shift-jis-2004

* Korean: cp949, euc-kr, johab, iso-2022-kr

* Some other new encodings were added: HP Roman8,  ISO_8859-11, ISO_8859-16,
  PCTP-154, and TIS-620.

* The UTF-8 and UTF-16 codecs now cope better with receiving partial input.
  Previously the :class:`StreamReader` class would try to read more data, making
  it impossible to resume decoding from the stream.  The :meth:`read` method will
  now return as much data as it can and future calls will resume decoding where
  previous ones left off.  (Implemented by Walter Dörwald.)

* There is a new :mod:`collections` module for  various specialized collection
  datatypes.   Currently it contains just one type, :class:`deque`,  a double-
  ended queue that supports efficiently adding and removing elements from either
  end::

     >>> from collections import deque
     >>> d = deque('ghi')        # make a new deque with three items
     >>> d.append('j')           # add a new entry to the right side
     >>> d.appendleft('f')       # add a new entry to the left side
     >>> d                       # show the representation of the deque
     deque(['f', 'g', 'h', 'i', 'j'])
     >>> d.pop()                 # return and remove the rightmost item
     'j'
     >>> d.popleft()             # return and remove the leftmost item
     'f'
     >>> list(d)                 # list the contents of the deque
     ['g', 'h', 'i']
     >>> 'h' in d                # search the deque
     True

  Several modules, such as the :mod:`Queue` and :mod:`threading` modules, now take
  advantage of :class:`collections.deque` for improved performance.  (Contributed
  by Raymond Hettinger.)

* The :mod:`ConfigParser` classes have been enhanced slightly. The :meth:`read`
  method now returns a list of the files that were successfully parsed, and the
  :meth:`set` method raises :exc:`TypeError` if passed a *value* argument that
  isn't a string.   (Contributed by John Belmonte and David Goodger.)

* The :mod:`curses` module now supports the ncurses extension
  :func:`use_default_colors`.  On platforms where the terminal supports
  transparency, this makes it possible to use a transparent background.
  (Contributed by Jörg Lehmann.)

* The :mod:`difflib` module now includes an :class:`HtmlDiff` class that creates
  an HTML table showing a side by side comparison of two versions of a text.
  (Contributed by Dan Gass.)

* The :mod:`email` package was updated to version 3.0,  which dropped various
  deprecated APIs and removes support for Python versions earlier than 2.3.  The
  3.0 version of the package uses a new incremental parser for MIME messages,
  available in the :mod:`email.FeedParser` module.  The new parser doesn't require
  reading the entire message into memory, and doesn't throw exceptions if a
  message is malformed; instead it records any problems in the  :attr:`defect`
  attribute of the message.  (Developed by Anthony Baxter, Barry Warsaw, Thomas
  Wouters, and others.)

* The :mod:`heapq` module has been converted to C.  The resulting tenfold
  improvement in speed makes the module suitable for handling high volumes of
  data.  In addition, the module has two new functions :func:`nlargest` and
  :func:`nsmallest` that use heaps to find the N largest or smallest values in a
  dataset without the expense of a full sort.  (Contributed by Raymond Hettinger.)

* The :mod:`httplib` module now contains constants for HTTP status codes defined
  in various HTTP-related RFC documents.  Constants have names such as
  :const:`OK`, :const:`CREATED`, :const:`CONTINUE`, and
  :const:`MOVED_PERMANENTLY`; use pydoc to get a full list.  (Contributed by
  Andrew Eland.)

* The :mod:`imaplib` module now supports IMAP's THREAD command (contributed by
  Yves Dionne) and new :meth:`deleteacl` and :meth:`myrights` methods (contributed
  by Arnaud Mazin).

* The :mod:`itertools` module gained a :func:`groupby(iterable[, *func*])`
  function. *iterable* is something that can be iterated over to return a stream
  of elements, and the optional *func* parameter is a function that takes an
  element and returns a key value; if omitted, the key is simply the element
  itself.  :func:`groupby` then groups the elements into subsequences which have
  matching values of the key, and returns a series of 2-tuples containing the key
  value and an iterator over the subsequence.

  Here's an example to make this clearer.  The *key* function simply returns
  whether a number is even or odd, so the result of :func:`groupby` is to return
  consecutive runs of odd or even numbers. ::

     >>> import itertools
     >>> L = [2, 4, 6, 7, 8, 9, 11, 12, 14]
     >>> for key_val, it in itertools.groupby(L, lambda x: x % 2):
     ...    print key_val, list(it)
     ...
     0 [2, 4, 6]
     1 [7]
     0 [8]
     1 [9, 11]
     0 [12, 14]
     >>>

  :func:`groupby` is typically used with sorted input.  The logic for
  :func:`groupby` is similar to the Unix ``uniq`` filter which makes it handy for
  eliminating, counting, or identifying duplicate elements::

     >>> word = 'abracadabra'
     >>> letters = sorted(word)   # Turn string into a sorted list of letters
     >>> letters
     ['a', 'a', 'a', 'a', 'a', 'b', 'b', 'c', 'd', 'r', 'r']
     >>> for k, g in itertools.groupby(letters):
     ...    print k, list(g)
     ...
     a ['a', 'a', 'a', 'a', 'a']
     b ['b', 'b']
     c ['c']
     d ['d']
     r ['r', 'r']
     >>> # List unique letters
     >>> [k for k, g in groupby(letters)]
     ['a', 'b', 'c', 'd', 'r']
     >>> # Count letter occurrences
     >>> [(k, len(list(g))) for k, g in groupby(letters)]
     [('a', 5), ('b', 2), ('c', 1), ('d', 1), ('r', 2)]

  (Contributed by Hye-Shik Chang.)

* :mod:`itertools` also gained a function named :func:`tee(iterator, N)` that
  returns *N* independent iterators that replicate *iterator*.  If *N* is omitted,
  the default is 2. ::

     >>> L = [1,2,3]
     >>> i1, i2 = itertools.tee(L)
     >>> i1,i2
     (<itertools.tee object at 0x402c2080>, <itertools.tee object at 0x402c2090>)
     >>> list(i1)               # Run the first iterator to exhaustion
     [1, 2, 3]
     >>> list(i2)               # Run the second iterator to exhaustion
     [1, 2, 3]

  Note that :func:`tee` has to keep copies of the values returned  by the
  iterator; in the worst case, it may need to keep all of them.   This should
  therefore be used carefully if the leading iterator can run far ahead of the
  trailing iterator in a long stream of inputs. If the separation is large, then
  you might as well use  :func:`list` instead.  When the iterators track closely
  with one another, :func:`tee` is ideal.  Possible applications include
  bookmarking, windowing, or lookahead iterators. (Contributed by Raymond
  Hettinger.)

* A number of functions were added to the :mod:`locale`  module, such as
  :func:`bind_textdomain_codeset` to specify a particular encoding and a family of
  :func:`l\*gettext` functions that return messages in the chosen encoding.
  (Contributed by Gustavo Niemeyer.)

* Some keyword arguments were added to the :mod:`logging` package's
  :func:`basicConfig` function to simplify log configuration.  The default
  behavior is to log messages to standard error, but various keyword arguments can
  be specified to log to a particular file, change the logging format, or set the
  logging level. For example::

     import logging
     logging.basicConfig(filename='/var/log/application.log',
         level=0,  # Log all messages
         format='%(levelname):%(process):%(thread):%(message)')

  Other additions to the :mod:`logging` package include a :meth:`log(level, msg)`
  convenience method, as well as a :class:`TimedRotatingFileHandler` class that
  rotates its log files at a timed interval.  The module already had
  :class:`RotatingFileHandler`, which rotated logs once the file exceeded a
  certain size.  Both classes derive from a new :class:`BaseRotatingHandler` class
  that can be used to implement other rotating handlers.

  (Changes implemented by Vinay Sajip.)

* The :mod:`marshal` module now shares interned strings on unpacking a  data
  structure.  This may shrink the size of certain pickle strings, but the primary
  effect is to make :file:`.pyc` files significantly smaller. (Contributed by
  Martin von Löwis.)

* The :mod:`nntplib` module's :class:`NNTP` class gained :meth:`description` and
  :meth:`descriptions` methods to retrieve  newsgroup descriptions for a single
  group or for a range of groups. (Contributed by Jürgen A. Erhard.)

* Two new functions were added to the :mod:`operator` module,
  :func:`attrgetter(attr)` and :func:`itemgetter(index)`. Both functions return
  callables that take a single argument and return the corresponding attribute or
  item; these callables make excellent data extractors when used with :func:`map`
  or :func:`sorted`.  For example::

     >>> L = [('c', 2), ('d', 1), ('a', 4), ('b', 3)]
     >>> map(operator.itemgetter(0), L)
     ['c', 'd', 'a', 'b']
     >>> map(operator.itemgetter(1), L)
     [2, 1, 4, 3]
     >>> sorted(L, key=operator.itemgetter(1)) # Sort list by second tuple item
     [('d', 1), ('c', 2), ('b', 3), ('a', 4)]

  (Contributed by Raymond Hettinger.)

* The :mod:`optparse` module was updated in various ways.  The module now passes
  its messages through :func:`gettext.gettext`, making it possible to
  internationalize Optik's help and error messages.  Help messages for options can
  now include the string ``'%default'``, which will be replaced by the option's
  default value.  (Contributed by Greg Ward.)

* The long-term plan is to deprecate the :mod:`rfc822` module in some future
  Python release in favor of the :mod:`email` package. To this end, the
  :func:`email.Utils.formatdate` function has been changed to make it usable as a
  replacement for :func:`rfc822.formatdate`.  You may want to write new e-mail
  processing code with this in mind.  (Change implemented by Anthony Baxter.)

* A new :func:`urandom(n)` function was added to the :mod:`os` module, returning
  a string containing *n* bytes of random data.  This function provides access to
  platform-specific sources of randomness such as :file:`/dev/urandom` on Linux or
  the Windows CryptoAPI.  (Contributed by Trevor Perrin.)

* Another new function: :func:`os.path.lexists(path)`  returns true if the file
  specified by *path* exists, whether or not it's a symbolic link.  This differs
  from the existing :func:`os.path.exists(path)` function, which returns false if
  *path* is a symlink that points to a destination that doesn't exist.
  (Contributed by Beni Cherniavsky.)

* A new :func:`getsid` function was added to the :mod:`posix` module that
  underlies the :mod:`os` module. (Contributed by J. Raynor.)

* The :mod:`poplib` module now supports POP over SSL.  (Contributed by Hector
  Urtubia.)

* The :mod:`profile` module can now profile C extension functions. (Contributed
  by Nick Bastin.)

* The :mod:`random` module has a new method called :meth:`getrandbits(N)` that
  returns a long integer *N* bits in length.  The existing :meth:`randrange`
  method now uses :meth:`getrandbits` where appropriate, making generation of
  arbitrarily large random numbers more efficient.  (Contributed by Raymond
  Hettinger.)

* The regular expression language accepted by the :mod:`re` module was extended
  with simple conditional expressions, written as ``(?(group)A|B)``.  *group* is
  either a numeric group ID or a group name defined with ``(?P<group>...)``
  earlier in the expression.  If the specified group matched, the regular
  expression pattern *A* will be tested against the string; if the group didn't
  match, the pattern *B* will be used instead. (Contributed by Gustavo Niemeyer.)

* The :mod:`re` module is also no longer recursive, thanks to a massive amount
  of work by Gustavo Niemeyer.  In a recursive regular expression engine, certain
  patterns result in a large amount of C stack space being consumed, and it was
  possible to overflow the stack. For example, if you matched a 30000-byte string
  of ``a`` characters against the expression ``(a|b)+``, one stack frame was
  consumed per character.  Python 2.3 tried to check for stack overflow and raise
  a :exc:`RuntimeError` exception, but certain patterns could sidestep the
  checking and if you were unlucky Python could segfault. Python 2.4's regular
  expression engine can match this pattern without problems.

* The :mod:`signal` module now performs tighter error-checking on the parameters
  to the :func:`signal.signal` function.  For example, you can't set a handler on
  the :const:`SIGKILL` signal; previous versions of Python would quietly accept
  this, but 2.4 will raise a :exc:`RuntimeError` exception.

* Two new functions were added to the :mod:`socket` module. :func:`socketpair`
  returns a pair of connected sockets and :func:`getservbyport(port)` looks up the
  service name for a given port number. (Contributed by Dave Cole and Barry
  Warsaw.)

* The :func:`sys.exitfunc` function has been deprecated.  Code should be using
  the existing :mod:`atexit` module, which correctly handles calling multiple exit
  functions.  Eventually :func:`sys.exitfunc` will become a purely internal
  interface, accessed only by :mod:`atexit`.

* The :mod:`tarfile` module now generates GNU-format tar files by default.
  (Contributed by Lars Gustaebel.)

* The :mod:`threading` module now has an elegantly simple way to support
  thread-local data.  The module contains a :class:`local` class whose attribute
  values are local to different threads. ::

     import threading

     data = threading.local()
     data.number = 42
     data.url = ('www.python.org', 80)

  Other threads can assign and retrieve their own values for the :attr:`number`
  and :attr:`url` attributes.  You can subclass :class:`local` to initialize
  attributes or to add methods. (Contributed by Jim Fulton.)

* The :mod:`timeit` module now automatically disables periodic garbage
  collection during the timing loop.  This change makes consecutive timings more
  comparable.  (Contributed by Raymond Hettinger.)

* The :mod:`weakref` module now supports a wider variety of objects including
  Python functions, class instances, sets, frozensets, deques, arrays, files,
  sockets, and regular expression pattern objects. (Contributed by Raymond
  Hettinger.)

* The :mod:`xmlrpclib` module now supports a multi-call extension for
  transmitting multiple XML-RPC calls in a single HTTP operation. (Contributed by
  Brian Quinlan.)

* The :mod:`mpz`, :mod:`rotor`, and :mod:`xreadlines` modules have  been
  removed.

.. ======================================================================
.. whole new modules get described in subsections here
.. =====================


cookielib
---------

The :mod:`cookielib` library supports client-side handling for HTTP cookies,
mirroring the :mod:`Cookie` module's server-side cookie support. Cookies are
stored in cookie jars; the library transparently stores cookies offered by the
web server in the cookie jar, and fetches the cookie from the jar when
connecting to the server. As in web browsers, policy objects control whether
cookies are accepted or not.

In order to store cookies across sessions, two implementations of cookie jars
are provided: one that stores cookies in the Netscape format so applications can
use the Mozilla or Lynx cookie files, and one that stores cookies in the same
format as the Perl libwww library.

:mod:`urllib2` has been changed to interact with :mod:`cookielib`:
:class:`HTTPCookieProcessor` manages a cookie jar that is used when accessing
URLs.

This module was contributed by John J. Lee.

.. ==================


doctest
-------

The :mod:`doctest` module underwent considerable refactoring thanks to Edward
Loper and Tim Peters.  Testing can still be as simple as running
:func:`doctest.testmod`, but the refactorings allow customizing the module's
operation in various ways

The new :class:`DocTestFinder` class extracts the tests from a given  object's
docstrings::

   def f (x, y):
       """>>> f(2,2)
   4
   >>> f(3,2)
   6
       """
       return x*y

   finder = doctest.DocTestFinder()

   # Get list of DocTest instances
   tests = finder.find(f)

The new :class:`DocTestRunner` class then runs individual tests and can produce
a summary of the results::

   runner = doctest.DocTestRunner()
   for t in tests:
       tried, failed = runner.run(t)

   runner.summarize(verbose=1)

The above example produces the following output::

   1 items passed all tests:
      2 tests in f
   2 tests in 1 items.
   2 passed and 0 failed.
   Test passed.

:class:`DocTestRunner` uses an instance of the :class:`OutputChecker` class to
compare the expected output with the actual output.  This class takes a number
of different flags that customize its behaviour; ambitious users can also write
a completely new subclass of :class:`OutputChecker`.

The default output checker provides a number of handy features. For example,
with the :const:`doctest.ELLIPSIS` option flag, an ellipsis (``...``) in the
expected output matches any substring,  making it easier to accommodate outputs
that vary in minor ways::

   def o (n):
       """>>> o(1)
   <__main__.C instance at 0x...>
   >>>
   """

Another special string, ``<BLANKLINE>``, matches a blank line::

   def p (n):
       """>>> p(1)
   <BLANKLINE>
   >>>
   """

Another new capability is producing a diff-style display of the output by
specifying the :const:`doctest.REPORT_UDIFF` (unified diffs),
:const:`doctest.REPORT_CDIFF` (context diffs), or :const:`doctest.REPORT_NDIFF`
(delta-style) option flags.  For example::

   def g (n):
       """>>> g(4)
   here
   is
   a
   lengthy
   >>>"""
       L = 'here is a rather lengthy list of words'.split()
       for word in L[:n]:
           print word

Running the above function's tests with :const:`doctest.REPORT_UDIFF` specified,
you get the following output::

   **********************************************************************
   File "t.py", line 15, in g
   Failed example:
       g(4)
   Differences (unified diff with -expected +actual):
       @@ -2,3 +2,3 @@
        is
        a
       -lengthy
       +rather
   **********************************************************************

.. ======================================================================


Build and C API Changes
=======================

Some of the changes to Python's build process and to the C API are:

* Three new convenience macros were added for common return values from
  extension functions: :cmacro:`Py_RETURN_NONE`, :cmacro:`Py_RETURN_TRUE`, and
  :cmacro:`Py_RETURN_FALSE`. (Contributed by Brett Cannon.)

* Another new macro, :cmacro:`Py_CLEAR(obj)`,  decreases the reference count of
  *obj* and sets *obj* to the null pointer.  (Contributed by Jim Fulton.)

* A new function, :cfunc:`PyTuple_Pack(N, obj1, obj2, ..., objN)`, constructs
  tuples from a variable length argument list of Python objects.  (Contributed by
  Raymond Hettinger.)

* A new function, :cfunc:`PyDict_Contains(d, k)`, implements fast dictionary
  lookups without masking exceptions raised during the look-up process.
  (Contributed by Raymond Hettinger.)

* The :cmacro:`Py_IS_NAN(X)` macro returns 1 if  its float or double argument
  *X* is a NaN.   (Contributed by Tim Peters.)

* C code can avoid unnecessary locking by using the new
  :cfunc:`PyEval_ThreadsInitialized` function to tell  if any thread operations
  have been performed.  If this function  returns false, no lock operations are
  needed. (Contributed by Nick Coghlan.)

* A new function, :cfunc:`PyArg_VaParseTupleAndKeywords`, is the same as
  :cfunc:`PyArg_ParseTupleAndKeywords` but takes a  :ctype:`va_list` instead of a
  number of arguments. (Contributed by Greg Chapman.)

* A new method flag, :const:`METH_COEXISTS`, allows a function defined in slots
  to co-exist with a :ctype:`PyCFunction` having the same name.  This can halve
  the access time for a method such as :meth:`set.__contains__`.  (Contributed by
  Raymond Hettinger.)

* Python can now be built with additional profiling for the interpreter itself,
  intended as an aid to people developing the Python core.  Providing
  :option:`----enable-profiling` to the :program:`configure` script will let you
  profile the interpreter with :program:`gprof`, and providing the
  :option:`----with-tsc` switch enables profiling using the Pentium's Time-Stamp-
  Counter register.  Note that the :option:`----with-tsc` switch is slightly
  misnamed, because the profiling feature also works on the PowerPC platform,
  though that processor architecture doesn't call that register "the TSC
  register".  (Contributed by Jeremy Hylton.)

* The :ctype:`tracebackobject` type has been renamed to
  :ctype:`PyTracebackObject`.

.. ======================================================================


Port-Specific Changes
---------------------

* The Windows port now builds under MSVC++ 7.1 as well as version 6.
  (Contributed by Martin von Löwis.)

.. ======================================================================


Porting to Python 2.4
=====================

This section lists previously described changes that may require changes to your
code:

* Left shifts and hexadecimal/octal constants that are too  large no longer
  trigger a :exc:`FutureWarning` and return  a value limited to 32 or 64 bits;
  instead they return a long integer.

* Integer operations will no longer trigger an :exc:`OverflowWarning`. The
  :exc:`OverflowWarning` warning will disappear in Python 2.5.

* The :func:`zip` built-in function and :func:`itertools.izip` now return  an
  empty list instead of raising a :exc:`TypeError` exception if called with no
  arguments.

* You can no longer compare the :class:`date` and :class:`datetime` instances
  provided by the :mod:`datetime` module.  Two  instances of different classes
  will now always be unequal, and  relative comparisons (``<``, ``>``) will raise
  a :exc:`TypeError`.

* :func:`dircache.listdir` now passes exceptions to the caller instead of
  returning empty lists.

* :func:`LexicalHandler.startDTD` used to receive the public and system IDs in
  the wrong order.  This has been corrected; applications relying on the wrong
  order need to be fixed.

* :func:`fcntl.ioctl` now warns if the *mutate*  argument is omitted and
  relevant.

* The :mod:`tarfile` module now generates GNU-format tar files by default.

* Encountering a failure while importing a module no longer leaves a partially-
  initialized module object in ``sys.modules``.

* :const:`None` is now a constant; code that binds a new value to  the name
  ``None`` is now a syntax error.

* The :func:`signals.signal` function now raises a :exc:`RuntimeError` exception
  for certain illegal values; previously these errors would pass silently.  For
  example, you can no longer set a handler on the :const:`SIGKILL` signal.

.. ======================================================================


.. _24acks:

Acknowledgements
================

The author would like to thank the following people for offering suggestions,
corrections and assistance with various drafts of this article: Koray Can, Hye-
Shik Chang, Michael Dyck, Raymond Hettinger, Brian Hurt, Hamish Lawson, Fredrik
Lundh, Sean Reifschneider, Sadruddin Rejeb.