Merged revisions 64002-64003,64012,64036-64037,64047,64050-64052,64054-64055,64066,64071 via svnmerge from

svn+ssh://pythondev@svn.python.org/python/trunk

........
  r64002 | travis.oliphant | 2008-06-07 00:33:21 +0200 (Sat, 07 Jun 2008) | 1 line

  Add long double check support to configure test.
........
  r64003 | travis.oliphant | 2008-06-07 00:39:47 +0200 (Sat, 07 Jun 2008) | 1 line

  Remove locking part of new buffer protocol.
........
  r64012 | facundo.batista | 2008-06-07 15:36:36 +0200 (Sat, 07 Jun 2008) | 4 lines


  Finished bug #2451.  Fixed the retrying part to make it
  more robust.
........
  r64036 | georg.brandl | 2008-06-08 10:54:40 +0200 (Sun, 08 Jun 2008) | 2 lines

  #3028: tokenize passes the physical line.
........
  r64037 | georg.brandl | 2008-06-08 10:59:38 +0200 (Sun, 08 Jun 2008) | 2 lines

  Argh, I read it wrong. Reverted 64036 and added a clarifying remark.
........
  r64047 | raymond.hettinger | 2008-06-09 03:28:30 +0200 (Mon, 09 Jun 2008) | 1 line

  Issue3065:  Fixed pickling of named tuples.  Added tests.
........
  r64050 | raymond.hettinger | 2008-06-09 08:54:45 +0200 (Mon, 09 Jun 2008) | 1 line

  Issue #2138: Add math.factorial().
........
  r64051 | raymond.hettinger | 2008-06-09 10:33:37 +0200 (Mon, 09 Jun 2008) | 1 line

  Let set.union() and set.update() accept multiple inputs.
........
  r64052 | raymond.hettinger | 2008-06-09 11:29:17 +0200 (Mon, 09 Jun 2008) | 1 line

  Address double-rounding scenarios by setting all variables to long doubles.
........
  r64054 | raymond.hettinger | 2008-06-09 13:24:47 +0200 (Mon, 09 Jun 2008) | 1 line

  Unhappy buildbots.  Revert 64052.  Long doubles have unexpected effects on some builds.
........
  r64055 | raymond.hettinger | 2008-06-09 15:07:27 +0200 (Mon, 09 Jun 2008) | 1 line

  Let set.intersection() and set.intersection_update() take multiple input arguments.
........
  r64066 | robert.schuppenies | 2008-06-10 12:10:31 +0200 (Tue, 10 Jun 2008) | 2 lines

  Issue 3048: Fixed sys.getsizeof for unicode objects.
........
  r64071 | thomas.heller | 2008-06-10 16:07:12 +0200 (Tue, 10 Jun 2008) | 3 lines

  NEWS entry for:
  Add an optional 'offset' parameter to byref, defaulting to zero.
........

1 files changed, 4 insertions, 6 deletions

diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst
index bbe73d0..b2caded 100644
--- a/Doc/library/tokenize.rst
+++ b/Doc/library/tokenize.rst
@@ -1,4 +1,3 @@
-
 :mod:`tokenize` --- Tokenizer for Python source
 ===============================================
 
@@ -15,7 +14,6 @@ colorizers for on-screen displays.
 
 The primary entry point is a :term:`generator`:
 
-
 .. function:: tokenize(readline)
 
    The :func:`tokenize` generator requires one argument, *readline*, which
@@ -28,11 +26,11 @@ The primary entry point is a :term:`generator`:
    token string; a 2-tuple ``(srow, scol)`` of ints specifying the row and 
    column where the token begins in the source; a 2-tuple ``(erow, ecol)`` of 
    ints specifying the row and column where the token ends in the source; and 
-   the line on which the token was found. The line passed is the *logical* 
-   line; continuation lines are included.
+   the line on which the token was found. The line passed (the last tuple item)
+   is the *logical* line; continuation lines are included.
    
-   tokenize determines the source encoding of the file by looking for a utf-8
-   bom or encoding cookie, according to :pep:`263`.
+   :func:`tokenize` determines the source encoding of the file by looking for a
+   UTF-8 BOM or encoding cookie, according to :pep:`263`.
 
 
 All constants from the :mod:`token` module are also exported from