Merged revisions 76235 via svnmerge from

svn+ssh://pythondev@svn.python.org/python/branches/py3k

................
  r76235 | benjamin.peterson | 2009-11-12 20:25:08 -0600 (Thu, 12 Nov 2009) | 170 lines

  Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ........
    r75149 | gregory.p.smith | 2009-09-29 16:56:31 -0500 (Tue, 29 Sep 2009) | 3 lines

    Mention issue6972 in extractall docs about overwriting things outside of
    the supplied path.
  ........
    r75260 | andrew.kuchling | 2009-10-05 16:24:20 -0500 (Mon, 05 Oct 2009) | 1 line

    Wording fix
  ........
    r75261 | andrew.kuchling | 2009-10-05 16:24:35 -0500 (Mon, 05 Oct 2009) | 1 line

    Fix narkup
  ........
    r75262 | andrew.kuchling | 2009-10-05 16:25:03 -0500 (Mon, 05 Oct 2009) | 1 line

    Document 'skip' parameter to constructor
  ........
    r75263 | andrew.kuchling | 2009-10-05 16:25:35 -0500 (Mon, 05 Oct 2009) | 1 line

    Note side benefit of socket.create_connection()
  ........
    r75265 | andrew.kuchling | 2009-10-05 17:31:11 -0500 (Mon, 05 Oct 2009) | 1 line

    Reword sentence
  ........
    r75266 | andrew.kuchling | 2009-10-05 17:32:48 -0500 (Mon, 05 Oct 2009) | 1 line

    Use standard comma punctuation; reword some sentences in the docs
  ........
    r75267 | andrew.kuchling | 2009-10-05 17:42:56 -0500 (Mon, 05 Oct 2009) | 1 line

    Backport r73983: Document the thousands separator.
  ........
    r75292 | benjamin.peterson | 2009-10-08 22:11:36 -0500 (Thu, 08 Oct 2009) | 1 line

    death to old CVS keyword
  ........
    r75300 | benjamin.peterson | 2009-10-09 16:48:14 -0500 (Fri, 09 Oct 2009) | 1 line

    fix some coding style
  ........
    r75376 | benjamin.peterson | 2009-10-11 20:26:07 -0500 (Sun, 11 Oct 2009) | 1 line

    platform we don't care about
  ........
    r75405 | neil.schemenauer | 2009-10-14 12:17:14 -0500 (Wed, 14 Oct 2009) | 4 lines

    Issue #1754094: Improve the stack depth calculation in the compiler.
    There should be no other effect than a small decrease in memory use.
    Patch by Christopher Tur Lesniewski-Laas.
  ........
    r75429 | benjamin.peterson | 2009-10-14 20:47:28 -0500 (Wed, 14 Oct 2009) | 1 line

    pep8ify if blocks
  ........
    r75430 | benjamin.peterson | 2009-10-14 20:49:37 -0500 (Wed, 14 Oct 2009) | 1 line

    use floor division and add a test that exercises the tabsize codepath
  ........
    r75431 | benjamin.peterson | 2009-10-14 20:56:25 -0500 (Wed, 14 Oct 2009) | 1 line

    change test to what I intended
  ........
    r75432 | benjamin.peterson | 2009-10-14 22:05:39 -0500 (Wed, 14 Oct 2009) | 1 line

    some cleanups
  ........
    r75433 | benjamin.peterson | 2009-10-14 22:06:55 -0500 (Wed, 14 Oct 2009) | 1 line

    make inspect.isabstract() always return a boolean; add a test for it, too #7069
  ........
    r75437 | benjamin.peterson | 2009-10-15 10:44:46 -0500 (Thu, 15 Oct 2009) | 1 line

    only clear a module's __dict__ if the module is the only one with a reference to it #7140
  ........
    r75445 | vinay.sajip | 2009-10-16 09:06:44 -0500 (Fri, 16 Oct 2009) | 1 line

    Issue #7120: logging: Removed import of multiprocessing which is causing crash in GAE.
  ........
    r75501 | antoine.pitrou | 2009-10-18 13:37:11 -0500 (Sun, 18 Oct 2009) | 3 lines

    Add a comment about unreachable code, and fix a typo
  ........
    r75551 | benjamin.peterson | 2009-10-19 22:14:10 -0500 (Mon, 19 Oct 2009) | 1 line

    use property api
  ........
    r75572 | benjamin.peterson | 2009-10-20 16:55:17 -0500 (Tue, 20 Oct 2009) | 1 line

    clarify buffer arg #7178
  ........
    r75589 | benjamin.peterson | 2009-10-21 21:26:47 -0500 (Wed, 21 Oct 2009) | 1 line

    whitespace
  ........
    r75590 | benjamin.peterson | 2009-10-21 21:36:47 -0500 (Wed, 21 Oct 2009) | 1 line

    rewrite to be nice to other implementations
  ........
    r75591 | benjamin.peterson | 2009-10-21 21:50:38 -0500 (Wed, 21 Oct 2009) | 4 lines

    rewrite for style, clarify, and comments

    Also, use the hasattr() like scheme of allowing BaseException exceptions through.
  ........
    r75657 | antoine.pitrou | 2009-10-24 07:41:27 -0500 (Sat, 24 Oct 2009) | 3 lines

    Fix compilation error in debug mode.
  ........
    r75742 | benjamin.peterson | 2009-10-26 17:51:16 -0500 (Mon, 26 Oct 2009) | 1 line

    use 'is' instead of id()
  ........
    r75868 | benjamin.peterson | 2009-10-27 15:59:18 -0500 (Tue, 27 Oct 2009) | 1 line

    test expect base classes
  ........
    r75952 | georg.brandl | 2009-10-29 15:38:32 -0500 (Thu, 29 Oct 2009) | 1 line

    Use the correct function name in docstring.
  ........
    r75953 | georg.brandl | 2009-10-29 15:39:50 -0500 (Thu, 29 Oct 2009) | 1 line

    Remove mention of the old -X command line switch.
  ........
    r75954 | georg.brandl | 2009-10-29 15:53:00 -0500 (Thu, 29 Oct 2009) | 1 line

    Use constants instead of magic integers for test result.  Do not re-run with --verbose3 for environment changing tests.
  ........
    r75955 | georg.brandl | 2009-10-29 15:54:03 -0500 (Thu, 29 Oct 2009) | 1 line

    Use a single style for all the docstrings in the math module.
  ........
    r75956 | georg.brandl | 2009-10-29 16:16:34 -0500 (Thu, 29 Oct 2009) | 1 line

    I do not think the "railroad" program mentioned is still available.
  ........
    r75957 | georg.brandl | 2009-10-29 16:44:56 -0500 (Thu, 29 Oct 2009) | 1 line

    Fix constant name.
  ........
    r76057 | benjamin.peterson | 2009-11-02 09:06:45 -0600 (Mon, 02 Nov 2009) | 1 line

    prevent a rather unlikely segfault
  ........
    r76105 | georg.brandl | 2009-11-04 01:38:12 -0600 (Wed, 04 Nov 2009) | 1 line

    #7259: show correct equivalent for operator.i* operations in docstring; fix minor issues in operator docs.
  ........
    r76139 | benjamin.peterson | 2009-11-06 19:04:38 -0600 (Fri, 06 Nov 2009) | 1 line

    spelling
  ........
    r76143 | georg.brandl | 2009-11-07 02:26:07 -0600 (Sat, 07 Nov 2009) | 1 line

    #7271: fix typo.
  ........
    r76162 | benjamin.peterson | 2009-11-08 22:10:53 -0600 (Sun, 08 Nov 2009) | 1 line

    discuss how to use -p
  ........
    r76223 | georg.brandl | 2009-11-12 02:29:46 -0600 (Thu, 12 Nov 2009) | 1 line

    Give the profile module a module directive.
  ........
................

1 files changed, 20 insertions, 13 deletions

diff --git a/Lib/tokenize.py b/Lib/tokenize.py
index fb58c6b..9d2a6bb 100644
--- a/Lib/tokenize.py
+++ b/Lib/tokenize.py
@@ -23,15 +23,15 @@ __author__ = 'Ka-Ping Yee <ping@lfw.org>'
 __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, '
                'Skip Montanaro, Raymond Hettinger, Trent Nelson, '
                'Michael Foord')
-
 import re, string, sys
 from token import *
 from codecs import lookup, BOM_UTF8
 cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
 
 import token
-__all__ = [x for x in dir(token) if x[0] != '_'] + ["COMMENT", "tokenize",
-           "detect_encoding", "NL", "untokenize", "ENCODING", "TokenInfo"]
+__all__ = [x for x in dir(token) if not x.startswith("_")]
+__all__.extend(["COMMENT", "tokenize", "detect_encoding", "NL", "untokenize",
+                "ENCODING", "TokenInfo"])
 del token
 
 COMMENT = N_TOKENS
@@ -407,7 +407,7 @@ def _tokenize(readline, encoding):
 
         if encoding is not None:
             line = line.decode(encoding)
-        lnum = lnum + 1
+        lnum += 1
         pos, max = 0, len(line)
 
         if contstr:                            # continued string
@@ -435,12 +435,17 @@ def _tokenize(readline, encoding):
             if not line: break
             column = 0
             while pos < max:                   # measure leading whitespace
-                if line[pos] == ' ': column = column + 1
-                elif line[pos] == '\t': column = (column/tabsize + 1)*tabsize
-                elif line[pos] == '\f': column = 0
-                else: break
-                pos = pos + 1
-            if pos == max: break
+                if line[pos] == ' ':
+                    column += 1
+                elif line[pos] == '\t':
+                    column = (column//tabsize + 1)*tabsize
+                elif line[pos] == '\f':
+                    column = 0
+                else:
+                    break
+                pos += 1
+            if pos == max:
+                break
 
             if line[pos] in '#\r\n':           # skip comments or blank lines
                 if line[pos] == '#':
@@ -516,13 +521,15 @@ def _tokenize(readline, encoding):
                 elif initial == '\\':                      # continued stmt
                     continued = 1
                 else:
-                    if initial in '([{': parenlev = parenlev + 1
-                    elif initial in ')]}': parenlev = parenlev - 1
+                    if initial in '([{':
+                        parenlev += 1
+                    elif initial in ')]}':
+                        parenlev -= 1
                     yield TokenInfo(OP, token, spos, epos, line)
             else:
                 yield TokenInfo(ERRORTOKEN, line[pos],
                            (lnum, pos), (lnum, pos+1), line)
-                pos = pos + 1
+                pos += 1
 
     for indent in indents[1:]:                 # pop remaining indent levels
         yield TokenInfo(DEDENT, '', (lnum, 0), (lnum, 0), '')