From ebd662d6f20816090d42aaa4ec8641769e4a2e53 Mon Sep 17 00:00:00 2001 From: Georg Brandl Date: Sun, 8 Jun 2008 08:54:40 +0000 Subject: #3028: tokenize passes the physical line. --- Doc/library/tokenize.rst | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 583e924..efe9eb9 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -15,21 +15,20 @@ for on-screen displays. The primary entry point is a :term:`generator`: - .. function:: generate_tokens(readline) - The :func:`generate_tokens` generator requires one argument, *readline*, which - must be a callable object which provides the same interface as the + The :func:`generate_tokens` generator requires one argument, *readline*, + which must be a callable object which provides the same interface as the :meth:`readline` method of built-in file objects (see section - :ref:`bltin-file-objects`). Each call to the function should return one line of - input as a string. + :ref:`bltin-file-objects`). Each call to the function should return one line + of input as a string. The generator produces 5-tuples with these members: the token type; the token - string; a 2-tuple ``(srow, scol)`` of ints specifying the row and column where - the token begins in the source; a 2-tuple ``(erow, ecol)`` of ints specifying - the row and column where the token ends in the source; and the line on which the - token was found. The line passed is the *logical* line; continuation lines are - included. + string; a 2-tuple ``(srow, scol)`` of ints specifying the row and column + where the token begins in the source; a 2-tuple ``(erow, ecol)`` of ints + specifying the row and column where the token ends in the source; and the + line on which the token was found. The line passed is the *physical* line, + that is, continuation lines are not handled specially. .. versionadded:: 2.2 -- cgit v0.12