diff options
| author | Georg Brandl <georg@python.org> | 2008-06-08 08:54:40 (GMT) |
|---|---|---|
| committer | Georg Brandl <georg@python.org> | 2008-06-08 08:54:40 (GMT) |
| commit | ebd662d6f20816090d42aaa4ec8641769e4a2e53 (patch) | |
| tree | 2bc05cad710a0b7267f48c3a6e38c06da64b7683 /Doc/library/tokenize.rst | |
| parent | 2fe3ef8750e7f06ebf250bb2264f71d5f96e2f9b (diff) | |
| download | cpython-ebd662d6f20816090d42aaa4ec8641769e4a2e53.zip cpython-ebd662d6f20816090d42aaa4ec8641769e4a2e53.tar.gz cpython-ebd662d6f20816090d42aaa4ec8641769e4a2e53.tar.bz2 | |
#3028: tokenize passes the physical line.
Diffstat (limited to 'Doc/library/tokenize.rst')
| -rw-r--r-- | Doc/library/tokenize.rst | 19 |
1 files changed, 9 insertions, 10 deletions
diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 583e924..efe9eb9 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -15,21 +15,20 @@ for on-screen displays. The primary entry point is a :term:`generator`: - .. function:: generate_tokens(readline) - The :func:`generate_tokens` generator requires one argument, *readline*, which - must be a callable object which provides the same interface as the + The :func:`generate_tokens` generator requires one argument, *readline*, + which must be a callable object which provides the same interface as the :meth:`readline` method of built-in file objects (see section - :ref:`bltin-file-objects`). Each call to the function should return one line of - input as a string. + :ref:`bltin-file-objects`). Each call to the function should return one line + of input as a string. The generator produces 5-tuples with these members: the token type; the token - string; a 2-tuple ``(srow, scol)`` of ints specifying the row and column where - the token begins in the source; a 2-tuple ``(erow, ecol)`` of ints specifying - the row and column where the token ends in the source; and the line on which the - token was found. The line passed is the *logical* line; continuation lines are - included. + string; a 2-tuple ``(srow, scol)`` of ints specifying the row and column + where the token begins in the source; a 2-tuple ``(erow, ecol)`` of ints + specifying the row and column where the token ends in the source; and the + line on which the token was found. The line passed is the *physical* line, + that is, continuation lines are not handled specially. .. versionadded:: 2.2 |