diff options
Diffstat (limited to 'Doc/library/tokenize.rst')
| -rw-r--r-- | Doc/library/tokenize.rst | 8 |
1 files changed, 1 insertions, 7 deletions
diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 61f2c4d..61b8497 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -31,11 +31,9 @@ The primary entry point is a generator: token was found. The line passed is the *logical* line; continuation lines are included. - .. versionadded:: 2.2 An older entry point is retained for backward compatibility: - .. function:: tokenize(readline[, tokeneater]) The :func:`tokenize` function accepts two parameters: one representing the input @@ -47,18 +45,15 @@ An older entry point is retained for backward compatibility: line of input as a string. Alternately, *readline* may be a callable object that signals completion by raising :exc:`StopIteration`. - .. versionchanged:: 2.5 - Added :exc:`StopIteration` support. - The second parameter, *tokeneater*, must also be a callable object. It is called once for each token, with five arguments, corresponding to the tuples generated by :func:`generate_tokens`. + All constants from the :mod:`token` module are also exported from :mod:`tokenize`, as are two additional token type values that might be passed to the *tokeneater* function by :func:`tokenize`: - .. data:: COMMENT Token value used to indicate a comment. @@ -87,7 +82,6 @@ back the modified script. type and token string as the spacing between tokens (column positions) may change. - .. versionadded:: 2.5 Example of a script re-writer that transforms float literals into Decimal objects:: |