summaryrefslogtreecommitdiffstats
path: root/Doc/library/tokenize.rst
diff options
context:
space:
mode:
authorGeorg Brandl <georg@python.org>2007-09-01 13:51:09 (GMT)
committerGeorg Brandl <georg@python.org>2007-09-01 13:51:09 (GMT)
commit55ac8f0f26efdbbcb5cc197f9369d23d50bee908 (patch)
treea0d5b7128c055d8c767652dc3948c3404be06396 /Doc/library/tokenize.rst
parent1617457cff847fed9fadb01f1acf6ba8bb621726 (diff)
downloadcpython-55ac8f0f26efdbbcb5cc197f9369d23d50bee908.zip
cpython-55ac8f0f26efdbbcb5cc197f9369d23d50bee908.tar.gz
cpython-55ac8f0f26efdbbcb5cc197f9369d23d50bee908.tar.bz2
Get rid of the remaining versionadded/versionchanged directives.
Diffstat (limited to 'Doc/library/tokenize.rst')
-rw-r--r--Doc/library/tokenize.rst8
1 files changed, 1 insertions, 7 deletions
diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst
index 61f2c4d..61b8497 100644
--- a/Doc/library/tokenize.rst
+++ b/Doc/library/tokenize.rst
@@ -31,11 +31,9 @@ The primary entry point is a generator:
token was found. The line passed is the *logical* line; continuation lines are
included.
- .. versionadded:: 2.2
An older entry point is retained for backward compatibility:
-
.. function:: tokenize(readline[, tokeneater])
The :func:`tokenize` function accepts two parameters: one representing the input
@@ -47,18 +45,15 @@ An older entry point is retained for backward compatibility:
line of input as a string. Alternately, *readline* may be a callable object that
signals completion by raising :exc:`StopIteration`.
- .. versionchanged:: 2.5
- Added :exc:`StopIteration` support.
-
The second parameter, *tokeneater*, must also be a callable object. It is
called once for each token, with five arguments, corresponding to the tuples
generated by :func:`generate_tokens`.
+
All constants from the :mod:`token` module are also exported from
:mod:`tokenize`, as are two additional token type values that might be passed to
the *tokeneater* function by :func:`tokenize`:
-
.. data:: COMMENT
Token value used to indicate a comment.
@@ -87,7 +82,6 @@ back the modified script.
type and token string as the spacing between tokens (column positions) may
change.
- .. versionadded:: 2.5
Example of a script re-writer that transforms float literals into Decimal
objects::