summaryrefslogtreecommitdiffstats
path: root/Doc/library/tokenize.rst
diff options
context:
space:
mode:
authorMeador Inge <meadori@gmail.com>2012-01-19 06:22:22 (GMT)
committerMeador Inge <meadori@gmail.com>2012-01-19 06:22:22 (GMT)
commit972cfb9169bdf7fa6c9758a5c49659429c84b5a8 (patch)
treecc8ef7b23439501cf4504f93492e502dab7f6195 /Doc/library/tokenize.rst
parent1b468af7be7a2f0af7730c64402a38cc11413f51 (diff)
downloadcpython-972cfb9169bdf7fa6c9758a5c49659429c84b5a8.zip
cpython-972cfb9169bdf7fa6c9758a5c49659429c84b5a8.tar.gz
cpython-972cfb9169bdf7fa6c9758a5c49659429c84b5a8.tar.bz2
Issue #2134: Clarify token.OP handling rationale in tokenize documentation.
Diffstat (limited to 'Doc/library/tokenize.rst')
-rw-r--r--Doc/library/tokenize.rst6
1 files changed, 6 insertions, 0 deletions
diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst
index 577d7cc..70919ca 100644
--- a/Doc/library/tokenize.rst
+++ b/Doc/library/tokenize.rst
@@ -15,6 +15,12 @@ implemented in Python. The scanner in this module returns comments as tokens
as well, making it useful for implementing "pretty-printers," including
colorizers for on-screen displays.
+To simplify token stream handling, all :ref:`operators` and :ref:`delimiters`
+tokens are returned using the generic :data:`token.OP` token type. The exact
+type can be determined by checking the token ``string`` field on the
+:term:`named tuple` returned from :func:`tokenize.tokenize` for the character
+sequence that identifies a specific operator token.
+
The primary entry point is a :term:`generator`:
.. function:: tokenize(readline)