diff options
| author | Terry Jan Reedy <tjreedy@udel.edu> | 2014-04-14 20:17:09 (GMT) |
|---|---|---|
| committer | Terry Jan Reedy <tjreedy@udel.edu> | 2014-04-14 20:17:09 (GMT) |
| commit | f7568e0a8a7902b303f442ab6544bdd009cfb3c9 (patch) | |
| tree | 8dba4f0b99607e9b39e7b3f8ac477923bca00558 | |
| parent | e03d3fdd204f6e702c9fb3129a50b290294e2a00 (diff) | |
| download | cpython-f7568e0a8a7902b303f442ab6544bdd009cfb3c9.zip cpython-f7568e0a8a7902b303f442ab6544bdd009cfb3c9.tar.gz cpython-f7568e0a8a7902b303f442ab6544bdd009cfb3c9.tar.bz2 | |
Closes #20956: 2.7 tokenize does not produce named tuples. Patch by Sam Kimbrel.
| -rw-r--r-- | Doc/library/tokenize.rst | 7 | ||||
| -rw-r--r-- | Misc/ACKS | 1 |
2 files changed, 5 insertions, 3 deletions
diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index d037452..0fc0d7e 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -17,9 +17,10 @@ for on-screen displays. To simplify token stream handling, all :ref:`operators` and :ref:`delimiters` tokens are returned using the generic :data:`token.OP` token type. The exact -type can be determined by checking the token ``string`` field on the -:term:`named tuple` returned from :func:`tokenize.tokenize` for the character -sequence that identifies a specific operator token. +type can be determined by checking the second field (containing the actual +token string matched) of the tuple returned from +:func:`tokenize.generate_tokens` for the character sequence that identifies a +specific operator token. The primary entry point is a :term:`generator`: @@ -670,6 +670,7 @@ Mads Kiilerich Jason Killen Jan Kim Taek Joo Kim +Sam Kimbrel W. Trevor King Paul Kippes Steve Kirsch |