diff options
| author | Trent Nelson <trent.nelson@snakebite.org> | 2008-03-18 22:41:35 (GMT) |
|---|---|---|
| committer | Trent Nelson <trent.nelson@snakebite.org> | 2008-03-18 22:41:35 (GMT) |
| commit | 428de65ca99492436130165bfbaeb56d6d1daec7 (patch) | |
| tree | d6c11516a28d8ca658e1f35ac6d7cc802958e336 /Misc | |
| parent | 112367a980481d54f8c21802ee2538a3485fdd41 (diff) | |
| download | cpython-428de65ca99492436130165bfbaeb56d6d1daec7.zip cpython-428de65ca99492436130165bfbaeb56d6d1daec7.tar.gz cpython-428de65ca99492436130165bfbaeb56d6d1daec7.tar.bz2 | |
- Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been
renamed tokenize and now works with bytes rather than strings. A new
detect_encoding function has been added for determining source file encoding
according to PEP-0263. Token sequences returned by tokenize always start
with an ENCODING token which specifies the encoding used to decode the file.
This token is used to encode the output of untokenize back to bytes.
Credit goes to Michael "I'm-going-to-name-my-first-child-unittest" Foord from Resolver Systems for this work.
Diffstat (limited to 'Misc')
| -rw-r--r-- | Misc/ACKS | 2 | ||||
| -rw-r--r-- | Misc/NEWS | 7 |
2 files changed, 8 insertions, 1 deletions
@@ -752,3 +752,5 @@ Artur Zaprzala Mike Zarnstorff Siebren van der Zee Uwe Zessin +Trent Nelson +Michael Foord @@ -41,6 +41,12 @@ Library - Issue #1202: zlib.crc32 and zlib.adler32 now return an unsigned value. +- Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been + renamed tokenize and now works with bytes rather than strings. A new + detect_encoding function has been added for determining source file encoding + according to PEP-0263. Token sequences returned by tokenize always start + with an ENCODING token which specifies the encoding used to decode the file. + This token is used to encode the output of untokenize back to bytes. What's New in Python 3.0a3? =========================== @@ -175,7 +181,6 @@ Library - Issue #1578: Problems in win_getpass. - Build ----- |