diff options
| author | Thomas Kluyver <takowl@gmail.com> | 2018-06-05 17:26:39 (GMT) |
|---|---|---|
| committer | Carol Willing <carolcode@willingconsulting.com> | 2018-06-05 17:26:39 (GMT) |
| commit | c56b17bd8c7a3fd03859822246633d2c9586f8bd (patch) | |
| tree | 346fb8b3a6614679232792b3f46398b33e5f3c0e /Misc | |
| parent | c2745d2d05546d76f655ab450eb23d1af39e0b1c (diff) | |
| download | cpython-c56b17bd8c7a3fd03859822246633d2c9586f8bd.zip cpython-c56b17bd8c7a3fd03859822246633d2c9586f8bd.tar.gz cpython-c56b17bd8c7a3fd03859822246633d2c9586f8bd.tar.bz2 | |
bpo-12486: Document tokenize.generate_tokens() as public API (#6957)
* Document tokenize.generate_tokens()
* Add news file
* Add test for generate_tokens
* Document behaviour around ENCODING token
* Add generate_tokens to __all__
Diffstat (limited to 'Misc')
| -rw-r--r-- | Misc/NEWS.d/next/Library/2018-05-17-22-14-58.bpo-12486.HBeh62.rst | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/Misc/NEWS.d/next/Library/2018-05-17-22-14-58.bpo-12486.HBeh62.rst b/Misc/NEWS.d/next/Library/2018-05-17-22-14-58.bpo-12486.HBeh62.rst new file mode 100644 index 0000000..89c88e2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-05-17-22-14-58.bpo-12486.HBeh62.rst @@ -0,0 +1,2 @@ +:func:`tokenize.generate_tokens` is now documented as a public API to +tokenize unicode strings. It was previously present but undocumented. |