diff options
| author | Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com> | 2023-06-09 16:00:59 (GMT) |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-06-09 16:00:59 (GMT) |
| commit | 45de31db9cc9be945702f3a7ca35bbb9f98476af (patch) | |
| tree | ba3b92e26d4d7dbb21f8c21d1bd8afb7679ef9e0 /Doc/library/tokenize.rst | |
| parent | a23ff66a5903ed7fcfc2419571684ea42e1b4e70 (diff) | |
| download | cpython-45de31db9cc9be945702f3a7ca35bbb9f98476af.zip cpython-45de31db9cc9be945702f3a7ca35bbb9f98476af.tar.gz cpython-45de31db9cc9be945702f3a7ca35bbb9f98476af.tar.bz2 | |
[3.10] Clarify the supported cases in the tokenize module (GH-105569) (#105575)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Diffstat (limited to 'Doc/library/tokenize.rst')
| -rw-r--r-- | Doc/library/tokenize.rst | 9 |
1 files changed, 9 insertions, 0 deletions
diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 11f569d..a903bff 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -22,6 +22,15 @@ the generic :data:`~token.OP` token type. The exact type can be determined by checking the ``exact_type`` property on the :term:`named tuple` returned from :func:`tokenize.tokenize`. + +.. warning:: + + Note that the functions in this module are only designed to parse + syntactically valid Python code (code that does not raise when parsed + using :func:`ast.parse`). The behavior of the functions in this module is + **undefined** when providing invalid Python code and it can change at any + point. + Tokenizing Input ---------------- |