summaryrefslogtreecommitdiffstats
path: root/Parser/tokenizer.c
Commit message (Collapse)AuthorAgeFilesLines
* [3.12] gh-107967: Fix infinite recursion on invalid escape sequence warning ↵Miss Islington (bot)2023-08-151-0/+3
| | | | | | | | (GH-107968) (#107970) gh-107967: Fix infinite recursion on invalid escape sequence warning (GH-107968) (cherry picked from commit d66bc9e8a7a8d6774d912a4b9d151885c4d8de1d) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* [3.12] gh-106989: Remove tok report warnings (GH-106993) (#107013)Miss Islington (bot)2023-07-221-5/+0
| | | | Co-authored-by: Menelaos Kotoglou <contact@menelaoskotoglou.com>
* [3.12] gh-105938: Emit a SyntaxWarning for escaped braces in an f-string ↵Miss Islington (bot)2023-06-201-3/+3
| | | | | | | (GH-105939) (#105941) (cherry picked from commit 6586cee27f32f0354fe4e77c7b8c6e399329b5e2) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* [3.12] gh-105820: Fix tok_mode expression buffer in file & readline ↵Miss Islington (bot)2023-06-151-3/+8
| | | | | | | tokenizer (GH-105828) (#105832) (cherry picked from commit d382ad49157b3802fc5619f68d96810def517869) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* [3.12] gh-105718: Fix buffer allocation in tokenizer with readline ↵Miss Islington (bot)2023-06-131-4/+0
| | | | | | (GH-105728) (#105729) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* [3.12] gh-105549: Tokenize separately NUMBER and NAME tokens and allow ↵Miss Islington (bot)2023-06-091-3/+10
| | | | | | 0-prefixed literals (GH-105555) (#105602) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* [3.12] gh-105259: Ensure we don't show newline characters for trailing ↵Miss Islington (bot)2023-06-061-0/+7
| | | | NEWLINE tokens (GH-105364) (#105367)
* [3.12] gh-105069: Add a readline-like callable to the tokenizer to consume ↵Miss Islington (bot)2023-05-311-3/+133
| | | | | | | | input iteratively (GH-105070) (#105119) gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (GH-105070) (cherry picked from commit 9216e69a87d16d871625721ed5a8aa302511f367) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* [3.12] gh-105042: Disable unmatched parens syntax error in python tokenize ↵Miss Islington (bot)2023-05-311-32/+33
| | | | | | | | (GH-105061) (#105120) gh-105042: Disable unmatched parens syntax error in python tokenize (GH-105061) (cherry picked from commit 70f315c2d6de87b0514ce16cc00a91a5b60a6098) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* [3.12] gh-105017: Include CRLF lines in strings and column numbers ↵Miss Islington (bot)2023-05-281-12/+26
| | | | | | | | | (GH-105030) (#105041) gh-105017: Include CRLF lines in strings and column numbers (GH-105030) (cherry picked from commit 96fff35325e519cc76ffacf22e57e4c393d4446f) Co-authored-by: Marta Gómez Macías <mgmacias@google.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
* [3.12] gh-105017: Fix including additional NL token when using CRLF ↵Miss Islington (bot)2023-05-271-1/+1
| | | | | | | (GH-105022) (#105023) Co-authored-by: Marta Gómez Macías <mgmacias@google.com> Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* [3.12] gh-104866: Tokenize should emit NEWLINE after exiting block with ↵Miss Islington (bot)2023-05-241-3/+6
| | | | | | | | | comment (GH-104870) (#104872) gh-104866: Tokenize should emit NEWLINE after exiting block with comment (GH-104870) (cherry picked from commit c90a862cdcf55dc1753c6466e5fa4a467a13ae24) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* gh-102856: Allow comments inside multi-line f-string expresions (#104006)Cristián Maureira-Fredes2023-05-221-4/+0
|
* gh-98836: Extend PyUnicode_FromFormat() (GH-98838)Serhiy Storchaka2023-05-211-8/+3
| | | | | | | | | * Support for conversion specifiers o (octal) and X (uppercase hexadecimal). * Support for length modifiers j (intmax_t) and t (ptrdiff_t). * Length modifiers are now applied to all integer conversions. * Support for wchar_t C strings (%ls and %lV). * Support for variable width and precision (*). * Support for flag - (left alignment).
* gh-102856: Python tokenizer implementation for PEP 701 (#104323)Marta Gómez Macías2023-05-211-2/+55
| | | | | | | | | | | This commit replaces the Python implementation of the tokenize module with an implementation that reuses the real C tokenizer via a private extension module. The tokenize module now implements a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward compatibility. As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation. Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
* gh-104658: Fix location of unclosed quote error for multiline f-strings ↵Pablo Galindo Salgado2023-05-201-2/+5
| | | | (#104660)
* Trim trailing whitespace and test on CI (#104275)Hugo van Kemenade2023-05-081-1/+1
| | | | Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* gh-104169: Ensure the tokenizer doesn't overwrite previous errors (#104170)Pablo Galindo Salgado2023-05-041-0/+6
|
* gh-97556: Raise null bytes syntax error upon null in multiline string ↵Lysandros Nikolaou2023-05-041-1/+8
| | | | (GH-104136)
* gh-104016: Fixed off by 1 error in f string tokenizer (#104047)jx1242023-05-011-2/+5
| | | | | | Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com> Co-authored-by: Ken Jin <kenjin@python.org> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
* gh-103824: fix use-after-free error in Parser/tokenizer.c (#103993)chgnrdv2023-05-011-0/+4
|
* gh-103656: Transfer f-string buffers to parser to avoid use-after-free ↵Lysandros Nikolaou2023-04-271-2/+32
| | | | | (GH-103896) Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
* gh-103718: Correctly set f-string buffers in all cases (GH-103815)Lysandros Nikolaou2023-04-251-8/+6
| | | | | | Turns out we always need to remember/restore fstring buffers in all of the stack of tokenizer modes, cause they might change to `TOK_REGULAR_MODE` and have newlines inside the braces (which is when we need to reallocate the buffer and restore the fstring ones).
* GH-103727: Avoid advancing tokenizer too far in f-string mode (GH-103775)Lysandros Nikolaou2023-04-241-8/+10
|
* GH-103718: Correctly cache and restore f-string buffers when needed (GH-103719)Lysandros Nikolaou2023-04-231-11/+27
|
* gh-102856: Clean some of the PEP 701 tokenizer implementation (#103634)Pablo Galindo Salgado2023-04-191-71/+65
|
* gh-102856: Initial implementation of PEP 701 (#102855)Pablo Galindo Salgado2023-04-191-6/+483
| | | | | | Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Batuhan Taskaya <isidentical@gmail.com> Co-authored-by: Marta Gómez Macías <mgmacias@google.com> Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
* gh-99891: Fix infinite recursion in the tokenizer when showing warnings ↵Pablo Galindo Salgado2022-11-301-0/+8
| | | | | (GH-99893) Automerge-Triggered-By: GH:pablogsal
* gh-99581: Fix a buffer overflow in the tokenizer when copying lines that ↵Pablo Galindo Salgado2022-11-201-1/+6
| | | | fill the available buffer (#99605)
* gh-99300: Use Py_NewRef() in Parser/ directory (#99330)Victor Stinner2022-11-101-2/+1
| | | | Replace Py_INCREF() with Py_NewRef() in C files of the Parser/ directory and in the PEG generator.
* gh-97997: Add col_offset field to tokenizer and use that for AST nodes (#98000)Lysandros Nikolaou2022-10-071-11/+41
|
* gh-97973: Return all necessary information from the tokenizer (GH-97984)Lysandros Nikolaou2022-10-061-103/+117
| | | | | Right now, the tokenizer only returns type and two pointers to the start and end of the token. This PR modifies the tokenizer to return the type and set all of the necessary information, so that the parser does not have to this.
* gh-96670: Raise SyntaxError when parsing NULL bytes (#97594)Pablo Galindo Salgado2022-09-271-4/+15
|
* gh-96678: Fix UB of null pointer arithmetic (GH-96782)Matthias Görgens2022-09-131-1/+1
| | | Automerge-Triggered-By: GH:pablogsal
* gh-96268: Fix loading invalid UTF-8 (#96270)Michael Droettboom2022-09-071-13/+45
| | | | | This makes tokenizer.c:valid_utf8 match stringlib/codecs.h:decode_utf8. It also fixes an off-by-one error introduced in 3.10 for the line number when the tokenizer reports bad UTF8.
* gh-96611: Fix error message for invalid UTF-8 in mid-multiline string (#96623)Michael Droettboom2022-09-061-0/+2
|
* gh-94360: Fix a tokenizer crash when reading encoded files with syntax ↵Pablo Galindo Salgado2022-07-051-1/+9
| | | | | | | | | | | errors from stdin (#94386) * gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin Signed-off-by: Pablo Galindo <pablogsal@gmail.com> * nitty nit Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* gh-93741: Add private C API _PyImport_GetModuleAttrString() (GH-93742)Serhiy Storchaka2022-06-141-5/+5
| | | | | | It combines PyImport_ImportModule() and PyObject_GetAttrString() and saves 4-6 lines of code on every use. Add also _PyImport_GetModuleAttr() which takes Python strings as arguments.
* GH-93207: Remove HAVE_STDARG_PROTOTYPES configure check for stdarg.h (#93215)Kumar Aditya2022-05-271-12/+0
|
* gh-93103: Parser uses PyConfig.parser_debug instead of Py_DebugFlag (#93106)Victor Stinner2022-05-241-1/+4
| | | | | | | * Replace deprecated Py_DebugFlag with PyConfig.parser_debug in the parser. * Add Parser.debug member. * Add tok_state.debug member. * Py_FrozenMain(): Replace Py_VerboseFlag with PyConfig.verbose.
* gh-92651: Remove the Include/token.h header file (#92652)Victor Stinner2022-05-111-3/+3
| | | | | | | | | | | | | | | Remove the token.h header file. There was never any public tokenizer C API. The token.h header file was only designed to be used by Python internals. Move Include/token.h to Include/internal/pycore_token.h. Including this header file now requires that the Py_BUILD_CORE macro is defined. It no longer checks for the Py_LIMITED_API macro. Rename functions: * PyToken_OneChar() => _PyToken_OneChar() * PyToken_TwoChars() => _PyToken_TwoChars() * PyToken_ThreeChars() => _PyToken_ThreeChars()
* gh-87999: Change warning type for numeric literal followed by keyword (GH-91980)Serhiy Storchaka2022-04-271-4/+6
| | | | | The warning emitted by the Python parser for a numeric literal immediately followed by keyword has been changed from deprecation warning to syntax warning.
* bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266)Christian Heimes2022-04-021-1/+1
|
* bpo-47126: Update to canonical PEP URLs specified by PEP 676 (GH-32124)Hugo van Kemenade2022-03-301-1/+1
|
* bpo-46315: Use fopencookie() to avoid dup() in ↵Christian Heimes2022-03-221-6/+34
| | | | | _PyTokenizer_FindEncodingFilename (GH-32033) WASI does not have dup() and Emscripten's emulation is slow.
* bpo-46920: Remove code that has explainers why it was disabled (GH-31813)Oleg Iarygin2022-03-141-24/+0
|
* bpo-46820: Fix a SyntaxError in a numeric literal followed by "not in" ↵Serhiy Storchaka2022-02-221-0/+3
| | | | | | | (GH-31479) Fix parsing a numeric literal immediately (without spaces) followed by "not in" keywords, like in "1not in x". Now the parser only emits a warning, not a syntax error.
* bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized ↵Eric Snow2022-02-081-9/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | global objects. (gh-30928) We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code. It is still used in a number of non-builtin stdlib modules. The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime. A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings). https://bugs.python.org/issue46541#msg411799 explains the rationale for this change. The core of the change is in: * (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros * Include/internal/pycore_runtime_init.h - added the static initializers for the global strings * Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState * Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings. That check is added to the PR CI config. The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()). This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *. The following are not changed (yet): * stop using _Py_IDENTIFIER() in the stdlib modules * (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API * (maybe) intern the strings during runtime init https://bugs.python.org/issue46541
* bpo-46521: Fix codeop to use a new partial-input mode of the parser (GH-31010)Pablo Galindo Salgado2022-02-081-10/+16
|
* bpo-14916: use specified tokenizer fd for file input (GH-31006)Paul m. p. P2022-02-011-1/+1
| | | | | | | | | | @pablogsal, sorry i failed to rebase to main, so i recreated https://github.com/python/cpython/pull/22190#issuecomment-1024633392 > PyRun_InteractiveOne\*() functions allow to explicitily set fd instead of stdin. but stdin was hardcoded in readline call. > This patch does not fix target file for prompt unlike original bpo one : prompt fd is unrelated to tokenizer source which could be read only. It is more of a bugfix regarding the docs : actual documentation say "prompt the user" so one would expect prompt to go on stdout not a file for both PyRun_InteractiveOne\*() and PyRun_InteractiveLoop\*(). Automerge-Triggered-By: GH:pablogsal
generated by cgit v0.12 at 2025-09-14 12:31:57 (GMT)