summaryrefslogtreecommitdiffstats
path: root/Python/Python-tokenize.c
Commit message (Collapse)AuthorAgeFilesLines
* gh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-104205)Eric Snow2023-05-051-0/+1
| | | Here we are doing no more than adding the value for Py_mod_multiple_interpreters and using it for stdlib modules. We will start checking for it in gh-104206 (once PyInterpreterState.ceval.own_gil is added in gh-104204).
* gh-102856: Initial implementation of PEP 701 (#102855)Pablo Galindo Salgado2023-04-191-2/+2
| | | | | | Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Batuhan Taskaya <isidentical@gmail.com> Co-authored-by: Marta Gómez Macías <mgmacias@google.com> Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
* gh-97973: Return all necessary information from the tokenizer (GH-97984)Lysandros Nikolaou2022-10-061-9/+8
| | | | | Right now, the tokenizer only returns type and two pointers to the start and end of the token. This PR modifies the tokenizer to return the type and set all of the necessary information, so that the parser does not have to this.
* gh-90928: Statically Initialize the Keywords Tuple in Clinic-Generated Code ↵Eric Snow2022-08-111-0/+1
| | | | | | | | | | | | | | | | (gh-95860) We only statically initialize for core code and builtin modules. Extension modules still create the tuple at runtime. We'll solve that part of interpreter isolation separately. This change includes generated code. The non-generated changes are in: * Tools/clinic/clinic.py * Python/getargs.c * Include/cpython/modsupport.h * Makefile.pre.in (re-generate global strings after running clinic) * very minor tweaks to Modules/_codecsmodule.c and Python/Python-tokenize.c All other changes are generated code (clinic, global strings).
* bpo-46613: Add PyType_GetModuleByDef to the public API (GH-31081)Petr Viktorin2022-02-111-1/+1
| | | | | * Make PyType_GetModuleByDef public (remove underscore) Co-authored-by: Victor Stinner <vstinner@python.org>
* bpo-45434: Mark the PyTokenizer C API as private (GH-28924)Victor Stinner2021-10-131-3/+3
| | | | | | | | | | | | | | Rename PyTokenize functions to mark them as private: * PyTokenizer_FindEncodingFilename() => _PyTokenizer_FindEncodingFilename() * PyTokenizer_FromString() => _PyTokenizer_FromString() * PyTokenizer_FromFile() => _PyTokenizer_FromFile() * PyTokenizer_FromUTF8() => _PyTokenizer_FromUTF8() * PyTokenizer_Free() => _PyTokenizer_Free() * PyTokenizer_Get() => _PyTokenizer_Get() Remove the unused PyTokenizer_FindEncoding() function. import.c: remove unused #include "errcode.h".
* Remove trailing spaces. (GH-28706)Serhiy Storchaka2021-10-031-1/+1
|
* Format the Python-tokenize module and fix exit path (GH-27935)Pablo Galindo Salgado2021-08-251-47/+46
|
* Add tests for the C tokenizer and expose it as a private module (GH-27924)Pablo Galindo Salgado2021-08-241-0/+195