summaryrefslogtreecommitdiffstats
path: root/Parser/tokenizer.c
Commit message (Collapse)AuthorAgeFilesLines
* bpo-36020: Remove snprintf macro in pyerrors.h (GH-20889)Miss Islington (bot)2020-06-151-1/+1
| | | | | | | | | | | | | On Windows, GH-include "pyerrors.h" no longer defines "snprintf" and "vsnprintf" macros. PyOS_snprintf() and PyOS_vsnprintf() should be used to get portable behavior. Replace snprintf() calls with PyOS_snprintf() and replace vsnprintf() calls with PyOS_vsnprintf(). (cherry picked from commit e822e37946f27c09953bb5733acf3b07c2db690f) Co-authored-by: Victor Stinner <vstinner@python.org>
* bpo-40847: Consider a line with only a LINECONT a blank line (GH-20769)Miss Islington (bot)2020-06-111-1/+2
| | | | | | | | | | | | | A line with only a line continuation character should be considered a blank line at tokenizer level so that only a single NEWLINE token gets emitted. The old parser was working around the issue, but the new parser threw a `SyntaxError` for valid input. For example, an empty line following a line continuation character was interpreted as a `SyntaxError`. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com> (cherry picked from commit 896f4cf63f9ab93e30572d879a5719d5aa2499fb) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* Fix peg_generator compiler warnings under MSVC (GH-20405)Miss Islington (bot)2020-05-261-4/+0
| | | | | (cherry picked from commit a2bbedc8b18c001d2f9e702e6e678efbb2990daa) Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
* bpo-40593: Improve syntax errors for invalid characters in source code. ↵Serhiy Storchaka2020-05-121-9/+37
| | | | (GH-20033)
* bpo-40246: Revert reporting of invalid string prefixes (GH-19888)Lysandros Nikolaou2020-05-041-4/+0
| | | | Due to backwards compatibility concerns regarding keywords immediately followed by a string without whitespace between them (like in `bg="#d00" if clear else"#fca"`) will fail to parse, commit 41d5b94af44e34ac05d4cd57460ed104ccf96628 has to be reverted.
* bpo-40335: Correctly handle multi-line strings in tokenize error scenarios ↵Pablo Galindo2020-04-211-3/+4
| | | | | (GH-19619) Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
* bpo-40246: Report a better error message for invalid string prefixes (GH-19476)Lysandros Nikolaou2020-04-121-0/+4
|
* bpo-39882: Add _Py_FatalErrorFormat() function (GH-19157)Victor Stinner2020-03-251-1/+1
|
* bpo-39882: Py_FatalError() logs the function name (GH-18819)Victor Stinner2020-03-061-3/+5
| | | | | | | | | | | | The Py_FatalError() function is replaced with a macro which logs automatically the name of the current function, unless the Py_LIMITED_API macro is defined. Changes: * Add _Py_FatalErrorFunc() function. * Remove the function name from the message of Py_FatalError() calls which included the function name. * Update tests.
* closes bpo-39721: Fix constness of members of tok_state struct. (GH-18600)Andy Lester2020-02-281-20/+30
| | | | | | | | | | | | | | | | | | | | | The function PyTokenizer_FromUTF8 from Parser/tokenizer.c had a comment: /* XXX: constify members. */ This patch addresses that. In the tok_state struct: * end and start were non-const but could be made const * str and input were const but should have been non-const Changes to support this include: * decode_str() now returns a char * since it is allocated. * PyTokenizer_FromString() and PyTokenizer_FromUTF8() each creates a new char * for an allocate string instead of reusing the input const char *. * PyTokenizer_Get() and tok_get() now take const char ** arguments. * Various local vars are const or non-const accordingly. I was able to remove five casts that cast away constness.
* bpo-39219: Fix SyntaxError attributes in the tokenizer. (GH-17828)Serhiy Storchaka2020-02-121-4/+32
| | | | * Always set the text attribute. * Correct the offset attribute for non-ascii sources.
* bpo-39500: Document PyUnicode_IsIdentifier() function (GH-18397)Victor Stinner2020-02-111-1/+2
| | | | PyUnicode_IsIdentifier() does not call Py_FatalError() anymore if the string is not ready.
* bpo-39209: Manage correctly multi-line tokens in interactive mode (GH-17860)Pablo Galindo2020-01-061-0/+2
|
* bpo-38673: dont switch to ps2 if the line starts with comment or whitespace ↵Batuhan Taşkaya2019-12-091-0/+6
| | | | | (GH-17421) https://bugs.python.org/issue38673
* Indent code inside if block. (GH-15284)Hansraj Das2019-08-151-1/+1
| | | Without indendation, seems like strcpy line is parallel to `if` condition.
* Fix `SyntaxError` indicator printing too many spaces for multi-line strings ↵Anthony Sottile2019-07-291-0/+2
| | | | (GH-14433)
* bpo-36878: Only allow text after `# type: ignore` if first character ASCII ↵Michael J. Sullivan2019-05-221-2/+3
| | | | | | | | | | | (GH-13504) This disallows things like `# type: ignoreé`, which seems wrong. Also switch to using Py_ISALNUM for the alnum check, for consistency with other code (and maybe correctness re: locale issues?). https://bugs.python.org/issue36878
* bpo-36878: Track extra text added to 'type: ignore' in the AST (GH-13479)Michael J. Sullivan2019-05-221-2/+6
| | | | | GH-13238 made extra text after a # type: ignore accepted by the parser. This finishes the job and actually plumbs the extra text through the parser and makes it available in the AST.
* bpo-2180: Treat line continuation at EOF as a `SyntaxError` (GH-13401)Anthony Sottile2019-05-181-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This makes the parser consistent with the tokenize module (already the case in `pypy`). sample ------ ```python x = 5\ ``` before ------ ```console $ python3 t.py $ python3 -mtokenize t.py t.py:2:0: error: EOF in multi-line statement ``` after ----- ```console $ ./python t.py File "t.py", line 3 x = 5\ ^ SyntaxError: unexpected EOF while parsing $ ./python -m tokenize t.py t.py:2:0: error: EOF in multi-line statement ``` https://bugs.python.org/issue2180
* bpo-36878: Allow extra text after `# type: ignore` comments (GH-13238)Michael J. Sullivan2019-05-111-8/+5
| | | | | | | In the parser, when using the type_comments=True option, recognize a TYPE_IGNORE as anything containing `# type: ignore` followed by a non-alphanumeric character. This is to allow ignores such as `# type: ignore[E1000]`.
* bpo-36623: Clean parser headers and include files (GH-12253)Pablo Galindo2019-04-131-1/+0
| | | After the removal of pgen, multiple header and function prototypes that lack implementation or are unused are still lying around.
* bpo-36459: Fix a possible double PyMem_FREE() due to tokenizer.c's ↵Zackery Spytz2019-03-281-1/+0
| | | | | | tok_nextc() (12601) Remove the PyMem_FREE() call added in cb90c89. The buffer will be freed when PyTokenizer_Free() is called on the tokenizer state.
* bpo-36367: Free buffer if realloc fails in tokenize.c (GH-12442)Pablo Galindo2019-03-191-2/+8
|
* bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)Guido van Rossum2019-03-071-0/+79
| | | | | | | This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.) https://bugs.python.org/issue35975
* bpo-35808: Retire pgen and use pgen2 to generate the parser (GH-11814)Pablo Galindo2019-03-011-56/+0
| | | | | Pgen is the oldest piece of technology in the CPython repository, building it requires various #if[n]def PGEN hacks in other parts of the code and it also depends more and more on CPython internals. This commit removes the old pgen C code and replaces it for a new version implemented in pure Python. This is a modified and adapted version of lib2to3/pgen2 that can generate grammar files compatibles with the current parser. This commit also eliminates all the #ifdef and code branches related to pgen, simplifying the code and making it more maintainable. The regen-grammar step now uses $(PYTHON_FOR_REGEN) that can be any version of the interpreter, so the new pgen code maintains compatibility with older versions of the interpreter (this also allows regenerating the grammar with the current CI solution that uses Python3.5). The new pgen Python module also makes use of the Grammar/Tokens file that holds the token specification, so is always kept in sync and avoids having to maintain duplicate token definitions.
* bpo-35766: Merge typed_ast back into CPython (GH-11645)Guido van Rossum2019-01-311-1/+56
|
* bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)Anthony Sottile2019-01-131-0/+7
|
* bpo-30455: Generate all token related code and docs from Grammar/Tokens. ↵Serhiy Storchaka2018-12-221-237/+0
| | | | | | | | | | | | | | | | | | | (GH-10370) "Include/token.h", "Lib/token.py" (containing now some data moved from "Lib/tokenize.py") and new files "Parser/token.c" (containing the code moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by "Tools/scripts/generate_token.py". The script overwrites files only if needed and can be used on the read-only sources tree. "Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py" instead of been executable itself. Added new make targets "regen-token" and "regen-symbol" which are now dependencies of "regen-all". The documentation contains now strings for operators and punctuation tokens.
* bpo-33306: Improve SyntaxError messages for unbalanced parentheses. (GH-6516)Serhiy Storchaka2018-12-171-0/+32
|
* bpo-35436: Add missing PyErr_NoMemory() calls and other minor bug fixes. ↵Zackery Spytz2018-12-071-0/+5
| | | | | | (GH-11015) Set MemoryError when appropriate, add missing failure checks, and fix some potential leaks.
* Remove unneeded PyUnicode_READY() in tokenizer.c (GH-9114)Zackery Spytz2018-09-101-1/+1
|
* Fix Windows compiler warning in tokenize.c (GH-8359)Victor Stinner2018-07-211-1/+1
| | | | | | Fix the following warning on Windows: parser\tokenizer.c(1297): warning C4244: 'function': conversion from '__int64' to 'int', possible loss of data.
* bpo-33305: Improve SyntaxError for invalid numerical literals. (GH-6517)Serhiy Storchaka2018-07-091-13/+52
|
* tokenizer: Remove unused tabs options (#4422)Victor Stinner2017-11-171-31/+11
| | | | | | | | | | Remove the following fields from tok_state structure which are now used unused: * altwarning: "Issue warning if alternate tabs don't match" * alterror: "Issue error if alternate tabs don't match" * alttabsize: "Alternate tab spacing" Replace alttabsize variable with ALTTABSIZE define.
* bpo-30406: Make async and await proper keywords (#1669)Jelle Zijlstra2017-10-061-63/+0
| | | Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
* bpo-25324: add missing comma in Parser/tokenizer.c (GH-1910)Albert-Jan Nijburg2017-06-011-1/+1
|
* bpo-25324: copy tok_name before changing it (#1608)Albert-Jan Nijburg2017-05-311-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | * add test to check if were modifying token * copy list so import tokenize doesnt have side effects on token * shorten line * add tokenize tokens to token.h to get them to show up in token * move ERRORTOKEN back to its previous location, and fix nitpick * copy comments from token.h automatically * fix whitespace and make more pythonic * change to fix comments from @haypo * update token.rst and Misc/NEWS * change wording * some more wording changes
* Issue #28489: Merge from 3.6Berker Peksag2017-02-051-1/+1
|\
| * Issue #28489: Fix comment in tokenizer.cBerker Peksag2017-02-051-1/+1
| | | | | | | | Patch by Ryan Gonzalez.
* | Use _PyObject_CallNoArg()Victor Stinner2016-12-061-3/+3
|/ | | | | | | Replace: PyObject_CallObject(callable, NULL) with: _PyObject_CallNoArg(callable)
* Replaced outdated macros _PyUnicode_AsString and _PyUnicode_AsStringAndSizeSerhiy Storchaka2016-11-201-1/+1
| | | | with PyUnicode_AsUTF8 and PyUnicode_AsUTF8AndSize.
* merge 3.5 (#24022)Benjamin Peterson2016-09-191-1/+1
|\
| * merge 3.4Benjamin Peterson2016-09-191-1/+1
| |\
| | * properly handle the single null-byte file (closes #24022)Benjamin Peterson2016-09-191-1/+1
| | |
* | | merge 3.5 (#27981)Benjamin Peterson2016-09-131-15/+17
|\ \ \ | |/ /
| * | restructure fp_setreadl so as to avoid refleaks (closes #27981)Benjamin Peterson2016-09-131-15/+17
| | |
* | | Issue #26331: Implement the parsing part of PEP 515.Brett Cannon2016-09-091-68/+162
| | | | | | | | | | | | Thanks to Georg Brandl for the patch.
* | | Skip unused value in tokenizer codeChristian Heimes2016-09-081-1/+1
| | | | | | | | | | | | | | | | | | | | | In the case of an escape character, c is never read. tok_next() is used to advance the pointer. CID 1225097
* | | Issue #22570: Renamed Py_SETREF to Py_XSETREF.Serhiy Storchaka2016-04-061-1/+1
|\ \ \ | |/ /
| * | Issue #22570: Renamed Py_SETREF to Py_XSETREF.Serhiy Storchaka2016-04-061-1/+1
| | |
generated by cgit v0.12 at 2025-03-11 16:50:42 (GMT)