summaryrefslogtreecommitdiffstats
path: root/Parser/tokenizer.c
Commit message (Collapse)AuthorAgeFilesLines
* bpo-39219: Fix SyntaxError attributes in the tokenizer. (GH-17828)Miss Islington (bot)2020-02-121-4/+32
| | | | | | | * Always set the text attribute. * Correct the offset attribute for non-ascii sources. (cherry picked from commit 0cc6b5e559b8303b18fdd56c2befd900fe7b5e35) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* bpo-39209: Manage correctly multi-line tokens in interactive mode (GH-17860)Miss Islington (bot)2020-01-061-0/+2
| | | | | (cherry picked from commit 5ec91f78d59d9c39b984f284e00cd04b96ddb5db) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-38673: dont switch to ps2 if the line starts with comment or whitespace ↵Miss Islington (bot)2019-12-091-0/+6
| | | | | | | | (GH-17421) https://bugs.python.org/issue38673 (cherry picked from commit 109fc2792a490ee5cd8a423e17d415fbdedec5c8) Co-authored-by: Batuhan Taşkaya <47358913+isidentical@users.noreply.github.com>
* Indent code inside if block. (GH-15284)Miss Islington (bot)2019-08-151-1/+1
| | | | | | Without indendation, seems like strcpy line is parallel to `if` condition. (cherry picked from commit 69f37bcb28d7cd78255828029f895958b5baf6ff) Co-authored-by: Hansraj Das <raj.das.136@gmail.com>
* Fix `SyntaxError` indicator printing too many spaces for multi-line strings ↵Miss Islington (bot)2019-07-291-0/+2
| | | | | | | (GH-14433) (cherry picked from commit 5b94f3578c662d5f1ee90c0e6b81481d9ec82d89) Co-authored-by: Anthony Sottile <asottile@umich.edu>
* bpo-36878: Only allow text after `# type: ignore` if first character ASCII ↵Michael J. Sullivan2019-05-221-2/+3
| | | | | | | | | | | (GH-13504) This disallows things like `# type: ignoreé`, which seems wrong. Also switch to using Py_ISALNUM for the alnum check, for consistency with other code (and maybe correctness re: locale issues?). https://bugs.python.org/issue36878
* bpo-36878: Track extra text added to 'type: ignore' in the AST (GH-13479)Michael J. Sullivan2019-05-221-2/+6
| | | | | GH-13238 made extra text after a # type: ignore accepted by the parser. This finishes the job and actually plumbs the extra text through the parser and makes it available in the AST.
* bpo-2180: Treat line continuation at EOF as a `SyntaxError` (GH-13401)Anthony Sottile2019-05-181-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This makes the parser consistent with the tokenize module (already the case in `pypy`). sample ------ ```python x = 5\ ``` before ------ ```console $ python3 t.py $ python3 -mtokenize t.py t.py:2:0: error: EOF in multi-line statement ``` after ----- ```console $ ./python t.py File "t.py", line 3 x = 5\ ^ SyntaxError: unexpected EOF while parsing $ ./python -m tokenize t.py t.py:2:0: error: EOF in multi-line statement ``` https://bugs.python.org/issue2180
* bpo-36878: Allow extra text after `# type: ignore` comments (GH-13238)Michael J. Sullivan2019-05-111-8/+5
| | | | | | | In the parser, when using the type_comments=True option, recognize a TYPE_IGNORE as anything containing `# type: ignore` followed by a non-alphanumeric character. This is to allow ignores such as `# type: ignore[E1000]`.
* bpo-36623: Clean parser headers and include files (GH-12253)Pablo Galindo2019-04-131-1/+0
| | | After the removal of pgen, multiple header and function prototypes that lack implementation or are unused are still lying around.
* bpo-36459: Fix a possible double PyMem_FREE() due to tokenizer.c's ↵Zackery Spytz2019-03-281-1/+0
| | | | | | tok_nextc() (12601) Remove the PyMem_FREE() call added in cb90c89. The buffer will be freed when PyTokenizer_Free() is called on the tokenizer state.
* bpo-36367: Free buffer if realloc fails in tokenize.c (GH-12442)Pablo Galindo2019-03-191-2/+8
|
* bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)Guido van Rossum2019-03-071-0/+79
| | | | | | | This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.) https://bugs.python.org/issue35975
* bpo-35808: Retire pgen and use pgen2 to generate the parser (GH-11814)Pablo Galindo2019-03-011-56/+0
| | | | | Pgen is the oldest piece of technology in the CPython repository, building it requires various #if[n]def PGEN hacks in other parts of the code and it also depends more and more on CPython internals. This commit removes the old pgen C code and replaces it for a new version implemented in pure Python. This is a modified and adapted version of lib2to3/pgen2 that can generate grammar files compatibles with the current parser. This commit also eliminates all the #ifdef and code branches related to pgen, simplifying the code and making it more maintainable. The regen-grammar step now uses $(PYTHON_FOR_REGEN) that can be any version of the interpreter, so the new pgen code maintains compatibility with older versions of the interpreter (this also allows regenerating the grammar with the current CI solution that uses Python3.5). The new pgen Python module also makes use of the Grammar/Tokens file that holds the token specification, so is always kept in sync and avoids having to maintain duplicate token definitions.
* bpo-35766: Merge typed_ast back into CPython (GH-11645)Guido van Rossum2019-01-311-1/+56
|
* bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)Anthony Sottile2019-01-131-0/+7
|
* bpo-30455: Generate all token related code and docs from Grammar/Tokens. ↵Serhiy Storchaka2018-12-221-237/+0
| | | | | | | | | | | | | | | | | | | (GH-10370) "Include/token.h", "Lib/token.py" (containing now some data moved from "Lib/tokenize.py") and new files "Parser/token.c" (containing the code moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by "Tools/scripts/generate_token.py". The script overwrites files only if needed and can be used on the read-only sources tree. "Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py" instead of been executable itself. Added new make targets "regen-token" and "regen-symbol" which are now dependencies of "regen-all". The documentation contains now strings for operators and punctuation tokens.
* bpo-33306: Improve SyntaxError messages for unbalanced parentheses. (GH-6516)Serhiy Storchaka2018-12-171-0/+32
|
* bpo-35436: Add missing PyErr_NoMemory() calls and other minor bug fixes. ↵Zackery Spytz2018-12-071-0/+5
| | | | | | (GH-11015) Set MemoryError when appropriate, add missing failure checks, and fix some potential leaks.
* Remove unneeded PyUnicode_READY() in tokenizer.c (GH-9114)Zackery Spytz2018-09-101-1/+1
|
* Fix Windows compiler warning in tokenize.c (GH-8359)Victor Stinner2018-07-211-1/+1
| | | | | | Fix the following warning on Windows: parser\tokenizer.c(1297): warning C4244: 'function': conversion from '__int64' to 'int', possible loss of data.
* bpo-33305: Improve SyntaxError for invalid numerical literals. (GH-6517)Serhiy Storchaka2018-07-091-13/+52
|
* tokenizer: Remove unused tabs options (#4422)Victor Stinner2017-11-171-31/+11
| | | | | | | | | | Remove the following fields from tok_state structure which are now used unused: * altwarning: "Issue warning if alternate tabs don't match" * alterror: "Issue error if alternate tabs don't match" * alttabsize: "Alternate tab spacing" Replace alttabsize variable with ALTTABSIZE define.
* bpo-30406: Make async and await proper keywords (#1669)Jelle Zijlstra2017-10-061-63/+0
| | | Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
* bpo-25324: add missing comma in Parser/tokenizer.c (GH-1910)Albert-Jan Nijburg2017-06-011-1/+1
|
* bpo-25324: copy tok_name before changing it (#1608)Albert-Jan Nijburg2017-05-311-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | * add test to check if were modifying token * copy list so import tokenize doesnt have side effects on token * shorten line * add tokenize tokens to token.h to get them to show up in token * move ERRORTOKEN back to its previous location, and fix nitpick * copy comments from token.h automatically * fix whitespace and make more pythonic * change to fix comments from @haypo * update token.rst and Misc/NEWS * change wording * some more wording changes
* Issue #28489: Merge from 3.6Berker Peksag2017-02-051-1/+1
|\
| * Issue #28489: Fix comment in tokenizer.cBerker Peksag2017-02-051-1/+1
| | | | | | | | Patch by Ryan Gonzalez.
* | Use _PyObject_CallNoArg()Victor Stinner2016-12-061-3/+3
|/ | | | | | | Replace: PyObject_CallObject(callable, NULL) with: _PyObject_CallNoArg(callable)
* Replaced outdated macros _PyUnicode_AsString and _PyUnicode_AsStringAndSizeSerhiy Storchaka2016-11-201-1/+1
| | | | with PyUnicode_AsUTF8 and PyUnicode_AsUTF8AndSize.
* merge 3.5 (#24022)Benjamin Peterson2016-09-191-1/+1
|\
| * merge 3.4Benjamin Peterson2016-09-191-1/+1
| |\
| | * properly handle the single null-byte file (closes #24022)Benjamin Peterson2016-09-191-1/+1
| | |
* | | merge 3.5 (#27981)Benjamin Peterson2016-09-131-15/+17
|\ \ \ | |/ /
| * | restructure fp_setreadl so as to avoid refleaks (closes #27981)Benjamin Peterson2016-09-131-15/+17
| | |
* | | Issue #26331: Implement the parsing part of PEP 515.Brett Cannon2016-09-091-68/+162
| | | | | | | | | | | | Thanks to Georg Brandl for the patch.
* | | Skip unused value in tokenizer codeChristian Heimes2016-09-081-1/+1
| | | | | | | | | | | | | | | | | | | | | In the case of an escape character, c is never read. tok_next() is used to advance the pointer. CID 1225097
* | | Issue #22570: Renamed Py_SETREF to Py_XSETREF.Serhiy Storchaka2016-04-061-1/+1
|\ \ \ | |/ /
| * | Issue #22570: Renamed Py_SETREF to Py_XSETREF.Serhiy Storchaka2016-04-061-1/+1
| | |
* | | remove duplicated check for fractions and complex numbers (closes #26076)Benjamin Peterson2016-03-251-4/+0
| | | | | | | | | | | | Patch by Oren Milman.
* | | Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-0/+1
|\ \ \ | |/ /
| * | Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-0/+1
| | |
* | | Issue #25923: Added more const qualifiers to signatures of static and ↵Serhiy Storchaka2015-12-251-3/+3
| | | | | | | | | | | | private functions.
* | | Issue #20440: Massive replacing unsafe attribute setting code with specialSerhiy Storchaka2015-12-241-2/+1
|\ \ \ | |/ / | | | | | | macro Py_SETREF.
| * | Issue #20440: Massive replacing unsafe attribute setting code with specialSerhiy Storchaka2015-12-241-2/+1
| | | | | | | | | | | | macro Py_SETREF.
* | | Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
|\ \ \ | |/ / | | | | | | with a null byte.
| * | Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
| |\ \ | | |/ | | | | | | with a null byte.
| | * Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
| | | | | | | | | | | | with a null byte.
* | | Issue #24965: Implement PEP 498 "Literal String Interpolation". ↵Eric V. Smith2015-09-191-3/+5
| | | | | | | | | | | | Documentation is still needed, I'll open an issue for that.
* | | Fixed indentation.Eric V. Smith2015-09-121-1/+1
|/ /
generated by cgit v0.12 at 2024-11-13 10:56:59 (GMT)