summaryrefslogtreecommitdiffstats
path: root/Parser
Commit message (Collapse)AuthorAgeFilesLines
* [3.9] bpo-42381: Allow walrus in set literals and set comprehensions ↵Pablo Galindo2020-11-181-1120/+955
| | | | | | | | | | | | | | | | (GH-23332) (GH-23333) Currently walruses are not allowerd in set literals and set comprehensions: >>> {y := 4, 4**2, 3**3} File "<stdin>", line 1 {y := 4, 4**2, 3**3} ^ SyntaxError: invalid syntax but they should be allowed as well per PEP 572. (cherry picked from commit b0aba1fcdc3da952698d99aec2334faa79a8b68c) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40998: Address compiler warnings found by ubsan (GH-20929)Miss Islington (bot)2020-11-181-0/+3
| | | | | | | | Signed-off-by: Christian Heimes <christian@python.org> Automerge-Triggered-By: GH:tiran (cherry picked from commit 07f2adedf0940b06d136208ec386d69b7d2d5b43) Co-authored-by: Christian Heimes <christian@python.org>
* bpo-42374: Allow unparenthesized walrus in genexps (GH-23319) (GH-23329)Lysandros Nikolaou2020-11-161-6/+6
| | | | | This fixes a regression that was introduced by the new parser. (cherry picked from commit cb3e5ed0716114393696ec7201e51fe0595eab4f)
* [3.9] bpo-42218: Correctly handle errors in left-recursive rules (GH-23065) ↵Lysandros Nikolaou2020-10-311-0/+18
| | | | | | | | | | | | (GH-23066) Left-recursive rules need to check for errors explicitly, since even if the rule returns NULL, the parsing might continue and lead to long-distance failures. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com> (cherry picked from commit 02cdfc93f82fecdb7eae97a868d4ee222b9875d9) Automerge-Triggered-By: GH:lysnikolaou
* [3.9] bpo-42214: Fix check for NOTEQUAL token in the PEG parser for the ↵Pablo Galindo2020-10-313-4/+3
| | | | | | | barry_as_flufl rule (GH-23048) (GH-23051) (cherry picked from commit 06f8c3328dcd81c84d1ee2b3a57b5381dcb38482) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* [3.9] bpo-42123: Run the parser two times and only enable invalid rules on ↵Lysandros Nikolaou2020-10-283-48/+62
| | | | | | | | | | | | the second run (GH-22111) (GH-23011) * Implement running the parser a second time for the errors messages The first parser run is only responsible for detecting whether there is a `SyntaxError` or not. If there isn't the AST gets returned. Otherwise, the parser is run a second time with all the `invalid_*` rules enabled so that all the customized error messages get produced. (cherry picked from commit bca701403253379409dece03053dbd739c0bd059)
* [3.9] bpo-41659: Disallow curly brace directly after primary (GH-22996) (#23006)Lysandros Nikolaou2020-10-271-167/+234
| | | (cherry picked from commit 15acc4eaba8519d7d5f2acaffde65446b44dcf79)
* bpo-42150: Avoid buffer overflow in the new parser (GH-22978)Miss Skeleton (bot)2020-10-251-1/+2
| | | | | (cherry picked from commit e68c67805e6a4c4ec80bea64be0e8373cc02d322) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* [3.9] bpo-41979: Accept star-unpacking on with-item targets (GH-22611) ↵Batuhan Taskaya2020-10-091-6/+9
| | | | | | | (GH-22612) Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com> Automerge-Triggered-By: @pablogsal
* [3.9] bpo-41631: _ast module uses again a global state (GH-21961) (GH-22258)Pablo Galindo2020-09-151-41/+20
| | | | | | | | | | | | | | | | | | | | Partially revert commit ac46eb4ad6662cf6d771b20d8963658b2186c48c: "bpo-38113: Update the Python-ast.c generator to PEP384 (gh-15957)". Using a module state per module instance is causing subtle practical problems. For example, the Mercurial project replaces the __import__() function to implement lazy import, whereas Python expected that "import _ast" always return a fully initialized _ast module. Add _PyAST_Fini() to clear the state at exit. The _ast module has no state (set _astmodule.m_size to 0). Remove astmodule_traverse(), astmodule_clear() and astmodule_free() functions.. (cherry picked from commit e5fbe0cbd4be99ced5f000ad382208ad2a561c90) Co-authored-by: Victor Stinner <vstinner@python.org>
* [3.9] bpo-41697: Correctly handle KeywordOrStarred when parsing arguments in ↵Pablo Galindo2020-09-033-10/+22
| | | | | | | the parser (GH-22077) (GH-22079) (cherry picked from commit 315a61f7a9418d904e0eea14b1f054fac3a90e9f) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* [3.9] bpo-41690: Use a loop to collect args in the parser instead of ↵Pablo Galindo2020-09-023-503/+625
| | | | | | | | | | | | | | | | | | | | | | | recursion (GH-22053) (GH-22067) This program can segfault the parser by stack overflow: ``` import ast code = "f(" + ",".join(['a' for _ in range(100000)]) + ")" print("Ready!") ast.parse(code) ``` the reason is that the rule for arguments has a simple recursion when collecting args: args[expr_ty]: [...] | a=named_expression b=[',' c=args { c }] { [...] }. (cherry picked from commit 4a97b1517a6b5ff22e2984b677a680b07ff0ce11) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* [3.9] bpo-41194: Convert _ast extension to PEP 489 (GH-21807)Victor Stinner2020-08-101-47/+75
| | | | | | | | | | | | * bpo-41194: Convert _ast extension to PEP 489 (GH-21293) Convert the _ast extension module to PEP 489 "Multiphase initialization". Replace the global _ast state with a module state. (cherry picked from commit b1cc6ba73a51d5cc3aeb113b5e7378fb50a0e20a) * bpo-41204: Fix compiler warning in ast_type_init() (GH-21307) (cherry picked from commit 1f76453173267887ed05bb3783e862cb22365ae8)
* bpo-38156: Fix compiler warning in PyOS_StdioReadline() (GH-21721)Miss Islington (bot)2020-08-041-1/+1
| | | | | | incr cannot be larger than INT_MAX: downcast to int explicitly. (cherry picked from commit bde48fd8110cc5f128d5db44810d17811e328a24) Co-authored-by: Victor Stinner <vstinner@python.org>
* closes bpo-38156: Always handle interrupts in PyOS_StdioReadline. (GH-21569)Miss Islington (bot)2020-07-291-30/+14
| | | | | | | | This consolidates the handling of my_fgets return values, so that interrupts are always handled, even if they come after EOF. I believe PyOS_StdioReadline is still buggy in that I/O errors will not result in a proper Python exception being set. However, that is a separate issue. (cherry picked from commit a74eea238f5baba15797e2e8b570d153bc8690a7) Co-authored-by: Benjamin Peterson <benjamin@python.org>
* [3.9] Validate the AST produced by the parser in debug mode (GH-21643) ↵Pablo Galindo2020-07-271-0/+9
| | | | | | | | (GH-21646) This will improve the debug experience if something fails in the produced AST. Previously, errors in the produced AST can be felt much later like in the garbage collector or the compiler, making debugging them much more difficult.. (cherry picked from commit 1332226b32da44087a55e1d71990ee6899dfd28a) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* Fix trivial typo in the PEG string parser (GH-21508)Miss Islington (bot)2020-07-161-1/+1
| | | | | (cherry picked from commit 0275e0452a773976827c2b9bd1e598ee08e2d7f5) Co-authored-by: Eric V. Smith <ericvsmith@users.noreply.github.com>
* Fix possibly-unitialized warning in string_parser.c. (GH-21503)Miss Islington (bot)2020-07-161-15/+16
| | | | | | | | | | | | | | | | | | | | | | | GCC says ``` ../cpython/Parser/string_parser.c: In function ‘fstring_find_expr’: ../cpython/Parser/string_parser.c:404:93: warning: ‘cols’ may be used uninitialized in this function [-Wmaybe-uninitialized] 404 | p2->starting_col_offset = p->tok->first_lineno == p->tok->lineno ? t->col_offset + cols : cols; | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~ ../cpython/Parser/string_parser.c:384:16: note: ‘cols’ was declared here 384 | int lines, cols; | ^~~~ ../cpython/Parser/string_parser.c:403:45: warning: ‘lines’ may be used uninitialized in this function [-Wmaybe-uninitialized] 403 | p2->starting_lineno = t->lineno + lines - 1; | ~~~~~~~~~~~~~~~~~~^~~ ../cpython/Parser/string_parser.c:384:9: note: ‘lines’ was declared here 384 | int lines, cols; | ^~~~~ ``` and, indeed, if `PyBytes_AsString` somehow fails, lines & cols will not be initialized. (cherry picked from commit 2ad7e9c011b7606c5c7307176df07419a0e60134) Co-authored-by: Benjamin Peterson <benjamin@python.org>
* bpo-41215: Make assertion in the new parser more strict (GH-21364)Miss Islington (bot)2020-07-061-1/+1
| | | | | (cherry picked from commit 782f44b8fb07ec33cee148b2b6b4cf53024fe0cd) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* [3.9] bpo-41215: Don't use NULL by default in the PEG parser keyword list ↵Pablo Galindo2020-07-062-9/+12
| | | | | | | | | (GH-21355) (GH-21356) (cherry picked from commit 39e76c0fb07e20acad454deb86a0457b279884a9) Co-authored-by: Pablo Galindo <pablogsal@gmail.com> Automerge-Triggered-By: @lysnikolaou
* [3.9] bpo-41194: The _ast module cannot be loaded more than once (GH-21290) ↵Victor Stinner2020-07-031-96/+111
| | | | | | | | | | | | | | | | | | | | | | | (GH-21292) * bpo-41194: Pass module state in Python-ast.c (GH-21284) Rework asdl_c.py to pass the module state to functions in Python-ast.c, instead of using astmodulestate_global. Handle also PyState_AddModule() failure in init_types(). (cherry picked from commit 74419f0c64959bb8392fcf3659058410423038e1) * bpo-41194: The _ast module cannot be loaded more than once (GH-21290) Fix a crash in the _ast module: it can no longer be loaded more than once. It now uses a global state rather than a module state. * Move _ast module state: use a global state instead. * Set _astmodule.m_size to -1, so the extension cannot be loaded more than once. (cherry picked from commit 91e1bc18bd467a13bceb62e16fbc435b33381c82)
* [3.9] bpo-35975: Only use cf_feature_version if PyCF_ONLY_AST in cf_flags ↵Guido van Rossum2020-06-281-2/+3
| | | | (#21022)
* [3.9] bpo-41076: Pre-feed the parser with the f-string expression location ↵Pablo Galindo2020-06-282-242/+25
| | | | | | (GH-21054) (GH-21190) This commit changes the parsing of f-string expressions with the new parser. The parser gets pre-fed with the location of the expression itself (not the f-string, which was what we were doing before). This allows us to completely skip the shifting of the AST nodes after the parsing is completed.. (cherry picked from commit 1f0f4abb110b9fbade6175842b6a26ab0b8df6dd)
* [3.9] bpo-40769: Allow extra surrounding parentheses for invalid annotated ↵Pablo Galindo2020-06-271-205/+261
| | | | | assignment rule (GH-20387) (GH-21186) (cherry picked from commit c8f29ad986f8274fc5fbf889bdd2a211878856b9)
* bpo-41084: Adjust message when an f-string expression causes a SyntaxError ↵Miss Islington (bot)2020-06-271-0/+21
| | | | | | | | (GH-21084) Prefix the error message with `fstring: `, when parsing an f-string expression throws a `SyntaxError`. (cherry picked from commit 2e0a920e9eb540654c0bb2298143b00637dc5961) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* [3.9] bpo-41132: Use pymalloc allocator in the f-string parser (GH-21173) ↵Lysandros Nikolaou2020-06-271-7/+7
| | | | | | | (GH-21183) (cherry picked from commit 6dcbc2422de9e2a7ff89a4689572d84001e230b2) Automerge-Triggered-By: @pablogsal
* [3.9] bpo-41119: Output correct error message for list/tuple followed by ↵Lysandros Nikolaou2020-06-261-320/+284
| | | | | colon (GH-21160) (GH-21172) (cherry picked from commit 4b85e60601489f9ee9dd2909e28d89a31566887c)
* [3.9] bpo-41060: Avoid SEGFAULT when calling GET_INVALID_TARGET in the ↵Lysandros Nikolaou2020-06-212-7/+26
| | | | | | | | | | | | | grammar (GH-21020) (GH-21024) `GET_INVALID_TARGET` might unexpectedly return `NULL`, which if not caught will cause a SEGFAULT. Therefore, this commit introduces a new inline function `RAISE_SYNTAX_ERROR_INVALID_TARGET` that always checks for `GET_INVALID_TARGET` returning NULL and can be used in the grammar, replacing the long C ternary operation used till now. (cherry picked from commit 6c4e0bd974f2895d42b63d9d004587e74b286c88) Automerge-Triggered-By: @pablogsal
* bpo-40958: Avoid 'possible loss of data' warning on Windows (GH-20970)Miss Islington (bot)2020-06-202-2/+2
| | | | | (cherry picked from commit 861efc6e8fe7f030b1e193989b13287b31385939) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* [3.9] bpo-40334: Produce better error messages on invalid targets (GH-20106) ↵Lysandros Nikolaou2020-06-193-1429/+1738
| | | | | | | | | | | | | | | | | | | (GH-20973) * bpo-40334: Produce better error messages on invalid targets (GH-20106) The following error messages get produced: - `cannot delete ...` for invalid `del` targets - `... is an illegal 'for' target` for invalid targets in for statements - `... is an illegal 'with' target` for invalid targets in with statements Additionally, a few `cut`s were added in various places before the invocation of the `invalid_*` rule, in order to speed things up. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com> (cherry picked from commit 01ece63d42b830df106948db0aefa6c1ba24416a)
* bpo-40958: Avoid buffer overflow in the parser when indexing the current ↵Miss Islington (bot)2020-06-162-13/+11
| | | | | | | line (GH-20875) (GH-20919) (cherry picked from commit 51c5896b6205911d29ac07f167ec7f3cf1cb600d) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-36020: Remove snprintf macro in pyerrors.h (GH-20889)Miss Islington (bot)2020-06-151-1/+1
| | | | | | | | | | | | | On Windows, GH-include "pyerrors.h" no longer defines "snprintf" and "vsnprintf" macros. PyOS_snprintf() and PyOS_vsnprintf() should be used to get portable behavior. Replace snprintf() calls with PyOS_snprintf() and replace vsnprintf() calls with PyOS_vsnprintf(). (cherry picked from commit e822e37946f27c09953bb5733acf3b07c2db690f) Co-authored-by: Victor Stinner <vstinner@python.org>
* [3.9] Improve readability and style in parser files (GH-20884) (GH-20885)Pablo Galindo2020-06-152-119/+160
| | | | | (cherry picked from commit fb61c42) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* [3.9] bpo-40939: Fix test_keyword for the old parser (GH-20814)Pablo Galindo2020-06-111-7/+7
|
* bpo-40847: Consider a line with only a LINECONT a blank line (GH-20769)Miss Islington (bot)2020-06-111-1/+2
| | | | | | | | | | | | | A line with only a line continuation character should be considered a blank line at tokenizer level so that only a single NEWLINE token gets emitted. The old parser was working around the issue, but the new parser threw a `SyntaxError` for valid input. For example, an empty line following a line continuation character was interpreted as a `SyntaxError`. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com> (cherry picked from commit 896f4cf63f9ab93e30572d879a5719d5aa2499fb) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* Raise specialised syntax error for invalid lambda parameters (GH-20776)Miss Islington (bot)2020-06-101-407/+712
| | | | | (cherry picked from commit c6483c989694cfa328dabd45eb191440da54bc68) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40903: Handle multiple '=' in invalid assignment rules in the PEG parser ↵Miss Islington (bot)2020-06-082-234/+429
| | | | | | | | (GH-20697) Automerge-Triggered-By: @pablogsal (cherry picked from commit 9f495908c5bd3645ed1af82d7bae6782720dab77) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40904: Fix segfault in the new parser with f-string containing yield ↵Miss Islington (bot)2020-06-081-0/+3
| | | | | | | statements with no value (GH-20701) (cherry picked from commit 972ab0327675e695373fc6272d5ac24e187579ad) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40880: Fix invalid read in newline_in_string in pegen.c (GH-20666)Miss Islington (bot)2020-06-061-2/+2
| | | | | | | | | | | | | | * bpo-40880: Fix invalid read in newline_in_string in pegen.c * Update Parser/pegen/pegen.c Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> * Add NEWS entry Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> (cherry picked from commit 2e6593db0086004a1ca7f7049218ff9573d473c2) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40883: Fix memory leak in fstring_compile_expr in parse_string.c (GH-20667)Miss Islington (bot)2020-06-061-0/+2
| | | | | (cherry picked from commit a54096e30523534e8eebb8dc1011b4536ed237a8) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* [3.9] bpo-40826: Fix GIL usage in PyOS_Readline() (GH-20613)Victor Stinner2020-06-031-33/+85
| | | | | | | | | | | | | | | | | | | | | | | | | * bpo-40826: Fix GIL usage in PyOS_Readline() (GH-20579) Fix GIL usage in PyOS_Readline(): lock the GIL to set an exception. Pass tstate to my_fgets() and _PyOS_WindowsConsoleReadline(). Cleanup these functions. (cherry picked from commit c353764fd564e401cf47a5d9efab18c72c60014e) * bpo-40826: Add _PyOS_InterruptOccurred(tstate) function (GH-20599) my_fgets() now calls _PyOS_InterruptOccurred(tstate) to check for pending signals, rather calling PyOS_InterruptOccurred(). my_fgets() is called with the GIL released, whereas PyOS_InterruptOccurred() must be called with the GIL held. test_repl: use text=True and avoid SuppressCrashReport in test_multiline_string_parsing(). Fix my_fgets() on Windows: fgets(fp) does crash if fileno(fp) is closed. (cherry picked from commit fa7ab6aa0f9a4f695e5525db5a113cd21fa93787)
* bpo-40217: Ensure Py_VISIT(Py_TYPE(self)) is always called for ↵Miss Islington (bot)2020-05-281-0/+1
| | | | | | | | | | | | PyType_FromSpec types (reverts GH-19414) (GH-20264) Heap types now always visit the type in tp_traverse. See added docs for details. This reverts commit 0169d3003be3d072751dd14a5c84748ab63a249f. Automerge-Triggered-By: @encukou (cherry picked from commit 1cf15af9a6f28750f37b08c028ada31d38e818dd) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* [3.9] bpo-40614: Respect feature version for f-string debug expressions ↵Pablo Galindo2020-05-271-0/+5
| | | | | | | | | (GH-20196) (GH-20464) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com> (cherry picked from commit c116c94) Co-authored-by: Shantanu <hauntsaninja@users.noreply.github.com>
* [3.9] Backport GH-20440: Set p->error_indicator in more places (GH-20457)Lysandros Nikolaou2020-05-271-0/+5
|
* [3.9] Backport GH-20370 and GH-20436: Soft keywords (GH-20458)Lysandros Nikolaou2020-05-272-1/+35
|
* Fix peg_generator compiler warnings under MSVC (GH-20405)Miss Islington (bot)2020-05-261-4/+0
| | | | | (cherry picked from commit a2bbedc8b18c001d2f9e702e6e678efbb2990daa) Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
* [3.9] bpo-38964: Print correct filename on a SyntaxError in an fstring ↵Lysandros Nikolaou2020-05-261-5/+2
| | | | | | | | | | | (GH-20399) (GH-20404) When a `SyntaxError` in the expression part of a fstring is found, the filename attribute of the `SyntaxError` is always `<fstring>`. With this commit, it gets changed to always have the name of the file the fstring resides in. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>. (cherry picked from commit f7b1e461567e5e3fa3ba46f589d9edc1b45b2dd0)
* bpo-40750: Do not expand the new parser debug flags if Py_BUILD_CORE is not ↵Miss Islington (bot)2020-05-251-1/+1
| | | | | | | defined (GH-20393) (cherry picked from commit deb4355a37e41edf1199920789fe9572c1fb43c2) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40750: Support -d flag in the new parser (GH-20340)Miss Islington (bot)2020-05-253-23/+4378
| | | | | (cherry picked from commit 800a35c623bbcdb5793c7d7a4974524286311479) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-36290: Fix keytword collision handling in AST node constructors (GH-12382)Miss Islington (bot)2020-05-241-3/+24
| | | | | (cherry picked from commit c73914a562580ae72048876cb42ed8e76e2c83f9) Co-authored-by: Rémi Lapeyre <remi.lapeyre@lenstra.fr>
generated by cgit v0.12 at 2025-02-16 19:01:58 (GMT)