summaryrefslogtreecommitdiffstats
path: root/Parser
Commit message (Collapse)AuthorAgeFilesLines
* bpo-42860: Remove type error from grammar (GH-24156)Lysandros Nikolaou2021-01-071-419/+430
| | | | | | This is only there so that alternative implementations written in statically-typed languages can use this grammar without having type errors in the way. Automerge-Triggered-By: GH:lysnikolaou
* bpo-42806: Fix ast locations of f-strings inside parentheses (GH-24067)Pablo Galindo2021-01-031-1/+1
|
* bpo-40631: Disallow single parenthesized star target (GH-24027)Lysandros Nikolaou2021-01-021-540/+787
|
* bpo-30858: Improve error location for expressions with assignments (GH-23753)Pablo Galindo2020-12-131-4/+4
| | | Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* bpo-42519: Replace PyMem_MALLOC() with PyMem_Malloc() (GH-23586)Victor Stinner2020-12-012-31/+31
| | | | | | | | | | | No longer use deprecated aliases to functions: * Replace PyMem_MALLOC() with PyMem_Malloc() * Replace PyMem_REALLOC() with PyMem_Realloc() * Replace PyMem_FREE() with PyMem_Free() * Replace PyMem_Del() with PyMem_Free() * Replace PyMem_DEL() with PyMem_Free() Modify also the PyMem_DEL() macro to use directly PyMem_Free().
* Refactor the grammar to match the language specification docs (GH-23574)Pablo Galindo2020-11-301-94/+94
|
* bpo-40998: Address compiler warnings found by ubsan (GH-20929)Christian Heimes2020-11-181-0/+3
| | | | | Signed-off-by: Christian Heimes <christian@python.org> Automerge-Triggered-By: GH:tiran
* bpo-42381: Allow walrus in set literals and set comprehensions (GH-23332)Pablo Galindo2020-11-171-1118/+953
| | | | | | | | | | | Currently walruses are not allowerd in set literals and set comprehensions: >>> {y := 4, 4**2, 3**3} File "<stdin>", line 1 {y := 4, 4**2, 3**3} ^ SyntaxError: invalid syntax but they should be allowed as well per PEP 572
* bpo-42316: Allow unparenthesized walrus operator in indexes (GH-23317)Lysandros Nikolaou2020-11-161-6/+6
|
* bpo-42374: Allow unparenthesized walrus in genexps (GH-23319)Lysandros Nikolaou2020-11-161-6/+6
| | | | | This fixes a regression that was introduced by the new parser. Automerge-Triggered-By: GH:lysnikolaou
* bpo-1635741: _ast uses PyModule_AddObjectRef() (GH-23146)Victor Stinner2020-11-041-4/+2
| | | | Replace PyModule_AddObject() with PyModule_AddObjectRef() in the _ast module (Python-ast.c).
* bpo-41796: Call _PyAST_Fini() earlier to fix a leak (GH-23131)Victor Stinner2020-11-031-24/+45
| | | | | | | | | | | | | | | Call _PyAST_Fini() on all interpreters, not only on the main interpreter. Also, call it ealier to fix a reference leak. Python types contain a reference to themselves in in their PyTypeObject.tp_mro member. _PyAST_Fini() must called before the last GC collection to destroy AST types. _PyInterpreterState_Clear() now calls _PyAST_Fini(). It now also calls _PyWarnings_Fini() on subinterpeters, not only on the main interpreter. Add an assertion in AST init_types() to ensure that the _ast module is no longer used after _PyAST_Fini() has been called.
* bpo-41796: Make _ast module state per interpreter (GH-23024)Victor Stinner2020-11-021-82/+161
| | | | | | | | | | | The ast module internal state is now per interpreter. * Rename "astmodulestate" to "struct ast_state" * Add pycore_ast.h internal header: the ast_state structure is now declared in pycore_ast.h. * Add PyInterpreterState.ast (struct ast_state) * Remove get_ast_state() * Rename get_global_ast_state() to get_ast_state() * PyAST_obj2mod() now handles get_ast_state() failures
* bpo-42218: Correctly handle errors in left-recursive rules (GH-23065)Lysandros Nikolaou2020-10-311-0/+18
| | | | | | | Left-recursive rules need to check for errors explicitly, since even if the rule returns NULL, the parsing might continue and lead to long-distance failures. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-42214: Fix check for NOTEQUAL token in the PEG parser for the ↵Pablo Galindo2020-10-303-4/+3
| | | | barry_as_flufl rule (GH-23048)
* bpo-42206: Propagate and raise errors from PyAST_Validate in the parser ↵Batuhan Taskaya2020-10-301-1/+3
| | | | (GH-23035)
* bpo-41659: Disallow curly brace directly after primary (GH-22996)Lysandros Nikolaou2020-10-271-167/+234
|
* bpo-42123: Run the parser two times and only enable invalid rules on the ↵Lysandros Nikolaou2020-10-263-47/+61
| | | | | | | | | | second run (GH-22111) * Implement running the parser a second time for the errors messages The first parser run is only responsible for detecting whether there is a `SyntaxError` or not. If there isn't the AST gets returned. Otherwise, the parser is run a second time with all the `invalid_*` rules enabled so that all the customized error messages get produced.
* bpo-42150: Avoid buffer overflow in the new parser (GH-22978)Pablo Galindo2020-10-251-1/+2
|
* bpo-41746: Cast to typed seqs in CHECK macros to avoid type erasure (GH-22864)Lysandros Nikolaou2020-10-212-41/+41
|
* bpo-42000: Cleanup the AST related C-code (GH-22641)Batuhan Taskaya2020-10-102-43/+1
| | | | | | | | - Use the proper asdl sequence when creating empty arguments - Remove reduntant casts (thanks to new typed asdl_sequences) - Remove MarshalPrototypeVisitor and some utilities from asdl generator - Fix the header of `Python/ast.c` (kept from pgen times) Automerge-Triggered-By: @pablogsal
* bpo-41979: Accept star-unpacking on with-item targets (GH-22611)Batuhan Taskaya2020-10-091-6/+9
| | | Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-41746: Add type information to asdl_seq objects (GH-22223)Pablo Galindo2020-09-165-461/+538
| | | | | | | | | | | | | * Add new capability to the PEG parser to type variable assignments. For instance: ``` | a[asdl_stmt_seq*]=';'.small_stmt+ [';'] NEWLINE { a } ``` * Add new sequence types from the asdl definition (automatically generated) * Make `asdl_seq` type a generic aliasing pointer type. * Create a new `asdl_generic_seq` for the generic case using `void*`. * The old `asdl_seq_GET`/`ast_seq_SET` macros now are typed. * New `asdl_seq_GET_UNTYPED`/`ast_seq_SET_UNTYPED` macros for dealing with generic sequences. * Changes all possible `asdl_seq` types to use specific versions everywhere.
* bpo-41631: _ast module uses again a global state (#21961)Victor Stinner2020-09-151-41/+20
| | | | | | | | | | | | | | | | | Partially revert commit ac46eb4ad6662cf6d771b20d8963658b2186c48c: "bpo-38113: Update the Python-ast.c generator to PEP384 (gh-15957)". Using a module state per module instance is causing subtle practical problems. For example, the Mercurial project replaces the __import__() function to implement lazy import, whereas Python expected that "import _ast" always return a fully initialized _ast module. Add _PyAST_Fini() to clear the state at exit. The _ast module has no state (set _astmodule.m_size to 0). Remove astmodule_traverse(), astmodule_clear() and astmodule_free() functions.
* bpo-41697: Correctly handle KeywordOrStarred when parsing arguments in the ↵Pablo Galindo2020-09-033-10/+22
| | | | parser (GH-22077)
* bpo-41690: Use a loop to collect args in the parser instead of recursion ↵Pablo Galindo2020-09-023-503/+625
| | | | | | | | | | | | | | | | | | | | | (GH-22053) This program can segfault the parser by stack overflow: ``` import ast code = "f(" + ",".join(['a' for _ in range(100000)]) + ")" print("Ready!") ast.parse(code) ``` the reason is that the rule for arguments has a simple recursion when collecting args: args[expr_ty]: [...] | a=named_expression b=[',' c=args { c }] { [...] }
* bpo-38156: Fix compiler warning in PyOS_StdioReadline() (GH-21721)Victor Stinner2020-08-041-1/+1
| | | incr cannot be larger than INT_MAX: downcast to int explicitly.
* closes bpo-38156: Always handle interrupts in PyOS_StdioReadline. (GH-21569)Benjamin Peterson2020-07-291-30/+14
| | | | | This consolidates the handling of my_fgets return values, so that interrupts are always handled, even if they come after EOF. I believe PyOS_StdioReadline is still buggy in that I/O errors will not result in a proper Python exception being set. However, that is a separate issue.
* Validate the AST produced by the parser in debug mode (GH-21643)Pablo Galindo2020-07-271-0/+9
| | | This will improve the debug experience if something fails in the produced AST. Previously, errors in the produced AST can be felt much later like in the garbage collector or the compiler, making debugging them much more difficult.
* Fix trivial typo in the PEG string parser (GH-21508)Eric V. Smith2020-07-161-1/+1
|
* Fix possibly-unitialized warning in string_parser.c. (GH-21503)Benjamin Peterson2020-07-161-15/+16
| | | | | | | | | | | | | | | | | | | | GCC says ``` ../cpython/Parser/string_parser.c: In function ‘fstring_find_expr’: ../cpython/Parser/string_parser.c:404:93: warning: ‘cols’ may be used uninitialized in this function [-Wmaybe-uninitialized] 404 | p2->starting_col_offset = p->tok->first_lineno == p->tok->lineno ? t->col_offset + cols : cols; | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~ ../cpython/Parser/string_parser.c:384:16: note: ‘cols’ was declared here 384 | int lines, cols; | ^~~~ ../cpython/Parser/string_parser.c:403:45: warning: ‘lines’ may be used uninitialized in this function [-Wmaybe-uninitialized] 403 | p2->starting_lineno = t->lineno + lines - 1; | ~~~~~~~~~~~~~~~~~~^~~ ../cpython/Parser/string_parser.c:384:9: note: ‘lines’ was declared here 384 | int lines, cols; | ^~~~~ ``` and, indeed, if `PyBytes_AsString` somehow fails, lines & cols will not be initialized.
* bpo-41215: Make assertion in the new parser more strict (GH-21364)Lysandros Nikolaou2020-07-061-1/+1
|
* bpo-41215: Don't use NULL by default in the PEG parser keyword list (GH-21355)Pablo Galindo2020-07-062-4/+7
| | | Automerge-Triggered-By: @lysnikolaou
* bpo-41204: Fix compiler warning in ast_type_init() (GH-21307)Victor Stinner2020-07-041-4/+5
|
* bpo-41194: Convert _ast extension to PEP 489 (GH-21293)Victor Stinner2020-07-031-46/+73
| | | | Convert the _ast extension module to PEP 489 "Multiphase initialization". Replace the global _ast state with a module state.
* bpo-41194: The _ast module cannot be loaded more than once (GH-21290)Victor Stinner2020-07-031-39/+32
| | | | | | | | Fix a crash in the _ast module: it can no longer be loaded more than once. It now uses a global state rather than a module state. * Move _ast module state: use a global state instead. * Set _astmodule.m_size to -1, so the extension cannot be loaded more than once.
* bpo-41194: Pass module state in Python-ast.c (GH-21284)Victor Stinner2020-07-031-80/+102
| | | | | | Rework asdl_c.py to pass the module state to functions in Python-ast.c, instead of using astmodulestate_global. Handle also PyState_AddModule() failure in init_types().
* bpo-35975: Only use cf_feature_version if PyCF_ONLY_AST in cf_flags (#21021)Guido van Rossum2020-06-281-2/+3
|
* bpo-41076: Pre-feed the parser with the f-string expression location (GH-21054)Lysandros Nikolaou2020-06-272-242/+25
| | | This commit changes the parsing of f-string expressions with the new parser. The parser gets pre-fed with the location of the expression itself (not the f-string, which was what we were doing before). This allows us to completely skip the shifting of the AST nodes after the parsing is completed.
* bpo-40769: Allow extra surrounding parentheses for invalid annotated ↵Batuhan Taskaya2020-06-271-205/+261
| | | | assignment rule (GH-20387)
* bpo-41132: Use pymalloc allocator in the f-string parser (GH-21173)Lysandros Nikolaou2020-06-272-10/+10
|
* bpo-41084: Adjust message when an f-string expression causes a SyntaxError ↵Lysandros Nikolaou2020-06-261-0/+21
| | | | | (GH-21084) Prefix the error message with `fstring: `, when parsing an f-string expression throws a `SyntaxError`.
* bpo-41119: Output correct error message for list/tuple followed by colon ↵Lysandros Nikolaou2020-06-251-320/+284
| | | | (GH-21160)
* bpo-40939: Rename PyPegen* functions to PyParser* (GH-21016)Lysandros Nikolaou2020-06-211-7/+22
| | | | | | Rename PyPegen* functions to PyParser*, so that we can remove the old set of PyParser* functions that were using the old parser.
* bpo-41060: Avoid SEGFAULT when calling GET_INVALID_TARGET in the grammar ↵Lysandros Nikolaou2020-06-212-7/+26
| | | | | | | | | (GH-21020) `GET_INVALID_TARGET` might unexpectedly return `NULL`, which if not caught will cause a SEGFAULT. Therefore, this commit introduces a new inline function `RAISE_SYNTAX_ERROR_INVALID_TARGET` that always checks for `GET_INVALID_TARGET` returning NULL and can be used in the grammar, replacing the long C ternary operation used till now.
* bpo-40939: Remove the old parser (Part 2) (GH-21005)Lysandros Nikolaou2020-06-201-189/+0
| | | Remove some remaining files and Makefile targets for the old parser
* bpo-40958: Avoid 'possible loss of data' warning on Windows (GH-20970)Lysandros Nikolaou2020-06-202-2/+2
|
* bpo-40334: Produce better error messages on invalid targets (GH-20106)Lysandros Nikolaou2020-06-183-1427/+1736
| | | | | | | | | | | | | | The following error messages get produced: - `cannot delete ...` for invalid `del` targets - `... is an illegal 'for' target` for invalid targets in for statements - `... is an illegal 'with' target` for invalid targets in with statements Additionally, a few `cut`s were added in various places before the invocation of the `invalid_*` rule, in order to speed things up. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40958: Avoid buffer overflow in the parser when indexing the current ↵Pablo Galindo2020-06-162-13/+11
| | | | line (GH-20875)
* Remove old comment in string_parser.c (GH-20906)Pablo Galindo2020-06-161-5/+0
|
generated by cgit v0.12 at 2024-11-23 08:50:19 (GMT)