summaryrefslogtreecommitdiffstats
path: root/Parser
Commit message (Collapse)AuthorAgeFilesLines
* bpo-42218: Correctly handle errors in left-recursive rules (GH-23065)Lysandros Nikolaou2020-10-311-0/+18
| | | | | | | Left-recursive rules need to check for errors explicitly, since even if the rule returns NULL, the parsing might continue and lead to long-distance failures. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-42214: Fix check for NOTEQUAL token in the PEG parser for the ↵Pablo Galindo2020-10-303-4/+3
| | | | barry_as_flufl rule (GH-23048)
* bpo-42206: Propagate and raise errors from PyAST_Validate in the parser ↵Batuhan Taskaya2020-10-301-1/+3
| | | | (GH-23035)
* bpo-41659: Disallow curly brace directly after primary (GH-22996)Lysandros Nikolaou2020-10-271-167/+234
|
* bpo-42123: Run the parser two times and only enable invalid rules on the ↵Lysandros Nikolaou2020-10-263-47/+61
| | | | | | | | | | second run (GH-22111) * Implement running the parser a second time for the errors messages The first parser run is only responsible for detecting whether there is a `SyntaxError` or not. If there isn't the AST gets returned. Otherwise, the parser is run a second time with all the `invalid_*` rules enabled so that all the customized error messages get produced.
* bpo-42150: Avoid buffer overflow in the new parser (GH-22978)Pablo Galindo2020-10-251-1/+2
|
* bpo-41746: Cast to typed seqs in CHECK macros to avoid type erasure (GH-22864)Lysandros Nikolaou2020-10-212-41/+41
|
* bpo-42000: Cleanup the AST related C-code (GH-22641)Batuhan Taskaya2020-10-102-43/+1
| | | | | | | | - Use the proper asdl sequence when creating empty arguments - Remove reduntant casts (thanks to new typed asdl_sequences) - Remove MarshalPrototypeVisitor and some utilities from asdl generator - Fix the header of `Python/ast.c` (kept from pgen times) Automerge-Triggered-By: @pablogsal
* bpo-41979: Accept star-unpacking on with-item targets (GH-22611)Batuhan Taskaya2020-10-091-6/+9
| | | Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-41746: Add type information to asdl_seq objects (GH-22223)Pablo Galindo2020-09-165-461/+538
| | | | | | | | | | | | | * Add new capability to the PEG parser to type variable assignments. For instance: ``` | a[asdl_stmt_seq*]=';'.small_stmt+ [';'] NEWLINE { a } ``` * Add new sequence types from the asdl definition (automatically generated) * Make `asdl_seq` type a generic aliasing pointer type. * Create a new `asdl_generic_seq` for the generic case using `void*`. * The old `asdl_seq_GET`/`ast_seq_SET` macros now are typed. * New `asdl_seq_GET_UNTYPED`/`ast_seq_SET_UNTYPED` macros for dealing with generic sequences. * Changes all possible `asdl_seq` types to use specific versions everywhere.
* bpo-41631: _ast module uses again a global state (#21961)Victor Stinner2020-09-151-41/+20
| | | | | | | | | | | | | | | | | Partially revert commit ac46eb4ad6662cf6d771b20d8963658b2186c48c: "bpo-38113: Update the Python-ast.c generator to PEP384 (gh-15957)". Using a module state per module instance is causing subtle practical problems. For example, the Mercurial project replaces the __import__() function to implement lazy import, whereas Python expected that "import _ast" always return a fully initialized _ast module. Add _PyAST_Fini() to clear the state at exit. The _ast module has no state (set _astmodule.m_size to 0). Remove astmodule_traverse(), astmodule_clear() and astmodule_free() functions.
* bpo-41697: Correctly handle KeywordOrStarred when parsing arguments in the ↵Pablo Galindo2020-09-033-10/+22
| | | | parser (GH-22077)
* bpo-41690: Use a loop to collect args in the parser instead of recursion ↵Pablo Galindo2020-09-023-503/+625
| | | | | | | | | | | | | | | | | | | | | (GH-22053) This program can segfault the parser by stack overflow: ``` import ast code = "f(" + ",".join(['a' for _ in range(100000)]) + ")" print("Ready!") ast.parse(code) ``` the reason is that the rule for arguments has a simple recursion when collecting args: args[expr_ty]: [...] | a=named_expression b=[',' c=args { c }] { [...] }
* bpo-38156: Fix compiler warning in PyOS_StdioReadline() (GH-21721)Victor Stinner2020-08-041-1/+1
| | | incr cannot be larger than INT_MAX: downcast to int explicitly.
* closes bpo-38156: Always handle interrupts in PyOS_StdioReadline. (GH-21569)Benjamin Peterson2020-07-291-30/+14
| | | | | This consolidates the handling of my_fgets return values, so that interrupts are always handled, even if they come after EOF. I believe PyOS_StdioReadline is still buggy in that I/O errors will not result in a proper Python exception being set. However, that is a separate issue.
* Validate the AST produced by the parser in debug mode (GH-21643)Pablo Galindo2020-07-271-0/+9
| | | This will improve the debug experience if something fails in the produced AST. Previously, errors in the produced AST can be felt much later like in the garbage collector or the compiler, making debugging them much more difficult.
* Fix trivial typo in the PEG string parser (GH-21508)Eric V. Smith2020-07-161-1/+1
|
* Fix possibly-unitialized warning in string_parser.c. (GH-21503)Benjamin Peterson2020-07-161-15/+16
| | | | | | | | | | | | | | | | | | | | GCC says ``` ../cpython/Parser/string_parser.c: In function ‘fstring_find_expr’: ../cpython/Parser/string_parser.c:404:93: warning: ‘cols’ may be used uninitialized in this function [-Wmaybe-uninitialized] 404 | p2->starting_col_offset = p->tok->first_lineno == p->tok->lineno ? t->col_offset + cols : cols; | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~ ../cpython/Parser/string_parser.c:384:16: note: ‘cols’ was declared here 384 | int lines, cols; | ^~~~ ../cpython/Parser/string_parser.c:403:45: warning: ‘lines’ may be used uninitialized in this function [-Wmaybe-uninitialized] 403 | p2->starting_lineno = t->lineno + lines - 1; | ~~~~~~~~~~~~~~~~~~^~~ ../cpython/Parser/string_parser.c:384:9: note: ‘lines’ was declared here 384 | int lines, cols; | ^~~~~ ``` and, indeed, if `PyBytes_AsString` somehow fails, lines & cols will not be initialized.
* bpo-41215: Make assertion in the new parser more strict (GH-21364)Lysandros Nikolaou2020-07-061-1/+1
|
* bpo-41215: Don't use NULL by default in the PEG parser keyword list (GH-21355)Pablo Galindo2020-07-062-4/+7
| | | Automerge-Triggered-By: @lysnikolaou
* bpo-41204: Fix compiler warning in ast_type_init() (GH-21307)Victor Stinner2020-07-041-4/+5
|
* bpo-41194: Convert _ast extension to PEP 489 (GH-21293)Victor Stinner2020-07-031-46/+73
| | | | Convert the _ast extension module to PEP 489 "Multiphase initialization". Replace the global _ast state with a module state.
* bpo-41194: The _ast module cannot be loaded more than once (GH-21290)Victor Stinner2020-07-031-39/+32
| | | | | | | | Fix a crash in the _ast module: it can no longer be loaded more than once. It now uses a global state rather than a module state. * Move _ast module state: use a global state instead. * Set _astmodule.m_size to -1, so the extension cannot be loaded more than once.
* bpo-41194: Pass module state in Python-ast.c (GH-21284)Victor Stinner2020-07-031-80/+102
| | | | | | Rework asdl_c.py to pass the module state to functions in Python-ast.c, instead of using astmodulestate_global. Handle also PyState_AddModule() failure in init_types().
* bpo-35975: Only use cf_feature_version if PyCF_ONLY_AST in cf_flags (#21021)Guido van Rossum2020-06-281-2/+3
|
* bpo-41076: Pre-feed the parser with the f-string expression location (GH-21054)Lysandros Nikolaou2020-06-272-242/+25
| | | This commit changes the parsing of f-string expressions with the new parser. The parser gets pre-fed with the location of the expression itself (not the f-string, which was what we were doing before). This allows us to completely skip the shifting of the AST nodes after the parsing is completed.
* bpo-40769: Allow extra surrounding parentheses for invalid annotated ↵Batuhan Taskaya2020-06-271-205/+261
| | | | assignment rule (GH-20387)
* bpo-41132: Use pymalloc allocator in the f-string parser (GH-21173)Lysandros Nikolaou2020-06-272-10/+10
|
* bpo-41084: Adjust message when an f-string expression causes a SyntaxError ↵Lysandros Nikolaou2020-06-261-0/+21
| | | | | (GH-21084) Prefix the error message with `fstring: `, when parsing an f-string expression throws a `SyntaxError`.
* bpo-41119: Output correct error message for list/tuple followed by colon ↵Lysandros Nikolaou2020-06-251-320/+284
| | | | (GH-21160)
* bpo-40939: Rename PyPegen* functions to PyParser* (GH-21016)Lysandros Nikolaou2020-06-211-7/+22
| | | | | | Rename PyPegen* functions to PyParser*, so that we can remove the old set of PyParser* functions that were using the old parser.
* bpo-41060: Avoid SEGFAULT when calling GET_INVALID_TARGET in the grammar ↵Lysandros Nikolaou2020-06-212-7/+26
| | | | | | | | | (GH-21020) `GET_INVALID_TARGET` might unexpectedly return `NULL`, which if not caught will cause a SEGFAULT. Therefore, this commit introduces a new inline function `RAISE_SYNTAX_ERROR_INVALID_TARGET` that always checks for `GET_INVALID_TARGET` returning NULL and can be used in the grammar, replacing the long C ternary operation used till now.
* bpo-40939: Remove the old parser (Part 2) (GH-21005)Lysandros Nikolaou2020-06-201-189/+0
| | | Remove some remaining files and Makefile targets for the old parser
* bpo-40958: Avoid 'possible loss of data' warning on Windows (GH-20970)Lysandros Nikolaou2020-06-202-2/+2
|
* bpo-40334: Produce better error messages on invalid targets (GH-20106)Lysandros Nikolaou2020-06-183-1427/+1736
| | | | | | | | | | | | | | The following error messages get produced: - `cannot delete ...` for invalid `del` targets - `... is an illegal 'for' target` for invalid targets in for statements - `... is an illegal 'with' target` for invalid targets in with statements Additionally, a few `cut`s were added in various places before the invocation of the `invalid_*` rule, in order to speed things up. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40958: Avoid buffer overflow in the parser when indexing the current ↵Pablo Galindo2020-06-162-13/+11
| | | | line (GH-20875)
* Remove old comment in string_parser.c (GH-20906)Pablo Galindo2020-06-161-5/+0
|
* bpo-36020: Remove snprintf macro in pyerrors.h (GH-20889)Victor Stinner2020-06-151-1/+1
| | | | | | | | | | On Windows, #include "pyerrors.h" no longer defines "snprintf" and "vsnprintf" macros. PyOS_snprintf() and PyOS_vsnprintf() should be used to get portable behavior. Replace snprintf() calls with PyOS_snprintf() and replace vsnprintf() calls with PyOS_vsnprintf().
* Improve readability and style in parser files (GH-20884)Pablo Galindo2020-06-152-119/+160
|
* bpo-40939: Remove the old parser (GH-20768)Pablo Galindo2020-06-1120-26697/+24309
| | | This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
* bpo-40939: Remove PEG parser easter egg (__new_parser__) (#20802)Lysandros Nikolaou2020-06-111-44/+10
| | | It no longer serves a purpose (there's only one parser) and having "new" in any name will eventually look odd. Also, it impinges on a potential sub-namespace, `__new_...__`.
* bpo-40847: Consider a line with only a LINECONT a blank line (GH-20769)Lysandros Nikolaou2020-06-101-1/+2
| | | | | | | | | | A line with only a line continuation character should be considered a blank line at tokenizer level so that only a single NEWLINE token gets emitted. The old parser was working around the issue, but the new parser threw a `SyntaxError` for valid input. For example, an empty line following a line continuation character was interpreted as a `SyntaxError`. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-39465: Use _PyInterpreterState_GET() (GH-20788)Victor Stinner2020-06-101-9/+9
| | | | | | | | | | | | Replace _PyThreadState_GET() with _PyInterpreterState_GET() in: * get_small_int() * gcmodule.c: add also get_gc_state() function * _PyTrash_deposit_object() * _PyTrash_destroy_chain() * warnings_get_state() * Py_GetRecursionLimit() Cleanup listnode.c: add 'parser' variable.
* Raise specialised syntax error for invalid lambda parameters (GH-20776)Pablo Galindo2020-06-101-407/+712
|
* bpo-40903: Handle multiple '=' in invalid assignment rules in the PEG parser ↵Pablo Galindo2020-06-082-234/+429
| | | | | (GH-20697) Automerge-Triggered-By: @pablogsal
* bpo-40904: Fix segfault in the new parser with f-string containing yield ↵Pablo Galindo2020-06-081-0/+3
| | | | statements with no value (GH-20701)
* bpo-40880: Fix invalid read in newline_in_string in pegen.c (#20666)Pablo Galindo2020-06-051-2/+2
| | | | | | | | | | | * bpo-40880: Fix invalid read in newline_in_string in pegen.c * Update Parser/pegen/pegen.c Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> * Add NEWS entry Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* bpo-40883: Fix memory leak in fstring_compile_expr in parse_string.c (GH-20667)Pablo Galindo2020-06-051-0/+2
|
* bpo-40826: Add _PyOS_InterruptOccurred(tstate) function (GH-20599)Victor Stinner2020-06-031-4/+13
| | | | | | | | | | | | my_fgets() now calls _PyOS_InterruptOccurred(tstate) to check for pending signals, rather calling PyOS_InterruptOccurred(). my_fgets() is called with the GIL released, whereas PyOS_InterruptOccurred() must be called with the GIL held. test_repl: use text=True and avoid SuppressCrashReport in test_multiline_string_parsing(). Fix my_fgets() on Windows: fgets(fp) does crash if fileno(fp) is closed.
* bpo-40826: Fix GIL usage in PyOS_Readline() (GH-20579)Victor Stinner2020-06-011-29/+72
| | | | | | Fix GIL usage in PyOS_Readline(): lock the GIL to set an exception. Pass tstate to my_fgets() and _PyOS_WindowsConsoleReadline(). Cleanup these functions.
generated by cgit v0.12 at 2024-12-05 01:06:58 (GMT)