summaryrefslogtreecommitdiffstats
path: root/Tools/peg_generator
Commit message (Collapse)AuthorAgeFilesLines
* Update pegen to use the latest upstream developments (GH-27586)Pablo Galindo Salgado2021-08-1219-397/+549
|
* bpo-44345: Fix 'generated by' comment in parser.c (GH-26615)Akira Nonaka2021-06-091-1/+1
|
* bpo-44180: Fix edge cases in invalid assigment rules in the parser (GH-26283)Pablo Galindo2021-05-211-0/+2
| | | | | | | | | | | | | | | The invalid assignment rules are very delicate since the parser can easily raise an invalid assignment when a keyword argument is provided. As they are very deep into the grammar tree, is very difficult to specify in which contexts these rules can be used and in which don't. For that, we need to use a different version of the rule that doesn't do error checking in those situations where we don't want the rule to raise (keyword arguments and generator expressions). We also need to check if we are in left-recursive rule, as those can try to eagerly advance the parser even if the parse will fail at the end of the expression. Failing to do this allows the parser to start parsing a call as a tuple and incorrectly identify a keyword argument as an invalid assignment, before it realizes that it was not a tuple after all.
* bpo-38605: Revert making 'from __future__ import annotations' the default ↵Pablo Galindo2021-04-212-2/+2
| | | | | | (GH-25490) This reverts commits 044a1048ca93d466965afc027b91a5a9eb9ce23c and 1be456ae9d53bb1cba2b24fc86175c282d1c2169, adapting the code to changes that happened after it.
* bpo-43822: Improve syntax errors for missing commas (GH-25377)Pablo Galindo2021-04-151-0/+12
|
* Sanitize macros and debug functions in pegen.c (GH-25291)Pablo Galindo2021-04-091-0/+8
|
* bpo-43244: Remove the pyarena.h header (GH-25007)Victor Stinner2021-03-241-4/+4
| | | | | | | | | | | | | | | | | | Remove the pyarena.h header file with functions: * PyArena_New() * PyArena_Free() * PyArena_Malloc() * PyArena_AddPyObject() These functions were undocumented, excluded from the limited C API, and were only used internally by the compiler. Add pycore_pyarena.h header. Rename functions: * PyArena_New() => _PyArena_New() * PyArena_Free() => _PyArena_Free() * PyArena_Malloc() => _PyArena_Malloc() * PyArena_AddPyObject() => _PyArena_AddPyObject()
* bpo-43244: Remove parser_interface.h header file (GH-25001)Victor Stinner2021-03-241-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | Remove parser functions using the "struct _mod" type, because the AST C API was removed: * PyParser_ASTFromFile() * PyParser_ASTFromFileObject() * PyParser_ASTFromFilename() * PyParser_ASTFromString() * PyParser_ASTFromStringObject() These functions were undocumented and excluded from the limited C API. Add pycore_parser.h internal header file. Rename functions: * PyParser_ASTFromFileObject() => _PyParser_ASTFromFile() * PyParser_ASTFromStringObject() => _PyParser_ASTFromString() These functions are no longer exported (replace PyAPI_FUNC() with extern). Remove also _PyPegen_run_parser_from_file() function. Update test_peg_generator to use _PyPegen_run_parser_from_file_pointer() instead.
* bpo-43244: Add pycore_compile.h header file (GH-25000)Victor Stinner2021-03-231-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | Remove the compiler functions using "struct _mod" type, because the public AST C API was removed: * PyAST_Compile() * PyAST_CompileEx() * PyAST_CompileObject() * PyFuture_FromAST() * PyFuture_FromASTObject() These functions were undocumented and excluded from the limited C API. Rename functions: * PyAST_CompileObject() => _PyAST_Compile() * PyFuture_FromASTObject() => _PyFuture_FromAST() Moreover, _PyFuture_FromAST() is no longer exported (replace PyAPI_FUNC() with extern). _PyAST_Compile() remains exported for test_peg_generator. Remove also compatibility functions: * PyAST_Compile() * PyAST_CompileEx() * PyFuture_FromAST()
* bpo-43244: Fix test_peg_generators on Windows (GH-24913)Victor Stinner2021-03-181-3/+2
| | | | Don't redefine Py_DebugFlag, it's already defined in pydebug.h which is included by Python.h
* bpo-43244: Fix test_peg_generator for PyAST_Validate() (GH-24912)Victor Stinner2021-03-181-0/+3
| | | | | | | | | test_peg_generator now defines _Py_TEST_PEGEN macro when building C code to not call PyAST_Validate() in Parser/pegen.c. Moreover, it defines Py_BUILD_CORE_MODULE macro to get access to the internal C API. Remove "global_ast_state" from Python-ast.c when it's built by test_peg_generator: always get the AST state from the current interpreter.
* Remove unnecessary imports in the grammar parser (GH-24904)Elisha Hollander2021-03-172-8/+2
|
* bpo-11717: fix ssize_t redefinition error when targeting 32bit Windows app ↵Jozef Grajciar2021-03-011-2/+2
| | | | (GH-24479)
* bpo-42997: Improve error message for missing : before suites (GH-24292)Pablo Galindo2021-02-024-2/+70
| | | | | | | | * Add to the peg generator a new directive ('&&') that allows to expect a token and hard fail the parsing if the token is not found. This allows to quickly emmit syntax errors for missing tokens. * Use the new grammar element to hard-fail if the ':' is missing before suites.
* Add small validator utility for PEG grammars (GH-23519)Pablo Galindo2020-12-262-0/+55
|
* bpo-42218: Correctly handle errors in left-recursive rules (GH-23065)Lysandros Nikolaou2020-10-311-0/+3
| | | | | | | Left-recursive rules need to check for errors explicitly, since even if the rule returns NULL, the parsing might continue and lead to long-distance failures. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-42123: Run the parser two times and only enable invalid rules on the ↵Lysandros Nikolaou2020-10-261-1/+4
| | | | | | | | | | second run (GH-22111) * Implement running the parser a second time for the errors messages The first parser run is only responsible for detecting whether there is a `SyntaxError` or not. If there isn't the AST gets returned. Otherwise, the parser is run a second time with all the `invalid_*` rules enabled so that all the customized error messages get produced.
* Revert "Fix all Python Cookbook links (#22205)" (GH-22424)Andre Delfino2020-09-271-2/+2
| | | This commit reverts commit ac0333e1e117b7f61ed7ef1dbcdb6e515ada603b as the original links are working again and they provide extended features such as comments and alternative versions.
* bpo-41746: Add type information to asdl_seq objects (GH-22223)Pablo Galindo2020-09-164-6/+56
| | | | | | | | | | | | | * Add new capability to the PEG parser to type variable assignments. For instance: ``` | a[asdl_stmt_seq*]=';'.small_stmt+ [';'] NEWLINE { a } ``` * Add new sequence types from the asdl definition (automatically generated) * Make `asdl_seq` type a generic aliasing pointer type. * Create a new `asdl_generic_seq` for the generic case using `void*`. * The old `asdl_seq_GET`/`ast_seq_SET` macros now are typed. * New `asdl_seq_GET_UNTYPED`/`ast_seq_SET_UNTYPED` macros for dealing with generic sequences. * Changes all possible `asdl_seq` types to use specific versions everywhere.
* Fix all Python Cookbook links (#22205)Andre Delfino2020-09-151-2/+2
|
* Fix 'gather' rules in the python parser generator (GH-22021)Pablo Galindo2020-09-031-0/+3
| | | | | | Currently, empty sequences in gather rules make the conditional for gather rules fail as empty sequences evaluate as "False". We need to explicitly check for "None" (the failure condition) to avoid false negatives.
* Delete remaining references to Grammar/Grammar from docs (#21624)Guido van Rossum2020-07-261-1/+1
| | | (Ironically, the file itself remains, see https://github.com/we-like-parsers/cpython/issues/135.)
* bpo-41215: Don't use NULL by default in the PEG parser keyword list (GH-21355)Pablo Galindo2020-07-061-1/+1
| | | Automerge-Triggered-By: @lysnikolaou
* bpo-41044: Generate valid PEG python parsers for opt+seq rules (GH-20995)Batuhan Taskaya2020-06-201-1/+7
| | | Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-41043: Escape literal part of the path for glob(). (GH-20994)Serhiy Storchaka2020-06-201-2/+2
|
* Fix trailing whitespace in keyword.py (GH-20881)Pablo Galindo2020-06-151-2/+2
|
* Include soft keywords in keyword.py (GH-20877)Pablo Galindo2020-06-152-4/+13
|
* bpo-40939: Clean and adapt the peg_generator directory after deleting the ↵Pablo Galindo2020-06-126-304/+30
| | | | old parser (GH-20822)
* bpo-40939: Remove the old parser (GH-20768)Pablo Galindo2020-06-111-3/+2
| | | This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
* bpo-40939: Generate keyword.py using the new parser (GH-20800)Lysandros Nikolaou2020-06-111-0/+73
|
* Refactor scripts in Tools/peg_generator/scripts (GH-20401)Lysandros Nikolaou2020-06-066-140/+131
|
* Fix lookahead of soft keywords in the PEG parser (GH-20436)Pablo Galindo2020-05-261-2/+9
| | | Automerge-Triggered-By: @gvanrossum
* Add soft keywords (GH-20370)Guido van Rossum2020-05-261-4/+20
| | | | | | | These are like keywords but they only work in context; they are not reserved except when there is an exact match. This would enable things like match statements without reserving `match` (which would be bad for the `re.match()` function and probably lots of other places). Automerge-Triggered-By: @gvanrossum
* Fix peg_generator compiler warnings under MSVC (GH-20405)Ammar Askar2020-05-261-4/+4
|
* bpo-40688: Use the correct parser in the peg_generator scripts (GH-20235)Lysandros Nikolaou2020-05-255-120/+55
| | | | | | | The scripts in `Tools/peg_generator/scripts` mostly assume that `ast.parse` and `compile` use the old parser, since this was the state of things, while we were developing them. They need to be updated to always use the correct parser. `_peg_parser` is being extended to support both parsing and compiling with both parsers.
* bpo-40750: Do not expand the new parser debug flags if Py_BUILD_CORE is not ↵Pablo Galindo2020-05-251-1/+1
| | | | defined (GH-20393)
* bpo-40750: Support -d flag in the new parser (GH-20340)Pablo Galindo2020-05-252-25/+61
|
* bpo-40334: Support suppressing of multiple optional variables in Pegen ↵Batuhan Taskaya2020-05-241-2/+2
| | | | (GH-20367)
* Fix debug output in PEG parser generator (GH-20308)Pablo Galindo2020-05-221-3/+3
|
* Fix typing problems reported by mypy in pegen (GH-20297)Pablo Galindo2020-05-212-27/+23
|
* bpo-40334: Correctly generate C parser when assigned var is None (GH-20296)Batuhan Taskaya2020-05-211-1/+4
| | | | | When there are 2 negative lookaheads in the same rule, let's say `!"(" blabla "," !")"`, there will the 2 `FunctionCall`'s where assigned value is None. Currently when the `add_var` is called the first one will be ignored but when the second lookahead's var is sent to dedupe it will be returned as `None_1` and this won't be ignored by the declaration generator in the `visit_Alt`. This patch adds an explicit check to `add_var` to distinguish whether if there is a variable or not.
* bpo-40669: Use requirements.pip when installing PEG dependencies (GH-20194)Pablo Galindo2020-05-181-1/+1
|
* bpo-40669: Install PEG benchmarking dependencies in a venv (GH-20183)Lysandros Nikolaou2020-05-183-15/+29
| | | | | | | | Create a `make venv` target, that creates a virtual environment and installs the dependency in that venv. `make time` and all the related targets are changed to use the virtual environment python. Automerge-Triggered-By: @pablogsal
* bpo-40661: Fix segfault when parsing invalid input (GH-20165)Lysandros Nikolaou2020-05-181-8/+9
| | | | | | Fix segfaults when parsing very complex invalid input, like `import äˆ ð£„¯ð¢·žð±‹á”€ð””ð‘©±å®ä±¬ð©¾\n𗶽`. Co-authored-by: Guido van Rossum <guido@python.org> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
* bpo-40334: Improvements to error-handling code in the PEG parser (GH-20003)Lysandros Nikolaou2020-05-171-21/+12
| | | | | | | | The following improvements are implemented in this commit: - `p->error_indicator` is set, in case malloc or realloc fail. - Avoid memory leaks in the case that realloc fails. - Call `PyErr_NoMemory()` instead of `PyErr_Format()`, because it requires no memory. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40334: Avoid collisions between parser variables and grammar variables ↵Pablo Galindo2020-05-102-78/+93
| | | | | | | | | | | | | | | (GH-19987) This is for the C generator: - Disallow rule and variable names starting with `_` - Rename most local variable names generated by the parser to start with `_` Exceptions: - Renaming `p` to `_p` will be a separate PR - There are still some names that might clash, e.g. - anything starting with `Py` - C reserved words (`if` etc.) - Macros like `EXTRA` and `CHECK`
* bpo-40555: Check for p->error_indicator in loop rules after the main loop is ↵Pablo Galindo2020-05-081-1/+1
| | | | done (GH-19986)
* bpo-40334: Generate comments in the parser code to improve debugging (GH-19966)Pablo Galindo2020-05-061-0/+14
|
* Clean up unused imports for the peg generator module (GH-19891)Anthony Shaw2020-05-049-17/+4
|
* bpo-40334: regenerate metaparser as part of regen-all (GH-19854)Pablo Galindo2020-05-021-1/+1
|
generated by cgit v0.12 at 2024-12-04 21:29:49 (GMT)