summaryrefslogtreecommitdiffstats
path: root/Tools/peg_generator
Commit message (Collapse)AuthorAgeFilesLines
* [3.9] bpo-45494: Fix parser crash when reporting errors involving invalid ↵Łukasz Langa2021-10-201-1/+1
| | | | | | | | | | | | | | | | | | | continuation characters (GH-28993) (#29071) There are two errors that this commit fixes: * The parser was not correctly computing the offset and the string source for E_LINECONT errors due to the incorrect usage of strtok(). * The parser was not correctly unwinding the call stack when a tokenizer exception happened in rules involving optionals ('?', [...]) as we always make them return valid results by using the comma operator. We need to check first if we don't have an error before continuing.. (cherry picked from commit a106343f632a99c8ebb0136fa140cf189b4a6a57) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> NOTE: unlike the cherry-picked original, this commit points at a crazy location due to a bug in the tokenizer that required a big refactor in 3.10 to fix. We are leaving as-is for 3.9.
* [3.9] Fix typos in the Tools directory (GH-28769) (GH-28800)Christian Clauss2021-10-073-3/+3
| | | | | | Like #28744 but for the Tools directory. Automerge-Triggered-By: GH:pablogsal (cherry picked from commit 682aecfdeba481c876bfc9f3796c635bd5b5df50)
* [3.9] bpo-42218: Correctly handle errors in left-recursive rules (GH-23065) ↵Lysandros Nikolaou2020-10-311-0/+3
| | | | | | | | | | | | (GH-23066) Left-recursive rules need to check for errors explicitly, since even if the rule returns NULL, the parsing might continue and lead to long-distance failures. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com> (cherry picked from commit 02cdfc93f82fecdb7eae97a868d4ee222b9875d9) Automerge-Triggered-By: GH:lysnikolaou
* [3.9] bpo-42123: Run the parser two times and only enable invalid rules on ↵Lysandros Nikolaou2020-10-281-1/+4
| | | | | | | | | | | | the second run (GH-22111) (GH-23011) * Implement running the parser a second time for the errors messages The first parser run is only responsible for detecting whether there is a `SyntaxError` or not. If there isn't the AST gets returned. Otherwise, the parser is run a second time with all the `invalid_*` rules enabled so that all the customized error messages get produced. (cherry picked from commit bca701403253379409dece03053dbd739c0bd059)
* [3.9] Remove git conflict lines from test_parse_directory in peg_generator ↵Lysandros Nikolaou2020-10-271-5/+0
| | | | (GH-23007)
* [3.9] Fix 'gather' rules in the python parser generator (GH-22021) (GH-22080)Pablo Galindo2020-09-031-0/+3
| | | | | | | | | Currently, empty sequences in gather rules make the conditional for gather rules fail as empty sequences evaluate as "False". We need to explicitly check for "None" (the failure condition) to avoid false negatives. (cherry picked from commit e55a0e9) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* [3.9] bpo-41215: Don't use NULL by default in the PEG parser keyword list ↵Pablo Galindo2020-07-061-1/+1
| | | | | | | | | (GH-21355) (GH-21356) (cherry picked from commit 39e76c0fb07e20acad454deb86a0457b279884a9) Co-authored-by: Pablo Galindo <pablogsal@gmail.com> Automerge-Triggered-By: @lysnikolaou
* [3.9] bpo-41043: Escape literal part of the path for glob(). (GH-20994). ↵Serhiy Storchaka2020-07-021-1/+5
| | | | | (GH-21275) (cherry picked from commit 935586845815f5b4c7814794413f6a812d4bd45f)
* bpo-41044: Generate valid PEG python parsers for opt+seq rules (GH-20995)Miss Islington (bot)2020-06-201-1/+7
| | | | | | Co-authored-by: Pablo Galindo <Pablogsal@gmail.com> (cherry picked from commit 55460ee6dc9a4f16bd68d6b6be3a8398c7d4a596) Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
* [3.9] Include soft keywords in keyword.py (GH-20877). (GH-20880)Pablo Galindo2020-06-152-7/+14
| | | | | (cherry picked from commit 78319e373d57cd4da67660f888aa7092efbd6f24) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40939: Generate keyword.py using the new parser (GH-20800)Miss Islington (bot)2020-06-111-0/+73
| | | | | (cherry picked from commit 9727694f08cad4b019d2939224e3416312b1c0e1) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* Refactor scripts in Tools/peg_generator/scripts (GH-20401)Miss Islington (bot)2020-06-066-140/+131
| | | | | (cherry picked from commit ba6fd87e41dceb01dcdacc57c722aca12cde42a9) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* [3.9] Backport GH-20370 and GH-20436: Soft keywords (GH-20458)Lysandros Nikolaou2020-05-271-5/+28
|
* Fix peg_generator compiler warnings under MSVC (GH-20405)Miss Islington (bot)2020-05-261-4/+4
| | | | | (cherry picked from commit a2bbedc8b18c001d2f9e702e6e678efbb2990daa) Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
* bpo-40688: Use the correct parser in the peg_generator scripts (GH-20235)Miss Islington (bot)2020-05-255-120/+55
| | | | | | | | | | The scripts in `Tools/peg_generator/scripts` mostly assume that `ast.parse` and `compile` use the old parser, since this was the state of things, while we were developing them. They need to be updated to always use the correct parser. `_peg_parser` is being extended to support both parsing and compiling with both parsers. (cherry picked from commit 9645930b5bc1833ef495891d22052d1ba65ab7ea) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
* bpo-40750: Do not expand the new parser debug flags if Py_BUILD_CORE is not ↵Miss Islington (bot)2020-05-251-1/+1
| | | | | | | defined (GH-20393) (cherry picked from commit deb4355a37e41edf1199920789fe9572c1fb43c2) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40750: Support -d flag in the new parser (GH-20340)Miss Islington (bot)2020-05-252-25/+61
| | | | | (cherry picked from commit 800a35c623bbcdb5793c7d7a4974524286311479) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40334: Support suppressing of multiple optional variables in Pegen ↵Miss Islington (bot)2020-05-241-2/+2
| | | | | | | (GH-20367) (cherry picked from commit cba503151056b448b7a3730dc36ef6655550ade5) Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
* Fix debug output in PEG parser generator (GH-20308)Miss Islington (bot)2020-05-221-3/+3
| | | | | (cherry picked from commit b831129123dbb4bfba49824ae893448b05398f27) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* [3.9] Fix typing problems reported by mypy in pegen (GH-20297) (GH-20300)Pablo Galindo2020-05-212-28/+27
| | | | | (cherry picked from commit d10fef35c6ce8a3193b974be7e8c8304b1146153) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40669: Use requirements.pip when installing PEG dependencies (GH-20194)Miss Islington (bot)2020-05-191-1/+1
| | | | | (cherry picked from commit 3764069f3ba2a7e932837ae19265059339dc86e3) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40669: Install PEG benchmarking dependencies in a venv (GH-20183)Lysandros Nikolaou2020-05-183-15/+29
| | | | | | | | Create a `make venv` target, that creates a virtual environment and installs the dependency in that venv. `make time` and all the related targets are changed to use the virtual environment python. Automerge-Triggered-By: @pablogsal
* bpo-40661: Fix segfault when parsing invalid input (GH-20165)Lysandros Nikolaou2020-05-181-8/+9
| | | | | | Fix segfaults when parsing very complex invalid input, like `import äˆ ð£„¯ð¢·žð±‹á”€ð””ð‘©±å®ä±¬ð©¾\n𗶽`. Co-authored-by: Guido van Rossum <guido@python.org> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
* bpo-40334: Improvements to error-handling code in the PEG parser (GH-20003)Lysandros Nikolaou2020-05-171-21/+12
| | | | | | | | The following improvements are implemented in this commit: - `p->error_indicator` is set, in case malloc or realloc fail. - Avoid memory leaks in the case that realloc fails. - Call `PyErr_NoMemory()` instead of `PyErr_Format()`, because it requires no memory. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-40334: Avoid collisions between parser variables and grammar variables ↵Pablo Galindo2020-05-102-78/+93
| | | | | | | | | | | | | | | (GH-19987) This is for the C generator: - Disallow rule and variable names starting with `_` - Rename most local variable names generated by the parser to start with `_` Exceptions: - Renaming `p` to `_p` will be a separate PR - There are still some names that might clash, e.g. - anything starting with `Py` - C reserved words (`if` etc.) - Macros like `EXTRA` and `CHECK`
* bpo-40555: Check for p->error_indicator in loop rules after the main loop is ↵Pablo Galindo2020-05-081-1/+1
| | | | done (GH-19986)
* bpo-40334: Generate comments in the parser code to improve debugging (GH-19966)Pablo Galindo2020-05-061-0/+14
|
* Clean up unused imports for the peg generator module (GH-19891)Anthony Shaw2020-05-049-17/+4
|
* bpo-40334: regenerate metaparser as part of regen-all (GH-19854)Pablo Galindo2020-05-021-1/+1
|
* Fix some scripts in the peg generator folder (GH-19853)Pablo Galindo2020-05-029-16/+16107
|
* bpo-40334: use the TOKENS file when checking dangling rules (GH-19849)Pablo Galindo2020-05-015-15/+31
|
* Fix the Tools/peg_generator/scripts/benchmark.py script (GH-19848)Pablo Galindo2020-05-012-3/+4
|
* bpo-40334: Simplify type handling in the PEG c_generator (GH-19818)Pablo Galindo2020-05-012-35/+25
|
* bpo-40334: refactor and cleanup for the PEG generators (GH-19775)Pablo Galindo2020-04-294-160/+269
|
* bpo-40334: Refactor peg_generator to receive a Tokens file when building c ↵Pablo Galindo2020-04-286-88/+215
| | | | code (GH-19745)
* bpo-40334: Support CO_FUTURE_BARRY_AS_BDFL in the new parser (GH-19721)Pablo Galindo2020-04-272-4/+13
| | | This commit also allows to pass flags to the new parser in all interfaces and fixes a bug in the parser generator that was causing to inline rules with actions, making them disappear.
* bpo-40334: Rewrite test_c_parser to avoid memory leaks (GH-19694)Lysandros Nikolaou2020-04-242-3/+2
| | | | | | | | | | Previously every test was building an extension module and loading it into sys.modules. The tearDown function was thus not able to clean up correctly, resulting in memory leaks. With this commit, every test function now builds the extension module and runs the actual test code in a new process (using assert_python_ok), so that sys.modules stays intact and no memory gets leaked.
* bpo-40334: Allow to run make regen-pegen without distutils (GH-19684)Pablo Galindo2020-04-232-11/+12
|
* bpo-40370: Use the same compile and link args as the interpreter used in ↵Pablo Galindo2020-04-231-3/+12
| | | | test_peg_generator (GH-19674)
* bpo-40334: Fix build errors and warnings in test_peg_generator (GH-19672)Pablo Galindo2020-04-232-1/+5
|
* Compile extensions in test_peg_generator with C99 (GH-19668)Pablo Galindo2020-04-231-0/+3
|
* bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503)Pablo Galindo2020-04-2234-0/+4796
Co-authored-by: Guido van Rossum <guido@python.org> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
generated by cgit v0.12 at 2026-01-04 23:09:56 (GMT)