summaryrefslogtreecommitdiffstats
path: root/Parser/tokenizer.h
Commit message (Collapse)AuthorAgeFilesLines
* [3.11] Fix typo in the tokenizer (GH-104950) (#104952)Miss Islington (bot)2023-05-261-1/+1
| | | | | (cherry picked from commit 705e387dd81b971cb1ee5727da54adfb565f61d0) Co-authored-by: Stepfen Shawn <m18824909883@163.com>
* [3.11] gh-99891: Fix infinite recursion in the tokenizer when showing ↵Pablo Galindo Salgado2022-12-011-0/+1
| | | | | | | | warnings (GH-99893) (GH-99896) Automerge-Triggered-By: GH:pablogsal. (cherry picked from commit 417206a05c4545bde96c2bbbea92b53e6cac0d48) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* Ensure the str member of the tokenizer is always initialised (GH-29681)Pablo Galindo Salgado2021-11-211-1/+1
|
* bpo-45434: Mark the PyTokenizer C API as private (GH-28924)Victor Stinner2021-10-131-5/+5
| | | | | | | | | | | | | | Rename PyTokenize functions to mark them as private: * PyTokenizer_FindEncodingFilename() => _PyTokenizer_FindEncodingFilename() * PyTokenizer_FromString() => _PyTokenizer_FromString() * PyTokenizer_FromFile() => _PyTokenizer_FromFile() * PyTokenizer_FromUTF8() => _PyTokenizer_FromUTF8() * PyTokenizer_Free() => _PyTokenizer_Free() * PyTokenizer_Get() => _PyTokenizer_Get() Remove the unused PyTokenizer_FindEncoding() function. import.c: remove unused #include "errcode.h".
* Fix typos in the Objects directory (GH-28766)Christian Clauss2021-10-061-1/+1
|
* bpo-44854: Remove trailing whitespaces (GH-27689)Serhiy Storchaka2021-08-091-1/+1
|
* bpo-44201: Avoid side effects of "invalid_*" rules in the REPL (GH-26298)Pablo Galindo2021-05-221-0/+10
| | | | | | | | | | | | When the parser does a second pass to check for errors, these rules can have some small side-effects as they may advance the parser more than the point reached in the first pass. This can cause the tokenizer to ask for extra tokens in interactive mode causing the tokenizer to show the prompt instead of failing instantly. To avoid this, add a new mode to the tokenizer that is activated in the second pass and deactivates asking for new tokens when the interactive line is finished. As the parsing should have reached the last line in the first pass, the second pass should not need to ask for more tokens.
* bpo-25643: Refactor the C tokenizer into smaller, logical units (GH-25050)Pablo Galindo2021-03-281-3/+2
|
* bpo-43410: Fix crash in the parser when producing syntax errors when reading ↵Pablo Galindo2021-03-141-1/+3
| | | | from stdin (GH-24763)
* bpo-42864: Improve error messages regarding unclosed parentheses (GH-24161)Pablo Galindo2021-01-191-0/+1
|
* bpo-42827: Fix crash on SyntaxError in multiline expressions (GH-24140)Lysandros Nikolaou2021-01-141-0/+1
| | | | | | | | | | | | | | | | | | | | | | | When trying to extract the error line for the error message there are two distinct cases: 1. The input comes from a file, which means that we can extract the error line by using `PyErr_ProgramTextObject` and which we already do. 2. The input does not come from a file, at which point we need to get the source code from the tokenizer: * If the tokenizer's current line number is the same with the line of the error, we get the line from `tok->buf` and we're ready. * Else, we can extract the error line from the source code in the following two ways: * If the input comes from a string we have all the input in `tok->str` and we can extract the error line from it. * If the input comes from stdin, i.e. the interactive prompt, we do not have access to the previous line. That's why a new field `tok->stdin_content` is added which holds the whole input for the current (multiline) statement or expression. We can then extract the error line from `tok->stdin_content` like we do in the string case above. Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* closes bpo-39721: Fix constness of members of tok_state struct. (GH-18600)Andy Lester2020-02-281-5/+5
| | | | | | | | | | | | | | | | | | | | | The function PyTokenizer_FromUTF8 from Parser/tokenizer.c had a comment: /* XXX: constify members. */ This patch addresses that. In the tok_state struct: * end and start were non-const but could be made const * str and input were const but should have been non-const Changes to support this include: * decode_str() now returns a char * since it is allocated. * PyTokenizer_FromString() and PyTokenizer_FromUTF8() each creates a new char * for an allocate string instead of reusing the input const char *. * PyTokenizer_Get() and tok_get() now take const char ** arguments. * Various local vars are const or non-const accordingly. I was able to remove five casts that cast away constness.
* bpo-36623: Clean parser headers and include files (GH-12253)Pablo Galindo2019-04-131-0/+2
| | | After the removal of pgen, multiple header and function prototypes that lack implementation or are unused are still lying around.
* bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)Guido van Rossum2019-03-071-0/+7
| | | | | | | This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.) https://bugs.python.org/issue35975
* bpo-35808: Retire pgen and use pgen2 to generate the parser (GH-11814)Pablo Galindo2019-03-011-8/+0
| | | | | Pgen is the oldest piece of technology in the CPython repository, building it requires various #if[n]def PGEN hacks in other parts of the code and it also depends more and more on CPython internals. This commit removes the old pgen C code and replaces it for a new version implemented in pure Python. This is a modified and adapted version of lib2to3/pgen2 that can generate grammar files compatibles with the current parser. This commit also eliminates all the #ifdef and code branches related to pgen, simplifying the code and making it more maintainable. The regen-grammar step now uses $(PYTHON_FOR_REGEN) that can be any version of the interpreter, so the new pgen code maintains compatibility with older versions of the interpreter (this also allows regenerating the grammar with the current CI solution that uses Python3.5). The new pgen Python module also makes use of the Grammar/Tokens file that holds the token specification, so is always kept in sync and avoids having to maintain duplicate token definitions.
* bpo-35766: Merge typed_ast back into CPython (GH-11645)Guido van Rossum2019-01-311-0/+2
|
* bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)Anthony Sottile2019-01-131-0/+5
|
* bpo-33306: Improve SyntaxError messages for unbalanced parentheses. (GH-6516)Serhiy Storchaka2018-12-171-1/+4
|
* tokenizer: Remove unused tabs options (#4422)Victor Stinner2017-11-171-3/+0
| | | | | | | | | | Remove the following fields from tok_state structure which are now used unused: * altwarning: "Issue warning if alternate tabs don't match" * alterror: "Issue error if alternate tabs don't match" * alttabsize: "Alternate tab spacing" Replace alttabsize variable with ALTTABSIZE define.
* bpo-30406: Make async and await proper keywords (#1669)Jelle Zijlstra2017-10-061-7/+0
| | | Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
* Remove obsolete declaration in tokenizer.h (#962)Jim Fasarakis-Hilliard2017-04-031-2/+0
|
* Issue #24619: Simplify async/await tokenization.Yury Selivanov2015-07-231-15/+6
| | | | | | | | | | This commit simplifies async/await tokenization in tokenizer.c, tokenize.py & lib2to3/tokenize.py. Previous solution was to keep a stack of async-def & def blocks, whereas the new approach is just to remember position of the outermost async-def block. This change won't bring any parsing performance improvements, but it makes the code much easier to read and validate.
* Issue #24619: New approach for tokenizing async/await.Yury Selivanov2015-07-221-6/+15
| | | | | | | | | | | | | | | | | | | | | This commit fixes how one-line async-defs and defs are tracked by tokenizer. It allows to correctly parse invalid code such as: >>> async def f(): ... def g(): pass ... async = 10 and valid code such as: >>> async def f(): ... async def g(): pass ... await z As a consequence, is is now possible to have one-line 'async def foo(): await ..' functions: >>> async def foo(): return await bar()
* PEP 0492 -- Coroutines with async and await syntax. Issue #24017.Yury Selivanov2015-05-121-0/+7
|
* Issue #1772673: The type of `char*` arguments now changed to `const char*`.Serhiy Storchaka2013-10-191-3/+3
|
* Issue #9319: Include the filename in "Non-UTF8 code ..." syntax error.Victor Stinner2011-04-041-1/+0
|
* Issue #10785: Store the filename as Unicode in the Python parser.Victor Stinner2011-04-041-1/+7
|
* #10222: fix for overzealous AIX compiler.Georg Brandl2010-10-291-1/+1
|
* Issue #9713, #10114: Parser functions (eg. PyParser_ASTFromFile) expectsVictor Stinner2010-10-161-1/+1
| | | | | filenames encoded to the filesystem encoding with surrogateescape error handler (to support undecodable bytes), instead of UTF-8 in strict mode.
* Issue #10095: fp_setreadl() doesn't reopen the file, reuse instead the fileVictor Stinner2010-10-141-1/+1
| | | | descriptor.
* Recorded merge of revisions 81029 via svnmerge fromAntoine Pitrou2010-05-091-45/+45
| | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r81029 | antoine.pitrou | 2010-05-09 16:46:46 +0200 (dim., 09 mai 2010) | 3 lines Untabify C files. Will watch buildbots. ........
* Merged revisions 76230 via svnmerge fromBenjamin Peterson2009-11-131-2/+3
| | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r76230 | benjamin.peterson | 2009-11-12 17:39:44 -0600 (Thu, 12 Nov 2009) | 2 lines fix several compile() issues by translating newlines in the tokenizer ........
* ignore the coding cookie in compile(), exec(), and eval() if the source is a ↵Benjamin Peterson2009-03-021-0/+1
| | | | string #4626
* Latin-1 source code was not being properly decoded when passed throughBrett Cannon2008-10-171-2/+2
| | | | | | | compile(). This was due to left-over special-casing before UTF-8 became the default source encoding. Closes issue #3574. Thanks to Victor Stinner for help with the patch.
* Issue 1267, continued.Guido van Rossum2007-10-221-1/+1
| | | | | | Additional patch by Christian Heimes to deal more cleanly with the FILE* vs file-descriptor issues. I cleaned up his code a bit, and moved the lseek() call into import.c.
* Patch 1267 by Christian Heimes.Guido van Rossum2007-10-191-0/+1
| | | | | | Move the initialization of sys.std{in,out,err} and __builtin__.open to C code. This solves the problem that "python -S" wouldn't work.
* Use an enum for decoding_state. It makes the code a little moreNeil Schemenauer2007-09-211-1/+7
| | | | understandable.
* Merged revisions 57778-58052 via svnmerge fromThomas Wouters2007-09-081-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r57820 | georg.brandl | 2007-08-31 08:59:27 +0200 (Fri, 31 Aug 2007) | 2 lines Document new shorthand notation for index entries. ........ r57827 | georg.brandl | 2007-08-31 10:47:51 +0200 (Fri, 31 Aug 2007) | 2 lines Fix subitem markup. ........ r57833 | martin.v.loewis | 2007-08-31 12:01:07 +0200 (Fri, 31 Aug 2007) | 1 line Mark registry components as 64-bit on Win64. ........ r57854 | bill.janssen | 2007-08-31 21:02:23 +0200 (Fri, 31 Aug 2007) | 1 line deprecate use of FakeSocket ........ r57855 | bill.janssen | 2007-08-31 21:02:46 +0200 (Fri, 31 Aug 2007) | 1 line remove mentions of socket.ssl in comments ........ r57856 | bill.janssen | 2007-08-31 21:03:31 +0200 (Fri, 31 Aug 2007) | 1 line remove use of non-existent SSLFakeSocket in apparently untested code ........ r57859 | martin.v.loewis | 2007-09-01 08:36:03 +0200 (Sat, 01 Sep 2007) | 3 lines Bug #1737210: Change Manufacturer of Windows installer to PSF. Will backport to 2.5. ........ r57865 | georg.brandl | 2007-09-01 09:51:24 +0200 (Sat, 01 Sep 2007) | 2 lines Fix RST link (backport from Py3k). ........ r57876 | georg.brandl | 2007-09-01 17:49:49 +0200 (Sat, 01 Sep 2007) | 2 lines Document sets' ">" and "<" operations (backport from py3k). ........ r57878 | skip.montanaro | 2007-09-01 19:40:03 +0200 (Sat, 01 Sep 2007) | 4 lines Added a note and examples to explain that re.split does not split on an empty pattern match. (issue 852532). ........ r57879 | walter.doerwald | 2007-09-01 20:18:09 +0200 (Sat, 01 Sep 2007) | 2 lines Fix wrong function names. ........ r57880 | walter.doerwald | 2007-09-01 20:34:05 +0200 (Sat, 01 Sep 2007) | 2 lines Fix typo. ........ r57889 | andrew.kuchling | 2007-09-01 22:31:59 +0200 (Sat, 01 Sep 2007) | 1 line Markup fix ........ r57892 | andrew.kuchling | 2007-09-01 22:43:36 +0200 (Sat, 01 Sep 2007) | 1 line Add various items ........ r57895 | andrew.kuchling | 2007-09-01 23:17:58 +0200 (Sat, 01 Sep 2007) | 1 line Wording change ........ r57896 | andrew.kuchling | 2007-09-01 23:18:31 +0200 (Sat, 01 Sep 2007) | 1 line Add more items ........ r57904 | ronald.oussoren | 2007-09-02 11:46:07 +0200 (Sun, 02 Sep 2007) | 3 lines Macosx: this patch ensures that the value of MACOSX_DEPLOYMENT_TARGET used by the Makefile is also used at configure-time. ........ r57925 | georg.brandl | 2007-09-03 09:16:46 +0200 (Mon, 03 Sep 2007) | 2 lines Fix #883466: don't allow Unicode as arguments to quopri and uu codecs. ........ r57936 | matthias.klose | 2007-09-04 01:33:04 +0200 (Tue, 04 Sep 2007) | 2 lines - Added support for linking the bsddb module against BerkeleyDB 4.6.x. ........ r57954 | mark.summerfield | 2007-09-04 10:16:15 +0200 (Tue, 04 Sep 2007) | 3 lines Added cross-references plus a note about dict & list shallow copying. ........ r57958 | martin.v.loewis | 2007-09-04 11:51:57 +0200 (Tue, 04 Sep 2007) | 3 lines Document that we rely on the OS to release the crypto context. Fixes #1626801. ........ r57960 | martin.v.loewis | 2007-09-04 15:13:14 +0200 (Tue, 04 Sep 2007) | 3 lines Patch #1388440: Add set_completion_display_matches_hook and get_completion_type to readline. ........ r57961 | martin.v.loewis | 2007-09-04 16:19:28 +0200 (Tue, 04 Sep 2007) | 3 lines Patch #1031213: Decode source line in SyntaxErrors back to its original source encoding. Will backport to 2.5. ........ r57972 | matthias.klose | 2007-09-04 20:17:36 +0200 (Tue, 04 Sep 2007) | 3 lines - Makefile.pre.in(buildbottest): Run an optional script pybuildbot.identify to include some information about the build environment. ........ r57973 | matthias.klose | 2007-09-04 21:05:38 +0200 (Tue, 04 Sep 2007) | 2 lines - Makefile.pre.in(buildbottest): Remove whitespace at eol. ........ r57975 | matthias.klose | 2007-09-04 22:46:02 +0200 (Tue, 04 Sep 2007) | 2 lines - Fix libffi configure for hppa*-*-linux* | parisc*-*-linux*. ........ r57980 | bill.janssen | 2007-09-05 02:46:27 +0200 (Wed, 05 Sep 2007) | 1 line SSL certificate distinguished names should be represented by tuples ........ r57985 | martin.v.loewis | 2007-09-05 08:39:17 +0200 (Wed, 05 Sep 2007) | 3 lines Patch #1105: Explain that one needs to build the solution to get dependencies right. ........ r57987 | armin.rigo | 2007-09-05 09:51:21 +0200 (Wed, 05 Sep 2007) | 4 lines PyDict_GetItem() returns a borrowed reference. There are probably a number of places that are open to attacks such as the following one, in bltinmodule.c:min_max(). ........ r57991 | martin.v.loewis | 2007-09-05 13:47:34 +0200 (Wed, 05 Sep 2007) | 3 lines Patch #786737: Allow building in a tree of symlinks pointing to a readonly source. ........ r57993 | georg.brandl | 2007-09-05 15:36:44 +0200 (Wed, 05 Sep 2007) | 2 lines Backport from Py3k: Bug #1684991: explain lookup semantics for __special__ methods (new-style classes only). ........ r58004 | armin.rigo | 2007-09-06 10:30:51 +0200 (Thu, 06 Sep 2007) | 4 lines Patch #1733973 by peaker: ptrace_enter_call() assumes no exception is currently set. This assumption is broken when throwing into a generator. ........ r58006 | armin.rigo | 2007-09-06 11:30:38 +0200 (Thu, 06 Sep 2007) | 4 lines PyDict_GetItem() returns a borrowed reference. This attack is against ceval.c:IMPORT_NAME, which calls an object (__builtin__.__import__) without holding a reference to it. ........ r58013 | georg.brandl | 2007-09-06 16:49:56 +0200 (Thu, 06 Sep 2007) | 2 lines Backport from 3k: #1116: fix reference to old filename. ........ r58021 | thomas.heller | 2007-09-06 22:26:20 +0200 (Thu, 06 Sep 2007) | 1 line Fix typo: c_float represents to C float type. ........ r58022 | skip.montanaro | 2007-09-07 00:29:06 +0200 (Fri, 07 Sep 2007) | 3 lines If this is correct for py3k branch and it's already in the release25-maint branch, seems like it ought to be on the trunk as well. ........ r58023 | gregory.p.smith | 2007-09-07 00:59:59 +0200 (Fri, 07 Sep 2007) | 4 lines Apply the fix from Issue1112 to make this test more robust and keep windows happy. ........ r58031 | brett.cannon | 2007-09-07 05:17:50 +0200 (Fri, 07 Sep 2007) | 4 lines Make uuid1 and uuid4 tests conditional on whether ctypes can be imported; implementation of either function depends on ctypes but uuid as a whole does not. ........ r58032 | brett.cannon | 2007-09-07 06:18:30 +0200 (Fri, 07 Sep 2007) | 6 lines Fix a crasher where Python code managed to infinitely recurse in C code without ever going back out to Python code in PyObject_Call(). Required introducing a static RuntimeError instance so that normalizing an exception there is no reliance on a recursive call that would put the exception system over the recursion check itself. ........ r58034 | thomas.heller | 2007-09-07 08:32:17 +0200 (Fri, 07 Sep 2007) | 1 line Add a 'c_longdouble' type to the ctypes module. ........ r58035 | thomas.heller | 2007-09-07 11:30:40 +0200 (Fri, 07 Sep 2007) | 1 line Remove unneeded #include. ........ r58036 | thomas.heller | 2007-09-07 11:33:24 +0200 (Fri, 07 Sep 2007) | 6 lines Backport from py3k branch: Add a workaround for a strange bug on win64, when _ctypes is compiled with the SDK compiler. This should fix the failing Lib\ctypes\test\test_as_parameter.py test. ........ r58037 | georg.brandl | 2007-09-07 16:14:40 +0200 (Fri, 07 Sep 2007) | 2 lines Fix a wrong indentation for sublists. ........ r58043 | georg.brandl | 2007-09-07 22:10:49 +0200 (Fri, 07 Sep 2007) | 2 lines #1095: ln -f doesn't work portably, fix in Makefile. ........ r58049 | skip.montanaro | 2007-09-08 02:34:17 +0200 (Sat, 08 Sep 2007) | 1 line be explicit about the actual location of the missing file ........
* Convert code from sys.stdin.encoding to UTF-8 inMartin v. Löwis2007-09-041-1/+2
| | | | interactive mode. Fixes #1100.
* Patch #1440601: Add col_offset attribute to AST nodes.Martin v. Löwis2006-03-011-0/+1
|
* Remove unused field.Martin v. Löwis2006-03-011-1/+0
|
* Constify filenames and scripts. Fixes #651362.Martin v. Löwis2002-12-111-2/+2
|
* Ignore encoding declarations inside strings. Fixes #603509.Martin v. Löwis2002-09-031-0/+1
|
* Make pgen compile with pydebug. Duplicate normalized names, as it mayMartin v. Löwis2002-08-041-0/+2
| | | | be longer than the old string.
* Patch #534304: Implement phase 1 of PEP 263.Martin v. Löwis2002-08-041-0/+11
|
* REMOVED all CWI, CNRI and BeOpen copyright markings.Guido van Rossum2000-09-011-9/+0
| | | | This should match the situation in the 1.6b1 tree.
* Nuke all remaining occurrences of Py_PROTO and Py_FPROTO.Tim Peters2000-07-091-5/+4
|
* Change copyright notice - 2nd try.Guido van Rossum2000-06-301-6/+0
|
* Change copyright notice.Guido van Rossum2000-06-301-21/+6
|
* Add checking for inconsistent tab usageGuido van Rossum1998-04-091-0/+6
|
generated by cgit v0.12 at 2024-11-23 20:54:15 (GMT)