summaryrefslogtreecommitdiffstats
path: root/Parser/tokenizer.h
Commit message (Collapse)AuthorAgeFilesLines
* Issue #24619: Simplify async/await tokenization.Yury Selivanov2015-07-231-15/+6
| | | | | | | | | | This commit simplifies async/await tokenization in tokenizer.c, tokenize.py & lib2to3/tokenize.py. Previous solution was to keep a stack of async-def & def blocks, whereas the new approach is just to remember position of the outermost async-def block. This change won't bring any parsing performance improvements, but it makes the code much easier to read and validate.
* Issue #24619: New approach for tokenizing async/await.Yury Selivanov2015-07-221-6/+15
| | | | | | | | | | | | | | | | | | | | | This commit fixes how one-line async-defs and defs are tracked by tokenizer. It allows to correctly parse invalid code such as: >>> async def f(): ... def g(): pass ... async = 10 and valid code such as: >>> async def f(): ... async def g(): pass ... await z As a consequence, is is now possible to have one-line 'async def foo(): await ..' functions: >>> async def foo(): return await bar()
* PEP 0492 -- Coroutines with async and await syntax. Issue #24017.Yury Selivanov2015-05-121-0/+7
|
* Issue #1772673: The type of `char*` arguments now changed to `const char*`.Serhiy Storchaka2013-10-191-3/+3
|
* Issue #9319: Include the filename in "Non-UTF8 code ..." syntax error.Victor Stinner2011-04-041-1/+0
|
* Issue #10785: Store the filename as Unicode in the Python parser.Victor Stinner2011-04-041-1/+7
|
* #10222: fix for overzealous AIX compiler.Georg Brandl2010-10-291-1/+1
|
* Issue #9713, #10114: Parser functions (eg. PyParser_ASTFromFile) expectsVictor Stinner2010-10-161-1/+1
| | | | | filenames encoded to the filesystem encoding with surrogateescape error handler (to support undecodable bytes), instead of UTF-8 in strict mode.
* Issue #10095: fp_setreadl() doesn't reopen the file, reuse instead the fileVictor Stinner2010-10-141-1/+1
| | | | descriptor.
* Recorded merge of revisions 81029 via svnmerge fromAntoine Pitrou2010-05-091-45/+45
| | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r81029 | antoine.pitrou | 2010-05-09 16:46:46 +0200 (dim., 09 mai 2010) | 3 lines Untabify C files. Will watch buildbots. ........
* Merged revisions 76230 via svnmerge fromBenjamin Peterson2009-11-131-2/+3
| | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r76230 | benjamin.peterson | 2009-11-12 17:39:44 -0600 (Thu, 12 Nov 2009) | 2 lines fix several compile() issues by translating newlines in the tokenizer ........
* ignore the coding cookie in compile(), exec(), and eval() if the source is a ↵Benjamin Peterson2009-03-021-0/+1
| | | | string #4626
* Latin-1 source code was not being properly decoded when passed throughBrett Cannon2008-10-171-2/+2
| | | | | | | compile(). This was due to left-over special-casing before UTF-8 became the default source encoding. Closes issue #3574. Thanks to Victor Stinner for help with the patch.
* Issue 1267, continued.Guido van Rossum2007-10-221-1/+1
| | | | | | Additional patch by Christian Heimes to deal more cleanly with the FILE* vs file-descriptor issues. I cleaned up his code a bit, and moved the lseek() call into import.c.
* Patch 1267 by Christian Heimes.Guido van Rossum2007-10-191-0/+1
| | | | | | Move the initialization of sys.std{in,out,err} and __builtin__.open to C code. This solves the problem that "python -S" wouldn't work.
* Use an enum for decoding_state. It makes the code a little moreNeil Schemenauer2007-09-211-1/+7
| | | | understandable.
* Merged revisions 57778-58052 via svnmerge fromThomas Wouters2007-09-081-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r57820 | georg.brandl | 2007-08-31 08:59:27 +0200 (Fri, 31 Aug 2007) | 2 lines Document new shorthand notation for index entries. ........ r57827 | georg.brandl | 2007-08-31 10:47:51 +0200 (Fri, 31 Aug 2007) | 2 lines Fix subitem markup. ........ r57833 | martin.v.loewis | 2007-08-31 12:01:07 +0200 (Fri, 31 Aug 2007) | 1 line Mark registry components as 64-bit on Win64. ........ r57854 | bill.janssen | 2007-08-31 21:02:23 +0200 (Fri, 31 Aug 2007) | 1 line deprecate use of FakeSocket ........ r57855 | bill.janssen | 2007-08-31 21:02:46 +0200 (Fri, 31 Aug 2007) | 1 line remove mentions of socket.ssl in comments ........ r57856 | bill.janssen | 2007-08-31 21:03:31 +0200 (Fri, 31 Aug 2007) | 1 line remove use of non-existent SSLFakeSocket in apparently untested code ........ r57859 | martin.v.loewis | 2007-09-01 08:36:03 +0200 (Sat, 01 Sep 2007) | 3 lines Bug #1737210: Change Manufacturer of Windows installer to PSF. Will backport to 2.5. ........ r57865 | georg.brandl | 2007-09-01 09:51:24 +0200 (Sat, 01 Sep 2007) | 2 lines Fix RST link (backport from Py3k). ........ r57876 | georg.brandl | 2007-09-01 17:49:49 +0200 (Sat, 01 Sep 2007) | 2 lines Document sets' ">" and "<" operations (backport from py3k). ........ r57878 | skip.montanaro | 2007-09-01 19:40:03 +0200 (Sat, 01 Sep 2007) | 4 lines Added a note and examples to explain that re.split does not split on an empty pattern match. (issue 852532). ........ r57879 | walter.doerwald | 2007-09-01 20:18:09 +0200 (Sat, 01 Sep 2007) | 2 lines Fix wrong function names. ........ r57880 | walter.doerwald | 2007-09-01 20:34:05 +0200 (Sat, 01 Sep 2007) | 2 lines Fix typo. ........ r57889 | andrew.kuchling | 2007-09-01 22:31:59 +0200 (Sat, 01 Sep 2007) | 1 line Markup fix ........ r57892 | andrew.kuchling | 2007-09-01 22:43:36 +0200 (Sat, 01 Sep 2007) | 1 line Add various items ........ r57895 | andrew.kuchling | 2007-09-01 23:17:58 +0200 (Sat, 01 Sep 2007) | 1 line Wording change ........ r57896 | andrew.kuchling | 2007-09-01 23:18:31 +0200 (Sat, 01 Sep 2007) | 1 line Add more items ........ r57904 | ronald.oussoren | 2007-09-02 11:46:07 +0200 (Sun, 02 Sep 2007) | 3 lines Macosx: this patch ensures that the value of MACOSX_DEPLOYMENT_TARGET used by the Makefile is also used at configure-time. ........ r57925 | georg.brandl | 2007-09-03 09:16:46 +0200 (Mon, 03 Sep 2007) | 2 lines Fix #883466: don't allow Unicode as arguments to quopri and uu codecs. ........ r57936 | matthias.klose | 2007-09-04 01:33:04 +0200 (Tue, 04 Sep 2007) | 2 lines - Added support for linking the bsddb module against BerkeleyDB 4.6.x. ........ r57954 | mark.summerfield | 2007-09-04 10:16:15 +0200 (Tue, 04 Sep 2007) | 3 lines Added cross-references plus a note about dict & list shallow copying. ........ r57958 | martin.v.loewis | 2007-09-04 11:51:57 +0200 (Tue, 04 Sep 2007) | 3 lines Document that we rely on the OS to release the crypto context. Fixes #1626801. ........ r57960 | martin.v.loewis | 2007-09-04 15:13:14 +0200 (Tue, 04 Sep 2007) | 3 lines Patch #1388440: Add set_completion_display_matches_hook and get_completion_type to readline. ........ r57961 | martin.v.loewis | 2007-09-04 16:19:28 +0200 (Tue, 04 Sep 2007) | 3 lines Patch #1031213: Decode source line in SyntaxErrors back to its original source encoding. Will backport to 2.5. ........ r57972 | matthias.klose | 2007-09-04 20:17:36 +0200 (Tue, 04 Sep 2007) | 3 lines - Makefile.pre.in(buildbottest): Run an optional script pybuildbot.identify to include some information about the build environment. ........ r57973 | matthias.klose | 2007-09-04 21:05:38 +0200 (Tue, 04 Sep 2007) | 2 lines - Makefile.pre.in(buildbottest): Remove whitespace at eol. ........ r57975 | matthias.klose | 2007-09-04 22:46:02 +0200 (Tue, 04 Sep 2007) | 2 lines - Fix libffi configure for hppa*-*-linux* | parisc*-*-linux*. ........ r57980 | bill.janssen | 2007-09-05 02:46:27 +0200 (Wed, 05 Sep 2007) | 1 line SSL certificate distinguished names should be represented by tuples ........ r57985 | martin.v.loewis | 2007-09-05 08:39:17 +0200 (Wed, 05 Sep 2007) | 3 lines Patch #1105: Explain that one needs to build the solution to get dependencies right. ........ r57987 | armin.rigo | 2007-09-05 09:51:21 +0200 (Wed, 05 Sep 2007) | 4 lines PyDict_GetItem() returns a borrowed reference. There are probably a number of places that are open to attacks such as the following one, in bltinmodule.c:min_max(). ........ r57991 | martin.v.loewis | 2007-09-05 13:47:34 +0200 (Wed, 05 Sep 2007) | 3 lines Patch #786737: Allow building in a tree of symlinks pointing to a readonly source. ........ r57993 | georg.brandl | 2007-09-05 15:36:44 +0200 (Wed, 05 Sep 2007) | 2 lines Backport from Py3k: Bug #1684991: explain lookup semantics for __special__ methods (new-style classes only). ........ r58004 | armin.rigo | 2007-09-06 10:30:51 +0200 (Thu, 06 Sep 2007) | 4 lines Patch #1733973 by peaker: ptrace_enter_call() assumes no exception is currently set. This assumption is broken when throwing into a generator. ........ r58006 | armin.rigo | 2007-09-06 11:30:38 +0200 (Thu, 06 Sep 2007) | 4 lines PyDict_GetItem() returns a borrowed reference. This attack is against ceval.c:IMPORT_NAME, which calls an object (__builtin__.__import__) without holding a reference to it. ........ r58013 | georg.brandl | 2007-09-06 16:49:56 +0200 (Thu, 06 Sep 2007) | 2 lines Backport from 3k: #1116: fix reference to old filename. ........ r58021 | thomas.heller | 2007-09-06 22:26:20 +0200 (Thu, 06 Sep 2007) | 1 line Fix typo: c_float represents to C float type. ........ r58022 | skip.montanaro | 2007-09-07 00:29:06 +0200 (Fri, 07 Sep 2007) | 3 lines If this is correct for py3k branch and it's already in the release25-maint branch, seems like it ought to be on the trunk as well. ........ r58023 | gregory.p.smith | 2007-09-07 00:59:59 +0200 (Fri, 07 Sep 2007) | 4 lines Apply the fix from Issue1112 to make this test more robust and keep windows happy. ........ r58031 | brett.cannon | 2007-09-07 05:17:50 +0200 (Fri, 07 Sep 2007) | 4 lines Make uuid1 and uuid4 tests conditional on whether ctypes can be imported; implementation of either function depends on ctypes but uuid as a whole does not. ........ r58032 | brett.cannon | 2007-09-07 06:18:30 +0200 (Fri, 07 Sep 2007) | 6 lines Fix a crasher where Python code managed to infinitely recurse in C code without ever going back out to Python code in PyObject_Call(). Required introducing a static RuntimeError instance so that normalizing an exception there is no reliance on a recursive call that would put the exception system over the recursion check itself. ........ r58034 | thomas.heller | 2007-09-07 08:32:17 +0200 (Fri, 07 Sep 2007) | 1 line Add a 'c_longdouble' type to the ctypes module. ........ r58035 | thomas.heller | 2007-09-07 11:30:40 +0200 (Fri, 07 Sep 2007) | 1 line Remove unneeded #include. ........ r58036 | thomas.heller | 2007-09-07 11:33:24 +0200 (Fri, 07 Sep 2007) | 6 lines Backport from py3k branch: Add a workaround for a strange bug on win64, when _ctypes is compiled with the SDK compiler. This should fix the failing Lib\ctypes\test\test_as_parameter.py test. ........ r58037 | georg.brandl | 2007-09-07 16:14:40 +0200 (Fri, 07 Sep 2007) | 2 lines Fix a wrong indentation for sublists. ........ r58043 | georg.brandl | 2007-09-07 22:10:49 +0200 (Fri, 07 Sep 2007) | 2 lines #1095: ln -f doesn't work portably, fix in Makefile. ........ r58049 | skip.montanaro | 2007-09-08 02:34:17 +0200 (Sat, 08 Sep 2007) | 1 line be explicit about the actual location of the missing file ........
* Convert code from sys.stdin.encoding to UTF-8 inMartin v. Löwis2007-09-041-1/+2
| | | | interactive mode. Fixes #1100.
* Patch #1440601: Add col_offset attribute to AST nodes.Martin v. Löwis2006-03-011-0/+1
|
* Remove unused field.Martin v. Löwis2006-03-011-1/+0
|
* Constify filenames and scripts. Fixes #651362.Martin v. Löwis2002-12-111-2/+2
|
* Ignore encoding declarations inside strings. Fixes #603509.Martin v. Löwis2002-09-031-0/+1
|
* Make pgen compile with pydebug. Duplicate normalized names, as it mayMartin v. Löwis2002-08-041-0/+2
| | | | be longer than the old string.
* Patch #534304: Implement phase 1 of PEP 263.Martin v. Löwis2002-08-041-0/+11
|
* REMOVED all CWI, CNRI and BeOpen copyright markings.Guido van Rossum2000-09-011-9/+0
| | | | This should match the situation in the 1.6b1 tree.
* Nuke all remaining occurrences of Py_PROTO and Py_FPROTO.Tim Peters2000-07-091-5/+4
|
* Change copyright notice - 2nd try.Guido van Rossum2000-06-301-6/+0
|
* Change copyright notice.Guido van Rossum2000-06-301-21/+6
|
* Add checking for inconsistent tab usageGuido van Rossum1998-04-091-0/+6
|
* Another directory quickly renamed.Guido van Rossum1997-04-291-4/+5
|
* New permission notice, includes CNRI.Guido van Rossum1996-10-251-12/+19
|
* Added 1995 to copyright message.Guido van Rossum1995-01-041-2/+2
|
* Merge back to main trunkGuido van Rossum1994-08-301-1/+2
|
* * Grammar: corrected old typo (class instead of 'class')Guido van Rossum1993-12-201-1/+1
| | | | | | | * dosmodule.c: MSDOS specific stuff from posixmodule.c. * posixmodule.c: removed all MSDOS specific stuff. * tokenizer.h, parsetok.h: in prototypes, don't mix named and unnamed parameters (MSC doesn't like this).
* * Added support for X11 modules.Guido van Rossum1993-07-281-0/+11
| | | | | | | * Makefile: change location of FORMS library. * posixmodule.c: turn #if 0 into #ifdef MSDOS (stuff in unistd.h or not) * Almost all .h files: added CPP magic to avoid duplicate inclusions and to support inclusion from C++.
* * selectmodule.c: fix (another!) two memory leaks -- this time in list2setGuido van Rossum1993-05-121-0/+2
| | | | * tokenizer.[ch]: allow continuation without \ inside () [] {}.
* * pythonrun.c: Print exception type+arg *after* stack trace instead ofGuido van Rossum1993-05-121-4/+5
| | | | | | | before it. * ceval.c, object.c: moved testbool() to object.c (now extern visible) * stringobject.c: fix bugs in and rationalize string resize in formatstring() * tokenizer.[ch]: fix non-working code for lines longer than BUFSIZ
* * Changed all copyright messages to include 1993.Guido van Rossum1993-03-291-2/+2
| | | | | | | | | | | | | | | | | * Stubs for faster implementation of local variables (not yet finished) * Added function name to code object. Print it for code and function objects. THIS MAKES THE .PYC FILE FORMAT INCOMPATIBLE (the version number has changed accordingly) * Print address of self for built-in methods * New internal functions getattro and setattro (getattr/setattr with string object arg) * Replaced "dictobject" with more powerful "mappingobject" * New per-type functio tp_hash to implement arbitrary object hashing, and hashobject() to interface to it * Added built-in functions hash(v) and hasattr(v, 'name') * classobject: made some functions static that accidentally weren't; added __hash__ special instance method to implement hash() * Added proper comparison for built-in methods and functions
* Copyright for 1992 addedGuido van Rossum1992-04-051-1/+1
|
* Added copyright notice.Guido van Rossum1991-02-191-0/+24
|
* Initial revisionGuido van Rossum1990-10-141-0/+29
generated by cgit v0.12 at 2024-11-23 12:22:17 (GMT)