summaryrefslogtreecommitdiffstats
path: root/Parser/tokenizer.c
Commit message (Collapse)AuthorAgeFilesLines
* Issue #24619: Simplify async/await tokenization.Yury Selivanov2015-07-231-87/+39
| | | | | | | | | | This commit simplifies async/await tokenization in tokenizer.c, tokenize.py & lib2to3/tokenize.py. Previous solution was to keep a stack of async-def & def blocks, whereas the new approach is just to remember position of the outermost async-def block. This change won't bring any parsing performance improvements, but it makes the code much easier to read and validate.
* Issue #24619: New approach for tokenizing async/await.Yury Selivanov2015-07-221-35/+70
| | | | | | | | | | | | | | | | | | | | | This commit fixes how one-line async-defs and defs are tracked by tokenizer. It allows to correctly parse invalid code such as: >>> async def f(): ... def g(): pass ... async = 10 and valid code such as: >>> async def f(): ... async def g(): pass ... await z As a consequence, is is now possible to have one-line 'async def foo(): await ..' functions: >>> async def foo(): return await bar()
* Issue 24226: Fix parsing of many sequential one-line 'def' statements.Yury Selivanov2015-05-181-8/+17
|
* PEP 0492 -- Coroutines with async and await syntax. Issue #24017.Yury Selivanov2015-05-121-0/+68
|
* merge 3.4 (#24022)Benjamin Peterson2015-04-211-4/+3
|\
| * do not call into python api if an exception is set (#24022)Benjamin Peterson2015-04-211-4/+3
| |
* | merge 3.4 (#21642)Benjamin Peterson2014-06-071-5/+14
|\ \ | |/
| * allow the keyword else immediately after (no space) an integer (closes #21642)Benjamin Peterson2014-06-071-5/+14
| |
* | PEP 465: a dedicated infix operator for matrix multiplication (closes #21176)Benjamin Peterson2014-04-101-1/+7
|/
* Merge with 3.3Martin v. Löwis2014-02-281-2/+12
|\
| * Issue #20731: Properly position in source code files even if theyMartin v. Löwis2014-02-281-2/+12
| | | | | | | | are opened in text mode. Patch by Serhiy Storchaka.
* | Do not reset the line number because we already set file position to correctSerhiy Storchaka2014-01-091-8/+0
|\ \ | |/ | | | | | | | | value. (fixes error in patch for issue #18960)
| * Do not reset the line number because we already set file position to correctSerhiy Storchaka2014-01-091-8/+0
| | | | | | | | | | | | value. (fixes error in patch for issue #18960)
* | Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-4/+22
|\ \ | |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
| * Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-4/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
* | Issue #1772673: The type of `char*` arguments now changed to `const char*`.Serhiy Storchaka2013-10-191-3/+4
| |
* | Issue #18571: Implementation of the PEP 446: file descriptors and file handlesVictor Stinner2013-08-271-0/+5
| | | | | | | | | | are now created non-inheritable; add functions os.get/set_inheritable(), os.get/set_handle_inheritable() and socket.socket.get/set_inheritable().
* | Issue #18722: Remove uses of the "register" keyword in C code.Antoine Pitrou2013-08-131-6/+6
| |
* | merge 3.3Benjamin Peterson2013-07-161-1/+2
|\ \ | |/
| * move declaration to top of blockBenjamin Peterson2013-07-161-1/+2
| |
* | merge 3.3 (closes #18470)Benjamin Peterson2013-07-161-36/+46
|\ \ | |/
| * check the return value of new_string() (closes #18470)Benjamin Peterson2013-07-161-36/+46
| |
* | Issue #18038: SyntaxError raised during compilation sources with illegalSerhiy Storchaka2013-06-091-7/+7
|\ \ | |/ | | | | encoding now always contains an encoding name.
| * Issue #18038: SyntaxError raised during compilation sources with illegalSerhiy Storchaka2013-06-091-7/+7
| | | | | | | | encoding now always contains an encoding name.
* | Issue #9566: Fix compiler warning on Windows 64-bitVictor Stinner2013-06-041-1/+2
| |
* | make _PyParser_TokenNames constBenjamin Peterson2012-10-241-1/+1
|/
* Issue #15096: Drop support for the ur string prefixChristian Heimes2012-06-201-2/+3
|
* Basic support for PEP 414 without docs or tests.Armin Ronacher2012-03-041-3/+7
|
* Issue #13748: Raw bytes literals can now be written with the `rb` prefix as ↵Antoine Pitrou2012-01-121-7/+9
| | | | well as `br`.
* Rename _Py_identifier to _Py_IDENTIFIER.Martin v. Löwis2011-10-141-2/+2
|
* Use identifier API for PyObject_GetAttrString.Martin v. Löwis2011-10-101-1/+2
|
* Add API for static strings, primarily good for identifiers.Martin v. Löwis2011-10-091-1/+2
| | | | Thanks to Konrad Schöbel and Jasper Schulz for helping with the mass-editing.
* Implement PEP 393.Martin v. Löwis2011-09-281-2/+4
|
* Revert bb62908896fe, but keep the testJesus Cea2011-04-251-13/+6
|
* Correctly merging #9319 into 3.3?Jesus Cea2011-04-251-6/+13
|\
| * Issue #9319: Fix a crash on parsing a Python source code without encodingVictor Stinner2011-04-221-1/+4
| | | | | | | | | | cookie and not valid in UTF-8: use "<file>" as the filename instead of reading from NULL.
* | Issue #9319: Include the filename in "Non-UTF8 code ..." syntax error.Victor Stinner2011-04-041-14/+27
| |
* | Issue #10785: Store the filename as Unicode in the Python parser.Victor Stinner2011-04-041-13/+22
|/
* Issue #10841: don't translate newlines for pgenVictor Stinner2011-01-071-1/+1
|
* Issue #10841: set binary mode on files; the parser translates newlinesVictor Stinner2011-01-071-0/+7
| | | | | | On Windows, set the binary mode on stdin, stdout, stderr and all io.FileIO objects (to not translate newlines, \r\n <=> \n). The Python parser translates newlines (\r\n => \n).
* Remove arbitrary string length limitsVictor Stinner2011-01-051-1/+1
| | | | | PyUnicode_FromFormat() and PyErr_Format() allocates a buffer of the needed size, it is no more a fixed-buffer of 500 bytes.
* Issue #10778: decoding_fgets() decodes the filename from the filesystemVictor Stinner2010-12-271-6/+11
| | | | encoding instead of UTF-8.
* Issue #10095: fp_setreadl() doesn't reopen the file, reuse instead the fileVictor Stinner2010-10-141-6/+9
| | | | descriptor.
* Recorded merge of revisions 81029 via svnmerge fromAntoine Pitrou2010-05-091-1384/+1384
| | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r81029 | antoine.pitrou | 2010-05-09 16:46:46 +0200 (dim., 09 mai 2010) | 3 lines Untabify C files. Will watch buildbots. ........
* Use PyErr_Format() in decoding_fgets()Victor Stinner2010-04-281-3/+1
| | | | Avoid a buffer of 500 bytes allocated on the stack.
* Merged revisions 79725 via svnmerge fromBenjamin Peterson2010-04-031-11/+1
| | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r79725 | benjamin.peterson | 2010-04-03 18:03:35 -0500 (Sat, 03 Apr 2010) | 4 lines use our own locale independent ctype macros requires building pyctype.o into pgen ........
* Merged revisions 79723 via svnmerge fromBenjamin Peterson2010-04-031-1/+11
| | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r79723 | benjamin.peterson | 2010-04-03 17:48:51 -0500 (Sat, 03 Apr 2010) | 1 line ensure that the locale does not affect the tokenization of identifiers ........
* Merged revisions 78826 via svnmerge fromVictor Stinner2010-03-121-5/+12
| | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r78826 | victor.stinner | 2010-03-10 23:30:19 +0100 (mer., 10 mars 2010) | 5 lines Issue #3137: Don't ignore errors at startup, especially a keyboard interrupt (SIGINT). If an error occurs while importing the site module, the error is printed and Python exits. Initialize the GIL before importing the site module. ........
* Merged revisions 78603 via svnmerge fromVictor Stinner2010-03-031-23/+28
| | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r78603 | victor.stinner | 2010-03-03 00:20:02 +0100 (mer., 03 mars 2010) | 5 lines Issue #7820: The parser tokenizer restores all bytes in the right if the BOM check fails. Fix an assertion in pydebug mode. ........
* Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge fromBenjamin Peterson2009-12-131-12/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r76052 | gregory.p.smith | 2009-11-01 20:02:38 -0600 (Sun, 01 Nov 2009) | 5 lines see issue1006238, this merges in the following patch to ease cross compiling the printf %zd check. http://sources.gentoo.org/viewcvs.py/gentoo-x86/dev-lang/python/files/python-2.5-cross-printf.patch?rev=1.1&view=markup ........ r76522 | barry.warsaw | 2009-11-25 12:38:32 -0600 (Wed, 25 Nov 2009) | 2 lines Add mktime_tz to __all__. It's documented as being available in email.utils. ........ r76591 | benjamin.peterson | 2009-11-29 16:26:26 -0600 (Sun, 29 Nov 2009) | 4 lines now that deepcopy can handle instance methods, this hack can be removed #7409 Thanks Robert Collins ........ r76689 | benjamin.peterson | 2009-12-06 11:37:48 -0600 (Sun, 06 Dec 2009) | 1 line rewrite translate_newlines for clarity ........ r76697 | benjamin.peterson | 2009-12-06 15:24:30 -0600 (Sun, 06 Dec 2009) | 2 lines fix test_parser from tokenizer tweak ........ r76733 | benjamin.peterson | 2009-12-09 21:37:59 -0600 (Wed, 09 Dec 2009) | 1 line substitute PyDict_Check() for PyObject_IsInstance ........
generated by cgit v0.12 at 2024-10-10 08:25:05 (GMT)