summaryrefslogtreecommitdiffstats
path: root/Parser/tokenizer.c
Commit message (Collapse)AuthorAgeFilesLines
* merge 3.4Benjamin Peterson2016-09-191-1/+1
|\
| * properly handle the single null-byte file (closes #24022)Benjamin Peterson2016-09-191-1/+1
| |
* | restructure fp_setreadl so as to avoid refleaks (closes #27981)Benjamin Peterson2016-09-131-15/+17
| |
* | Issue #22570: Renamed Py_SETREF to Py_XSETREF.Serhiy Storchaka2016-04-061-1/+1
| |
* | Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-0/+1
| |
* | Issue #20440: Massive replacing unsafe attribute setting code with specialSerhiy Storchaka2015-12-241-2/+1
| | | | | | | | macro Py_SETREF.
* | Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
|\ \ | |/ | | | | with a null byte.
| * Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
| | | | | | | | with a null byte.
* | Issue #24619: Simplify async/await tokenization.Yury Selivanov2015-07-231-87/+39
| | | | | | | | | | | | | | | | | | | | This commit simplifies async/await tokenization in tokenizer.c, tokenize.py & lib2to3/tokenize.py. Previous solution was to keep a stack of async-def & def blocks, whereas the new approach is just to remember position of the outermost async-def block. This change won't bring any parsing performance improvements, but it makes the code much easier to read and validate.
* | Issue #24619: New approach for tokenizing async/await.Yury Selivanov2015-07-221-35/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit fixes how one-line async-defs and defs are tracked by tokenizer. It allows to correctly parse invalid code such as: >>> async def f(): ... def g(): pass ... async = 10 and valid code such as: >>> async def f(): ... async def g(): pass ... await z As a consequence, is is now possible to have one-line 'async def foo(): await ..' functions: >>> async def foo(): return await bar()
* | Issue 24226: Fix parsing of many sequential one-line 'def' statements.Yury Selivanov2015-05-181-8/+17
| |
* | PEP 0492 -- Coroutines with async and await syntax. Issue #24017.Yury Selivanov2015-05-121-0/+68
| |
* | merge 3.4 (#24022)Benjamin Peterson2015-04-211-4/+3
|\ \ | |/
| * do not call into python api if an exception is set (#24022)Benjamin Peterson2015-04-211-4/+3
| |
* | merge 3.4 (#21642)Benjamin Peterson2014-06-071-5/+14
|\ \ | |/
| * allow the keyword else immediately after (no space) an integer (closes #21642)Benjamin Peterson2014-06-071-5/+14
| |
* | PEP 465: a dedicated infix operator for matrix multiplication (closes #21176)Benjamin Peterson2014-04-101-1/+7
|/
* Merge with 3.3Martin v. Löwis2014-02-281-2/+12
|\
| * Issue #20731: Properly position in source code files even if theyMartin v. Löwis2014-02-281-2/+12
| | | | | | | | are opened in text mode. Patch by Serhiy Storchaka.
* | Do not reset the line number because we already set file position to correctSerhiy Storchaka2014-01-091-8/+0
|\ \ | |/ | | | | | | | | value. (fixes error in patch for issue #18960)
| * Do not reset the line number because we already set file position to correctSerhiy Storchaka2014-01-091-8/+0
| | | | | | | | | | | | value. (fixes error in patch for issue #18960)
* | Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-4/+22
|\ \ | |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
| * Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-4/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
* | Issue #1772673: The type of `char*` arguments now changed to `const char*`.Serhiy Storchaka2013-10-191-3/+4
| |
* | Issue #18571: Implementation of the PEP 446: file descriptors and file handlesVictor Stinner2013-08-271-0/+5
| | | | | | | | | | are now created non-inheritable; add functions os.get/set_inheritable(), os.get/set_handle_inheritable() and socket.socket.get/set_inheritable().
* | Issue #18722: Remove uses of the "register" keyword in C code.Antoine Pitrou2013-08-131-6/+6
| |
* | merge 3.3Benjamin Peterson2013-07-161-1/+2
|\ \ | |/
| * move declaration to top of blockBenjamin Peterson2013-07-161-1/+2
| |
* | merge 3.3 (closes #18470)Benjamin Peterson2013-07-161-36/+46
|\ \ | |/
| * check the return value of new_string() (closes #18470)Benjamin Peterson2013-07-161-36/+46
| |
* | Issue #18038: SyntaxError raised during compilation sources with illegalSerhiy Storchaka2013-06-091-7/+7
|\ \ | |/ | | | | encoding now always contains an encoding name.
| * Issue #18038: SyntaxError raised during compilation sources with illegalSerhiy Storchaka2013-06-091-7/+7
| | | | | | | | encoding now always contains an encoding name.
* | Issue #9566: Fix compiler warning on Windows 64-bitVictor Stinner2013-06-041-1/+2
| |
* | make _PyParser_TokenNames constBenjamin Peterson2012-10-241-1/+1
|/
* Issue #15096: Drop support for the ur string prefixChristian Heimes2012-06-201-2/+3
|
* Basic support for PEP 414 without docs or tests.Armin Ronacher2012-03-041-3/+7
|
* Issue #13748: Raw bytes literals can now be written with the `rb` prefix as ↵Antoine Pitrou2012-01-121-7/+9
| | | | well as `br`.
* Rename _Py_identifier to _Py_IDENTIFIER.Martin v. Löwis2011-10-141-2/+2
|
* Use identifier API for PyObject_GetAttrString.Martin v. Löwis2011-10-101-1/+2
|
* Add API for static strings, primarily good for identifiers.Martin v. Löwis2011-10-091-1/+2
| | | | Thanks to Konrad Schöbel and Jasper Schulz for helping with the mass-editing.
* Implement PEP 393.Martin v. Löwis2011-09-281-2/+4
|
* Revert bb62908896fe, but keep the testJesus Cea2011-04-251-13/+6
|
* Correctly merging #9319 into 3.3?Jesus Cea2011-04-251-6/+13
|\
| * Issue #9319: Fix a crash on parsing a Python source code without encodingVictor Stinner2011-04-221-1/+4
| | | | | | | | | | cookie and not valid in UTF-8: use "<file>" as the filename instead of reading from NULL.
* | Issue #9319: Include the filename in "Non-UTF8 code ..." syntax error.Victor Stinner2011-04-041-14/+27
| |
* | Issue #10785: Store the filename as Unicode in the Python parser.Victor Stinner2011-04-041-13/+22
|/
* Issue #10841: don't translate newlines for pgenVictor Stinner2011-01-071-1/+1
|
* Issue #10841: set binary mode on files; the parser translates newlinesVictor Stinner2011-01-071-0/+7
| | | | | | On Windows, set the binary mode on stdin, stdout, stderr and all io.FileIO objects (to not translate newlines, \r\n <=> \n). The Python parser translates newlines (\r\n => \n).
* Remove arbitrary string length limitsVictor Stinner2011-01-051-1/+1
| | | | | PyUnicode_FromFormat() and PyErr_Format() allocates a buffer of the needed size, it is no more a fixed-buffer of 500 bytes.
* Issue #10778: decoding_fgets() decodes the filename from the filesystemVictor Stinner2010-12-271-6/+11
| | | | encoding instead of UTF-8.
generated by cgit v0.12 at 2024-11-28 00:51:24 (GMT)