summaryrefslogtreecommitdiffstats
path: root/Parser/tokenizer.c
Commit message (Collapse)AuthorAgeFilesLines
* bpo-25324: add missing comma in Parser/tokenizer.c (GH-1910)Albert-Jan Nijburg2017-06-011-1/+1
|
* bpo-25324: copy tok_name before changing it (#1608)Albert-Jan Nijburg2017-05-311-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | * add test to check if were modifying token * copy list so import tokenize doesnt have side effects on token * shorten line * add tokenize tokens to token.h to get them to show up in token * move ERRORTOKEN back to its previous location, and fix nitpick * copy comments from token.h automatically * fix whitespace and make more pythonic * change to fix comments from @haypo * update token.rst and Misc/NEWS * change wording * some more wording changes
* Issue #28489: Merge from 3.6Berker Peksag2017-02-051-1/+1
|\
| * Issue #28489: Fix comment in tokenizer.cBerker Peksag2017-02-051-1/+1
| | | | | | | | Patch by Ryan Gonzalez.
* | Use _PyObject_CallNoArg()Victor Stinner2016-12-061-3/+3
|/ | | | | | | Replace: PyObject_CallObject(callable, NULL) with: _PyObject_CallNoArg(callable)
* Replaced outdated macros _PyUnicode_AsString and _PyUnicode_AsStringAndSizeSerhiy Storchaka2016-11-201-1/+1
| | | | with PyUnicode_AsUTF8 and PyUnicode_AsUTF8AndSize.
* merge 3.5 (#24022)Benjamin Peterson2016-09-191-1/+1
|\
| * merge 3.4Benjamin Peterson2016-09-191-1/+1
| |\
| | * properly handle the single null-byte file (closes #24022)Benjamin Peterson2016-09-191-1/+1
| | |
* | | merge 3.5 (#27981)Benjamin Peterson2016-09-131-15/+17
|\ \ \ | |/ /
| * | restructure fp_setreadl so as to avoid refleaks (closes #27981)Benjamin Peterson2016-09-131-15/+17
| | |
* | | Issue #26331: Implement the parsing part of PEP 515.Brett Cannon2016-09-091-68/+162
| | | | | | | | | | | | Thanks to Georg Brandl for the patch.
* | | Skip unused value in tokenizer codeChristian Heimes2016-09-081-1/+1
| | | | | | | | | | | | | | | | | | | | | In the case of an escape character, c is never read. tok_next() is used to advance the pointer. CID 1225097
* | | Issue #22570: Renamed Py_SETREF to Py_XSETREF.Serhiy Storchaka2016-04-061-1/+1
|\ \ \ | |/ /
| * | Issue #22570: Renamed Py_SETREF to Py_XSETREF.Serhiy Storchaka2016-04-061-1/+1
| | |
* | | remove duplicated check for fractions and complex numbers (closes #26076)Benjamin Peterson2016-03-251-4/+0
| | | | | | | | | | | | Patch by Oren Milman.
* | | Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-0/+1
|\ \ \ | |/ /
| * | Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-0/+1
| | |
* | | Issue #25923: Added more const qualifiers to signatures of static and ↵Serhiy Storchaka2015-12-251-3/+3
| | | | | | | | | | | | private functions.
* | | Issue #20440: Massive replacing unsafe attribute setting code with specialSerhiy Storchaka2015-12-241-2/+1
|\ \ \ | |/ / | | | | | | macro Py_SETREF.
| * | Issue #20440: Massive replacing unsafe attribute setting code with specialSerhiy Storchaka2015-12-241-2/+1
| | | | | | | | | | | | macro Py_SETREF.
* | | Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
|\ \ \ | |/ / | | | | | | with a null byte.
| * | Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
| |\ \ | | |/ | | | | | | with a null byte.
| | * Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
| | | | | | | | | | | | with a null byte.
* | | Issue #24965: Implement PEP 498 "Literal String Interpolation". ↵Eric V. Smith2015-09-191-3/+5
| | | | | | | | | | | | Documentation is still needed, I'll open an issue for that.
* | | Fixed indentation.Eric V. Smith2015-09-121-1/+1
|/ /
* | Issue #24619: Simplify async/await tokenization.Yury Selivanov2015-07-231-87/+39
| | | | | | | | | | | | | | | | | | | | This commit simplifies async/await tokenization in tokenizer.c, tokenize.py & lib2to3/tokenize.py. Previous solution was to keep a stack of async-def & def blocks, whereas the new approach is just to remember position of the outermost async-def block. This change won't bring any parsing performance improvements, but it makes the code much easier to read and validate.
* | Issue #24619: New approach for tokenizing async/await.Yury Selivanov2015-07-221-35/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit fixes how one-line async-defs and defs are tracked by tokenizer. It allows to correctly parse invalid code such as: >>> async def f(): ... def g(): pass ... async = 10 and valid code such as: >>> async def f(): ... async def g(): pass ... await z As a consequence, is is now possible to have one-line 'async def foo(): await ..' functions: >>> async def foo(): return await bar()
* | Issue 24226: Fix parsing of many sequential one-line 'def' statements.Yury Selivanov2015-05-181-8/+17
| |
* | PEP 0492 -- Coroutines with async and await syntax. Issue #24017.Yury Selivanov2015-05-121-0/+68
| |
* | merge 3.4 (#24022)Benjamin Peterson2015-04-211-4/+3
|\ \ | |/
| * do not call into python api if an exception is set (#24022)Benjamin Peterson2015-04-211-4/+3
| |
* | merge 3.4 (#21642)Benjamin Peterson2014-06-071-5/+14
|\ \ | |/
| * allow the keyword else immediately after (no space) an integer (closes #21642)Benjamin Peterson2014-06-071-5/+14
| |
* | PEP 465: a dedicated infix operator for matrix multiplication (closes #21176)Benjamin Peterson2014-04-101-1/+7
|/
* Merge with 3.3Martin v. Löwis2014-02-281-2/+12
|\
| * Issue #20731: Properly position in source code files even if theyMartin v. Löwis2014-02-281-2/+12
| | | | | | | | are opened in text mode. Patch by Serhiy Storchaka.
* | Do not reset the line number because we already set file position to correctSerhiy Storchaka2014-01-091-8/+0
|\ \ | |/ | | | | | | | | value. (fixes error in patch for issue #18960)
| * Do not reset the line number because we already set file position to correctSerhiy Storchaka2014-01-091-8/+0
| | | | | | | | | | | | value. (fixes error in patch for issue #18960)
* | Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-4/+22
|\ \ | |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
| * Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-4/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
* | Issue #1772673: The type of `char*` arguments now changed to `const char*`.Serhiy Storchaka2013-10-191-3/+4
| |
* | Issue #18571: Implementation of the PEP 446: file descriptors and file handlesVictor Stinner2013-08-271-0/+5
| | | | | | | | | | are now created non-inheritable; add functions os.get/set_inheritable(), os.get/set_handle_inheritable() and socket.socket.get/set_inheritable().
* | Issue #18722: Remove uses of the "register" keyword in C code.Antoine Pitrou2013-08-131-6/+6
| |
* | merge 3.3Benjamin Peterson2013-07-161-1/+2
|\ \ | |/
| * move declaration to top of blockBenjamin Peterson2013-07-161-1/+2
| |
* | merge 3.3 (closes #18470)Benjamin Peterson2013-07-161-36/+46
|\ \ | |/
| * check the return value of new_string() (closes #18470)Benjamin Peterson2013-07-161-36/+46
| |
* | Issue #18038: SyntaxError raised during compilation sources with illegalSerhiy Storchaka2013-06-091-7/+7
|\ \ | |/ | | | | encoding now always contains an encoding name.
| * Issue #18038: SyntaxError raised during compilation sources with illegalSerhiy Storchaka2013-06-091-7/+7
| | | | | | | | encoding now always contains an encoding name.
generated by cgit v0.12 at 2024-12-02 10:58:38 (GMT)