summaryrefslogtreecommitdiffstats
path: root/Lib/test/test_tokenize.py
Commit message (Collapse)AuthorAgeFilesLines
* Issue #24619: Simplify async/await tokenization.Yury Selivanov2015-07-231-0/+73
| | | | | | | | | | This commit simplifies async/await tokenization in tokenizer.c, tokenize.py & lib2to3/tokenize.py. Previous solution was to keep a stack of async-def & def blocks, whereas the new approach is just to remember position of the outermost async-def block. This change won't bring any parsing performance improvements, but it makes the code much easier to read and validate.
* Issue #24619: New approach for tokenizing async/await.Yury Selivanov2015-07-221-2/+13
| | | | | | | | | | | | | | | | | | | | | This commit fixes how one-line async-defs and defs are tracked by tokenizer. It allows to correctly parse invalid code such as: >>> async def f(): ... def g(): pass ... async = 10 and valid code such as: >>> async def f(): ... async def g(): pass ... await z As a consequence, is is now possible to have one-line 'async def foo(): await ..' functions: >>> async def foo(): return await bar()
* Issue #20387: Merge test and patch from 3.4.4Jason R. Coombs2015-06-281-1/+20
|\
| * Issue #20387: Correct test to properly capture expectation.Jason R. Coombs2015-06-261-2/+2
| |
| * Issue #20387: Add test capturing failure to roundtrip indented code in ↵Jason R. Coombs2015-06-201-0/+17
| | | | | | | | tokenize module.
| * Remove unused import and remove doctest-only import into doctests.Jason R. Coombs2015-06-201-1/+3
| |
* | (Merge 3.5) Issue #23840: tokenize.open() now closes the temporary binary fileVictor Stinner2015-05-251-1/+9
|\ \ | |/ | | | | on error to fix a resource warning.
| * Issue #23840: tokenize.open() now closes the temporary binary file on error toVictor Stinner2015-05-251-1/+9
| | | | | | | | fix a resource warning.
* | Issue 24226: Fix parsing of many sequential one-line 'def' statements.Yury Selivanov2015-05-181-0/+11
| |
* | PEP 0492 -- Coroutines with async and await syntax. Issue #24017.Yury Selivanov2015-05-121-0/+186
| |
* | Issue #23681: Fixed Python 2 to 3 poring bugs.Serhiy Storchaka2015-03-201-3/+4
|\ \ | |/ | | | | Indexing bytes retiurns an integer, not bytes.
| * Issue #23681: Fixed Python 2 to 3 poring bugs.Serhiy Storchaka2015-03-201-3/+4
| | | | | | | | Indexing bytes retiurns an integer, not bytes.
* | PEP 465: a dedicated infix operator for matrix multiplication (closes #21176)Benjamin Peterson2014-04-101-1/+4
|/
* Issue #9974: When untokenizing, use row info to insert backslash+newline.Terry Jan Reedy2014-02-241-1/+16
| | | | Original patches by A. Kuchling and G. Rees (#12691).
* Issue #20750, Enable roundtrip tests for new 5-tuple untokenize. TheTerry Jan Reedy2014-02-231-14/+38
| | | | | | constructed examples and all but 7 of the test/test_*.py files (run with -ucpu) pass. Remove those that fail the new test from the selection list. Patch partly based on patches by G. Brandl (#8478) and G. Rees (#12691).
* Issue #8478: Untokenizer.compat now processes first token from iterator input.Terry Jan Reedy2014-02-181-0/+13
| | | | Patch based on lines from Georg Brandl, Eric Snow, and Gareth Rees.
* whitespaceTerry Jan Reedy2014-02-171-2/+2
|
* Untokenize: An logically incorrect assert tested user input validity.Terry Jan Reedy2014-02-171-1/+15
| | | | | | Replace it with correct logic that raises ValueError for bad input. Issues #8478 and #12691 reported the incorrect logic. Add an Untokenize test case and an initial test method.
* Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-0/+33
| | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
* Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py scriptSerhiy Storchaka2013-09-161-0/+7
| | | | now detect Python source code encoding only in comment lines.
* #16152: merge with 3.2.Ezio Melotti2012-11-031-0/+4
|\
| * #16152: fix tokenize to ignore whitespace at the end of the code when no ↵Ezio Melotti2012-11-031-0/+5
| | | | | | | | newline is found. Patch by Ned Batchelder.
* | Merge branchFlorent Xicluna2012-07-071-0/+4
|\ \ | |/
| * Issue #14990: tokenize: correctly fail with SyntaxError on invalid encoding ↵Florent Xicluna2012-07-071-0/+4
| | | | | | | | declaration.
* | Issue #15096: Drop support for the ur string prefixChristian Heimes2012-06-201-20/+2
| |
* | Issue #15054: Fix incorrect tokenization of 'b' string literals.Meador Inge2012-06-171-0/+76
| | | | | | | | Patch by Serhiy Storchaka.
* | Issue #14629: Mention the filename in SyntaxError exceptions fromBrett Cannon2012-04-201-0/+29
| | | | | | | | tokenizer.detect_encoding() (when available).
* | merge 3.2: issue 14629Martin v. Löwis2012-04-201-0/+10
|\ \ | |/
| * Issue #14629: Raise SyntaxError in tokenizer.detect_encodingMartin v. Löwis2012-04-201-0/+10
| | | | | | | | if the first two lines have non-UTF-8 characters without an encoding declaration.
* | Updated tokenize to support the inverse byte literals new in 3.3Armin Ronacher2012-03-041-0/+12
| |
* | Issue #2134: Add support for tokenize.TokenInfo.exact_type.Meador Inge2012-01-191-1/+74
| |
* | #13012: use splitlines(keepends=True/False) instead of splitlines(0/1).Ezio Melotti2011-09-281-1/+1
|/
* tokenize is just broken on test_pep3131.pyBenjamin Peterson2011-08-131-0/+3
|
* Issue #12587: Correct faulty test file and reference in test_tokenize.Ned Deily2011-07-191-1/+1
| | | | (Patch by Robert Xiao)
* #9424: Replace deprecated assert* methods in the Python test suite.Ezio Melotti2010-11-201-29/+29
|
* test_tokenize: use self.assertEqual() instead of plain assertVictor Stinner2010-11-091-4/+4
|
* Issue #10335: Add tokenize.open(), detect the file encoding usingVictor Stinner2010-11-091-1/+22
| | | | tokenize.detect_encoding() and open it in read only mode.
* Fix #10258 - clean up resource warningBrian Curtin2010-10-301-2/+4
|
* Replace the "compiler" resource with the more generic "cpu", soAntoine Pitrou2010-10-141-2/+2
| | | | as to mark CPU-heavy tests.
* handle names starting with non-ascii characters correctly #9712Benjamin Peterson2010-08-301-0/+13
|
* remove pointless coding cookieBenjamin Peterson2010-08-301-2/+0
|
* Issue #9337: Make float.__str__ identical to float.__repr__.Mark Dickinson2010-08-041-2/+2
| | | | (And similarly for complex numbers.)
* Merged revisions 82356 via svnmerge fromMark Dickinson2010-06-291-1/+1
| | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r82356 | mark.dickinson | 2010-06-29 08:37:25 +0100 (Tue, 29 Jun 2010) | 1 line Spelling. ........
* in tokenize.detect_encoding(), return utf-8-sig when a BOM is foundBenjamin Peterson2010-03-181-5/+5
|
* Merged revisions ↵Benjamin Peterson2009-11-131-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r75149 | gregory.p.smith | 2009-09-29 16:56:31 -0500 (Tue, 29 Sep 2009) | 3 lines Mention issue6972 in extractall docs about overwriting things outside of the supplied path. ........ r75260 | andrew.kuchling | 2009-10-05 16:24:20 -0500 (Mon, 05 Oct 2009) | 1 line Wording fix ........ r75261 | andrew.kuchling | 2009-10-05 16:24:35 -0500 (Mon, 05 Oct 2009) | 1 line Fix narkup ........ r75262 | andrew.kuchling | 2009-10-05 16:25:03 -0500 (Mon, 05 Oct 2009) | 1 line Document 'skip' parameter to constructor ........ r75263 | andrew.kuchling | 2009-10-05 16:25:35 -0500 (Mon, 05 Oct 2009) | 1 line Note side benefit of socket.create_connection() ........ r75265 | andrew.kuchling | 2009-10-05 17:31:11 -0500 (Mon, 05 Oct 2009) | 1 line Reword sentence ........ r75266 | andrew.kuchling | 2009-10-05 17:32:48 -0500 (Mon, 05 Oct 2009) | 1 line Use standard comma punctuation; reword some sentences in the docs ........ r75267 | andrew.kuchling | 2009-10-05 17:42:56 -0500 (Mon, 05 Oct 2009) | 1 line Backport r73983: Document the thousands separator. ........ r75292 | benjamin.peterson | 2009-10-08 22:11:36 -0500 (Thu, 08 Oct 2009) | 1 line death to old CVS keyword ........ r75300 | benjamin.peterson | 2009-10-09 16:48:14 -0500 (Fri, 09 Oct 2009) | 1 line fix some coding style ........ r75376 | benjamin.peterson | 2009-10-11 20:26:07 -0500 (Sun, 11 Oct 2009) | 1 line platform we don't care about ........ r75405 | neil.schemenauer | 2009-10-14 12:17:14 -0500 (Wed, 14 Oct 2009) | 4 lines Issue #1754094: Improve the stack depth calculation in the compiler. There should be no other effect than a small decrease in memory use. Patch by Christopher Tur Lesniewski-Laas. ........ r75429 | benjamin.peterson | 2009-10-14 20:47:28 -0500 (Wed, 14 Oct 2009) | 1 line pep8ify if blocks ........ r75430 | benjamin.peterson | 2009-10-14 20:49:37 -0500 (Wed, 14 Oct 2009) | 1 line use floor division and add a test that exercises the tabsize codepath ........ r75431 | benjamin.peterson | 2009-10-14 20:56:25 -0500 (Wed, 14 Oct 2009) | 1 line change test to what I intended ........ r75432 | benjamin.peterson | 2009-10-14 22:05:39 -0500 (Wed, 14 Oct 2009) | 1 line some cleanups ........ r75433 | benjamin.peterson | 2009-10-14 22:06:55 -0500 (Wed, 14 Oct 2009) | 1 line make inspect.isabstract() always return a boolean; add a test for it, too #7069 ........ r75437 | benjamin.peterson | 2009-10-15 10:44:46 -0500 (Thu, 15 Oct 2009) | 1 line only clear a module's __dict__ if the module is the only one with a reference to it #7140 ........ r75445 | vinay.sajip | 2009-10-16 09:06:44 -0500 (Fri, 16 Oct 2009) | 1 line Issue #7120: logging: Removed import of multiprocessing which is causing crash in GAE. ........ r75501 | antoine.pitrou | 2009-10-18 13:37:11 -0500 (Sun, 18 Oct 2009) | 3 lines Add a comment about unreachable code, and fix a typo ........ r75551 | benjamin.peterson | 2009-10-19 22:14:10 -0500 (Mon, 19 Oct 2009) | 1 line use property api ........ r75572 | benjamin.peterson | 2009-10-20 16:55:17 -0500 (Tue, 20 Oct 2009) | 1 line clarify buffer arg #7178 ........ r75589 | benjamin.peterson | 2009-10-21 21:26:47 -0500 (Wed, 21 Oct 2009) | 1 line whitespace ........ r75590 | benjamin.peterson | 2009-10-21 21:36:47 -0500 (Wed, 21 Oct 2009) | 1 line rewrite to be nice to other implementations ........ r75591 | benjamin.peterson | 2009-10-21 21:50:38 -0500 (Wed, 21 Oct 2009) | 4 lines rewrite for style, clarify, and comments Also, use the hasattr() like scheme of allowing BaseException exceptions through. ........ r75657 | antoine.pitrou | 2009-10-24 07:41:27 -0500 (Sat, 24 Oct 2009) | 3 lines Fix compilation error in debug mode. ........ r75742 | benjamin.peterson | 2009-10-26 17:51:16 -0500 (Mon, 26 Oct 2009) | 1 line use 'is' instead of id() ........ r75868 | benjamin.peterson | 2009-10-27 15:59:18 -0500 (Tue, 27 Oct 2009) | 1 line test expect base classes ........ r75952 | georg.brandl | 2009-10-29 15:38:32 -0500 (Thu, 29 Oct 2009) | 1 line Use the correct function name in docstring. ........ r75953 | georg.brandl | 2009-10-29 15:39:50 -0500 (Thu, 29 Oct 2009) | 1 line Remove mention of the old -X command line switch. ........ r75954 | georg.brandl | 2009-10-29 15:53:00 -0500 (Thu, 29 Oct 2009) | 1 line Use constants instead of magic integers for test result. Do not re-run with --verbose3 for environment changing tests. ........ r75955 | georg.brandl | 2009-10-29 15:54:03 -0500 (Thu, 29 Oct 2009) | 1 line Use a single style for all the docstrings in the math module. ........ r75956 | georg.brandl | 2009-10-29 16:16:34 -0500 (Thu, 29 Oct 2009) | 1 line I do not think the "railroad" program mentioned is still available. ........ r75957 | georg.brandl | 2009-10-29 16:44:56 -0500 (Thu, 29 Oct 2009) | 1 line Fix constant name. ........ r76057 | benjamin.peterson | 2009-11-02 09:06:45 -0600 (Mon, 02 Nov 2009) | 1 line prevent a rather unlikely segfault ........ r76105 | georg.brandl | 2009-11-04 01:38:12 -0600 (Wed, 04 Nov 2009) | 1 line #7259: show correct equivalent for operator.i* operations in docstring; fix minor issues in operator docs. ........ r76139 | benjamin.peterson | 2009-11-06 19:04:38 -0600 (Fri, 06 Nov 2009) | 1 line spelling ........ r76143 | georg.brandl | 2009-11-07 02:26:07 -0600 (Sat, 07 Nov 2009) | 1 line #7271: fix typo. ........ r76162 | benjamin.peterson | 2009-11-08 22:10:53 -0600 (Sun, 08 Nov 2009) | 1 line discuss how to use -p ........ r76223 | georg.brandl | 2009-11-12 02:29:46 -0600 (Thu, 12 Nov 2009) | 1 line Give the profile module a module directive. ........
* normalize latin-1 and utf-8 variant encodings like the builtin tokenizer doesBenjamin Peterson2009-10-091-1/+29
|
* convert old fail* assertions to assert*Benjamin Peterson2009-06-301-1/+1
|
* raise a SyntaxError in detect_encoding() when a codec lookup fails like the ↵Benjamin Peterson2008-12-121-0/+2
| | | | builtin parser #4021
* #2621 rename test.test_support to test.supportBenjamin Peterson2008-05-201-8/+8
|
* Convert a lot of print statements to print functions in docstrings,Neal Norwitz2008-05-131-1/+1
| | | | documentation, and unused/rarely used functions.
generated by cgit v0.12 at 2025-11-18 10:02:32 (GMT)