summaryrefslogtreecommitdiffstats
path: root/Lib/tokenize.py
Commit message (Collapse)AuthorAgeFilesLines
* Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.Ka-Ping Yee2001-01-151-9/+25
|
* Whitespace normalization.Tim Peters2001-01-151-1/+0
|
* Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).Tim Peters2000-10-071-12/+20
| | | | | | | | | | | tokenize.py has always used naive regexps for matching string literals, and that appears to trigger the sre recursion limit on Skip's platform (he has very long single-line string literals). Replaced all of tokenize.py's string regexps with the "unrolled" forms used in IDLE, where they're known to handle even absurd (multi-megabyte!) string literals without trouble. See Friedl's book for explanation (at heart, the naive regexps create a backtracking choice point for each character in the literal, while the unrolled forms create none).
* Update for augmented assignment, tested & approved by Guido.Thomas Wouters2000-08-241-2/+5
|
* Convert some old-style string exceptions to class exceptions.Fred Drake2000-08-171-1/+4
|
* Differentiate between NEWLINE token (an official newline) and NL tokenGuido van Rossum1998-04-031-5/+15
| | | | (a newline that the grammar ignores).
* New, fixed version with proper r"..." and R"..." support from Ka-Ping.Guido van Rossum1997-10-271-7/+10
|
* Redone (by Ka-Ping) using the new re module, and adding recognitionGuido van Rossum1997-10-271-57/+55
| | | | for r"..." raw strings. (And R"..." string support added by Guido.)
* Correct typo in last line (test program invocation).Guido van Rossum1997-06-031-1/+1
|
* Ping's latest. Fixes triple quoted strings ending in oddGuido van Rossum1997-04-091-20/+31
| | | | #backslashes, and other stuff I don't know.
* Ka-Ping's muich improved version of March 26, 1997:Guido van Rossum1997-04-081-74/+98
| | | | | | # Ignore now accepts \f as whitespace. Operator now includes '**'. # Ignore and Special now accept \n or \r\n at the end of a line. # Imagnumber is new. Expfloat is corrected to reject '0e4'.
* Added support for imaginary constants (e.g. 0j, 1j, 1.0j).Guido van Rossum1997-03-101-4/+5
|
* Fixed doc string, added __version__, fixed 1 bug.Guido van Rossum1997-03-071-11/+18
|
* Ka-Ping's version.Guido van Rossum1997-03-071-45/+132
|
* * test_select.py: (some) tests for built-in select moduleGuido van Rossum1993-11-111-1/+3
| | | | | | * test_grammar.py, testall.out: added test for funny things in string literals * token.py, symbol.py: definitions used with built-in parser module. * tokenize.py: added double-quote recognition
* Change the order in which Floatnumber and Intnumber are triedGuido van Rossum1992-03-161-2/+3
| | | | | so it will correctly recognize floats. Fix the test program so it works again.
* Initial revisionGuido van Rossum1992-01-011-0/+60