summaryrefslogtreecommitdiffstats
path: root/Lib/tokenize.py
Commit message (Collapse)AuthorAgeFilesLines
* Add name to credits (for untokenize).Raymond Hettinger2006-12-021-1/+1
|
* Replace dead code with an assert.Jeremy Hylton2006-08-231-4/+1
| | | | | Now that COMMENT tokens are reliably followed by NL or NEWLINE, there is never a need to add extra newlines in untokenize.
* Bug fixes large and small for tokenize.Jeremy Hylton2006-08-231-31/+79
| | | | | | | | | | | | | | | | | | | | Small: Always generate a NL or NEWLINE token following a COMMENT token. The old code did not generate an NL token if the comment was on a line by itself. Large: The output of untokenize() will now match the input exactly if it is passed the full token sequence. The old, crufty output is still generated if a limited input sequence is provided, where limited means that it does not include position information for tokens. Remaining bug: There is no CONTINUATION token (\) so there is no way for untokenize() to handle such code. Also, expanded the number of doctests in hopes of eventually removing the old-style tests that compare against a golden file. Bug fix candidate for Python 2.5.1. (Sigh.)
* Make tabnanny recognize IndentationErrors raised by tokenize.Georg Brandl2006-08-141-1/+2
| | | | | Add a test to test_inspect to make sure indented source is recognized correctly. (fixes #1224621)
* Insert a safety space after numbers as well as names in untokenize().Guido van Rossum2006-03-301-1/+1
|
* SF bug #1224621: tokenize module does not detect inconsistent dedentsRaymond Hettinger2005-06-211-0/+3
|
* Add untokenize() function to allow full round-trip tokenization.Raymond Hettinger2005-06-101-3/+49
| | | | | | Should significantly enhance the utility of the module by supporting the creation of tools that modify the token stream and writeback the modified result.
* PEP-0318, @decorator-style. In Guido's words:Anthony Baxter2004-08-021-1/+1
| | | | | "@ seems the syntax that everybody can hate equally" Implementation by Mark Russell, from SF #979728.
* Get rid of many apply() calls.Guido van Rossum2003-02-271-3/+3
|
* SF 633560: tokenize.__all__ needs "generate_tokens"Raymond Hettinger2002-11-051-1/+2
|
* Speed up the most egregious "if token in (long tuple)" cases by usingGuido van Rossum2002-08-241-10/+19
| | | | | a dict instead. (Alas, using a Set would be slower instead of faster.)
* Whitespace normalization.Tim Peters2002-05-231-5/+5
|
* Added docstrings excerpted from Python Library Reference.Raymond Hettinger2002-05-151-0/+25
| | | | Closes patch 556161.
* Remove some now-obsolete generator future statements.Tim Peters2002-04-011-2/+0
| | | | | I left the email pkg alone; I'm not sure how Barry would like to handle that.
* Cleanup x so it is not left in moduleNeal Norwitz2002-03-261-0/+1
|
* SF patch #455966: Allow leading 0 in float/imag literals.Tim Peters2001-08-301-2/+2
| | | | Consequences for Jython still unknown (but raised on Jython-Dev).
* Add new tokens // and //=, in support of PEP 238.Guido van Rossum2001-08-081-0/+1
|
* Use string.ascii_letters instead of string.letters (SF bug #226706).Fred Drake2001-07-201-1/+1
|
* Preliminary support for "from __future__ import generators" to enableGuido van Rossum2001-07-151-0/+2
| | | | | | | | the yield statement. I figure we have to have this in before I can release 2.2a1 on Wednesday. Note: test_generators is currently broken, I'm counting on Tim to fix this.
* Turns out Neil didn't intend for *all* of his gen-branch work to getTim Peters2001-06-291-8/+21
| | | | | | | | | | | | | | | | | | | | committed. tokenize.py: I like these changes, and have tested them extensively without even realizing it, so I just updated the docstring and the docs. tabnanny.py: Also liked this, but did a little code fiddling. I should really rewrite this to *exploit* generators, but that's near the bottom of my effort/benefit scale so doubt I'll get to it anytime soon (it would be most useful as a non-trivial example of ideal use of generators; but test_generators.py has already grown plenty of food-for-thought examples). inspect.py: I'm sure Ping intended for this to continue running even under 1.5.2, so I reverted this to the last pre-gen-branch version. The "bugfix" I checked in in-between was actually repairing a bug *introduced* by the conversion to generators, so it's OK that the reverted version doesn't reflect that checkin.
* Merging the gen-branch into the main line, at Guido's direction. Yay!Tim Peters2001-06-181-15/+20
| | | | | Bugfix candidate in inspect.py: it was referencing "self" outside of a method.
* Provide a StopTokenizing exception for conveniently exiting the loop.Ka-Ping Yee2001-03-231-10/+11
|
* Better __credits__.Ka-Ping Yee2001-03-011-1/+2
|
* Add __author__ and __credits__ variables.Ka-Ping Yee2001-03-011-1/+2
|
* final round of __all__ lists (I hope) - skipped urllib2 because Moshe may beSkip Montanaro2001-03-011-1/+5
| | | | giving it a slight facelift
* String method conversion.Eric S. Raymond2001-02-091-1/+1
|
* Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.Ka-Ping Yee2001-01-151-9/+25
|
* Whitespace normalization.Tim Peters2001-01-151-1/+0
|
* Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).Tim Peters2000-10-071-12/+20
| | | | | | | | | | | tokenize.py has always used naive regexps for matching string literals, and that appears to trigger the sre recursion limit on Skip's platform (he has very long single-line string literals). Replaced all of tokenize.py's string regexps with the "unrolled" forms used in IDLE, where they're known to handle even absurd (multi-megabyte!) string literals without trouble. See Friedl's book for explanation (at heart, the naive regexps create a backtracking choice point for each character in the literal, while the unrolled forms create none).
* Update for augmented assignment, tested & approved by Guido.Thomas Wouters2000-08-241-2/+5
|
* Convert some old-style string exceptions to class exceptions.Fred Drake2000-08-171-1/+4
|
* Differentiate between NEWLINE token (an official newline) and NL tokenGuido van Rossum1998-04-031-5/+15
| | | | (a newline that the grammar ignores).
* New, fixed version with proper r"..." and R"..." support from Ka-Ping.Guido van Rossum1997-10-271-7/+10
|
* Redone (by Ka-Ping) using the new re module, and adding recognitionGuido van Rossum1997-10-271-57/+55
| | | | for r"..." raw strings. (And R"..." string support added by Guido.)
* Correct typo in last line (test program invocation).Guido van Rossum1997-06-031-1/+1
|
* Ping's latest. Fixes triple quoted strings ending in oddGuido van Rossum1997-04-091-20/+31
| | | | #backslashes, and other stuff I don't know.
* Ka-Ping's muich improved version of March 26, 1997:Guido van Rossum1997-04-081-74/+98
| | | | | | # Ignore now accepts \f as whitespace. Operator now includes '**'. # Ignore and Special now accept \n or \r\n at the end of a line. # Imagnumber is new. Expfloat is corrected to reject '0e4'.
* Added support for imaginary constants (e.g. 0j, 1j, 1.0j).Guido van Rossum1997-03-101-4/+5
|
* Fixed doc string, added __version__, fixed 1 bug.Guido van Rossum1997-03-071-11/+18
|
* Ka-Ping's version.Guido van Rossum1997-03-071-45/+132
|
* * test_select.py: (some) tests for built-in select moduleGuido van Rossum1993-11-111-1/+3
| | | | | | * test_grammar.py, testall.out: added test for funny things in string literals * token.py, symbol.py: definitions used with built-in parser module. * tokenize.py: added double-quote recognition
* Change the order in which Floatnumber and Intnumber are triedGuido van Rossum1992-03-161-2/+3
| | | | | so it will correctly recognize floats. Fix the test program so it works again.
* Initial revisionGuido van Rossum1992-01-011-0/+60
generated by cgit v0.12 at 2025-01-19 04:59:26 (GMT)