summaryrefslogtreecommitdiffstats
path: root/Parser/tokenizer.c
Commit message (Collapse)AuthorAgeFilesLines
* Merge with 3.3Martin v. Löwis2014-02-281-2/+12
|\
| * Issue #20731: Properly position in source code files even if theyMartin v. Löwis2014-02-281-2/+12
| | | | | | | | are opened in text mode. Patch by Serhiy Storchaka.
* | Do not reset the line number because we already set file position to correctSerhiy Storchaka2014-01-091-8/+0
|\ \ | |/ | | | | | | | | value. (fixes error in patch for issue #18960)
| * Do not reset the line number because we already set file position to correctSerhiy Storchaka2014-01-091-8/+0
| | | | | | | | | | | | value. (fixes error in patch for issue #18960)
* | Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-4/+22
|\ \ | |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
| * Issue #18960: Fix bugs with Python source code encoding in the second line.Serhiy Storchaka2014-01-091-4/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * The first line of Python script could be executed twice when the source encoding (not equal to 'utf-8') was specified on the second line. * Now the source encoding declaration on the second line isn't effective if the first line contains anything except a comment. * As a consequence, 'python -x' works now again with files with the source encoding declarations specified on the second file, and can be used again to make Python batch files on Windows. * The tokenize module now ignore the source encoding declaration on the second line if the first line contains anything except a comment. * IDLE now ignores the source encoding declaration on the second line if the first line contains anything except a comment. * 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment.
* | Issue #1772673: The type of `char*` arguments now changed to `const char*`.Serhiy Storchaka2013-10-191-3/+4
| |
* | Issue #18571: Implementation of the PEP 446: file descriptors and file handlesVictor Stinner2013-08-271-0/+5
| | | | | | | | | | are now created non-inheritable; add functions os.get/set_inheritable(), os.get/set_handle_inheritable() and socket.socket.get/set_inheritable().
* | Issue #18722: Remove uses of the "register" keyword in C code.Antoine Pitrou2013-08-131-6/+6
| |
* | merge 3.3Benjamin Peterson2013-07-161-1/+2
|\ \ | |/
| * move declaration to top of blockBenjamin Peterson2013-07-161-1/+2
| |
* | merge 3.3 (closes #18470)Benjamin Peterson2013-07-161-36/+46
|\ \ | |/
| * check the return value of new_string() (closes #18470)Benjamin Peterson2013-07-161-36/+46
| |
* | Issue #18038: SyntaxError raised during compilation sources with illegalSerhiy Storchaka2013-06-091-7/+7
|\ \ | |/ | | | | encoding now always contains an encoding name.
| * Issue #18038: SyntaxError raised during compilation sources with illegalSerhiy Storchaka2013-06-091-7/+7
| | | | | | | | encoding now always contains an encoding name.
* | Issue #9566: Fix compiler warning on Windows 64-bitVictor Stinner2013-06-041-1/+2
| |
* | make _PyParser_TokenNames constBenjamin Peterson2012-10-241-1/+1
|/
* Issue #15096: Drop support for the ur string prefixChristian Heimes2012-06-201-2/+3
|
* Basic support for PEP 414 without docs or tests.Armin Ronacher2012-03-041-3/+7
|
* Issue #13748: Raw bytes literals can now be written with the `rb` prefix as ↵Antoine Pitrou2012-01-121-7/+9
| | | | well as `br`.
* Rename _Py_identifier to _Py_IDENTIFIER.Martin v. Löwis2011-10-141-2/+2
|
* Use identifier API for PyObject_GetAttrString.Martin v. Löwis2011-10-101-1/+2
|
* Add API for static strings, primarily good for identifiers.Martin v. Löwis2011-10-091-1/+2
| | | | Thanks to Konrad Schöbel and Jasper Schulz for helping with the mass-editing.
* Implement PEP 393.Martin v. Löwis2011-09-281-2/+4
|
* Revert bb62908896fe, but keep the testJesus Cea2011-04-251-13/+6
|
* Correctly merging #9319 into 3.3?Jesus Cea2011-04-251-6/+13
|\
| * Issue #9319: Fix a crash on parsing a Python source code without encodingVictor Stinner2011-04-221-1/+4
| | | | | | | | | | cookie and not valid in UTF-8: use "<file>" as the filename instead of reading from NULL.
* | Issue #9319: Include the filename in "Non-UTF8 code ..." syntax error.Victor Stinner2011-04-041-14/+27
| |
* | Issue #10785: Store the filename as Unicode in the Python parser.Victor Stinner2011-04-041-13/+22
|/
* Issue #10841: don't translate newlines for pgenVictor Stinner2011-01-071-1/+1
|
* Issue #10841: set binary mode on files; the parser translates newlinesVictor Stinner2011-01-071-0/+7
| | | | | | On Windows, set the binary mode on stdin, stdout, stderr and all io.FileIO objects (to not translate newlines, \r\n <=> \n). The Python parser translates newlines (\r\n => \n).
* Remove arbitrary string length limitsVictor Stinner2011-01-051-1/+1
| | | | | PyUnicode_FromFormat() and PyErr_Format() allocates a buffer of the needed size, it is no more a fixed-buffer of 500 bytes.
* Issue #10778: decoding_fgets() decodes the filename from the filesystemVictor Stinner2010-12-271-6/+11
| | | | encoding instead of UTF-8.
* Issue #10095: fp_setreadl() doesn't reopen the file, reuse instead the fileVictor Stinner2010-10-141-6/+9
| | | | descriptor.
* Recorded merge of revisions 81029 via svnmerge fromAntoine Pitrou2010-05-091-1384/+1384
| | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r81029 | antoine.pitrou | 2010-05-09 16:46:46 +0200 (dim., 09 mai 2010) | 3 lines Untabify C files. Will watch buildbots. ........
* Use PyErr_Format() in decoding_fgets()Victor Stinner2010-04-281-3/+1
| | | | Avoid a buffer of 500 bytes allocated on the stack.
* Merged revisions 79725 via svnmerge fromBenjamin Peterson2010-04-031-11/+1
| | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r79725 | benjamin.peterson | 2010-04-03 18:03:35 -0500 (Sat, 03 Apr 2010) | 4 lines use our own locale independent ctype macros requires building pyctype.o into pgen ........
* Merged revisions 79723 via svnmerge fromBenjamin Peterson2010-04-031-1/+11
| | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r79723 | benjamin.peterson | 2010-04-03 17:48:51 -0500 (Sat, 03 Apr 2010) | 1 line ensure that the locale does not affect the tokenization of identifiers ........
* Merged revisions 78826 via svnmerge fromVictor Stinner2010-03-121-5/+12
| | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r78826 | victor.stinner | 2010-03-10 23:30:19 +0100 (mer., 10 mars 2010) | 5 lines Issue #3137: Don't ignore errors at startup, especially a keyboard interrupt (SIGINT). If an error occurs while importing the site module, the error is printed and Python exits. Initialize the GIL before importing the site module. ........
* Merged revisions 78603 via svnmerge fromVictor Stinner2010-03-031-23/+28
| | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r78603 | victor.stinner | 2010-03-03 00:20:02 +0100 (mer., 03 mars 2010) | 5 lines Issue #7820: The parser tokenizer restores all bytes in the right if the BOM check fails. Fix an assertion in pydebug mode. ........
* Merged revisions 76052,76522,76591,76689,76697,76733 via svnmerge fromBenjamin Peterson2009-12-131-12/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r76052 | gregory.p.smith | 2009-11-01 20:02:38 -0600 (Sun, 01 Nov 2009) | 5 lines see issue1006238, this merges in the following patch to ease cross compiling the printf %zd check. http://sources.gentoo.org/viewcvs.py/gentoo-x86/dev-lang/python/files/python-2.5-cross-printf.patch?rev=1.1&view=markup ........ r76522 | barry.warsaw | 2009-11-25 12:38:32 -0600 (Wed, 25 Nov 2009) | 2 lines Add mktime_tz to __all__. It's documented as being available in email.utils. ........ r76591 | benjamin.peterson | 2009-11-29 16:26:26 -0600 (Sun, 29 Nov 2009) | 4 lines now that deepcopy can handle instance methods, this hack can be removed #7409 Thanks Robert Collins ........ r76689 | benjamin.peterson | 2009-12-06 11:37:48 -0600 (Sun, 06 Dec 2009) | 1 line rewrite translate_newlines for clarity ........ r76697 | benjamin.peterson | 2009-12-06 15:24:30 -0600 (Sun, 06 Dec 2009) | 2 lines fix test_parser from tokenizer tweak ........ r76733 | benjamin.peterson | 2009-12-09 21:37:59 -0600 (Wed, 09 Dec 2009) | 1 line substitute PyDict_Check() for PyObject_IsInstance ........
* Merged revisions ↵Benjamin Peterson2009-11-131-11/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r75149 | gregory.p.smith | 2009-09-29 16:56:31 -0500 (Tue, 29 Sep 2009) | 3 lines Mention issue6972 in extractall docs about overwriting things outside of the supplied path. ........ r75260 | andrew.kuchling | 2009-10-05 16:24:20 -0500 (Mon, 05 Oct 2009) | 1 line Wording fix ........ r75261 | andrew.kuchling | 2009-10-05 16:24:35 -0500 (Mon, 05 Oct 2009) | 1 line Fix narkup ........ r75262 | andrew.kuchling | 2009-10-05 16:25:03 -0500 (Mon, 05 Oct 2009) | 1 line Document 'skip' parameter to constructor ........ r75263 | andrew.kuchling | 2009-10-05 16:25:35 -0500 (Mon, 05 Oct 2009) | 1 line Note side benefit of socket.create_connection() ........ r75265 | andrew.kuchling | 2009-10-05 17:31:11 -0500 (Mon, 05 Oct 2009) | 1 line Reword sentence ........ r75266 | andrew.kuchling | 2009-10-05 17:32:48 -0500 (Mon, 05 Oct 2009) | 1 line Use standard comma punctuation; reword some sentences in the docs ........ r75267 | andrew.kuchling | 2009-10-05 17:42:56 -0500 (Mon, 05 Oct 2009) | 1 line Backport r73983: Document the thousands separator. ........ r75292 | benjamin.peterson | 2009-10-08 22:11:36 -0500 (Thu, 08 Oct 2009) | 1 line death to old CVS keyword ........ r75300 | benjamin.peterson | 2009-10-09 16:48:14 -0500 (Fri, 09 Oct 2009) | 1 line fix some coding style ........ r75376 | benjamin.peterson | 2009-10-11 20:26:07 -0500 (Sun, 11 Oct 2009) | 1 line platform we don't care about ........ r75405 | neil.schemenauer | 2009-10-14 12:17:14 -0500 (Wed, 14 Oct 2009) | 4 lines Issue #1754094: Improve the stack depth calculation in the compiler. There should be no other effect than a small decrease in memory use. Patch by Christopher Tur Lesniewski-Laas. ........ r75429 | benjamin.peterson | 2009-10-14 20:47:28 -0500 (Wed, 14 Oct 2009) | 1 line pep8ify if blocks ........ r75430 | benjamin.peterson | 2009-10-14 20:49:37 -0500 (Wed, 14 Oct 2009) | 1 line use floor division and add a test that exercises the tabsize codepath ........ r75431 | benjamin.peterson | 2009-10-14 20:56:25 -0500 (Wed, 14 Oct 2009) | 1 line change test to what I intended ........ r75432 | benjamin.peterson | 2009-10-14 22:05:39 -0500 (Wed, 14 Oct 2009) | 1 line some cleanups ........ r75433 | benjamin.peterson | 2009-10-14 22:06:55 -0500 (Wed, 14 Oct 2009) | 1 line make inspect.isabstract() always return a boolean; add a test for it, too #7069 ........ r75437 | benjamin.peterson | 2009-10-15 10:44:46 -0500 (Thu, 15 Oct 2009) | 1 line only clear a module's __dict__ if the module is the only one with a reference to it #7140 ........ r75445 | vinay.sajip | 2009-10-16 09:06:44 -0500 (Fri, 16 Oct 2009) | 1 line Issue #7120: logging: Removed import of multiprocessing which is causing crash in GAE. ........ r75501 | antoine.pitrou | 2009-10-18 13:37:11 -0500 (Sun, 18 Oct 2009) | 3 lines Add a comment about unreachable code, and fix a typo ........ r75551 | benjamin.peterson | 2009-10-19 22:14:10 -0500 (Mon, 19 Oct 2009) | 1 line use property api ........ r75572 | benjamin.peterson | 2009-10-20 16:55:17 -0500 (Tue, 20 Oct 2009) | 1 line clarify buffer arg #7178 ........ r75589 | benjamin.peterson | 2009-10-21 21:26:47 -0500 (Wed, 21 Oct 2009) | 1 line whitespace ........ r75590 | benjamin.peterson | 2009-10-21 21:36:47 -0500 (Wed, 21 Oct 2009) | 1 line rewrite to be nice to other implementations ........ r75591 | benjamin.peterson | 2009-10-21 21:50:38 -0500 (Wed, 21 Oct 2009) | 4 lines rewrite for style, clarify, and comments Also, use the hasattr() like scheme of allowing BaseException exceptions through. ........ r75657 | antoine.pitrou | 2009-10-24 07:41:27 -0500 (Sat, 24 Oct 2009) | 3 lines Fix compilation error in debug mode. ........ r75742 | benjamin.peterson | 2009-10-26 17:51:16 -0500 (Mon, 26 Oct 2009) | 1 line use 'is' instead of id() ........ r75868 | benjamin.peterson | 2009-10-27 15:59:18 -0500 (Tue, 27 Oct 2009) | 1 line test expect base classes ........ r75952 | georg.brandl | 2009-10-29 15:38:32 -0500 (Thu, 29 Oct 2009) | 1 line Use the correct function name in docstring. ........ r75953 | georg.brandl | 2009-10-29 15:39:50 -0500 (Thu, 29 Oct 2009) | 1 line Remove mention of the old -X command line switch. ........ r75954 | georg.brandl | 2009-10-29 15:53:00 -0500 (Thu, 29 Oct 2009) | 1 line Use constants instead of magic integers for test result. Do not re-run with --verbose3 for environment changing tests. ........ r75955 | georg.brandl | 2009-10-29 15:54:03 -0500 (Thu, 29 Oct 2009) | 1 line Use a single style for all the docstrings in the math module. ........ r75956 | georg.brandl | 2009-10-29 16:16:34 -0500 (Thu, 29 Oct 2009) | 1 line I do not think the "railroad" program mentioned is still available. ........ r75957 | georg.brandl | 2009-10-29 16:44:56 -0500 (Thu, 29 Oct 2009) | 1 line Fix constant name. ........ r76057 | benjamin.peterson | 2009-11-02 09:06:45 -0600 (Mon, 02 Nov 2009) | 1 line prevent a rather unlikely segfault ........ r76105 | georg.brandl | 2009-11-04 01:38:12 -0600 (Wed, 04 Nov 2009) | 1 line #7259: show correct equivalent for operator.i* operations in docstring; fix minor issues in operator docs. ........ r76139 | benjamin.peterson | 2009-11-06 19:04:38 -0600 (Fri, 06 Nov 2009) | 1 line spelling ........ r76143 | georg.brandl | 2009-11-07 02:26:07 -0600 (Sat, 07 Nov 2009) | 1 line #7271: fix typo. ........ r76162 | benjamin.peterson | 2009-11-08 22:10:53 -0600 (Sun, 08 Nov 2009) | 1 line discuss how to use -p ........ r76223 | georg.brandl | 2009-11-12 02:29:46 -0600 (Thu, 12 Nov 2009) | 1 line Give the profile module a module directive. ........
* Merged revisions 76230 via svnmerge fromBenjamin Peterson2009-11-131-18/+74
| | | | | | | | | | svn+ssh://pythondev@svn.python.org/python/trunk ........ r76230 | benjamin.peterson | 2009-11-12 17:39:44 -0600 (Thu, 12 Nov 2009) | 2 lines fix several compile() issues by translating newlines in the tokenizer ........
* Issue 7147 - remove ability to attempt to build Python without complex ↵Skip Montanaro2009-10-181-6/+0
| | | | number support (was broken anyway)
* Merged revisions ↵Benjamin Peterson2009-06-281-4/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 73004,73439,73496,73509,73529,73564,73576-73577,73595-73596,73605 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r73004 | jeffrey.yasskin | 2009-05-28 22:44:31 -0500 (Thu, 28 May 2009) | 5 lines Fix nearly all compilation warnings under Apple gcc-4.0. Tested with OPT="-g -Wall -Wstrict-prototypes -Werror" in both --with-pydebug mode and --without. There's still a batch of non-prototype warnings in Xlib.h that I don't know how to fix. ........ r73439 | benjamin.peterson | 2009-06-15 19:29:31 -0500 (Mon, 15 Jun 2009) | 1 line don't mask encoding errors when decoding a string #6289 ........ r73496 | vinay.sajip | 2009-06-21 12:37:27 -0500 (Sun, 21 Jun 2009) | 1 line Issue #6314: logging.basicConfig() performs extra checks on the "level" argument. ........ r73509 | amaury.forgeotdarc | 2009-06-22 14:33:48 -0500 (Mon, 22 Jun 2009) | 2 lines #4490 Fix sample code run by "python -m xml.sax.xmlreader" ........ r73529 | r.david.murray | 2009-06-23 13:02:46 -0500 (Tue, 23 Jun 2009) | 4 lines Fix issue 5230 by having pydoc's safeimport check to see if the import error was thrown from itself in order to decide if the module can't be found. Thanks to Lucas Prado Melo for collaborating on the fix and tests. ........ r73564 | amaury.forgeotdarc | 2009-06-25 17:29:29 -0500 (Thu, 25 Jun 2009) | 6 lines #2016 Fix a crash in function call when the **kwargs dictionary is mutated during the function call setup. This even gives a slight speedup, probably because tuple allocation is faster than PyMem_NEW. ........ r73576 | benjamin.peterson | 2009-06-26 18:37:06 -0500 (Fri, 26 Jun 2009) | 1 line document is_declared_global() ........ r73577 | benjamin.peterson | 2009-06-27 09:16:23 -0500 (Sat, 27 Jun 2009) | 1 line link to extensive generator docs in the reference manual ........ r73595 | ezio.melotti | 2009-06-27 18:45:39 -0500 (Sat, 27 Jun 2009) | 1 line stmt and setup can contain multiple statements, see #5896 ........ r73596 | ezio.melotti | 2009-06-27 19:07:45 -0500 (Sat, 27 Jun 2009) | 1 line Fixed a wrong apostrophe ........ r73605 | georg.brandl | 2009-06-28 07:10:18 -0500 (Sun, 28 Jun 2009) | 1 line Remove stray pychecker directive. ........
* The BDFL has retired! Long live the FLUFL (Friendly Language Uncle For Life)!Brett Cannon2009-04-011-0/+1
|
* ignore the coding cookie in compile(), exec(), and eval() if the source is a ↵Benjamin Peterson2009-03-021-0/+22
| | | | string #4626
* Fix an issue in the tokenizer, where a file is opened by fd, but the ↵Kristján Valur Jónsson2008-12-181-2/+2
| | | | | | | | | underlying PyFileIO object wasn created with the closefd attribute true. Also fix error handling for close() int _fileio.c . It was incorrect, looking for a negative refcount, and so errors weren't raised. This is why this issue wasn't caught. There is a second reason why it isn't seen: Class IOBase in io.py has a try:/except: around the close() funtion in the __del__() method. This also masks these error conditions. This issue was discovered by removing the _set_invalid_parameter_handler() fiddling, thus enabling the C runtime checks on windows.
* Latin-1 source code was not being properly decoded when passed throughBrett Cannon2008-10-171-2/+2
| | | | | | | compile(). This was due to left-over special-casing before UTF-8 became the default source encoding. Closes issue #3574. Thanks to Victor Stinner for help with the patch.
* Issues #2384 and #3975: Tracebacks were not correctly printed when the ↵Amaury Forgeot d'Arc2008-10-091-0/+8
| | | | | | | | source file contains a ``coding:`` header: the wrong line was displayed, and the encoding was not respected. Patch by Victor Stinner.
generated by cgit v0.12 at 2024-11-29 04:11:51 (GMT)