Merged revisions 57778-58052 via svnmerge from

svn+ssh://pythondev@svn.python.org/python/trunk

........
  r57820 | georg.brandl | 2007-08-31 08:59:27 +0200 (Fri, 31 Aug 2007) | 2 lines

  Document new shorthand notation for index entries.
........
  r57827 | georg.brandl | 2007-08-31 10:47:51 +0200 (Fri, 31 Aug 2007) | 2 lines

  Fix subitem markup.
........
  r57833 | martin.v.loewis | 2007-08-31 12:01:07 +0200 (Fri, 31 Aug 2007) | 1 line

  Mark registry components as 64-bit on Win64.
........
  r57854 | bill.janssen | 2007-08-31 21:02:23 +0200 (Fri, 31 Aug 2007) | 1 line

  deprecate use of FakeSocket
........
  r57855 | bill.janssen | 2007-08-31 21:02:46 +0200 (Fri, 31 Aug 2007) | 1 line

  remove mentions of socket.ssl in comments
........
  r57856 | bill.janssen | 2007-08-31 21:03:31 +0200 (Fri, 31 Aug 2007) | 1 line

  remove use of non-existent SSLFakeSocket in apparently untested code
........
  r57859 | martin.v.loewis | 2007-09-01 08:36:03 +0200 (Sat, 01 Sep 2007) | 3 lines

  Bug #1737210: Change Manufacturer of Windows installer to PSF.
  Will backport to 2.5.
........
  r57865 | georg.brandl | 2007-09-01 09:51:24 +0200 (Sat, 01 Sep 2007) | 2 lines

  Fix RST link (backport from Py3k).
........
  r57876 | georg.brandl | 2007-09-01 17:49:49 +0200 (Sat, 01 Sep 2007) | 2 lines

  Document sets' ">" and "<" operations (backport from py3k).
........
  r57878 | skip.montanaro | 2007-09-01 19:40:03 +0200 (Sat, 01 Sep 2007) | 4 lines

  Added a note and examples to explain that re.split does not split on an
  empty pattern match. (issue 852532).
........
  r57879 | walter.doerwald | 2007-09-01 20:18:09 +0200 (Sat, 01 Sep 2007) | 2 lines

  Fix wrong function names.
........
  r57880 | walter.doerwald | 2007-09-01 20:34:05 +0200 (Sat, 01 Sep 2007) | 2 lines

  Fix typo.
........
  r57889 | andrew.kuchling | 2007-09-01 22:31:59 +0200 (Sat, 01 Sep 2007) | 1 line

  Markup fix
........
  r57892 | andrew.kuchling | 2007-09-01 22:43:36 +0200 (Sat, 01 Sep 2007) | 1 line

  Add various items
........
  r57895 | andrew.kuchling | 2007-09-01 23:17:58 +0200 (Sat, 01 Sep 2007) | 1 line

  Wording change
........
  r57896 | andrew.kuchling | 2007-09-01 23:18:31 +0200 (Sat, 01 Sep 2007) | 1 line

  Add more items
........
  r57904 | ronald.oussoren | 2007-09-02 11:46:07 +0200 (Sun, 02 Sep 2007) | 3 lines

  Macosx: this patch ensures that the value of MACOSX_DEPLOYMENT_TARGET used
  by the Makefile is also used at configure-time.
........
  r57925 | georg.brandl | 2007-09-03 09:16:46 +0200 (Mon, 03 Sep 2007) | 2 lines

  Fix #883466: don't allow Unicode as arguments to quopri and uu codecs.
........
  r57936 | matthias.klose | 2007-09-04 01:33:04 +0200 (Tue, 04 Sep 2007) | 2 lines

  - Added support for linking the bsddb module against BerkeleyDB 4.6.x.
........
  r57954 | mark.summerfield | 2007-09-04 10:16:15 +0200 (Tue, 04 Sep 2007) | 3 lines

  Added cross-references plus a note about dict & list shallow copying.
........
  r57958 | martin.v.loewis | 2007-09-04 11:51:57 +0200 (Tue, 04 Sep 2007) | 3 lines

  Document that we rely on the OS to release the crypto
  context. Fixes #1626801.
........
  r57960 | martin.v.loewis | 2007-09-04 15:13:14 +0200 (Tue, 04 Sep 2007) | 3 lines

  Patch #1388440: Add set_completion_display_matches_hook and
  get_completion_type to readline.
........
  r57961 | martin.v.loewis | 2007-09-04 16:19:28 +0200 (Tue, 04 Sep 2007) | 3 lines

  Patch #1031213: Decode source line in SyntaxErrors back to its original
  source encoding. Will backport to 2.5.
........
  r57972 | matthias.klose | 2007-09-04 20:17:36 +0200 (Tue, 04 Sep 2007) | 3 lines

  - Makefile.pre.in(buildbottest): Run an optional script pybuildbot.identify
    to include some information about the build environment.
........
  r57973 | matthias.klose | 2007-09-04 21:05:38 +0200 (Tue, 04 Sep 2007) | 2 lines

  - Makefile.pre.in(buildbottest): Remove whitespace at eol.
........
  r57975 | matthias.klose | 2007-09-04 22:46:02 +0200 (Tue, 04 Sep 2007) | 2 lines

  - Fix libffi configure for hppa*-*-linux* | parisc*-*-linux*.
........
  r57980 | bill.janssen | 2007-09-05 02:46:27 +0200 (Wed, 05 Sep 2007) | 1 line

  SSL certificate distinguished names should be represented by tuples
........
  r57985 | martin.v.loewis | 2007-09-05 08:39:17 +0200 (Wed, 05 Sep 2007) | 3 lines

  Patch #1105: Explain that one needs to build the solution
  to get dependencies right.
........
  r57987 | armin.rigo | 2007-09-05 09:51:21 +0200 (Wed, 05 Sep 2007) | 4 lines

  PyDict_GetItem() returns a borrowed reference.
  There are probably a number of places that are open to attacks
  such as the following one, in bltinmodule.c:min_max().
........
  r57991 | martin.v.loewis | 2007-09-05 13:47:34 +0200 (Wed, 05 Sep 2007) | 3 lines

  Patch #786737: Allow building in a tree of symlinks pointing to
  a readonly source.
........
  r57993 | georg.brandl | 2007-09-05 15:36:44 +0200 (Wed, 05 Sep 2007) | 2 lines

  Backport from Py3k: Bug #1684991: explain lookup semantics for __special__ methods (new-style classes only).
........
  r58004 | armin.rigo | 2007-09-06 10:30:51 +0200 (Thu, 06 Sep 2007) | 4 lines

  Patch #1733973 by peaker:
  ptrace_enter_call() assumes no exception is currently set.
  This assumption is broken when throwing into a generator.
........
  r58006 | armin.rigo | 2007-09-06 11:30:38 +0200 (Thu, 06 Sep 2007) | 4 lines

  PyDict_GetItem() returns a borrowed reference.
  This attack is against ceval.c:IMPORT_NAME, which calls an
  object (__builtin__.__import__) without holding a reference to it.
........
  r58013 | georg.brandl | 2007-09-06 16:49:56 +0200 (Thu, 06 Sep 2007) | 2 lines

  Backport from 3k: #1116: fix reference to old filename.
........
  r58021 | thomas.heller | 2007-09-06 22:26:20 +0200 (Thu, 06 Sep 2007) | 1 line

  Fix typo:  c_float represents to C float type.
........
  r58022 | skip.montanaro | 2007-09-07 00:29:06 +0200 (Fri, 07 Sep 2007) | 3 lines

  If this is correct for py3k branch and it's already in the release25-maint
  branch, seems like it ought to be on the trunk as well.
........
  r58023 | gregory.p.smith | 2007-09-07 00:59:59 +0200 (Fri, 07 Sep 2007) | 4 lines

  Apply the fix from Issue1112 to make this test more robust and keep
  windows happy.
........
  r58031 | brett.cannon | 2007-09-07 05:17:50 +0200 (Fri, 07 Sep 2007) | 4 lines

  Make uuid1 and uuid4 tests conditional on whether ctypes can be imported;
  implementation of either function depends on ctypes but uuid as a whole does
  not.
........
  r58032 | brett.cannon | 2007-09-07 06:18:30 +0200 (Fri, 07 Sep 2007) | 6 lines

  Fix a crasher where Python code managed to infinitely recurse in C code without
  ever going back out to Python code in PyObject_Call().  Required introducing a
  static RuntimeError instance so that normalizing an exception there is no
  reliance on a recursive call that would put the exception system over the
  recursion check itself.
........
  r58034 | thomas.heller | 2007-09-07 08:32:17 +0200 (Fri, 07 Sep 2007) | 1 line

  Add a 'c_longdouble' type to the ctypes module.
........
  r58035 | thomas.heller | 2007-09-07 11:30:40 +0200 (Fri, 07 Sep 2007) | 1 line

  Remove unneeded #include.
........
  r58036 | thomas.heller | 2007-09-07 11:33:24 +0200 (Fri, 07 Sep 2007) | 6 lines

  Backport from py3k branch:

  Add a workaround for a strange bug on win64, when _ctypes is compiled
  with the SDK compiler.  This should fix the failing
  Lib\ctypes\test\test_as_parameter.py test.
........
  r58037 | georg.brandl | 2007-09-07 16:14:40 +0200 (Fri, 07 Sep 2007) | 2 lines

  Fix a wrong indentation for sublists.
........
  r58043 | georg.brandl | 2007-09-07 22:10:49 +0200 (Fri, 07 Sep 2007) | 2 lines

  #1095: ln -f doesn't work portably, fix in Makefile.
........
  r58049 | skip.montanaro | 2007-09-08 02:34:17 +0200 (Sat, 08 Sep 2007) | 1 line

  be explicit about the actual location of the missing file
........

3 files changed, 77 insertions, 5 deletions

diff --git a/Parser/parsetok.c b/Parser/parsetok.c
index 708c26d..8d0704a 100644
--- a/Parser/parsetok.c
+++ b/Parser/parsetok.c
@@ -213,16 +213,24 @@ parsetok(struct tok_state *tok, grammar *g, int start, perrdetail *err_ret,
 			err_ret->error = E_EOF;
 		err_ret->lineno = tok->lineno;
 		if (tok->buf != NULL) {
+			char *text = NULL;
 			size_t len;
 			assert(tok->cur - tok->buf < INT_MAX);
 			err_ret->offset = (int)(tok->cur - tok->buf);
 			len = tok->inp - tok->buf;
-			err_ret->text = (char *) PyObject_MALLOC(len + 1);
-			if (err_ret->text != NULL) {
-				if (len > 0)
-					strncpy(err_ret->text, tok->buf, len);
-				err_ret->text[len] = '\0';
+#ifdef Py_USING_UNICODE
+			text = PyTokenizer_RestoreEncoding(tok, len, &err_ret->offset);
+
+#endif
+			if (text == NULL) {
+				text = (char *) PyObject_MALLOC(len + 1);
+				if (text != NULL) {
+					if (len > 0)
+						strncpy(text, tok->buf, len);
+					text[len] = '\0';
+				}
 			}
+			err_ret->text = text;
 		}
 	} else if (tok->encoding != NULL) {
 		node* r = PyNode_New(encoding_decl);
diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c
index 7f51e14..e01c03e 100644
--- a/Parser/tokenizer.c
+++ b/Parser/tokenizer.c
@@ -1536,6 +1536,68 @@ PyTokenizer_Get(struct tok_state *tok, char **p_start, char **p_end)
 	return result;
 }
 
+/* This function is only called from parsetok. However, it cannot live
+   there, as it must be empty for PGEN, and we can check for PGEN only
+   in this file. */
+
+#ifdef PGEN
+char*
+PyTokenizer_RestoreEncoding(struct tok_state* tok, int len, int* offset)
+{
+	return NULL;
+}
+#else
+static PyObject *
+dec_utf8(const char *enc, const char *text, size_t len) {
+	PyObject *ret = NULL;	
+	PyObject *unicode_text = PyUnicode_DecodeUTF8(text, len, "replace");
+	if (unicode_text) {
+		ret = PyUnicode_AsEncodedString(unicode_text, enc, "replace");
+		Py_DECREF(unicode_text);
+	}
+	if (!ret) {
+		PyErr_Print();
+	}
+	return ret;
+}
+
+char *
+PyTokenizer_RestoreEncoding(struct tok_state* tok, int len, int *offset)
+{
+	char *text = NULL;
+	if (tok->encoding) {
+		/* convert source to original encondig */
+		PyObject *lineobj = dec_utf8(tok->encoding, tok->buf, len);
+		if (lineobj != NULL) {
+			int linelen = PyString_Size(lineobj);
+			const char *line = PyString_AsString(lineobj);
+			text = PyObject_MALLOC(linelen + 1);
+			if (text != NULL && line != NULL) {
+				if (linelen)
+					strncpy(text, line, linelen);
+				text[linelen] = '\0';
+			}
+			Py_DECREF(lineobj);
+					
+			/* adjust error offset */
+			if (*offset > 1) {
+				PyObject *offsetobj = dec_utf8(tok->encoding, 
+							       tok->buf, *offset-1);
+				if (offsetobj) {
+					*offset = PyString_Size(offsetobj) + 1;
+					Py_DECREF(offsetobj);
+				}
+			}
+			
+		}
+	}
+	return text;
+
+}
+#endif
+
+			   
+
 #ifdef Py_DEBUG
 
 void
diff --git a/Parser/tokenizer.h b/Parser/tokenizer.h
index ba90a5f..f38ad60 100644
--- a/Parser/tokenizer.h
+++ b/Parser/tokenizer.h
@@ -59,6 +59,8 @@ extern struct tok_state *PyTokenizer_FromFile(FILE *, char*,
 					      char *, char *);
 extern void PyTokenizer_Free(struct tok_state *);
 extern int PyTokenizer_Get(struct tok_state *, char **, char **);
+extern char * PyTokenizer_RestoreEncoding(struct tok_state* tok, 
+					  int len, int *offset);
 
 #ifdef __cplusplus
 }