Merge p3yk branch with the trunk up to revision 45595. This breaks a fair

number of tests, all because of the codecs/_multibytecodecs issue described
here (it's not a Py3K issue, just something Py3K discovers):
http://mail.python.org/pipermail/python-dev/2006-April/064051.html

Hye-Shik Chang promised to look for a fix, so no need to fix it here. The
tests that are expected to break are:

test_codecencodings_cn
test_codecencodings_hk
test_codecencodings_jp
test_codecencodings_kr
test_codecencodings_tw
test_codecs
test_multibytecodec

This merge fixes an actual test failure (test_weakref) in this branch,
though, so I believe merging is the right thing to do anyway.

1 files changed, 40 insertions, 1 deletions

diff --git a/Include/object.h b/Include/object.h
index 9198007..cdbddfe 100644
--- a/Include/object.h
+++ b/Include/object.h
@@ -327,6 +327,7 @@ typedef struct _typeobject {
 	Py_ssize_t tp_allocs;
 	Py_ssize_t tp_frees;
 	Py_ssize_t tp_maxalloc;
+	struct _typeobject *tp_prev;
 	struct _typeobject *tp_next;
 #endif
 } PyTypeObject;
@@ -566,6 +567,9 @@ environment the global variable trick is not safe.)
 PyAPI_DATA(Py_ssize_t) _Py_RefTotal;
 PyAPI_FUNC(void) _Py_NegativeRefcount(const char *fname,
 					    int lineno, PyObject *op);
+PyAPI_FUNC(PyObject *) _PyDict_Dummy(void);
+PyAPI_FUNC(PyObject *) _PySet_Dummy(void);
+PyAPI_FUNC(Py_ssize_t) _Py_GetRefTotal(void);
 #define _Py_INC_REFTOTAL	_Py_RefTotal++
 #define _Py_DEC_REFTOTAL	_Py_RefTotal--
 #define _Py_REF_DEBUG_COMMA	,
@@ -583,8 +587,9 @@ PyAPI_FUNC(void) _Py_NegativeRefcount(const char *fname,
 
 #ifdef COUNT_ALLOCS
 PyAPI_FUNC(void) inc_count(PyTypeObject *);
+PyAPI_FUNC(void) dec_count(PyTypeObject *);
 #define _Py_INC_TPALLOCS(OP)	inc_count((OP)->ob_type)
-#define _Py_INC_TPFREES(OP)	(OP)->ob_type->tp_frees++
+#define _Py_INC_TPFREES(OP)	dec_count((OP)->ob_type)
 #define _Py_DEC_TPFREES(OP)	(OP)->ob_type->tp_frees--
 #define _Py_COUNT_ALLOCS_COMMA	,
 #else
@@ -630,6 +635,40 @@ PyAPI_FUNC(void) _Py_AddToAllObjects(PyObject *, int force);
 	else						\
 		_Py_Dealloc((PyObject *)(op))
 
+/* Safely decref `op` and set `op` to NULL, especially useful in tp_clear
+ * and tp_dealloc implementatons.
+ *
+ * Note that "the obvious" code can be deadly:
+ *
+ *     Py_XDECREF(op);
+ *     op = NULL;
+ *
+ * Typically, `op` is something like self->containee, and `self` is done
+ * using its `containee` member.  In the code sequence above, suppose
+ * `containee` is non-NULL with a refcount of 1.  Its refcount falls to
+ * 0 on the first line, which can trigger an arbitrary amount of code,
+ * possibly including finalizers (like __del__ methods or weakref callbacks)
+ * coded in Python, which in turn can release the GIL and allow other threads
+ * to run, etc.  Such code may even invoke methods of `self` again, or cause
+ * cyclic gc to trigger, but-- oops! --self->containee still points to the
+ * object being torn down, and it may be in an insane state while being torn
+ * down.  This has in fact been a rich historic source of miserable (rare &
+ * hard-to-diagnose) segfaulting (and other) bugs.
+ *
+ * The safe way is:
+ *
+ *      Py_CLEAR(op);
+ *
+ * That arranges to set `op` to NULL _before_ decref'ing, so that any code
+ * triggered as a side-effect of `op` getting torn down no longer believes
+ * `op` points to a valid object.
+ *
+ * There are cases where it's safe to use the naive code, but they're brittle.
+ * For example, if `op` points to a Python integer, you know that destroying
+ * one of those can't cause problems -- but in part that relies on that
+ * Python integers aren't currently weakly referencable.  Best practice is
+ * to use Py_CLEAR() even if you can't think of a reason for why you need to.
+ */
 #define Py_CLEAR(op)				\
         do {                            	\
                 if (op) {			\