Issue #18112: PEP 442 implementation (safe object finalization).

7 files changed, 114 insertions, 43 deletions

diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst
index ea1a0ad..ea787de 100644
--- a/Doc/c-api/typeobj.rst
+++ b/Doc/c-api/typeobj.rst
@@ -465,6 +465,14 @@ type objects) *must* have the :attr:`ob_size` field.
       :const:`Py_TPFLAGS_HAVE_VERSION_TAG`.
 
 
+   .. data:: Py_TPFLAGS_HAVE_FINALIZE
+
+      This bit is set when the :attr:`tp_finalize` slot is present in the
+      type structure.
+
+      .. versionadded:: 3.4
+
+
 .. c:member:: char* PyTypeObject.tp_doc
 
    An optional pointer to a NUL-terminated C string giving the docstring for this
@@ -968,6 +976,47 @@ type objects) *must* have the :attr:`ob_size` field.
    This field is not inherited; it is calculated fresh by :c:func:`PyType_Ready`.
 
 
+.. c:member:: destructor PyTypeObject.tp_finalize
+
+   An optional pointer to an instance finalization function.  Its signature is
+   :c:type:`destructor`::
+
+      void tp_finalize(PyObject *)
+
+   If :attr:`tp_finalize` is set, the interpreter calls it once when
+   finalizing an instance.  It is called either from the garbage
+   collector (if the instance is part of an isolated reference cycle) or
+   just before the object is deallocated.  Either way, it is guaranteed
+   to be called before attempting to break reference cycles, ensuring
+   that it finds the object in a sane state.
+
+   :attr:`tp_finalize` should not mutate the current exception status;
+   therefore, a recommended way to write a non-trivial finalizer is::
+
+      static void
+      local_finalize(PyObject *self)
+      {
+          PyObject *error_type, *error_value, *error_traceback;
+
+          /* Save the current exception, if any. */
+          PyErr_Fetch(&error_type, &error_value, &error_traceback);
+
+          /* ... */
+
+          /* Restore the saved exception. */
+          PyErr_Restore(error_type, error_value, error_traceback);
+      }
+
+   For this field to be taken into account (even through inheritance),
+   you must also set the :const:`Py_TPFLAGS_HAVE_FINALIZE` flags bit.
+
+   This field is inherited by subtypes.
+
+   .. versionadded:: 3.4
+
+   .. seealso:: "Safe object finalization" (:pep:`442`)
+
+
 .. c:member:: PyObject* PyTypeObject.tp_cache
 
    Unused.  Not inherited.  Internal use only.
diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst
index cb20bce..d5ee877 100644
--- a/Doc/extending/newtypes.rst
+++ b/Doc/extending/newtypes.rst
@@ -157,7 +157,8 @@ to :const:`Py_TPFLAGS_DEFAULT`. ::
    Py_TPFLAGS_DEFAULT,        /* tp_flags */
 
 All types should include this constant in their flags.  It enables all of the
-members defined by the current version of Python.
+members defined until at least Python 3.3.  If you need further members,
+you will need to OR the corresponding flags.
 
 We provide a doc string for the type in :attr:`tp_doc`. ::
 
@@ -928,8 +929,9 @@ Finalization and De-allocation
 
 This function is called when the reference count of the instance of your type is
 reduced to zero and the Python interpreter wants to reclaim it.  If your type
-has memory to free or other clean-up to perform, put it here.  The object itself
-needs to be freed here as well.  Here is an example of this function::
+has memory to free or other clean-up to perform, you can put it here.  The
+object itself needs to be freed here as well.  Here is an example of this
+function::
 
    static void
    newdatatype_dealloc(newdatatypeobject * obj)
@@ -981,6 +983,22 @@ done.  This can be done using the :c:func:`PyErr_Fetch` and
        Py_TYPE(obj)->tp_free((PyObject*)self);
    }
 
+.. note::
+   There are limitations to what you can safely do in a deallocator function.
+   First, if your type supports garbage collection (using :attr:`tp_traverse`
+   and/or :attr:`tp_clear`), some of the object's members can have been
+   cleared or finalized by the time :attr:`tp_dealloc` is called.  Second, in
+   :attr:`tp_dealloc`, your object is in an unstable state: its reference
+   count is equal to zero.  Any call to a non-trivial object or API (as in the
+   example above) might end up calling :attr:`tp_dealloc` again, causing a
+   double free and a crash.
+
+   Starting with Python 3.4, it is recommended not to put any complex
+   finalization code in :attr:`tp_dealloc`, and instead use the new
+   :c:member:`~PyTypeObject.tp_finalize` type method.
+
+   .. seealso::
+      :pep:`442` explains the new finalization scheme.
 
 .. index::
    single: string; object representation
diff --git a/Doc/includes/typestruct.h b/Doc/includes/typestruct.h
index 32647c0..fcb846a 100644
--- a/Doc/includes/typestruct.h
+++ b/Doc/includes/typestruct.h
@@ -70,4 +70,11 @@ typedef struct _typeobject {
     PyObject *tp_subclasses;
     PyObject *tp_weaklist;
 
+    destructor tp_del;
+
+    /* Type attribute cache version tag. Added in version 2.6 */
+    unsigned int tp_version_tag;
+
+    destructor tp_finalize;
+
 } PyTypeObject;
diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst
index 95df2f8..c578690 100644
--- a/Doc/library/gc.rst
+++ b/Doc/library/gc.rst
@@ -176,24 +176,13 @@ values but should not rebind them):
 
 .. data:: garbage
 
-   A list of objects which the collector found to be unreachable but could not be
-   freed (uncollectable objects).  By default, this list contains only objects with
-   :meth:`__del__` methods. Objects that have :meth:`__del__` methods and are
-   part of a reference cycle cause the entire reference cycle to be uncollectable,
-   including objects not necessarily in the cycle but reachable only from it.
-   Python doesn't collect such cycles automatically because, in general, it isn't
-   possible for Python to guess a safe order in which to run the :meth:`__del__`
-   methods.  If you know a safe order, you can force the issue by examining the
-   *garbage* list, and explicitly breaking cycles due to your objects within the
-   list.  Note that these objects are kept alive even so by virtue of being in the
-   *garbage* list, so they should be removed from *garbage* too.  For example,
-   after breaking cycles, do ``del gc.garbage[:]`` to empty the list.  It's
-   generally better to avoid the issue by not creating cycles containing objects
-   with :meth:`__del__` methods, and *garbage* can be examined in that case to
-   verify that no such cycles are being created.
-
-   If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be added
-   to this list rather than freed.
+   A list of objects which the collector found to be unreachable but could
+   not be freed (uncollectable objects).  Starting with Python 3.4, this
+   list should be empty most of the time, except when using instances of
+   C extension types with a non-NULL ``tp_del`` slot.
+
+   If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be
+   added to this list rather than freed.
 
    .. versionchanged:: 3.2
       If this list is non-empty at interpreter shutdown, a
@@ -201,6 +190,10 @@ values but should not rebind them):
       :const:`DEBUG_UNCOLLECTABLE` is set, in addition all uncollectable objects
       are printed.
 
+   .. versionchanged:: 3.4
+      Following :pep:`442`, objects with a :meth:`__del__` method don't end
+      up in :attr:`gc.garbage` anymore.
+
 .. data:: callbacks
 
    A list of callbacks that will be invoked by the garbage collector before and
diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst
index 5b5e460..818a6fa 100644
--- a/Doc/library/weakref.rst
+++ b/Doc/library/weakref.rst
@@ -529,22 +529,13 @@ follows::
         def __del__(self):
             self.remove()
 
-This solution has a couple of serious problems:
-
-* There is no guarantee that the object will be garbage collected
-  before the program exists, so the directory might be left.  This is
-  because reference cycles containing an object with a :meth:`__del__`
-  method can never be collected.  And even if the :class:`TempDir`
-  object is not itself part of a reference cycle, it may still be kept
-  alive by some unkown uncollectable reference cycle.
-
-* The :meth:`__del__` method may be called at shutdown after the
-  :mod:`shutil` module has been cleaned up, in which case
-  :attr:`shutil.rmtree` will have been replaced by :const:`None`.
-  This will cause the :meth:`__del__` method to fail and the directory
-  will not be removed.
-
-Using finalizers we can avoid these problems::
+This solution has a serious problem: the :meth:`__del__` method may be
+called at shutdown after the :mod:`shutil` module has been cleaned up,
+in which case :attr:`shutil.rmtree` will have been replaced by :const:`None`.
+This will cause the :meth:`__del__` method to fail and the directory
+will not be removed.
+
+Using finalizers we can avoid this problem::
 
     class TempDir:
         def __init__(self):
diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
index 1a48c1f..95028c2 100644
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -1120,12 +1120,10 @@ Basic customization
       ``sys.last_traceback`` keeps the stack frame alive).  The first situation
       can only be remedied by explicitly breaking the cycles; the latter two
       situations can be resolved by storing ``None`` in ``sys.last_traceback``.
-      Circular references which are garbage are detected when the option cycle
-      detector is enabled (it's on by default), but can only be cleaned up if
-      there are no Python- level :meth:`__del__` methods involved. Refer to the
-      documentation for the :mod:`gc` module for more information about how
-      :meth:`__del__` methods are handled by the cycle detector, particularly
-      the description of the ``garbage`` value.
+      Circular references which are garbage are detected and cleaned up when
+      the cyclic garbage collector is enabled (it's on by default). Refer to the
+      documentation for the :mod:`gc` module for more information about this
+      topic.
 
    .. warning::
 
diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst
index b5be568..17cec3f 100644
--- a/Doc/whatsnew/3.4.rst
+++ b/Doc/whatsnew/3.4.rst
@@ -119,6 +119,21 @@ The :pep:`445` adds new Application Programming Interfaces (API) to customize
 Python memory allocators.
 
 
+.. _pep-442:
+
+PEP 442: Safe object finalization
+=================================
+
+This PEP removes the current limitations and quirks of object finalization.
+With it, objects with :meth:`__del__` methods, as well as generators
+with :keyword:`finally` clauses, can be finalized when they are part of a
+reference cycle.
+
+.. seealso::
+
+   :pep:`442` - Safe object finalization
+      PEP written and implemented by Antoine Pitrou
+
 
 Other Language Changes
 ======================