gh-47146: Soft-deprecate structmember.h, expose its contents via Python.h (GH-99014)

The ``structmember.h`` header is deprecated, though it continues to be available
and there are no plans to remove it. There are no deprecation warnings. Old code
can stay unchanged (unless the extra include and non-namespaced macros bother
you greatly). Specifically, no uses in CPython are updated -- that would just be
unnecessary churn.
The ``structmember.h`` header is deprecated, though it continues to be
available and there are no plans to remove it.

Its contents are now available just by including ``Python.h``,
with a ``Py`` prefix added if it was missing:

- `PyMemberDef`, `PyMember_GetOne` and`PyMember_SetOne`
- Type macros like `Py_T_INT`, `Py_T_DOUBLE`, etc.
  (previously ``T_INT``, ``T_DOUBLE``, etc.)
- The flags `Py_READONLY` (previously ``READONLY``) and
  `Py_AUDIT_READ` (previously all uppercase)

Several items are not exposed from ``Python.h``:

- `T_OBJECT` (use `Py_T_OBJECT_EX`)
- `T_NONE` (previously undocumented, and pretty quirky)
- The macro ``WRITE_RESTRICTED`` which does nothing.
- The macros ``RESTRICTED`` and ``READ_RESTRICTED``, equivalents of
  `Py_AUDIT_READ`.
- In some configurations, ``<stddef.h>`` is not included from ``Python.h``.
  It should be included manually when using ``offsetof()``.

The deprecated header continues to provide its original
contents under the original names.
Your old code can stay unchanged, unless the extra include and non-namespaced
macros bother you greatly.

There is discussion on the issue to rename `T_PYSSIZET` to `PY_T_SSIZE` or
similar. I chose not to do that -- users will probably copy/paste that with any
spelling, and not renaming it makes migration docs simpler.


Co-Authored-By: Alexander Belopolsky <abalkin@users.noreply.github.com>
Co-Authored-By: Matthias Braun <MatzeB@users.noreply.github.com>

8 files changed, 233 insertions, 109 deletions

diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst
index 5a20f07..827d624 100644
--- a/Doc/c-api/structures.rst
+++ b/Doc/c-api/structures.rst
@@ -385,86 +385,67 @@ Accessing attributes of extension types
 .. c:type:: PyMemberDef
 
    Structure which describes an attribute of a type which corresponds to a C
-   struct member.  Its fields are:
+   struct member.  Its fields are, in order:
 
-   .. c:member:: const char* PyMemberDef.name
+   .. c:member:: const char* name
 
-         Name of the member
+         Name of the member.
+         A NULL value marks the end of a ``PyMemberDef[]`` array.
 
-   .. c:member:: int PyMemberDef.type
-
-      The type of the member in the C struct.
+         The string should be static, no copy is made of it.
 
    .. c:member:: Py_ssize_t PyMemberDef.offset
 
       The offset in bytes that the member is located on the type’s object struct.
 
-   .. c:member:: int PyMemberDef.flags
-
-      Flag bits indicating if the field should be read-only or writable.
-
-   .. c:member:: const char* PyMemberDef.doc
-
-      Points to the contents of the docstring.
-
-   :c:member:`PyMemberDef.type` can be one of many ``T_`` macros corresponding to various C
-   types.  When the member is accessed in Python, it will be converted to the
-   equivalent Python type.
-
-   =============== ==================
-   Macro name      C type
-   =============== ==================
-   T_SHORT         short
-   T_INT           int
-   T_LONG          long
-   T_FLOAT         float
-   T_DOUBLE        double
-   T_STRING        const char \*
-   T_OBJECT        PyObject \*
-   T_OBJECT_EX     PyObject \*
-   T_CHAR          char
-   T_BYTE          char
-   T_UBYTE         unsigned char
-   T_UINT          unsigned int
-   T_USHORT        unsigned short
-   T_ULONG         unsigned long
-   T_BOOL          char
-   T_LONGLONG      long long
-   T_ULONGLONG     unsigned long long
-   T_PYSSIZET      Py_ssize_t
-   =============== ==================
-
-   :c:macro:`T_OBJECT` and :c:macro:`T_OBJECT_EX` differ in that
-   :c:macro:`T_OBJECT` returns ``None`` if the member is ``NULL`` and
-   :c:macro:`T_OBJECT_EX` raises an :exc:`AttributeError`.  Try to use
-   :c:macro:`T_OBJECT_EX` over :c:macro:`T_OBJECT` because :c:macro:`T_OBJECT_EX`
-   handles use of the :keyword:`del` statement on that attribute more correctly
-   than :c:macro:`T_OBJECT`.
-
-   :c:member:`PyMemberDef.flags` can be ``0`` for write and read access or :c:macro:`READONLY` for
-   read-only access.  Using :c:macro:`T_STRING` for :attr:`type` implies
-   :c:macro:`READONLY`.  :c:macro:`T_STRING` data is interpreted as UTF-8.
-   Only :c:macro:`T_OBJECT` and :c:macro:`T_OBJECT_EX`
-   members can be deleted.  (They are set to ``NULL``).
+   .. c:member:: int type
+
+      The type of the member in the C struct.
+      See :ref:`PyMemberDef-types` for the possible values.
+
+   .. c:member:: int flags
+
+      Zero or more of the :ref:`PyMemberDef-flags`, combined using bitwise OR.
+
+   .. c:member:: const char* doc
+
+      The docstring, or NULL.
+      The string should be static, no copy is made of it.
+      Typically, it is defined using :c:macro:`PyDoc_STR`.
+
+   By default (when :c:member:`flags` is ``0``), members allow
+   both read and write access.
+   Use the :c:macro:`Py_READONLY` flag for read-only access.
+   Certain types, like :c:macro:`Py_T_STRING`, imply :c:macro:`Py_READONLY`.
+   Only :c:macro:`Py_T_OBJECT_EX` (and legacy :c:macro:`T_OBJECT`) members can
+   be deleted.
 
    .. _pymemberdef-offsets:
 
-   Heap allocated types (created using :c:func:`PyType_FromSpec` or similar),
-   ``PyMemberDef`` may contain definitions for the special member
-   ``__vectorcalloffset__``, corresponding to
+   For heap-allocated types (created using :c:func:`PyType_FromSpec` or similar),
+   ``PyMemberDef`` may contain a definition for the special member
+   ``"__vectorcalloffset__"``, corresponding to
    :c:member:`~PyTypeObject.tp_vectorcall_offset` in type objects.
-   These must be defined with ``T_PYSSIZET`` and ``READONLY``, for example::
+   These must be defined with ``Py_T_PYSSIZET`` and ``Py_READONLY``, for example::
 
       static PyMemberDef spam_type_members[] = {
-          {"__vectorcalloffset__", T_PYSSIZET, offsetof(Spam_object, vectorcall), READONLY},
+          {"__vectorcalloffset__", Py_T_PYSSIZET,
+           offsetof(Spam_object, vectorcall), Py_READONLY},
           {NULL}  /* Sentinel */
       };
 
+   (You may need to ``#include <stddef.h>`` for :c:func:`!offsetof`.)
+
    The legacy offsets :c:member:`~PyTypeObject.tp_dictoffset` and
-   :c:member:`~PyTypeObject.tp_weaklistoffset` are still supported, but extensions are
-   strongly encouraged to use ``Py_TPFLAGS_MANAGED_DICT`` and
-   ``Py_TPFLAGS_MANAGED_WEAKREF`` instead.
+   :c:member:`~PyTypeObject.tp_weaklistoffset` can be defined similarly using
+   ``"__dictoffset__"`` and ``"__weaklistoffset__"`` members, but extensions
+   are strongly encouraged to use :const:`Py_TPFLAGS_MANAGED_DICT` and
+   :const:`Py_TPFLAGS_MANAGED_WEAKREF` instead.
 
+   .. versionchanged:: 3.12
+
+      ``PyMemberDef`` is always available.
+      Previously, it required including ``"structmember.h"``.
 
 .. c:function:: PyObject* PyMember_GetOne(const char *obj_addr, struct PyMemberDef *m)
 
@@ -472,6 +453,10 @@ Accessing attributes of extension types
    attribute is described by ``PyMemberDef`` *m*.  Returns ``NULL``
    on error.
 
+   .. versionchanged:: 3.12
+
+      ``PyMember_GetOne`` is always available.
+      Previously, it required including ``"structmember.h"``.
 
 .. c:function:: int PyMember_SetOne(char *obj_addr, struct PyMemberDef *m, PyObject *o)
 
@@ -479,6 +464,144 @@ Accessing attributes of extension types
    The attribute to set is described by ``PyMemberDef`` *m*.  Returns ``0``
    if successful and a negative value on failure.
 
+   .. versionchanged:: 3.12
+
+      ``PyMember_SetOne`` is always available.
+      Previously, it required including ``"structmember.h"``.
+
+.. _PyMemberDef-flags:
+
+Member flags
+^^^^^^^^^^^^
+
+The following flags can be used with :c:member:`PyMemberDef.flags`:
+
+.. c:macro:: Py_READONLY
+
+   Not writable.
+
+.. c:macro:: Py_AUDIT_READ
+
+   Emit an ``object.__getattr__`` :ref:`audit event <audit-events>`
+   before reading.
+
+.. index::
+   single: READ_RESTRICTED
+   single: WRITE_RESTRICTED
+   single: RESTRICTED
+
+.. versionchanged:: 3.10
+
+   The :const:`!RESTRICTED`, :const:`!READ_RESTRICTED` and
+   :const:`!WRITE_RESTRICTED` macros available with
+   ``#include "structmember.h"`` are deprecated.
+   :const:`!READ_RESTRICTED` and :const:`!RESTRICTED` are equivalent to
+   :const:`Py_AUDIT_READ`; :const:`!WRITE_RESTRICTED` does nothing.
+
+.. index::
+   single: READONLY
+
+.. versionchanged:: 3.12
+
+   The :const:`!READONLY` macro was renamed to :const:`Py_READONLY`.
+   The :const:`!PY_AUDIT_READ` macro was renamed with the ``Py_`` prefix.
+   The new names are now always available.
+   Previously, these required ``#include "structmember.h"``.
+   The header is still available and it provides the old names.
+
+.. _PyMemberDef-types:
+
+Member types
+^^^^^^^^^^^^
+
+:c:member:`PyMemberDef.type` can be one of the following macros corresponding
+to various C types.
+When the member is accessed in Python, it will be converted to the
+equivalent Python type.
+When it is set from Python, it will be converted back to the C type.
+If that is not possible, an exception such as :exc:`TypeError` or
+:exc:`ValueError` is raised.
+
+Unless marked (D), attributes defined this way cannot be deleted
+using e.g. :keyword:`del` or :py:func:`delattr`.
+
+================================ ============================= ======================
+Macro name                       C type                        Python type
+================================ ============================= ======================
+.. c:macro:: Py_T_BYTE           :c:expr:`char`                :py:class:`int`
+.. c:macro:: Py_T_SHORT          :c:expr:`short`               :py:class:`int`
+.. c:macro:: Py_T_INT            :c:expr:`int`                 :py:class:`int`
+.. c:macro:: Py_T_LONG           :c:expr:`long`                :py:class:`int`
+.. c:macro:: Py_T_LONGLONG       :c:expr:`long long`           :py:class:`int`
+.. c:macro:: Py_T_UBYTE          :c:expr:`unsigned char`       :py:class:`int`
+.. c:macro:: Py_T_UINT           :c:expr:`unsigned int`        :py:class:`int`
+.. c:macro:: Py_T_USHORT         :c:expr:`unsigned short`      :py:class:`int`
+.. c:macro:: Py_T_ULONG          :c:expr:`unsigned long`       :py:class:`int`
+.. c:macro:: Py_T_ULONGLONG      :c:expr:`unsigned long long`  :py:class:`int`
+.. c:macro:: Py_T_PYSSIZET       :c:expr:`Py_ssize_t`          :py:class:`int`
+.. c:macro:: Py_T_FLOAT          :c:expr:`float`               :py:class:`float`
+.. c:macro:: Py_T_DOUBLE         :c:expr:`double`              :py:class:`float`
+.. c:macro:: Py_T_BOOL           :c:expr:`char`                :py:class:`bool`
+                                 (written as 0 or 1)
+.. c:macro:: Py_T_STRING         :c:expr:`const char *` (*)    :py:class:`str` (RO)
+.. c:macro:: Py_T_STRING_INPLACE :c:expr:`const char[]` (*)    :py:class:`str` (RO)
+.. c:macro:: Py_T_CHAR           :c:expr:`char` (0-127)        :py:class:`str` (**)
+.. c:macro:: Py_T_OBJECT_EX      :c:expr:`PyObject *`          :py:class:`object` (D)
+================================ ============================= ======================
+
+   (*): Zero-terminated, UTF8-encoded C string.
+   With :c:macro:`!Py_T_STRING` the C representation is a pointer;
+   with :c:macro:`!Py_T_STRING_INLINE` the string is stored directly
+   in the structure.
+
+   (**): String of length 1. Only ASCII is accepted.
+
+   (RO): Implies :c:macro:`Py_READONLY`.
+
+   (D): Can be deleted, in which case the pointer is set to ``NULL``.
+   Reading a ``NULL`` pointer raises :py:exc:`AttributeError`.
+
+.. index::
+   single: T_BYTE
+   single: T_SHORT
+   single: T_INT
+   single: T_LONG
+   single: T_LONGLONG
+   single: T_UBYTE
+   single: T_USHORT
+   single: T_UINT
+   single: T_ULONG
+   single: T_ULONGULONG
+   single: T_PYSSIZET
+   single: T_FLOAT
+   single: T_DOUBLE
+   single: T_BOOL
+   single: T_CHAR
+   single: T_STRING
+   single: T_STRING_INPLACE
+   single: T_OBJECT_EX
+   single: structmember.h
+
+.. versionadded:: 3.12
+
+   In previous versions, the macros were only available with
+   ``#include "structmember.h"`` and were named without the ``Py_`` prefix
+   (e.g. as ``T_INT``).
+   The header is still available and contains the old names, along with
+   the following deprecated types:
+
+   .. c:macro:: T_OBJECT
+
+      Like ``Py_T_OBJECT_EX``, but ``NULL`` is converted to ``None``.
+      This results in surprising behavior in Python: deleting the attribute
+      effectively sets it to ``None``.
+
+   .. c:macro:: T_NONE
+
+      Always ``None``. Must be used with :c:macro:`Py_READONLY`.
+
+Defining Getters and Setters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. c:type:: PyGetSetDef
 
diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat
index db8fc15..53895bb 100644
--- a/Doc/data/stable_abi.dat
+++ b/Doc/data/stable_abi.dat
@@ -386,6 +386,8 @@ function,PyMem_Malloc,3.2,,
 function,PyMem_Realloc,3.2,,
 type,PyMemberDef,3.2,,full-abi
 var,PyMemberDescr_Type,3.2,,
+function,PyMember_GetOne,3.2,,
+function,PyMember_SetOne,3.2,,
 function,PyMemoryView_FromBuffer,3.11,,
 function,PyMemoryView_FromMemory,3.7,,
 function,PyMemoryView_FromObject,3.2,,
diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst
index 3de849a..80a1387 100644
--- a/Doc/extending/newtypes.rst
+++ b/Doc/extending/newtypes.rst
@@ -286,36 +286,11 @@ be read-only or read-write.  The structures in the table are defined as::
 
 For each entry in the table, a :term:`descriptor` will be constructed and added to the
 type which will be able to extract a value from the instance structure.  The
-:attr:`type` field should contain one of the type codes defined in the
-:file:`structmember.h` header; the value will be used to determine how to
+:attr:`type` field should contain a type code like :c:macro:`Py_T_INT` or
+:c:macro:`Py_T_DOUBLE`; the value will be used to determine how to
 convert Python values to and from C values.  The :attr:`flags` field is used to
-store flags which control how the attribute can be accessed.
-
-The following flag constants are defined in :file:`structmember.h`; they may be
-combined using bitwise-OR.
-
-+---------------------------+----------------------------------------------+
-| Constant                  | Meaning                                      |
-+===========================+==============================================+
-| :const:`READONLY`         | Never writable.                              |
-+---------------------------+----------------------------------------------+
-| :const:`PY_AUDIT_READ`    | Emit an ``object.__getattr__``               |
-|                           | :ref:`audit events <audit-events>` before    |
-|                           | reading.                                     |
-+---------------------------+----------------------------------------------+
-
-.. versionchanged:: 3.10
-   :const:`RESTRICTED`, :const:`READ_RESTRICTED` and :const:`WRITE_RESTRICTED`
-   are deprecated. However, :const:`READ_RESTRICTED` is an alias for
-   :const:`PY_AUDIT_READ`, so fields that specify either :const:`RESTRICTED`
-   or :const:`READ_RESTRICTED` will also raise an audit event.
-
-.. index::
-   single: READONLY
-   single: READ_RESTRICTED
-   single: WRITE_RESTRICTED
-   single: RESTRICTED
-   single: PY_AUDIT_READ
+store flags which control how the attribute can be accessed: you can set it to
+:c:macro:`Py_READONLY` to prevent Python code from setting it.
 
 An interesting advantage of using the :c:member:`~PyTypeObject.tp_members` table to build
 descriptors that are used at runtime is that any attribute defined this way can
diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst
index 5d4a3f0..54de3fd 100644
--- a/Doc/extending/newtypes_tutorial.rst
+++ b/Doc/extending/newtypes_tutorial.rst
@@ -239,13 +239,6 @@ adds these capabilities:
 
 This version of the module has a number of changes.
 
-We've added an extra include::
-
-   #include <structmember.h>
-
-This include provides declarations that we use to handle attributes, as
-described a bit later.
-
 The  :class:`Custom` type now has three data attributes in its C struct,
 *first*, *last*, and *number*.  The *first* and *last* variables are Python
 strings containing first and last names.  The *number* attribute is a C integer.
@@ -436,11 +429,11 @@ We want to expose our instance variables as attributes. There are a
 number of ways to do that. The simplest way is to define member definitions::
 
    static PyMemberDef Custom_members[] = {
-       {"first", T_OBJECT_EX, offsetof(CustomObject, first), 0,
+       {"first", Py_T_OBJECT_EX, offsetof(CustomObject, first), 0,
         "first name"},
-       {"last", T_OBJECT_EX, offsetof(CustomObject, last), 0,
+       {"last", Py_T_OBJECT_EX, offsetof(CustomObject, last), 0,
         "last name"},
-       {"number", T_INT, offsetof(CustomObject, number), 0,
+       {"number", Py_T_INT, offsetof(CustomObject, number), 0,
         "custom number"},
        {NULL}  /* Sentinel */
    };
@@ -609,7 +602,7 @@ above.  In this case, we aren't using a closure, so we just pass ``NULL``.
 We also remove the member definitions for these attributes::
 
    static PyMemberDef Custom_members[] = {
-       {"number", T_INT, offsetof(CustomObject, number), 0,
+       {"number", Py_T_INT, offsetof(CustomObject, number), 0,
         "custom number"},
        {NULL}  /* Sentinel */
    };
diff --git a/Doc/includes/custom2.c b/Doc/includes/custom2.c
index aee9e1b..6638b9f 100644
--- a/Doc/includes/custom2.c
+++ b/Doc/includes/custom2.c
@@ -1,6 +1,6 @@
 #define PY_SSIZE_T_CLEAN
 #include <Python.h>
-#include "structmember.h"
+#include <stddef.h> /* for offsetof() */
 
 typedef struct {
     PyObject_HEAD
@@ -63,11 +63,11 @@ Custom_init(CustomObject *self, PyObject *args, PyObject *kwds)
 }
 
 static PyMemberDef Custom_members[] = {
-    {"first", T_OBJECT_EX, offsetof(CustomObject, first), 0,
+    {"first", Py_T_OBJECT_EX, offsetof(CustomObject, first), 0,
      "first name"},
-    {"last", T_OBJECT_EX, offsetof(CustomObject, last), 0,
+    {"last", Py_T_OBJECT_EX, offsetof(CustomObject, last), 0,
      "last name"},
-    {"number", T_INT, offsetof(CustomObject, number), 0,
+    {"number", Py_T_INT, offsetof(CustomObject, number), 0,
      "custom number"},
     {NULL}  /* Sentinel */
 };
diff --git a/Doc/includes/custom3.c b/Doc/includes/custom3.c
index 8d88bc2..0faf2bd 100644
--- a/Doc/includes/custom3.c
+++ b/Doc/includes/custom3.c
@@ -1,6 +1,6 @@
 #define PY_SSIZE_T_CLEAN
 #include <Python.h>
-#include "structmember.h"
+#include <stddef.h> /* for offsetof() */
 
 typedef struct {
     PyObject_HEAD
@@ -63,7 +63,7 @@ Custom_init(CustomObject *self, PyObject *args, PyObject *kwds)
 }
 
 static PyMemberDef Custom_members[] = {
-    {"number", T_INT, offsetof(CustomObject, number), 0,
+    {"number", Py_T_INT, offsetof(CustomObject, number), 0,
      "custom number"},
     {NULL}  /* Sentinel */
 };
diff --git a/Doc/includes/custom4.c b/Doc/includes/custom4.c
index ad240ae..b725bc0 100644
--- a/Doc/includes/custom4.c
+++ b/Doc/includes/custom4.c
@@ -1,6 +1,6 @@
 #define PY_SSIZE_T_CLEAN
 #include <Python.h>
-#include "structmember.h"
+#include <stddef.h> /* for offsetof() */
 
 typedef struct {
     PyObject_HEAD
@@ -79,7 +79,7 @@ Custom_init(CustomObject *self, PyObject *args, PyObject *kwds)
 }
 
 static PyMemberDef Custom_members[] = {
-    {"number", T_INT, offsetof(CustomObject, number), 0,
+    {"number", Py_T_INT, offsetof(CustomObject, number), 0,
      "custom number"},
     {NULL}  /* Sentinel */
 };
diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst
index f8786c1..8e9a4f0 100644
--- a/Doc/whatsnew/3.12.rst
+++ b/Doc/whatsnew/3.12.rst
@@ -849,6 +849,37 @@ Deprecated
 * Creating :c:data:`immutable types <Py_TPFLAGS_IMMUTABLETYPE>` with mutable
   bases is deprecated and will be disabled in Python 3.14.
 
+* The ``structmember.h`` header is deprecated, though it continues to be
+  available and there are no plans to remove it.
+
+  Its contents are now available just by including ``Python.h``,
+  with a ``Py`` prefix added if it was missing:
+
+  - :c:struct:`PyMemberDef`, :c:func:`PyMember_GetOne` and
+    :c:func:`PyMember_SetOne`
+  - Type macros like :c:macro:`Py_T_INT`, :c:macro:`Py_T_DOUBLE`, etc.
+    (previously ``T_INT``, ``T_DOUBLE``, etc.)
+  - The flags :c:macro:`Py_READONLY` (previously ``READONLY``) and
+    :c:macro:`Py_AUDIT_READ` (previously all uppercase)
+
+  Several items are not exposed from ``Python.h``:
+
+  - :c:macro:`T_OBJECT` (use :c:macro:`Py_T_OBJECT_EX`)
+  - :c:macro:`T_NONE` (previously undocumented, and pretty quirky)
+  - The macro ``WRITE_RESTRICTED`` which does nothing.
+  - The macros ``RESTRICTED`` and ``READ_RESTRICTED``, equivalents of
+    :c:macro:`Py_AUDIT_READ`.
+  - In some configurations, ``<stddef.h>`` is not included from ``Python.h``.
+    It should be included manually when using ``offsetof()``.
+
+  The deprecated header continues to provide its original
+  contents under the original names.
+  Your old code can stay unchanged, unless the extra include and non-namespaced
+  macros bother you greatly.
+
+  (Contributed in :gh:`47146` by Petr Viktorin, based on
+  earlier work by Alexander Belopolsky and Matthias Braun.)
+
 
 Removed
 -------