add documentation for the new buffer interface based on PEP 3118; I hope it's at least 60% right...

1 files changed, 256 insertions, 67 deletions

diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
index 70e526c..1d880c7 100644
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -6,19 +6,20 @@ Buffer Objects
 --------------
 
 .. sectionauthor:: Greg Stein <gstein@lyra.org>
+.. sectionauthor:: Benjamin Peterson
 
 
 .. index::
    object: buffer
    single: buffer interface
 
-Python objects implemented in C can export a group of functions called the
-"buffer interface."  These functions can be used by an object to expose its data
-in a raw, byte-oriented format. Clients of the object can use the buffer
-interface to access the object data directly, without needing to copy it first.
+Python objects implemented in C can export a "buffer interface."  These
+functions can be used by an object to expose its data in a raw, byte-oriented
+format. Clients of the object can use the buffer interface to access the object
+data directly, without needing to copy it first.
 
-Two examples of objects that support the buffer interface are strings and
-arrays. The string object exposes the character contents in the buffer
+Two examples of objects that support the buffer interface are bytes and
+arrays. The bytes object exposes the character contents in the buffer
 interface's byte-oriented form. An array can also expose its contents, but it
 should be noted that array elements may be multi-byte values.
 
@@ -33,87 +34,275 @@ returning data from the target object.
 More information on the buffer interface is provided in the section 
 :ref:`buffer-structs`, under the description for :ctype:`PyBufferProcs`.
 
-A "buffer object" is defined in the :file:`bufferobject.h` header (included by
-:file:`Python.h`). These objects look very similar to string objects at the
-Python programming level: they support slicing, indexing, concatenation, and
-some other standard string operations. However, their data can come from one of
-two sources: from a block of memory, or from another object which exports the
-buffer interface.
-
 Buffer objects are useful as a way to expose the data from another object's
-buffer interface to the Python programmer. They can also be used as a zero-copy
-slicing mechanism. Using their ability to reference a block of memory, it is
-possible to expose any data to the Python programmer quite easily. The memory
+buffer interface to the Python programmer.  They can also be used as a zero-copy
+slicing mechanism.  Using their ability to reference a block of memory, it is
+possible to expose any data to the Python programmer quite easily.  The memory
 could be a large, constant array in a C extension, it could be a raw block of
 memory for manipulation before passing to an operating system library, or it
 could be used to pass around structured data in its native, in-memory format.
 
 
-.. ctype:: PyBufferObject
-
-   This subtype of :ctype:`PyObject` represents a buffer object.
-
-
-.. cvar:: PyTypeObject PyBuffer_Type
-
-   .. index:: single: BufferType (in module types)
-
-   The instance of :ctype:`PyTypeObject` which represents the Python buffer type;
-   it is the same object as ``buffer`` and  ``types.BufferType`` in the Python
-   layer. .
-
-
-.. cvar:: int Py_END_OF_BUFFER
-
-   This constant may be passed as the *size* parameter to
-   :cfunc:`PyBuffer_FromObject` or :cfunc:`PyBuffer_FromReadWriteObject`.  It
-   indicates that the new :ctype:`PyBufferObject` should refer to *base* object
-   from the specified *offset* to the end of its exported buffer.  Using this
-   enables the caller to avoid querying the *base* object for its length.
-
+.. ctype:: Py_buffer
 
-.. cfunction:: int PyBuffer_Check(PyObject *p)
+   .. cmember:: void *buf
 
-   Return true if the argument has type :cdata:`PyBuffer_Type`.
+      A pointer to the start of the memory for the object.
 
+   .. cmember:: Py_ssize_t len
 
-.. cfunction:: PyObject* PyBuffer_FromObject(PyObject *base, Py_ssize_t offset, Py_ssize_t size)
+      The total length of the memory in bytes.
 
-   Return a new read-only buffer object.  This raises :exc:`TypeError` if *base*
-   doesn't support the read-only buffer protocol or doesn't provide exactly one
-   buffer segment, or it raises :exc:`ValueError` if *offset* is less than zero.
-   The buffer will hold a reference to the *base* object, and the buffer's contents
-   will refer to the *base* object's buffer interface, starting as position
-   *offset* and extending for *size* bytes. If *size* is :const:`Py_END_OF_BUFFER`,
-   then the new buffer's contents extend to the length of the *base* object's
-   exported buffer data.
+   .. cmember:: int readonly
 
+      An indicator of whether the buffer is read only.
 
-.. cfunction:: PyObject* PyBuffer_FromReadWriteObject(PyObject *base, Py_ssize_t offset, Py_ssize_t size)
+   .. cmember:: const char *format
 
-   Return a new writable buffer object.  Parameters and exceptions are similar to
-   those for :cfunc:`PyBuffer_FromObject`.  If the *base* object does not export
-   the writable buffer protocol, then :exc:`TypeError` is raised.
+      A *NULL* terminated string in :mod:`struct` module style syntax giving the
+      contents of the elements available through the buffer.  If this is *NULL*,
+      ``"B"`` (unsigned bytes) is assumed.
 
+   .. cmember:: int ndim
 
-.. cfunction:: PyObject* PyBuffer_FromMemory(void *ptr, Py_ssize_t size)
+      The number of dimensions the memory represents as a multi-dimensional
+      array.  If it is 0, :cdata:`strides` and :cdata:`suboffsets` must be
+      *NULL*.
 
-   Return a new read-only buffer object that reads from a specified location in
-   memory, with a specified size.  The caller is responsible for ensuring that the
-   memory buffer, passed in as *ptr*, is not deallocated while the returned buffer
-   object exists.  Raises :exc:`ValueError` if *size* is less than zero.  Note that
-   :const:`Py_END_OF_BUFFER` may *not* be passed for the *size* parameter;
-   :exc:`ValueError` will be raised in that case.
+   .. cmember:: Py_ssize_t *shape
 
+      An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim` giving the
+      shape of the memory as a multi-dimensional array.  Note that
+      ``((*shape)[0] * ... * (*shape)[ndims-1])*itemsize`` should be equal to
+      :cdata:`len`.
 
-.. cfunction:: PyObject* PyBuffer_FromReadWriteMemory(void *ptr, Py_ssize_t size)
+   .. cmember:: Py_ssize_t *strides
 
-   Similar to :cfunc:`PyBuffer_FromMemory`, but the returned buffer is writable.
+      An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim` giving the
+      number of bytes to skip to get to a new element in each dimension.
 
+   .. cmember:: Py_ssize_t *suboffsets
 
-.. cfunction:: PyObject* PyBuffer_New(Py_ssize_t size)
+      An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim`.  If these
+      suboffset numbers are greater than or equal to 0, then the value stored
+      along the indicated dimension is a pointer and the suboffset value
+      dictates how many bytes to add to the pointer after de-referencing. A
+      suboffset value that it negative indicates that no de-referencing should
+      occur (striding in a contiguous memory block).
 
-   Return a new writable buffer object that maintains its own memory buffer of
-   *size* bytes.  :exc:`ValueError` is returned if *size* is not zero or positive.
-   Note that the memory buffer (as returned by :cfunc:`PyObject_AsWriteBuffer`) is
-   not specifically aligned.
+      Here is a function that returns a pointer to the element in an N-D array
+      pointed to by an N-dimesional index when there are both non-NULL strides
+      and suboffsets::
+
+          void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides,
+              Py_ssize_t *suboffsets, Py_ssize_t *indices) {
+              char *pointer = (char*)buf;
+              int i;
+              for (i = 0; i < ndim; i++) {
+                  pointer += strides[i] * indices[i];
+                  if (suboffsets[i] >=0 ) {
+                      pointer = *((char**)pointer) + suboffsets[i];
+                  }                     
+              }
+              return (void*)pointer;
+           }
+
+
+   .. cmember:: Py_ssize_t itemsize
+
+      This is a storage for the itemsize (in bytes) of each element of the
+      shared memory. It is technically un-necessary as it can be obtained using
+      :cfunc:`PyBuffer_SizeFromFormat`, however an exporter may know this
+      information without parsing the format string and it is necessary to know
+      the itemsize for proper interpretation of striding. Therefore, storing it
+      is more convenient and faster.
+
+   .. cmember:: void *internal
+
+      This is for use internally by the exporting object. For example, this
+      might be re-cast as an integer by the exporter and used to store flags
+      about whether or not the shape, strides, and suboffsets arrays must be
+      freed when the buffer is released. The consumer should never alter this
+      value.
+
+
+Buffer related functions
+========================
+
+
+.. cfunction:: int PyObject_CheckBuffer(PyObject *obj)
+
+   Return 1 if *obj* supports the buffer interface otherwise 0.
+
+
+.. cfunction:: int PyObject_GetBuffer(PyObject *obj, PyObject *view, int flags)
+
+      Export *obj* into a :ctype:`Py_buffer`, *view*.  These arguments must
+      never be *NULL*.  The *flags* argument is a bit field indicating what kind
+      of buffer the caller is prepared to deal with and therefore what kind of
+      buffer the exporter is allowed to return.  The buffer interface allows for
+      complicated memory sharing possibilities, but some caller may not be able
+      to handle all the complexibity but may want to see if the exporter will
+      let them take a simpler view to its memory.
+
+      Some exporters may not be able to share memory in every possible way and
+      may need to raise errors to signal to some consumers that something is
+      just not possible. These errors should be a :exc:`BufferError` unless
+      there is another error that is actually causing the problem. The exporter
+      can use flags information to simplify how much of the :cdata:`Py_buffer`
+      structure is filled in with non-default values and/or raise an error if
+      the object can't support a simpler view of its memory.
+
+      0 is returned on success and -1 on error.
+
+      The following table gives possible values to the *flags* arguments.
+
+      +------------------------------+-----------------------------------------------+
+      | Flag                         | Description                                   |
+      +==============================+===============================================+
+      | :cmacro:`PyBUF_SIMPLE`       |This is the default flag state.  The returned  |
+      |                              |buffer may or may not have writable memory.    |
+      |                              |The format will be assumed to be unsigned bytes|
+      |                              |.  This is a "stand-alone" flag constant. It   |
+      |                              |never needs to be |'d to the others. The       |
+      |                              |exporter will raise an error if it cannot      |
+      |                              |provide such a contiguous buffer of bytes.     |
+      |                              |                                               |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_WRITABLE`     |The returned buffer must be writable. If it is |
+      |                              |not writable, then raise an error.             |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_STRIDES`      |This implies :cmacro:`PyBUF_ND`. The returned  |
+      |                              |buffer must provide strides information        |
+      |                              |(i.e. the strides cannot be NULL). This would  |
+      |                              |be used when the consumer can handle strided,  |
+      |                              |discontiguous arrays. Handling strides         |
+      |                              |automatically assumes you can handle shape. The|
+      |                              |exporter may raise an error if cannot provide a|
+      |                              |strided-only representation of the data        |
+      |                              |(i.e. without the suboffsets).                 |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_ND`           |The returned buffer must provide shape         |
+      |                              |information. The memory will be assumed C-style|
+      |                              |contiguous (last dimension varies the          |
+      |                              |fastest). The exporter may raise an error if it|
+      |                              |cannot provide this kind of contiguous         |
+      |                              |buffer. If this is not given then shape will be|
+      |                              |*NULL*.                                        |
+      |                              |                                               |
+      |                              |                                               |
+      +------------------------------+-----------------------------------------------+
+      |:cmacro:`PyBUF_C_CONTIGUOUS`  |These flags indicate that the contiguoity      |
+      |:cmacro:`PyBUF_F_CONTIGUOUS`  |returned buffer must be respectively,          |
+      |:cmacro:`PyBUF_ANY_CONTIGUOUS`|C-contiguous (last dimension varies the        |
+      |                              |fastest), Fortran contiguous (first dimension  |
+      |                              |varies the fastest) or either one.  All of     |
+      |                              |these flags imply :cmacro:`PyBUF_STRIDES` and  |
+      |                              |guarantee that the strides buffer info         |
+      |                              |structure will be filled in correctly.         |
+      |                              |                                               |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_INDIRECT`     |This implies :cmacro:`PyBUF_STRIDES`. The      |
+      |                              |returned buffer must have suboffsets           |
+      |                              |information (which can be NULL if no suboffsets|
+      |                              |are needed). This would be used when the       |
+      |                              |consumer can handle indirect array referencing |
+      |                              |implied by these suboffsets.                   |
+      |                              |                                               |
+      |                              |                                               |
+      |                              |                                               |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_FORMAT`       |The returned buffer must have true format      |
+      |                              |information if this flag is provided. This     |
+      |                              |would be used when the consumer is going to be |
+      |                              |checking for what 'kind' of data is actually   |
+      |                              |stored. An exporter should always be able to   |
+      |                              |provide this information if requested. If      |
+      |                              |format is not explicitly requested then the    |
+      |                              |format must be returned as *NULL* (which means |
+      |                              |``'B'``, or unsigned bytes)                    |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_STRIDED`      |This is equivalent to ``(PyBUF_STRIDES |       |
+      |                              |PyBUF_WRITABLE)``.                             |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_STRIDED_RO`   |This is equivalent to ``(PyBUF_STRIDES)``.     |
+      |                              |                                               |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_RECORDS`      |This is equivalent to ``(PyBUF_STRIDES |       |
+      |                              |PyBUF_FORMAT | PyBUF_WRITABLE)``.              |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_RECORDS_RO`   |This is equivalent to ``(PyBUF_STRIDES |       |
+      |                              |PyBUF_FORMAT)``.                               |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_FULL`         |This is equivalent to ``(PyBUF_INDIRECT |      |
+      |                              |PyBUF_FORMAT | PyBUF_WRITABLE)``.              |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_FULL_RO``     |This is equivalent to ``(PyBUF_INDIRECT |      |
+      |                              |PyBUF_FORMAT)``.                               |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_CONTIG`       |This is equivalent to ``(PyBUF_ND |            |
+      |                              |PyBUF_WRITABLE)``.                             |
+      +------------------------------+-----------------------------------------------+
+      | :cmacro:`PyBUF_CONTIG_RO`    |This is equivalent to ``(PyBUF_ND)``.          |
+      |                              |                                               |
+      +------------------------------+-----------------------------------------------+
+
+
+.. cfunction:: void PyBuffer_Release(PyObject *obj, Py_buffer *view)
+
+   Release the buffer *view* over *obj*.  This shouldd be called when the buffer
+   is no longer being used as it may free memory from it.
+
+
+.. cfunction:: Py_ssize_t PyBuffer_SizeFromFormat(const char *)
+
+   Return the implied :cdata:`~Py_buffer.itemsize` from the struct-stype
+   :cdata:`~Py_buffer.format`.
+
+
+.. cfunction:: int PyObject_CopyToObject(PyObject *obj, void *buf, Py_ssize_t len, char fortran)
+
+   Copy *len* bytes of data pointed to by the contiguous chunk of memory pointed
+   to by *buf* into the buffer exported by obj.  The buffer must of course be
+   writable.  Return 0 on success and return -1 and raise an error on failure.
+   If the object does not have a writable buffer, then an error is raised.  If
+   *fortran* is ``'F'``, then if the object is multi-dimensional, then the data
+   will be copied into the array in Fortran-style (first dimension varies the
+   fastest).  If *fortran* is ``'C'``, then the data will be copied into the
+   array in C-style (last dimension varies the fastest).  If *fortran* is
+   ``'A'``, then it does not matter and the copy will be made in whatever way is
+   more efficient.
+
+
+.. cfunction:: int PyBuffer_IsContiguous(Py_buffer *view, char fortran)
+
+   Return 1 if the memory defined by the *view* is C-style (*fortran* is
+   ``'C'``) or Fortran-style (*fortran* is ``'F'``) contiguous or either one
+   (*fortran* is ``'A'``).  Return 0 otherwise.
+
+
+.. cfunction:: void PyBuffer_FillContiguousStrides(int ndim, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char fortran)
+
+   Fill the *strides* array with byte-strides of a contiguous (C-style if
+   *fortran* is ``'C'`` or Fortran-style if *fortran* is ``'F'`` array of the
+   given shape with the given number of bytes per element.
+
+
+.. cfunction:: int PyBuffer_FillInfo(Py_buffer *view, void *buf, Py_ssize_t len, int readonly, int infoflags)
+
+   Fill in a buffer-info structure, *view*, correctly for an exporter that can
+   only share a contiguous chunk of memory of "unsigned bytes" of the given
+   length.  Return 0 on success and -1 (with raising an error) on error.
+
+
+MemoryView objects
+==================
+
+A memoryview object is an extended buffer object that could replace the buffer
+object (but doesn't have to as that could be kept as a simple 1-d memoryview
+object).  It, unlike :ctype:`Py_buffer`, is a Python object (exposed as
+:class:`memoryview` in :mod:`builtins`), so it can be used with Python code.
+
+.. cfunction:: PyObject* PyMemoryView_FromObject(PyObject *obj)
+
+   Return a memoryview object from an object that defines the buffer interface.