Issue #19659: Added documentation for Argument Clinic.

5 files changed, 954 insertions, 10 deletions

diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst
new file mode 100644
index 0000000..afbeb41
--- /dev/null
+++ b/Doc/howto/clinic.rst
@@ -0,0 +1,900 @@
+======================
+Argument Clinic How-To
+======================
+
+:author: Larry Hastings
+
+
+.. topic:: Abstract
+
+  Argument Clinic is a preprocessor for CPython C files.
+  Its purpose is to automate all the boilerplate involved
+  with writing argument parsing code for "builtins".
+  This document shows you how to convert your first C
+  function to work with Argument Clinic, and then introduces
+  some advanced topics on Argument Clinic usage.
+
+  Argument Clinic is currently considered an internal
+  tool for the CPython code tree.  Its use is not supported
+  for files outside the CPython code tree, and no guarantees
+  are made regarding backwards compatibility for future
+  versions.  In other words: if you maintain an external C
+  extension for CPython, you're welcome to experiment with
+  Argument Clinic in your own code.  But the version of Argument
+  Clinic that ships with CPython 3.5 *could* be totally
+  incompatible and break all your code.
+
+========================
+Basic Concepts And Usage
+========================
+
+Argument Clinic ships with CPython.  You can find it in ``Tools/clinic/clinic.py``.
+If you run that script, specifying a C file as an argument::
+
+    % python3 Tools/clinic/clinic.py foo.c
+
+Argument Clinic will scan over the file looking for lines that
+look exactly like this::
+
+    /*[clinic]
+
+When it finds one, it reads everything up to a line that looks
+like this::
+
+    [clinic]*/
+
+Everything in between these two lines is input for Argument Clinic.
+All of these lines, including the beginning and ending comment
+lines, are collectively called an Argument Clinic "input block",
+or "block" for short.
+
+When Argument Clinic parses one of these blocks, it
+generates output.  This output is rewritten into the C file
+immediately after the block, followed by a comment containing a checksum.
+The resulting Argument Clinic block looks like this::
+
+    /*[clinic]
+    ... clinic input goes here ...
+    [clinic]*/
+    ... clinic output goes here ...
+    /*[clinic checksum:...]*/
+
+If you run Argument Clinic on the same file a second time, Argument Clinic
+will discard the old output and write out the new output with a fresh checksum
+line.  However, if the input hasn't changed, the output won't change either.
+
+You should never modify the output portion of an Argument Clinic block.  Instead,
+change the input until it produces the output you want.  (That's the purpose of the
+checksum--to detect and warn you in case someone accidentally modifies the output.)
+
+For the sake of clarity, here's the terminology we'll use with Argument Clinic:
+
+* The first line of the comment (``/*[clinic]``) is the *start line*.
+* The last line of the initial comment (``[clinic]*/``) is the *end line*.
+* The last line (``/*[clinic checksum:...]*/``) is the *checksum line*.
+* In between the start line and the end line is the *input*.
+* In between the end line and the checksum line is the *output*.
+* All the text collectively, from the start line to the checksum line inclusively,
+  is the *block*.  (A block that hasn't been successfully processed by Argument
+  Clinic yet doesn't have output or a checksum line, but it's still considered
+  a block.)
+
+
+==============================
+Converting Your First Function
+==============================
+
+The best way to get a sense of how Argument Clinic works is to
+convert a function to work with it.  Let's dive in!
+
+0. Make sure you're working with a freshly updated trunk.
+
+1. Find a Python builtin that calls either ``PyArg_ParseTuple()``
+   or ``PyArg_ParseTupleAndKeywords()``, and hasn't been converted yet.
+   For my example I'm using ``pickle.Pickler.dump()``.
+
+2. If the call to the ``PyArg_Parse`` function uses any of the
+   following format units::
+
+       O&
+       O!
+       es
+       es#
+       et
+       et#
+
+   or if it has multiple calls to ``PyArg_ParseTuple()``,
+   you should choose a different function.  Argument Clinic *does*
+   support all of these scenarios.  But these are advanced
+   topics--let's do something simpler for your first function.
+
+3. Add the following boilerplate above the function, creating our block::
+
+    /*[clinic]
+    [clinic]*/
+
+4. Cut the docstring and paste it in between the ``[clinic]`` lines,
+   removing all the junk that makes it a properly quoted C string.
+   When you're done you should have just the text, based at the left
+   margin, with no line wider than 80 characters.
+   (Argument Clinic will preserve indents inside the docstring.)
+
+   Sample::
+
+    /*[clinic]
+    Write a pickled representation of obj to the open file.
+    [clinic]*/
+
+5. If your docstring doesn't have a "summary" line, Argument Clinic will
+   complain.  So let's make sure it has one.  The "summary" line should
+   be a paragraph consisting of a single 80-column line
+   at the beginning of the docstring.
+
+   (Our docstring consists solely of the summary line, so the sample
+   code doesn't have to change for this step.)
+
+6. Above the docstring, enter the name of the function, followed
+   by a blank line.  This should be the Python name of the function,
+   and should be the full dotted path
+   to the function--it should start with the name of the module,
+   include any sub-modules, and if the function is a method on
+   a class it should include the class name too.
+
+   Sample::
+
+    /*[clinic]
+    pickle.Pickler.dump
+
+    Write a pickled representation of obj to the open file.
+    [clinic]*/
+
+7. If this is the first time that module or class has been used with Argument
+   Clinic in this C file,
+   you must declare the module and/or class.  Proper Argument Clinic hygiene
+   prefers declaring these in a separate block somewhere near the
+   top of the C file, in the same way that include files and statics go at
+   the top.  (In our sample code we'll just show the two blocks next to
+   each other.)
+
+   Sample::
+
+    /*[clinic]
+    module pickle
+    class pickle.Pickler
+    [clinic]*/
+
+    /*[clinic]
+    pickle.Pickler.dump
+
+    Write a pickled representation of obj to the open file.
+    [clinic]*/
+
+
+8. Declare each of the parameters to the function.  Each parameter
+   should get its own line.  All the parameter lines should be
+   indented from the function name and the docstring.
+
+   The general form of these parameter lines is as follows::
+
+       name_of_parameter: converter
+
+   If the parameter has a default value, add that after the
+   converter::
+
+       name_of_parameter: converter = default_value
+
+   Add a blank line below the parameters.
+
+   What's a "converter"?  It establishes both the type
+   of the variable used in C, and the method to convert the Python
+   value into a C value at runtime.
+   For now you're going to use what's called a "legacy converter"--a
+   convenience syntax intended to make porting old code into Argument
+   Clinic easier.
+
+   For each parameter, copy the "format unit" for that
+   parameter from the ``PyArg_Parse()`` format argument and
+   specify *that* as its converter, as a quoted
+   string.  ("format unit" is the formal name for the one-to-three
+   character substring of the ``format`` parameter that tells
+   the argument parsing function what the type of the variable
+   is and how to convert it.)
+
+   For multicharacter format units like ``z#``, use the
+   entire two-or-three character string.
+
+   Sample::
+
+       /*[clinic]
+       module pickle
+       class pickle.Pickler
+       [clinic]*/
+
+       /*[clinic]
+       pickle.Pickler.dump
+
+           obj: 'O'
+
+       Write a pickled representation of obj to the open file.
+       [clinic]*/
+
+9. If your function has ``|`` in the format string, meaning some
+   parameters have default values, you can ignore it.  Argument
+   Clinic infers which parameters are optional based on whether
+   or not they have default values.
+
+   If your function has ``$`` in the format string, meaning it
+   takes keyword-only arguments, specify ``*`` on a line by
+   itself before the first keyword-only argument, indented the
+   same as the parameter lines.
+
+   (``pickle.Pickler.dump`` has neither, so our sample is unchanged.)
+
+
+10. If the existing C function uses ``PyArg_ParseTuple()``
+    (instead of ``PyArg_ParseTupleAndKeywords()``), then all its
+    arguments are positional-only.
+
+    To mark all parameters as positional-only in Argument Clinic,
+    add a ``/`` on a line by itself after the last parameter,
+    indented the same as the parameter lines.
+
+    Sample::
+
+        /*[clinic]
+        module pickle
+        class pickle.Pickler
+        [clinic]*/
+
+        /*[clinic]
+        pickle.Pickler.dump
+
+            obj: 'O'
+            /
+
+        Write a pickled representation of obj to the open file.
+        [clinic]*/
+
+11. It's helpful to write a per-parameter docstring, indented
+    another level past the parameter declaration.  But per-parameter
+    docstrings are optional; you can skip this step if you prefer.
+
+    Here's how per-parameter docstrings work.  The first line
+    of the per-parameter docstring must be indented further than the
+    parameter definition.  This left margin establishes the left margin
+    for the whole per-parameter docstring; all the text you write will
+    be outdented by this amount.  You can write as much as you like,
+    across multiple lines if you wish.
+
+    Sample::
+
+        /*[clinic]
+        module pickle
+        class pickle.Pickler
+        [clinic]*/
+
+        /*[clinic]
+        pickle.Pickler.dump
+
+            obj: 'O'
+                The object to be pickled.
+            /
+
+        Write a pickled representation of obj to the open file.
+        [clinic]*/
+
+12. Save and close the file, then run ``Tools/clinic/clinic.py`` on it.
+    With luck everything worked and your block now has output!  Reopen
+    the file in your text editor to see::
+
+       /*[clinic]
+       module pickle
+       class pickle.Pickler
+       [clinic]*/
+       /*[clinic checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709]*/
+
+       /*[clinic]
+       pickle.Pickler.dump
+
+           obj: 'O'
+               The object to be pickled.
+           /
+
+       Write a pickled representation of obj to the open file.
+       [clinic]*/
+
+       PyDoc_STRVAR(pickle_Pickler_dump__doc__,
+       "Write a pickled representation of obj to the open file.\n"
+       "\n"
+       ...
+       static PyObject *
+       pickle_Pickler_dump_impl(PyObject *self, PyObject *obj)
+       /*[clinic checksum: 3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/
+
+13. Double-check that the argument-parsing code Argument Clinic generated
+    looks basically the same as the existing code.
+
+    First, ensure both places use the same argument-parsing function.
+    The existing code must call either
+    ``PyArg_ParseTuple()`` or ``PyArg_ParseTupleAndKeywords()``;
+    ensure that the code generated by Argument Clinic calls the
+    same function.
+
+    Second, the format string passed in to ``PyArg_ParseTuple()`` or
+    ``PyArg_ParseTupleAndKeywords()`` should be *exactly* the same
+    as the hand-written one in the existing function.
+
+    Well, there's one way that Argument Clinic's output is permitted
+    to be different.  Argument Clinic always generates a format string
+    ending with ``:`` followed by the name of the function.  If the
+    format string originally ended with ``;`` (to specify usage help),
+    this is harmless--don't worry about this difference.
+
+    Apart from that, if either of these things differ in *any way*,
+    fix your input to Argument Clinic and rerun ``Tools/clinic/clinic.py``
+    until they are the same.
+
+
+14. Notice that the last line of its output is the declaration
+    of your "impl" function.  This is where the builtin's implementation goes.
+    Delete the existing prototype of the function you're modifying, but leave
+    the opening curly brace.  Now delete its argument parsing code and the
+    declarations of all the variables it dumps the arguments into.
+    Notice how the Python arguments are now arguments to this impl function;
+    if the implementation used different names for these variables, fix it.
+    The result should be a function that handles just the implementation
+    of the Python function without any argument-parsing code.
+
+    Sample::
+
+        /*[clinic]
+        module pickle
+        class pickle.Pickler
+        [clinic]*/
+        /*[clinic checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709]*/
+
+        /*[clinic]
+        pickle.Pickler.dump
+
+            obj: 'O'
+                The object to be pickled.
+            /
+
+        Write a pickled representation of obj to the open file.
+        [clinic]*/
+
+        PyDoc_STRVAR(pickle_Pickler_dump__doc__,
+        "Write a pickled representation of obj to the open file.\n"
+        "\n"
+        ...
+        static PyObject *
+        pickle_Pickler_dump_impl(PyObject *self, PyObject *obj)
+        /*[clinic checksum: 3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/
+        {
+            /* Check whether the Pickler was initialized correctly (issue3664).
+               Developers often forget to call __init__() in their subclasses, which
+               would trigger a segfault without this check. */
+            if (self->write == NULL) {
+                PyErr_Format(PicklingError,
+                             "Pickler.__init__() was not called by %s.__init__()",
+                             Py_TYPE(self)->tp_name);
+                return NULL;
+            }
+
+            if (_Pickler_ClearBuffer(self) < 0)
+                return NULL;
+
+            ...
+
+15. Compile and run the relevant portions of the regression-test suite.
+    This change should not introduce any new compile-time warnings or errors,
+    and there should be no externally-visible change to Python's behavior.
+
+    Well, except for one difference: ``inspect.signature()`` run on your function
+    should now provide a valid signature!
+
+    Congratulations, you've ported your first function to work with Argument Clinic!
+
+===============
+Advanced Topics
+===============
+
+
+Renaming the C functions generated by Argument Clinic
+-----------------------------------------------------
+
+Argument Clinic automatically names the functions it generates for you.
+Occasionally this may cause a problem, if the generated name collides with
+the name of an existing C function.  There's an easy solution: you can explicitly
+specify the base name to use for the C functions.  Just add the keyword ``"as"``
+to your function declaration line, followed by the function name you wish to use.
+Argument Clinic will use the function name you use for the base (generated) function,
+and then add ``"_impl"`` to the end for the name of the impl function.
+
+For example, if we wanted to rename the C function names generated for
+``pickle.Pickler.dump``, it'd look like this::
+
+    /*[clinic]
+    pickle.Pickler.dump as pickler_dumper
+
+    ...
+
+The base function would now be named ``pickler_dumper()``,
+and the impl function would be named ``pickler_dumper_impl()``.
+
+
+Optional Groups
+---------------
+
+Some legacy functions have a tricky approach to parsing their arguments:
+they count the number of positional arguments, then use a ``switch`` statement
+to call one of several different ``PyArg_ParseTuple()`` calls depending on
+how many positional arguments there are.  (These functions cannot accept
+keyword-only arguments.)  This approach was used to simulate optional
+arguments back before ``PyArg_ParseTupleAndKeywords()`` was created.
+
+Functions using this approach can often be converted to
+use ``PyArg_ParseTupleAndKeywords()``, optional arguments, and default values.
+But it's not always possible, because some of these legacy functions have
+behaviors ``PyArg_ParseTupleAndKeywords()`` can't directly support.
+The most obvious example is the builtin function ``range()``, which has
+an optional argument on the *left* side of its required argument!
+Another example is ``curses.window.addch()``, which has a group of two
+arguments that must always be specified together.  (The arguments are
+called ``x`` and ``y``; if you call the function passing in ``x``,
+you must also pass in ``y``--and if you don't pass in ``x`` you may not
+pass in ``y`` either.)
+
+For the sake of backwards compatibility, Argument Clinic supports this
+alternate approach to parsing, using what are called *optional groups*.
+Optional groups are groups of arguments that can only be specified together.
+They can be to the left or the right of the required arguments.  They
+can *only* be used with positional-only parameters.
+
+To specify an optional group, add a ``[`` on a line by itself before
+the parameters you wish to be
+in a group together, and a ``]`` on a line by itself after the
+parameters.  As an example, here's how ``curses.window.addch``
+uses optional groups to make the first two parameters and the last
+parameter optional::
+
+    /*[clinic]
+
+    curses.window.addch
+
+        [
+        x: int
+          X-coordinate.
+        y: int
+          Y-coordinate.
+        ]
+
+        ch: object
+          Character to add.
+
+        [
+        attr: long
+          Attributes for the character.
+        ]
+        /
+
+    ...
+
+
+Notes:
+
+* For every optional group, one additional parameter will be passed into the
+  impl function representing the group.  The parameter will be an int, and it will
+  be named ``group_{direction}_{number}``,
+  where ``{direction}`` is either ``right`` or ``left`` depending on whether the group
+  is before or after the required parameters, and ``{number}`` is a monotonically
+  increasing number (starting at 1) indicating how far away the group is from
+  the required parameters.  When the impl is called, this parameter will be set
+  to zero if this group was unused, and set to non-zero if this group was used.
+  (By used or unused, I mean whether or not the parameters received arguments
+  in this invocation.)
+
+* If there are no required arguments, the optional groups will behave
+  as if they are to the right of the required arguments.
+
+* In the case of ambiguity, the argument parsing code
+  favors parameters on the left (before the required parameters).
+
+* Optional groups are *only* intended for legacy code.  Please do not
+  use optional groups for new code.
+
+
+Using real Argument Clinic converters, instead of "legacy converters"
+---------------------------------------------------------------------
+
+To save time, and to minimize how much you need to learn
+to achieve your first port to Argument Clinic, the walkthrough above tells
+you to use the "legacy converters".  "Legacy converters" are a convenience,
+designed explicitly to make porting existing code to Argument Clinic
+easier.  And to be clear, their use is entirely acceptable when porting
+code for Python 3.4.
+
+However, in the long term we probably want all our blocks to
+use Argument Clinic's real syntax for converters.  Why?  A couple
+reasons:
+
+* The proper converters are far easier to read and clearer in their intent.
+* There are some format units that are unsupported as "legacy converters",
+  because they require arguments, and the legacy converter syntax doesn't
+  support specifying arguments.
+* In the future we may have a new argument parsing library that isn't
+  restricted to what ``PyArg_ParseTuple()`` supports.
+
+So if you want
+to go that extra effort, you should consider using normal
+converters instead of the legacy converters.
+
+In a nutshell, the syntax for Argument Clinic (non-legacy) converters
+looks like a Python function call.  However, if there are no explicit
+arguments to the function (all functions take their default values),
+you may omit the parentheses.  Thus ``bool`` and ``bool()`` are exactly
+the same.  All parameters to Argument Clinic converters are keyword-only.
+
+All Argument Clinic converters accept the following arguments:
+
+``doc_default``
+  If the parameter takes a default value, normally this value is also
+  provided in the ``inspect.Signature`` metadata, and displayed in the
+  docstring.  ``doc_default`` lets you override the value used in these
+  two places: pass in a string representing the Python value you wish
+  to use in these two contexts.
+
+``required``
+  If a parameter takes a default value, Argument Clinic infers that the
+  parameter is optional.  However, you may want a parameter to take a
+  default value in C, but not behave in Python as if the parameter is
+  optional.  Passing in ``required=True`` to a converter tells Argument
+  Clinic that this parameter is not optional, even if it has a default
+  value.
+
+``annotation``
+  The annotation value for this parameter.  Not currently supported,
+  because PEP 8 mandates that the Python library may not use
+  annotations.
+
+Below is a table showing the mapping of legacy converters into real
+Argument Clinic converters.  On the left is the legacy converter,
+on the right is the text you'd replace it with.
+
+=========   =================================================================================
+``'B'``     ``byte(bitwise=True)``
+``'b'``     ``byte``
+``'c'``     ``char``
+``'C'``     ``int(types='str')``
+``'d'``     ``double``
+``'D'``     ``Py_complex``
+``'es#'``   ``str(encoding='name_of_encoding', length=True, zeroes=True)``
+``'es'``    ``str(encoding='name_of_encoding')``
+``'et#'``   ``str(encoding='name_of_encoding', types='bytes bytearray str', length=True)``
+``'et'``    ``str(encoding='name_of_encoding', types='bytes bytearray str')``
+``'f'``     ``float``
+``'h'``     ``short``
+``'H'``     ``unsigned_short``
+``'i'``     ``int``
+``'I'``     ``unsigned_int``
+``'K'``     ``unsigned_PY_LONG_LONG``
+``'L'``     ``PY_LONG_LONG``
+``'n'``     ``Py_ssize_t``
+``'O!'``    ``object(type='name_of_Python_type')``
+``'O&'``    ``object(converter='name_of_c_function')``
+``'O'``     ``object``
+``'p'``     ``bool``
+``'s#'``    ``str(length=True)``
+``'S'``     ``PyBytesObject``
+``'s'``     ``str``
+``'s*'``    ``Py_buffer(types='str bytes bytearray buffer')``
+``'u#'``    ``Py_UNICODE(length=True)``
+``'u'``     ``Py_UNICODE``
+``'U'``     ``unicode``
+``'w*'``    ``Py_buffer(types='bytearray rwbuffer')``
+``'y#'``    ``str(type='bytes', length=True)``
+``'Y'``     ``PyByteArrayObject``
+``'y'``     ``str(type='bytes')``
+``'y*'``    ``Py_buffer``
+``'Z#'``    ``Py_UNICODE(nullable=True, length=True)``
+``'z#'``    ``str(nullable=True, length=True)``
+``'Z'``     ``Py_UNICODE(nullable=True)``
+``'z'``     ``str(nullable=True)``
+``'z*'``    ``Py_buffer(types='str bytes bytearray buffer', nullable=True)``
+=========   =================================================================================
+
+As an example, here's our sample ``pickle.Pickler.dump`` using the proper
+converter::
+
+    /*[clinic]
+    pickle.Pickler.dump
+
+        obj: object
+            The object to be pickled.
+        /
+
+    Write a pickled representation of obj to the open file.
+    [clinic]*/
+
+Argument Clinic will show you all the converters it has
+available.  For each converter it'll show you all the parameters
+it accepts, along with the default value for each parameter.
+Just run ``Tools/clinic/clinic.py --converters`` to see the full list.
+
+
+Advanced converters
+-------------------
+
+Remeber those format units you skipped for your first
+time because they were advanced?  Here's how to handle those too.
+
+The trick is, all those format units take arguments--either
+conversion functions, or types, or strings specifying an encoding.
+(But "legacy converters" don't support arguments.  That's why we
+skipped them for your first function.)  The argument you specified
+to the format unit is now an argument to the converter; this
+argument is either ``converter`` (for ``O&``), ``type`` (for ``O!``),
+or ``encoding`` (for all the format units that start with ``e``).
+
+Note that ``object()`` must explicitly support each Python type you specify
+for the ``type`` argument.  Currently it only supports ``str``.  It should be
+easy to add more, just edit ``Tools/clinic/clinic.py``, search for ``O!`` in
+the text, and add more entries to the dict mapping types to strings just above it.
+
+Note also that this approach takes away some possible flexibility for the format
+units starting with ``e``.  It used to be possible to decide at runtime what
+encoding string to pass in to ``PyArg_ParseTuple()``.   But now this string must
+be hard-coded at compile-time.  This limitation is deliberate; it made supporting
+this format unit much easier, and may allow for future compile-time optimizations.
+This restriction does not seem unreasonable; CPython itself always passes in static
+hard-coded strings when using format units starting with ``e``.
+
+
+Using a return converter
+------------------------
+
+By default the impl function Argument Clinic generates for you returns ``PyObject *``.
+But your C function often computes some C type, then converts it into the ``PyObject *``
+at the last moment.  Argument Clinic handles converting your inputs from Python types
+into native C types--why not have it convert your return value from a native C type
+into a Python type too?
+
+That's what a "return converter" does.  It changes your impl function to return
+some C type, then adds code to the generated (non-impl) function to handle converting
+that value into the appropriate ``PyObject *``.
+
+The syntax for return converters is similar to that of parameter converters.
+You specify the return converter like it was a return annotation on the
+function itself.  Return converters behave much the same as parameter converters;
+they take arguments, the arguments are all keyword-only, and if you're not changing
+any of the default arguments you can omit the parentheses.
+
+(If you use both ``"as"`` *and* a return converter for your function,
+the ``"as"`` should come before the return converter.)
+
+There's one additional complication when using return converters: how do you
+indicate an error has occured?  Normally, a function returns a valid (non-``NULL``)
+pointer for success, and ``NULL`` for failure.  But if you use an integer return converter,
+all integers are valid.  How can Argument Clinic detect an error?  Its solution: each return
+converter implicitly looks for a special value that indicates an error.  If you return
+that value, and an error has been set (``PyErr_Occurred()`` returns a true
+value), then the generated code will propogate the error.  Otherwise it will
+encode the value you return like normal.
+
+Currently Argument Clinic supports only a few return converters::
+
+    int
+    long
+    Py_ssize_t
+    DecodeFSDefault
+
+None of these take parameters.  For the first three, return -1 to indicate
+error.  For ``DecodeFSDefault``, the return type is ``char *``; return a NULL
+pointer to indicate an error.
+
+Calling Python code
+-------------------
+
+The rest of the advanced topics require you to write Python code
+which lives inside your C file and modifies Argument Clinic at
+runtime.  This is simple; you simply define a Python block.
+
+A Python block uses different delimiter lines than an Argument
+Clinic function block.  It looks like this::
+
+    /*[python]
+    # python code goes here
+    [python]*/
+
+All the code inside the Python block is executed at the
+time it's parsed.  All text written to stdout inside the block
+is redirected into the "output" after the block.
+
+As an example, here's a Python block that adds a static integer
+variable to the C code::
+
+    /*[python]
+    print('static int __ignored_unused_variable__ = 0;')
+    [python]*/
+    static int __ignored_unused_variable__ = 0;
+    /*[python checksum:...]*/
+
+
+Using a "self converter"
+------------------------
+
+Argument Clinic automatically adds a "self" parameter for you
+using a default converter.  However, you can override
+Argument Clinic's converter and specify one yourself.
+Just add your own ``self`` parameter as the first parameter in a
+block, and ensure that its converter is an instance of
+``self_converter`` or a subclass thereof.
+
+What's the point?  This lets you automatically cast ``self``
+from ``PyObject *`` to a custom type.
+
+How do you specify the custom type you want to cast ``self`` to?
+If you only have one or two functions with the same type for ``self``,
+you can directly use Argument Clinic's existing ``self`` converter,
+passing in the type you want to use as the ``type`` parameter::
+
+    /*[clinic]
+
+    _pickle.Pickler.dump
+
+      self: self(type="PicklerObject *")
+      obj: object
+      /
+
+    Write a pickled representation of the given object to the open file.
+    [clinic]*/
+
+On the other hand, if you have a lot of functions that will use the same
+type for ``self``, it's best to create your own converter, subclassing
+``self_converter`` but overwriting the ``type`` member::
+
+    /*[clinic]
+    class PicklerObject_converter(self_converter):
+        type = "PicklerObject *"
+    [clinic]*/
+
+    /*[clinic]
+
+    _pickle.Pickler.dump
+
+      self: PicklerObject
+      obj: object
+      /
+
+    Write a pickled representation of the given object to the open file.
+    [clinic]*/
+
+
+
+Writing a custom converter
+--------------------------
+
+As we hinted at in the previous section... you can write your own converters!
+A converter is simply a Python class that inherits from ``CConverter``.
+The main purpose of a custom converter is if you have a parameter using
+the ``O&`` format unit--parsing this parameter means calling
+a ``PyArg_ParseTuple()`` "converter function".
+
+Your converter class should be named ``*something*_converter``.
+If the name follows this convention, then your converter class
+will be automatically registered with Argument Clinic; its name
+will be the name of your class with the ``_converter`` suffix
+stripped off.  (This is done automatically for you with a metaclass.)
+
+You shouldn't subclass ``CConverter.__init__``.  Instead, you should
+write a ``converter_init()`` function.  ``converter_init()``
+always accepts a ``self`` parameter; after that, all additional
+parameters *must* be keyword-only.  Any arguments passed in to
+the converter in Argument Clinic will be passed along to your
+``converter_init()``.
+
+There are some additional members of ``CConverter`` you may wish
+to specify in your subclass.  Here's the current list:
+
+``type``
+    The C type to use for this variable.
+    ``type`` should be a Python string specifying the type, e.g. ``int``.
+    If this is a pointer type, the type string should end with ``' *'``.
+
+``default``
+    The Python default value for this parameter, as a Python value.
+    Or the magic value ``unspecified`` if there is no default.
+
+``doc_default``
+    ``default`` as it should appear in the documentation,
+    as a string.
+    Or ``None`` if there is no default.
+    This string, when run through ``eval()``, should produce
+    a Python value.
+
+``py_default``
+    ``default`` as it should appear in Python code,
+    as a string.
+    Or ``None`` if there is no default.
+
+``c_default``
+    ``default`` as it should appear in C code,
+    as a string.
+    Or ``None`` if there is no default.
+
+``c_ignored_default``
+    The default value used to initialize the C variable when
+    there is no default, but not specifying a default may
+    result in an "uninitialized variable" warning.  This is
+    easily happen when using option groups--although
+    properly-written code won't actually use the variable,
+    the variable does get passed in to the _impl, and the
+    C compiler will complain about the "use" of the uninitialized
+    value.  This value should be a string.
+
+``converter``
+    The name of the C converter function, as a string.
+
+``impl_by_reference``
+    A boolean value.  If true,
+    Argument Clinic will add a ``&`` in front of the name of
+    the variable when passing it into the impl function.
+
+``parse_by_reference``
+    A boolean value.  If true,
+    Argument Clinic will add a ``&`` in front of the name of
+    the variable when passing it into ``PyArg_ParseTuple()``.
+
+
+Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c``::
+
+    /*[python]
+
+    class uint_converter(CConverter):
+        type = 'unsigned int'
+        converter = 'uint_converter'
+
+    [python]*/
+    /*[python checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709]*/
+
+This block adds a ``uint`` converter to Argument Clinic.  Parameters
+declared as ``uint`` will be declared as type ``unsigned int``, and will
+be parsed by calling the ``uint_converter`` converter function in C.
+``uint`` variables automatically support default values.
+
+More sophisticated custom converters can insert custom C code to
+handle initialization and cleanup.
+You can see more examples of custom converters in the CPython
+source tree; grep the C files for the string ``CConverter``.
+
+Writing a custom return converter
+---------------------------------
+
+Writing a custom return converter is much like writing
+a custom converter.  Except it's much simpler, because return
+converters are themselves much simpler.
+
+Return converters must subclass ``CReturnConverter``.
+There are no examples yet of custom return converters,
+because they are not widely used yet.  If you wish to
+write your own return converter, please read ``Tools/clinic/clinic.py``,
+specifically the implementation of ``CReturnConverter`` and
+all its subclasses.
+
+
+Using Argument Clinic in Python files
+-------------------------------------
+
+It's actually possible to use Argument Clinic to preprocess Python files.
+There's no point to using Argument Clinic blocks, of course, as the output
+wouldn't make any sense to the Python interpreter.  But using Argument Clinic
+to run Python blocks lets you use Python as a Python preprocessor!
+
+Since Python comments are different from C comments, Argument Clinic
+blocks embedded in Python files look slightly different.  They look like this::
+
+    #/*[python]
+    #print("def foo(): pass")
+    #[python]*/
+    def foo(): pass
+    #/*[python checksum:...]*/
diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst
index 81a4f8b..2c9d699 100644
--- a/Doc/howto/index.rst
+++ b/Doc/howto/index.rst
@@ -28,4 +28,5 @@ Currently, the HOWTOs are:
    webservers.rst
    argparse.rst
    ipaddress.rst
+   clinic.rst
 
diff --git a/Misc/NEWS b/Misc/NEWS
index feeace5..fcc423a 100644
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -345,6 +345,8 @@ Documentation
 Tools/Demos
 -----------
 
+- Issue #19659: Added documentation for Argument Clinic.
+
 - Issue #19976: Argument Clinic METH_NOARGS functions now always
   take two parameters.
 
diff --git a/Modules/zlibmodule.c b/Modules/zlibmodule.c
index 72137db..d197d88 100644
--- a/Modules/zlibmodule.c
+++ b/Modules/zlibmodule.c
@@ -312,9 +312,6 @@ class uint_converter(CConverter):
     type = 'unsigned int'
     converter = 'uint_converter'
 
-class compobject_converter(self_converter):
-    type = "compobject *"
-
 [python]*/
 /*[python checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709]*/
 
@@ -750,7 +747,7 @@ save_unconsumed_input(compobject *self, int err)
 
 zlib.Decompress.decompress
 
-    self: compobject
+    self: self(type="compobject *")
 
     data: Py_buffer
         The binary data to decompress.
@@ -1032,7 +1029,7 @@ PyZlib_flush(compobject *self, PyObject *args)
 /*[clinic]
 zlib.Compress.copy
 
-    self: compobject
+    self: self(type="compobject *")
 
 Return a copy of the compression object.
 [clinic]*/
diff --git a/Tools/clinic/clinic.py b/Tools/clinic/clinic.py
index 7dd6215..5480add 100755
--- a/Tools/clinic/clinic.py
+++ b/Tools/clinic/clinic.py
@@ -1296,11 +1296,13 @@ class CConverter(metaclass=CConverterAutoRegister):
     must be keyword-only.
     """
 
+    # The C type to use for this variable.
+    # 'type' should be a Python string specifying the type, e.g. "int".
+    # If this is a pointer type, the type string should end with ' *'.
     type = None
-    format_unit = 'O&'
 
     # The Python default value for this parameter, as a Python value.
-    # Or "unspecified" if there is no default.
+    # Or the magic value "unspecified" if there is no default.
     default = unspecified
 
     # "default" as it should appear in the documentation, as a string.
@@ -1330,9 +1332,32 @@ class CConverter(metaclass=CConverterAutoRegister):
     # (If this is not None, format_unit must be 'O&'.)
     converter = None
 
-    encoding = None
+    # Should Argument Clinic add a '&' before the name of
+    # the variable when passing it into the _impl function?
     impl_by_reference = False
+
+    # Should Argument Clinic add a '&' before the name of
+    # the variable when passing it into PyArg_ParseTuple (AndKeywords)?
     parse_by_reference = True
+
+    #############################################################
+    #############################################################
+    ## You shouldn't need to read anything below this point to ##
+    ## write your own converter functions.                     ##
+    #############################################################
+    #############################################################
+
+    # The "format unit" to specify for this variable when
+    # parsing arguments using PyArg_ParseTuple (AndKeywords).
+    # Custom converters should always use the default value of 'O&'.
+    format_unit = 'O&'
+
+    # What encoding do we want for this variable?  Only used
+    # by format units starting with 'e'.
+    encoding = None
+
+    # Do we want an adjacent '_length' variable for this variable?
+    # Only used by format units ending with '#'.
     length = False
 
     def __init__(self, name, function, default=unspecified, *, doc_default=None, required=False, annotation=unspecified, **kwargs):
@@ -1751,7 +1776,7 @@ class self_converter(CConverter):
     this is the default converter used for "self".
     """
     type = "PyObject *"
-    def converter_init(self):
+    def converter_init(self, *, type=None):
         f = self.function
         if f.kind == CALLABLE:
             if f.cls:
@@ -1766,6 +1791,9 @@ class self_converter(CConverter):
             self.name = "cls"
             self.type = "PyTypeObject *"
 
+        if type:
+            self.type = type
+
     def render(self, parameter, data):
         fail("render() should never be called on self_converter instances")
 
@@ -1787,7 +1815,13 @@ class CReturnConverterAutoRegister(type):
 
 class CReturnConverter(metaclass=CReturnConverterAutoRegister):
 
+    # The C type to use for this variable.
+    # 'type' should be a Python string specifying the type, e.g. "int".
+    # If this is a pointer type, the type string should end with ' *'.
     type = 'PyObject *'
+
+    # The Python default value for this parameter, as a Python value.
+    # Or the magic value "unspecified" if there is no default.
     default = None
 
     def __init__(self, *, doc_default=None, **kwargs):
@@ -1826,6 +1860,16 @@ class CReturnConverter(metaclass=CReturnConverterAutoRegister):
 
 add_c_return_converter(CReturnConverter, 'object')
 
+class NoneType_return_converter(CReturnConverter):
+    def render(self, function, data):
+        self.declare(data)
+        data.return_conversion.append('''
+if (_return_value != Py_None)
+    goto exit;
+return_value = Py_None;
+Py_INCREF(Py_None);
+'''.strip())
+
 class int_return_converter(CReturnConverter):
     type = 'int'
 
@@ -2680,7 +2724,7 @@ def main(argv):
 
                 # print("   ", short_name + "".join(parameters))
             print()
-        print("All converters also accept (doc_default=None, required=False).")
+        print("All converters also accept (doc_default=None, required=False, annotation=None).")
         print("All return converters also accept (doc_default=None).")
         sys.exit(0)