#24277: The new email API is no longer provisional.

This is a wholesale reorganization and editing of the email documentation to
make the new API the standard one, and the old API the 'legacy' one.  The
default is still the compat32 policy, for backward compatibility.  We will
change that eventually.

1 files changed, 208 insertions, 168 deletions

diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst
index d596ed8..c1d94ca 100644
--- a/Doc/library/email.generator.rst
+++ b/Doc/library/email.generator.rst
@@ -8,210 +8,243 @@
 
 --------------
 
-One of the most common tasks is to generate the flat text of the email message
-represented by a message object structure.  You will need to do this if you want
-to send your message via the :mod:`smtplib` module or the :mod:`nntplib` module,
-or print the message on the console.  Taking a message object structure and
-producing a flat text document is the job of the :class:`Generator` class.
-
-Again, as with the :mod:`email.parser` module, you aren't limited to the
-functionality of the bundled generator; you could write one from scratch
-yourself.  However the bundled generator knows how to generate most email in a
-standards-compliant way, should handle MIME and non-MIME email messages just
-fine, and is designed so that the transformation from flat text, to a message
-structure via the :class:`~email.parser.Parser` class, and back to flat text,
-is idempotent (the input is identical to the output) [#]_.  On the other hand,
-using the Generator on a :class:`~email.message.Message` constructed by program
-may result in changes to the :class:`~email.message.Message` object as defaults
-are filled in.
-
-:class:`bytes` output can be generated using the :class:`BytesGenerator` class.
-If the message object structure contains non-ASCII bytes, this generator's
-:meth:`~BytesGenerator.flatten` method will emit the original bytes.  Parsing a
-binary message and then flattening it with :class:`BytesGenerator` should be
-idempotent for standards compliant messages.
-
-Here are the public methods of the :class:`Generator` class, imported from the
-:mod:`email.generator` module:
-
-
-.. class:: Generator(outfp, mangle_from_=True, maxheaderlen=78, *, policy=None)
-
-   The constructor for the :class:`Generator` class takes a :term:`file-like object`
-   called *outfp* for an argument.  *outfp* must support the :meth:`write` method
-   and be usable as the output file for the :func:`print` function.
-
-   Optional *mangle_from_* is a flag that, when ``True``, puts a ``>`` character in
-   front of any line in the body that starts exactly as ``From``, i.e. ``From``
-   followed by a space at the beginning of the line.  This is the only guaranteed
-   portable way to avoid having such lines be mistaken for a Unix mailbox format
-   envelope header separator (see `WHY THE CONTENT-LENGTH FORMAT IS BAD
-   <https://www.jwz.org/doc/content-length.html>`_ for details).  *mangle_from_*
-   defaults to ``True``, but you might want to set this to ``False`` if you are not
-   writing Unix mailbox format files.
-
-   Optional *maxheaderlen* specifies the longest length for a non-continued header.
-   When a header line is longer than *maxheaderlen* (in characters, with tabs
-   expanded to 8 spaces), the header will be split as defined in the
-   :class:`~email.header.Header` class.  Set to zero to disable header wrapping.
-   The default is 78, as recommended (but not required) by :rfc:`2822`.
-
-   The *policy* keyword specifies a :mod:`~email.policy` object that controls a
-   number of aspects of the generator's operation.  If no *policy* is specified,
-   then the *policy* attached to the message object passed to :attr:`flatten`
-   is used.
+One of the most common tasks is to generate the flat (serialized) version of
+the email message represented by a message object structure.  You will need to
+do this if you want to send your message via :meth:`smtplib.SMTP.sendmail` or
+the :mod:`nntplib` module, or print the message on the console.  Taking a
+message object structure and producing a serialized representation is the job
+of the generator classes.
+
+As with the :mod:`email.parser` module, you aren't limited to the functionality
+of the bundled generator; you could write one from scratch yourself.  However
+the bundled generator knows how to generate most email in a standards-compliant
+way, should handle MIME and non-MIME email messages just fine, and is designed
+so that the bytes-oriented parsing and generation operations are inverses,
+assuming the same non-transforming :mod:`~email.policy` is used for both.  That
+is, parsing the serialized byte stream via the
+:class:`~email.parser.BytesParser` class and then regenerating the serialized
+byte stream using :class:`BytesGenerator` should produce output identical to
+the input [#]_.  (On the other hand, using the generator on an
+:class:`~email.message.EmailMessage` constructed by program may result in
+changes to the :class:`~email.message.EmailMessage` object as defaults are
+filled in.)
+
+The :class:`Generator` class can be used to flatten a message into a text (as
+opposed to binary) serialized representation, but since Unicode cannot
+represent binary data directly, the message is of necessity transformed into
+something that contains only ASCII characters, using the standard email RFC
+Content Transfer Encoding techniques for encoding email messages for transport
+over channels that are not "8 bit clean".
+
+
+.. class:: BytesGenerator(outfp, mangle_from_=None, maxheaderlen=None, *, \
+                          policy=None)
 
-   .. versionchanged:: 3.3 Added the *policy* keyword.
+   Return a :class:`BytesGenerator` object that will write any message provided
+   to the :meth:`flatten` method, or any surrogateescape encoded text provided
+   to the :meth:`write` method, to the :term:`file-like object` *outfp*.
+   *outfp* must support a ``write`` method that accepts binary data.
+
+   If optional *mangle_from_* is ``True``, put a ``>`` character in front of
+   any line in the body that starts with the exact string ``"From "``, that is
+   ``From`` followed by a space at the beginning of a line.  *mangle_from_*
+   defaults to the value of the :attr:`~email.policy.Policy.mangle_from_`
+   setting of the *policy* (which is ``True`` for the
+   :data:`~email.policy.compat32` policy and ``False`` for all others).
+   *mangle_from_* is intended for use when messages are stored in unix mbox
+   format (see :mod:`mailbox` and `WHY THE CONTENT-LENGTH FORMAT IS BAD
+   <http://www.jwz.org/doc/content-length.html>`_).
+
+   If *maxheaderlen* is not ``None``, refold any header lines that are longer
+   than *maxheaderlen*, or if ``0``, do not rewrap any headers.  If
+   *manheaderlen* is ``None`` (the default), wrap headers and other message
+   lines according to the *policy* settings.
+
+   If *policy* is specified, use that policy to control message generation.  If
+   *policy* is ``None`` (the default), use the policy associated with the
+   :class:`~email.message.Message` or :class:`~email.message.EmailMessage`
+   object passed to ``flatten`` to control the message generation.  See
+   :mod:`email.policy` for details on what *policy* controls.
 
-   The other public :class:`Generator` methods are:
+   .. versionadded:: 3.2
 
+   .. versionchanged:: 3.3 Added the *policy* keyword.
 
-   .. method:: flatten(msg, unixfrom=False, linesep=None)
+   .. versionchanged:: 3.6 The default behavior of the *mangle_from_*
+      and *maxheaderlen* parameters is to follow the policy.
 
-      Print the textual representation of the message object structure rooted at
-      *msg* to the output file specified when the :class:`Generator` instance
-      was created.  Subparts are visited depth-first and the resulting text will
-      be properly MIME encoded.
 
-      Optional *unixfrom* is a flag that forces the printing of the envelope
-      header delimiter before the first :rfc:`2822` header of the root message
-      object.  If the root object has no envelope header, a standard one is
-      crafted.  By default, this is set to ``False`` to inhibit the printing of
-      the envelope delimiter.
+   .. method:: flatten(msg, unixfrom=False, linesep=None)
 
+      Print the textual representation of the message object structure rooted
+      at *msg* to the output file specified when the :class:`BytesGenerator`
+      instance was created.
+
+      If the :mod:`~email.policy` option :attr:`~email.policy.Policy.cte_type`
+      is ``8bit`` (the default), copy any headers in the original parsed
+      message that have not been modified to the output with any bytes with the
+      high bit set reproduced as in the original, and preserve the non-ASCII
+      :mailheader:`Content-Transfer-Encoding` of any body parts that have them.
+      If ``cte_type`` is ``7bit``, convert the bytes with the high bit set as
+      needed using an ASCII-compatible :mailheader:`Content-Transfer-Encoding`.
+      That is, transform parts with non-ASCII
+      :mailheader:`Cotnent-Transfer-Encoding`
+      (:mailheader:`Content-Transfer-Encoding: 8bit`) to an ASCII compatibile
+      :mailheader:`Content-Transfer-Encoding`, and encode RFC-invalid non-ASCII
+      bytes in headers using the MIME ``unknown-8bit`` character set, thus
+      rendering them RFC-compliant.
+
+      .. XXX: There should be an option that just does the RFC
+         compliance transformation on headers but leaves CTE 8bit parts alone.
+
+      If *unixfrom* is ``True``, print the envelope header delimiter used by
+      the Unix mailbox format (see :mod:`mailbox`) before the first of the
+      :rfc:`5322` headers of the root message object.  If the root object has
+      no envelope header, craft a standard one.  The default is ``False``.
       Note that for subparts, no envelope header is ever printed.
 
-      Optional *linesep* specifies the line separator character used to
-      terminate lines in the output.  If specified it overrides the value
-      specified by the *msg*\'s or ``Generator``\'s ``policy``.
+      If *linesep* is not ``None``, use it as the separator character between
+      all the lines of the flattened message.  If *linesep* is ``None`` (the
+      default), use the value specified in the *policy*.
 
-      Because strings cannot represent non-ASCII bytes, if the policy that
-      applies when ``flatten`` is run has :attr:`~email.policy.Policy.cte_type`
-      set to ``8bit``, ``Generator`` will operate as if it were set to
-      ``7bit``.  This means that messages parsed with a Bytes parser that have
-      a :mailheader:`Content-Transfer-Encoding` of ``8bit`` will be converted
-      to a use a ``7bit`` Content-Transfer-Encoding.  Non-ASCII bytes in the
-      headers will be :rfc:`2047` encoded with a charset of ``unknown-8bit``.
+      .. XXX: flatten should take a *policy* keyword.
 
-      .. versionchanged:: 3.2
-         Added support for re-encoding ``8bit`` message bodies, and the
-         *linesep* argument.
 
    .. method:: clone(fp)
 
-      Return an independent clone of this :class:`Generator` instance with the
-      exact same options.
-
-   .. method:: write(s)
-
-      Write the string *s* to the underlying file object, i.e. *outfp* passed to
-      :class:`Generator`'s constructor.  This provides just enough file-like API
-      for :class:`Generator` instances to be used in the :func:`print` function.
+      Return an independent clone of this :class:`BytesGenerator` instance with
+      the exact same option settings, and *fp* as the new *outfp*.
 
-As a convenience, see the :class:`~email.message.Message` methods
-:meth:`~email.message.Message.as_string` and ``str(aMessage)``, a.k.a.
-:meth:`~email.message.Message.__str__`, which simplify the generation of a
-formatted string representation of a message object.  For more detail, see
-:mod:`email.message`.
 
-.. class:: BytesGenerator(outfp, mangle_from_=True, maxheaderlen=78, *, \
-                          policy=None)
+   .. method:: write(s)
 
-   The constructor for the :class:`BytesGenerator` class takes a binary
-   :term:`file-like object` called *outfp* for an argument.  *outfp* must
-   support a :meth:`write` method that accepts binary data.
+      Encode *s* using the ``ASCII`` codec and the ``surrogateescape`` error
+      handler, and pass it to the *write* method of the *outfp* passed to the
+      :class:`BytesGenerator`'s constructor.
 
-   Optional *mangle_from_* is a flag that, when ``True``, puts a ``>``
-   character in front of any line in the body that starts exactly as ``From``,
-   i.e. ``From`` followed by a space at the beginning of the line.  This is the
-   only guaranteed portable way to avoid having such lines be mistaken for a
-   Unix mailbox format envelope header separator (see `WHY THE CONTENT-LENGTH
-   FORMAT IS BAD <https://www.jwz.org/doc/content-length.html>`_ for details).
-   *mangle_from_* defaults to ``True``, but you might want to set this to
-   ``False`` if you are not writing Unix mailbox format files.
 
-   Optional *maxheaderlen* specifies the longest length for a non-continued
-   header.  When a header line is longer than *maxheaderlen* (in characters,
-   with tabs expanded to 8 spaces), the header will be split as defined in the
-   :class:`~email.header.Header` class.  Set to zero to disable header
-   wrapping.  The default is 78, as recommended (but not required) by
-   :rfc:`2822`.
+As a convenience, :class:`~email.message.EmailMessage` provides the methods
+:meth:`~email.message.EmailMessage.as_bytes` and ``bytes(aMessage)`` (a.k.a.
+:meth:`~email.message.EmailMessage.__bytes__`), which simplify the generation of
+a serialized binary representation of a message object.  For more detail, see
+:mod:`email.message`.
 
 
-   The *policy* keyword specifies a :mod:`~email.policy` object that controls a
-   number of aspects of the generator's operation.  If no *policy* is specified,
-   then the *policy* attached to the message object passed to :attr:`flatten`
-   is used.
+Because strings cannot represent binary data, the :class:`Generator` class must
+convert any binary data in any message it flattens to an ASCII compatible
+format, by converting them to an ASCII compatible
+:mailheader:`Content-Transfer_Encoding`.  Using the terminology of the email
+RFCs, you can think of this as :class:`Generator` serializing to an I/O stream
+that is not "8 bit clean".  In other words, most applications will want
+to be using :class:`BytesGenerator`, and not :class:`Generator`.
+
+.. class:: Generator(outfp, mangle_from_=None, maxheaderlen=None, *, \
+                     policy=None)
+
+   Return a :class:`Generator` object that will write any message provided
+   to the :meth:`flatten` method, or any text provided to the :meth:`write`
+   method, to the :term:`file-like object` *outfp*.  *outfp* must support a
+   ``write`` method that accepts string data.
+
+   If optional *mangle_from_* is ``True``, put a ``>`` character in front of
+   any line in the body that starts with the exact string ``"From "``, that is
+   ``From`` followed by a space at the beginning of a line.  *mangle_from_*
+   defaults to the value of the :attr:`~email.policy.Policy.mangle_from_`
+   setting of the *policy* (which is ``True`` for the
+   :data:`~email.policy.compat32` policy and ``False`` for all others).
+   *mangle_from_* is intended for use when messages are stored in unix mbox
+   format (see :mod:`mailbox` and `WHY THE CONTENT-LENGTH FORMAT IS BAD
+   <http://www.jwz.org/doc/content-length.html>`_).
+
+   If *maxheaderlen* is not ``None``, refold any header lines that are longer
+   than *maxheaderlen*, or if ``0``, do not rewrap any headers.  If
+   *manheaderlen* is ``None`` (the default), wrap headers and other message
+   lines according to the *policy* settings.
+
+   If *policy* is specified, use that policy to control message generation.  If
+   *policy* is ``None`` (the default), use the policy associated with the
+   :class:`~email.message.Message` or :class:`~email.message.EmailMessage`
+   object passed to ``flatten`` to control the message generation.  See
+   :mod:`email.policy` for details on what *policy* controls.
 
    .. versionchanged:: 3.3 Added the *policy* keyword.
 
-   The other public :class:`BytesGenerator` methods are:
+   .. versionchanged:: 3.6 The default behavior of the *mangle_from_*
+      and *maxheaderlen* parameters is to follow the policy.
 
 
    .. method:: flatten(msg, unixfrom=False, linesep=None)
 
       Print the textual representation of the message object structure rooted
-      at *msg* to the output file specified when the :class:`BytesGenerator`
-      instance was created.  Subparts are visited depth-first and the resulting
-      text will be properly MIME encoded.  If the :mod:`~email.policy` option
-      :attr:`~email.policy.Policy.cte_type` is ``8bit`` (the default),
-      then any bytes with the high bit set in the original parsed message that
-      have not been modified will be copied faithfully to the output.  If
-      ``cte_type`` is ``7bit``, the bytes will be converted as needed
-      using an ASCII-compatible Content-Transfer-Encoding.  In particular,
-      RFC-invalid non-ASCII bytes in headers will be encoded using the MIME
-      ``unknown-8bit`` character set, thus rendering them RFC-compliant.
-
-      .. XXX: There should be a complementary option that just does the RFC
-         compliance transformation but leaves CTE 8bit parts alone.
-
-      Messages parsed with a Bytes parser that have a
-      :mailheader:`Content-Transfer-Encoding` of 8bit will be reconstructed
-      as 8bit if they have not been modified.
-
-      Optional *unixfrom* is a flag that forces the printing of the envelope
-      header delimiter before the first :rfc:`2822` header of the root message
-      object.  If the root object has no envelope header, a standard one is
-      crafted.  By default, this is set to ``False`` to inhibit the printing of
-      the envelope delimiter.
-
+      at *msg* to the output file specified when the :class:`Generator`
+      instance was created.
+
+      If the :mod:`~email.policy` option :attr:`~email.policy.Policy.cte_type`
+      is ``8bit``, generate the message as if the option were set to ``7bit``.
+      (This is required because strings cannot represent non-ASCII bytes.)
+      Convert any bytes with the high bit set as needed using an
+      ASCII-compatible :mailheader:`Content-Transfer-Encoding`.  That is,
+      transform parts with non-ASCII :mailheader:`Cotnent-Transfer-Encoding`
+      (:mailheader:`Content-Transfer-Encoding: 8bit`) to an ASCII compatibile
+      :mailheader:`Content-Transfer-Encoding`, and encode RFC-invalid non-ASCII
+      bytes in headers using the MIME ``unknown-8bit`` character set, thus
+      rendering them RFC-compliant.
+
+      If *unixfrom* is ``True``, print the envelope header delimiter used by
+      the Unix mailbox format (see :mod:`mailbox`) before the first of the
+      :rfc:`5322` headers of the root message object.  If the root object has
+      no envelope header, craft a standard one.  The default is ``False``.
       Note that for subparts, no envelope header is ever printed.
 
-      Optional *linesep* specifies the line separator character used to
-      terminate lines in the output.  If specified it overrides the value
-      specified by the ``Generator``\ or *msg*\ 's ``policy``.
+      If *linesep* is not ``None``, use it as the separator character between
+      all the lines of the flattened message.  If *linesep* is ``None`` (the
+      default), use the value specified in the *policy*.
+
+      .. XXX: flatten should take a *policy* keyword.
+
+      .. versionchanged:: 3.2
+         Added support for re-encoding ``8bit`` message bodies, and the
+         *linesep* argument.
+
 
    .. method:: clone(fp)
 
-      Return an independent clone of this :class:`BytesGenerator` instance with
-      the exact same options.
+      Return an independent clone of this :class:`Generator` instance with the
+      exact same options, and *fp* as the new *outfp*.
+
 
    .. method:: write(s)
 
-      Write the string *s* to the underlying file object.  *s* is encoded using
-      the ``ASCII`` codec and written to the *write* method of the  *outfp*
-      *outfp* passed to the :class:`BytesGenerator`'s constructor.  This
-      provides just enough file-like API for :class:`BytesGenerator` instances
-      to be used in the :func:`print` function.
+      Write *s* to the *write* method of the *outfp* passed to the
+      :class:`Generator`'s constructor.  This provides just enough file-like
+      API for :class:`Generator` instances to be used in the :func:`print`
+      function.
 
-   .. versionadded:: 3.2
 
-The :mod:`email.generator` module also provides a derived class, called
-:class:`DecodedGenerator` which is like the :class:`Generator` base class,
-except that non-\ :mimetype:`text` parts are substituted with a format string
-representing the part.
+As a convenience, :class:`~email.message.EmailMessage` provides the methods
+:meth:`~email.message.EmailMessage.as_string` and ``str(aMessage)`` (a.k.a.
+:meth:`~email.message.EmailMessage.__str__`), which simplify the generation of
+a formatted string representation of a message object.  For more detail, see
+:mod:`email.message`.
+
 
+The :mod:`email.generator` module also provides a derived class,
+:class:`DecodedGenerator`, which is like the :class:`Generator` base class,
+except that non-\ :mimetype:`text` parts are not serialized, but are instead
+represented in the output stream by a string derived from a template filled
+in with information about the part.
 
-.. class:: DecodedGenerator(outfp, mangle_from_=True, maxheaderlen=78, fmt=None)
+.. class:: DecodedGenerator(outfp, mangle_from_=None, maxheaderlen=78, fmt=None)
 
-   This class, derived from :class:`Generator` walks through all the subparts of a
-   message.  If the subpart is of main type :mimetype:`text`, then it prints the
-   decoded payload of the subpart. Optional *_mangle_from_* and *maxheaderlen* are
-   as with the :class:`Generator` base class.
+   Act like :class:`Generator`, except that for any subpart of the message
+   passed to :meth:`Generator.flatten`, if the subpart is of main type
+   :mimetype:`text`, print the decoded payload of the subpart, and if the main
+   type is not :mimetype:`text`, instead of printing it fill in the string
+   *fmt* using information from the part and print the resulting
+   filled-in string.
 
-   If the subpart is not of main type :mimetype:`text`, optional *fmt* is a format
-   string that is used instead of the message payload. *fmt* is expanded with the
-   following keywords, ``%(keyword)s`` format:
+   To fill in *fmt*, execute ``fmt % part_info``, where ``part_info``
+   is a dictionary composed of the following keys and values:
 
    * ``type`` -- Full MIME type of the non-\ :mimetype:`text` part
 
@@ -225,15 +258,22 @@ representing the part.
 
    * ``encoding`` -- Content transfer encoding of the non-\ :mimetype:`text` part
 
-   The default value for *fmt* is ``None``, meaning ::
+   If *fmt* is ``None``, use the following default *fmt*:
+
+      "[Non-text (%(type)s) part of message omitted, filename %(filename)s]"
 
-      [Non-text (%(type)s) part of message omitted, filename %(filename)s]
+   Optional *_mangle_from_* and *maxheaderlen* are as with the
+   :class:`Generator` base class, except that the default value for
+   *maxheaderlen* is ``78`` (the RFC standard default header length).
 
 
 .. rubric:: Footnotes
 
-.. [#] This statement assumes that you use the appropriate setting for the
-       ``unixfrom`` argument, and that you set maxheaderlen=0 (which will
-       preserve whatever the input line lengths were).  It is also not strictly
-       true, since in many cases runs of whitespace in headers are collapsed
-       into single blanks.  The latter is a bug that will eventually be fixed.
+.. [#] This statement assumes that you use the appropriate setting for
+       ``unixfrom``, and that there are no :mod:`policy` settings calling for
+       automatic adjustments (for example,
+       :attr:`~email.policy.Policy.refold_source` must be ``none``, which is
+       *not* the default).  It is also not 100% true, since if the message
+       does not conform to the RFC standards occasionally information about the
+       exact original text is lost during parsing error recovery.  It is a goal
+       to fix these latter edge cases when possible.