summaryrefslogtreecommitdiffstats
path: root/Doc/library/email.charset.rst
blob: 161d86a3b70d2fb8ae81d55ad463a385fc1c4ef8 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
:mod:`email.charset`: Representing character sets
-------------------------------------------------

.. module:: email.charset
   :synopsis: Character Sets

**Source code:** :source:`Lib/email/charset.py`

--------------

This module provides a class :class:`Charset` for representing character sets
and character set conversions in email messages, as well as a character set
registry and several convenience methods for manipulating this registry.
Instances of :class:`Charset` are used in several other modules within the
:mod:`email` package.

Import this class from the :mod:`email.charset` module.


.. class:: Charset(input_charset=DEFAULT_CHARSET)

   Map character sets to their email properties.

   This class provides information about the requirements imposed on email for a
   specific character set.  It also provides convenience routines for converting
   between character sets, given the availability of the applicable codecs.  Given
   a character set, it will do its best to provide information on how to use that
   character set in an email message in an RFC-compliant way.

   Certain character sets must be encoded with quoted-printable or base64 when used
   in email headers or bodies.  Certain character sets must be converted outright,
   and are not allowed in email.

   Optional *input_charset* is as described below; it is always coerced to lower
   case.  After being alias normalized it is also used as a lookup into the
   registry of character sets to find out the header encoding, body encoding, and
   output conversion codec to be used for the character set.  For example, if
   *input_charset* is ``iso-8859-1``, then headers and bodies will be encoded using
   quoted-printable and no output conversion codec is necessary.  If
   *input_charset* is ``euc-jp``, then headers will be encoded with base64, bodies
   will not be encoded, but output text will be converted from the ``euc-jp``
   character set to the ``iso-2022-jp`` character set.

   :class:`Charset` instances have the following data attributes:

   .. attribute:: input_charset

      The initial character set specified.  Common aliases are converted to
      their *official* email names (e.g. ``latin_1`` is converted to
      ``iso-8859-1``).  Defaults to 7-bit ``us-ascii``.


   .. attribute:: header_encoding

      If the character set must be encoded before it can be used in an email
      header, this attribute will be set to ``Charset.QP`` (for
      quoted-printable), ``Charset.BASE64`` (for base64 encoding), or
      ``Charset.SHORTEST`` for the shortest of QP or BASE64 encoding. Otherwise,
      it will be ``None``.


   .. attribute:: body_encoding

      Same as *header_encoding*, but describes the encoding for the mail
      message's body, which indeed may be different than the header encoding.
      ``Charset.SHORTEST`` is not allowed for *body_encoding*.


   .. attribute:: output_charset

      Some character sets must be converted before they can be used in email
      headers or bodies.  If the *input_charset* is one of them, this attribute
      will contain the name of the character set output will be converted to.
      Otherwise, it will be ``None``.


   .. attribute:: input_codec

      The name of the Python codec used to convert the *input_charset* to
      Unicode.  If no conversion codec is necessary, this attribute will be
      ``None``.


   .. attribute:: output_codec

      The name of the Python codec used to convert Unicode to the
      *output_charset*.  If no conversion codec is necessary, this attribute
      will have the same value as the *input_codec*.


   :class:`Charset` instances also have the following methods:

   .. method:: get_body_encoding()

      Return the content transfer encoding used for body encoding.

      This is either the string ``quoted-printable`` or ``base64`` depending on
      the encoding used, or it is a function, in which case you should call the
      function with a single argument, the Message object being encoded.  The
      function should then set the :mailheader:`Content-Transfer-Encoding`
      header itself to whatever is appropriate.

      Returns the string ``quoted-printable`` if *body_encoding* is ``QP``,
      returns the string ``base64`` if *body_encoding* is ``BASE64``, and
      returns the string ``7bit`` otherwise.


   .. XXX to_splittable and from_splittable are not there anymore!

   .. to_splittable(s)

      Convert a possibly multibyte string to a safely splittable format. *s* is
      the string to split.

      Uses the *input_codec* to try and convert the string to Unicode, so it can
      be safely split on character boundaries (even for multibyte characters).

      Returns the string as-is if it isn't known how to convert *s* to Unicode
      with the *input_charset*.

      Characters that could not be converted to Unicode will be replaced with
      the Unicode replacement character ``'U+FFFD'``.


   .. from_splittable(ustr[, to_output])

      Convert a splittable string back into an encoded string.  *ustr* is a
      Unicode string to "unsplit".

      This method uses the proper codec to try and convert the string from
      Unicode back into an encoded format.  Return the string as-is if it is not
      Unicode, or if it could not be converted from Unicode.

      Characters that could not be converted from Unicode will be replaced with
      an appropriate character (usually ``'?'``).

      If *to_output* is ``True`` (the default), uses *output_codec* to convert
      to an encoded format.  If *to_output* is ``False``, it uses *input_codec*.


   .. method:: get_output_charset()

      Return the output character set.

      This is the *output_charset* attribute if that is not ``None``, otherwise
      it is *input_charset*.


   .. method:: header_encode(string)

      Header-encode the string *string*.

      The type of encoding (base64 or quoted-printable) will be based on the
      *header_encoding* attribute.


   .. method:: header_encode_lines(string, maxlengths)

      Header-encode a *string* by converting it first to bytes.

      This is similar to :meth:`header_encode` except that the string is fit
      into maximum line lengths as given by the argument *maxlengths*, which
      must be an iterator: each element returned from this iterator will provide
      the next maximum line length.


   .. method:: body_encode(string)

      Body-encode the string *string*.

      The type of encoding (base64 or quoted-printable) will be based on the
      *body_encoding* attribute.

   The :class:`Charset` class also provides a number of methods to support
   standard operations and built-in functions.


   .. method:: __str__()

      Returns *input_charset* as a string coerced to lower
      case. :meth:`__repr__` is an alias for :meth:`__str__`.


   .. method:: __eq__(other)

      This method allows you to compare two :class:`Charset` instances for
      equality.


   .. method:: __ne__(other)

      This method allows you to compare two :class:`Charset` instances for
      inequality.

The :mod:`email.charset` module also provides the following functions for adding
new entries to the global character set, alias, and codec registries:


.. function:: add_charset(charset, header_enc=None, body_enc=None, output_charset=None)

   Add character properties to the global registry.

   *charset* is the input character set, and must be the canonical name of a
   character set.

   Optional *header_enc* and *body_enc* is either ``Charset.QP`` for
   quoted-printable, ``Charset.BASE64`` for base64 encoding,
   ``Charset.SHORTEST`` for the shortest of quoted-printable or base64 encoding,
   or ``None`` for no encoding.  ``SHORTEST`` is only valid for
   *header_enc*. The default is ``None`` for no encoding.

   Optional *output_charset* is the character set that the output should be in.
   Conversions will proceed from input charset, to Unicode, to the output charset
   when the method :meth:`Charset.convert` is called.  The default is to output in
   the same character set as the input.

   Both *input_charset* and *output_charset* must have Unicode codec entries in the
   module's character set-to-codec mapping; use :func:`add_codec` to add codecs the
   module does not know about.  See the :mod:`codecs` module's documentation for
   more information.

   The global character set registry is kept in the module global dictionary
   ``CHARSETS``.


.. function:: add_alias(alias, canonical)

   Add a character set alias.  *alias* is the alias name, e.g. ``latin-1``.
   *canonical* is the character set's canonical name, e.g. ``iso-8859-1``.

   The global charset alias registry is kept in the module global dictionary
   ``ALIASES``.


.. function:: add_codec(charset, codecname)

   Add a codec that map characters in the given character set to and from Unicode.

   *charset* is the canonical name of a character set. *codecname* is the name of a
   Python codec, as appropriate for the second argument to the :class:`str`'s
   :meth:`~str.encode` method.