summaryrefslogtreecommitdiffstats
path: root/Doc/lib/libcodecs.tex
blob: bb444a272e1efa38d4c9f2701a3e4e1d405d1b52 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
\section{\module{codecs} ---
         Codec registry and base classes}

\declaremodule{standard}{codecs}
\modulesynopsis{Encode and decode data and streams.}
\moduleauthor{Marc-Andre Lemburg}{mal@lemburg.com}
\sectionauthor{Marc-Andre Lemburg}{mal@lemburg.com}


\index{Unicode}
\index{Codecs}
\indexii{Codecs}{encode}
\indexii{Codecs}{decode}
\index{streams}
\indexii{stackable}{streams}


This module defines base classes for standard Python codecs (encoders
and decoders) and provides access to the internal Python codec
registry which manages the codec lookup process.

It defines the following functions:

\begin{funcdesc}{register}{search_function}
Register a codec search function. Search functions are expected to
take one argument, the encoding name in all lower case letters, and
return a tuple of functions \code{(\var{encoder}, \var{decoder}, \var{stream_reader},
\var{stream_writer})} taking the following arguments:

  \var{encoder} and \var{decoder}: These must be functions or methods
  which have the same interface as the
  \method{encode()}/\method{decode()} methods of Codec instances (see
  Codec Interface). The functions/methods are expected to work in a
  stateless mode.

  \var{stream_reader} and \var{stream_writer}: These have to be
  factory functions providing the following interface:

        \code{factory(\var{stream}, \var{errors}='strict')}

  The factory functions must return objects providing the interfaces
  defined by the base classes \class{StreamWriter} and
  \class{StreamReader}, respectively. Stream codecs can maintain
  state.

  Possible values for errors are \code{'strict'} (raise an exception
  in case of an encoding error), \code{'replace'} (replace malformed
  data with a suitable replacement marker, such as \character{?}) and
  \code{'ignore'} (ignore malformed data and continue without further
  notice).

In case a search function cannot find a given encoding, it should
return \code{None}.
\end{funcdesc}

\begin{funcdesc}{lookup}{encoding}
Looks up a codec tuple in the Python codec registry and returns the
function tuple as defined above.

Encodings are first looked up in the registry's cache. If not found,
the list of registered search functions is scanned. If no codecs tuple
is found, a \exception{LookupError} is raised. Otherwise, the codecs
tuple is stored in the cache and returned to the caller.
\end{funcdesc}

To simplify working with encoded files or stream, the module
also defines these utility functions:

\begin{funcdesc}{open}{filename, mode\optional{, encoding\optional{,
                       errors\optional{, buffering}}}}
Open an encoded file using the given \var{mode} and return
a wrapped version providing transparent encoding/decoding.

\strong{Note:} The wrapped version will only accept the object format
defined by the codecs, i.e.\ Unicode objects for most built-in
codecs.  Output is also codec-dependent and will usually be Unicode as
well.

\var{encoding} specifies the encoding which is to be used for the
the file.

\var{errors} may be given to define the error handling. It defaults
to \code{'strict'} which causes a \exception{ValueError} to be raised
in case an encoding error occurs.

\var{buffering} has the same meaning as for the built-in
\function{open()} function.  It defaults to line buffered.
\end{funcdesc}

\begin{funcdesc}{EncodedFile}{file, input\optional{,
                              output\optional{, errors}}}
Return a wrapped version of file which provides transparent
encoding translation.

Strings written to the wrapped file are interpreted according to the
given \var{input} encoding and then written to the original file as
strings using the \var{output} encoding. The intermediate encoding will
usually be Unicode but depends on the specified codecs.

If \var{output} is not given, it defaults to \var{input}.

\var{errors} may be given to define the error handling. It defaults to
\code{'strict'}, which causes \exception{ValueError} to be raised in case
an encoding error occurs.
\end{funcdesc}

The module also provides the following constants which are useful
for reading and writing to platform dependent files:

\begin{datadesc}{BOM}
\dataline{BOM_BE}
\dataline{BOM_LE}
\dataline{BOM32_BE}
\dataline{BOM32_LE}
\dataline{BOM64_BE}
\dataline{BOM64_LE}
These constants define the byte order marks (BOM) used in data
streams to indicate the byte order used in the stream or file.
\constant{BOM} is either \constant{BOM_BE} or \constant{BOM_LE}
depending on the platform's native byte order, while the others
represent big endian (\samp{_BE} suffix) and little endian
(\samp{_LE} suffix) byte order using 32-bit and 64-bit encodings.
\end{datadesc}

\subsection{Codec Base Classes}

The \module{codecs} defines a set of base classes which define the
interface and can also be used to easily write you own codecs for use
in Python.

Each codec has to define four interfaces to make it usable as codec in
Python: stateless encoder, stateless decoder, stream reader and stream
writer. The stream reader and writers typically reuse the stateless
encoder/decoder to implement the file protocols.

The \class{Codec} class defines the interface for stateless
encoders/decoders.

To simplify and standardize error handling, the \method{encode()} and
\method{decode()} methods may implement different error handling
schemes by providing the \var{errors} string argument.  The following
string values are defined and implemented by all standard Python
codecs:

\begin{itemize}
  \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
                      this is the default.
  \item \code{'ignore'} Ignore the character and continue with the next.
  \item \code{'replace'} Replace with a suitable replacement character;
                      Python will use the official U+FFFD REPLACEMENT
                      CHARACTER for the builtin Unicode codecs.
\end{itemize}


\subsubsection{Codec Objects \label{codec-objects}}

The \class{Codec} class defines these methods which also define the
function interfaces of the stateless encoder and decoder:

\begin{methoddesc}{encode}{input\optional{, errors}}
  Encodes the object \var{input} and returns a tuple (output object,
  length consumed).

  \var{errors} defines the error handling to apply. It defaults to
  \code{'strict'} handling.

  The method may not store state in the \class{Codec} instance. Use
  \class{StreamCodec} for codecs which have to keep state in order to
  make encoding/decoding efficient.

  The encoder must be able to handle zero length input and return an
  empty object of the output object type in this situation.
\end{methoddesc}

\begin{methoddesc}{decode}{input\optional{, errors}}
  Decodes the object \var{input} and returns a tuple (output object,
  length consumed).

  \var{input} must be an object which provides the \code{bf_getreadbuf}
  buffer slot.  Python strings, buffer objects and memory mapped files
  are examples of objects providing this slot.

  \var{errors} defines the error handling to apply. It defaults to
  \code{'strict'} handling.

  The method may not store state in the \class{Codec} instance. Use
  \class{StreamCodec} for codecs which have to keep state in order to
  make encoding/decoding efficient.

  The decoder must be able to handle zero length input and return an
  empty object of the output object type in this situation.
\end{methoddesc}

The \class{StreamWriter} and \class{StreamReader} classes provide
generic working interfaces which can be used to implement new
encodings submodules very easily. See \module{encodings.utf_8} for an
example on how this is done.


\subsubsection{StreamWriter Objects \label{stream-writer-objects}}

The \class{StreamWriter} class is a subclass of \class{Codec} and
defines the following methods which every stream writer must define in
order to be compatible to the Python codec registry.

\begin{classdesc}{StreamWriter}{stream\optional{, errors}}
  Constructor for a \class{StreamWriter} instance. 

  All stream writers must provide this constructor interface. They are
  free to add additional keyword arguments, but only the ones defined
  here are used by the Python codec registry.

  \var{stream} must be a file-like object open for writing (binary)
  data.

  The \class{StreamWriter} may implement different error handling
  schemes by providing the \var{errors} keyword argument. These
  parameters are defined:

  \begin{itemize}
    \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
                          this is the default.
    \item \code{'ignore'} Ignore the character and continue with the next.
    \item \code{'replace'} Replace with a suitable replacement character
  \end{itemize}
\end{classdesc}

\begin{methoddesc}{write}{object}
  Writes the object's contents encoded to the stream.
\end{methoddesc}

\begin{methoddesc}{writelines}{list}
  Writes the concatenated list of strings to the stream (possibly by
  reusing the \method{write()} method).
\end{methoddesc}

\begin{methoddesc}{reset}{}
  Flushes and resets the codec buffers used for keeping state.

  Calling this method should ensure that the data on the output is put
  into a clean state, that allows appending of new fresh data without
  having to rescan the whole stream to recover state.
\end{methoddesc}

In addition to the above methods, the \class{StreamWriter} must also
inherit all other methods and attribute from the underlying stream.


\subsubsection{StreamReader Objects \label{stream-reader-objects}}

The \class{StreamReader} class is a subclass of \class{Codec} and
defines the following methods which every stream reader must define in
order to be compatible to the Python codec registry.

\begin{classdesc}{StreamReader}{stream\optional{, errors}}
  Constructor for a \class{StreamReader} instance. 

  All stream readers must provide this constructor interface. They are
  free to add additional keyword arguments, but only the ones defined
  here are used by the Python codec registry.

  \var{stream} must be a file-like object open for reading (binary)
  data.

  The \class{StreamReader} may implement different error handling
  schemes by providing the \var{errors} keyword argument. These
  parameters are defined:

  \begin{itemize}
    \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
                          this is the default.
    \item \code{'ignore'} Ignore the character and continue with the next.
    \item \code{'replace'} Replace with a suitable replacement character.
  \end{itemize}
\end{classdesc}

\begin{methoddesc}{read}{\optional{size}}
  Decodes data from the stream and returns the resulting object.

  \var{size} indicates the approximate maximum number of bytes to read
  from the stream for decoding purposes. The decoder can modify this
  setting as appropriate. The default value -1 indicates to read and
  decode as much as possible.  \var{size} is intended to prevent having
  to decode huge files in one step.

  The method should use a greedy read strategy meaning that it should
  read as much data as is allowed within the definition of the encoding
  and the given size, e.g.  if optional encoding endings or state
  markers are available on the stream, these should be read too.
\end{methoddesc}

\begin{methoddesc}{readline}{[size]}
  Read one line from the input stream and return the
  decoded data.

  Note: Unlike the \method{readlines()} method, this method inherits
  the line breaking knowledge from the underlying stream's
  \method{readline()} method -- there is currently no support for line
  breaking using the codec decoder due to lack of line buffering.
  Sublcasses should however, if possible, try to implement this method
  using their own knowledge of line breaking.

  \var{size}, if given, is passed as size argument to the stream's
  \method{readline()} method.
\end{methoddesc}

\begin{methoddesc}{readlines}{[sizehint]}
  Read all lines available on the input stream and return them as list
  of lines.

  Line breaks are implemented using the codec's decoder method and are
  included in the list entries.

  \var{sizehint}, if given, is passed as \var{size} argument to the
  stream's \method{read()} method.
\end{methoddesc}

\begin{methoddesc}{reset}{}
  Resets the codec buffers used for keeping state.

  Note that no stream repositioning should take place.  This method is
  primarily intended to be able to recover from decoding errors.
\end{methoddesc}

In addition to the above methods, the \class{StreamReader} must also
inherit all other methods and attribute from the underlying stream.

The next two base classes are included for convenience. They are not
needed by the codec registry, but may provide useful in practice.


\subsubsection{StreamReaderWriter Objects \label{stream-reader-writer}}

The \class{StreamReaderWriter} allows wrapping streams which work in
both read and write modes.

The design is such that one can use the factory functions returned by
the \function{lookup()} function to construct the instance.

\begin{classdesc}{StreamReaderWriter}{stream, Reader, Writer, errors}
  Creates a \class{StreamReaderWriter} instance.
  \var{stream} must be a file-like object.
  \var{Reader} and \var{Writer} must be factory functions or classes
  providing the \class{StreamReader} and \class{StreamWriter} interface
  resp.
  Error handling is done in the same way as defined for the
  stream readers and writers.
\end{classdesc}

\class{StreamReaderWriter} instances define the combined interfaces of
\class{StreamReader} and \class{StreamWriter} classes. They inherit
all other methods and attribute from the underlying stream.


\subsubsection{StreamRecoder Objects \label{stream-recoder-objects}}

The \class{StreamRecoder} provide a frontend - backend view of
encoding data which is sometimes useful when dealing with different
encoding environments.

The design is such that one can use the factory functions returned by
the \function{lookup()} function to construct the instance.

\begin{classdesc}{StreamRecoder}{stream, encode, decode,
                                 Reader, Writer, errors}
  Creates a \class{StreamRecoder} instance which implements a two-way
  conversion: \var{encode} and \var{decode} work on the frontend (the
  input to \method{read()} and output of \method{write()}) while
  \var{Reader} and \var{Writer} work on the backend (reading and
  writing to the stream).

  You can use these objects to do transparent direct recodings from
  e.g.\ Latin-1 to UTF-8 and back.

  \var{stream} must be a file-like object.

  \var{encode}, \var{decode} must adhere to the \class{Codec}
  interface, \var{Reader}, \var{Writer} must be factory functions or
  classes providing objects of the the \class{StreamReader} and
  \class{StreamWriter} interface respectively.

  \var{encode} and \var{decode} are needed for the frontend
  translation, \var{Reader} and \var{Writer} for the backend
  translation.  The intermediate format used is determined by the two
  sets of codecs, e.g. the Unicode codecs will use Unicode as
  intermediate encoding.

  Error handling is done in the same way as defined for the
  stream readers and writers.
\end{classdesc}

\class{StreamRecoder} instances define the combined interfaces of
\class{StreamReader} and \class{StreamWriter} classes. They inherit
all other methods and attribute from the underlying stream.