Patch #1177307: UTF-8-Sig codec.

1 files changed, 125 insertions, 0 deletions

diff --git a/Doc/lib/libcodecs.tex b/Doc/lib/libcodecs.tex
index be2385e..71d6fe8 100644
--- a/Doc/lib/libcodecs.tex
+++ b/Doc/lib/libcodecs.tex
@@ -522,6 +522,113 @@ the \function{lookup()} function to construct the instance.
 \class{StreamReader} and \class{StreamWriter} classes. They inherit
 all other methods and attribute from the underlying stream.
 
+\subsection{Encodings and Unicode\label{encodings-overview}}
+
+Unicode strings are stored internally as sequences of codepoints (to
+be precise as Py_UNICODE arrays). Depending on the way Python is
+compiled (either via --enable-unicode=ucs2 or --enable-unicode=ucs4,
+with the former being the default) Py_UNICODE is either a 16-bit or
+32-bit data type. Once a Unicode object is used outside of CPU and
+memory, CPU endianness and how these arrays are stored as bytes become
+an issue. Transforming a unicode object into a sequence of bytes is
+called encoding and recreating the unicode object from the sequence of
+bytes is known as decoding. There are many different methods how this
+transformation can be done (these methods are also called encodings).
+The simplest method is to map the codepoints 0-255 to the bytes
+0x0-0xff. This means that a unicode object that contains codepoints
+above U+00FF can't be encoded with this method (which is called
+'latin-1' or 'iso-8859-1'). unicode.encode() will raise a
+UnicodeEncodeError that looks like this: UnicodeEncodeError: 'latin-1'
+codec can't encode character u'\u1234' in position 3: ordinal not in
+range(256)
+
+There's another group of encodings (the so called charmap encodings)
+that choose a different subset of all unicode code points and how
+these codepoints are mapped to the bytes 0x0-0xff. To see how this is
+done simply open e.g.  encodings/cp1252.py (which is an encoding that
+is used primarily on Windows).  There's string constant with 256
+characters that shows you which character is mapped to which byte
+value.
+
+All of these encodings can only encode 256 of the 65536 (or 1114111)
+codepoints defined in unicode. A simple and straightforward way that
+can store each Unicode code point, is to store each codepoint as two
+consecutive bytes. There are two possibilities: Store the bytes in big
+endian or in little endian order. These two encodings are called
+UTF-16-BE and UTF-16-LE respectively. Their disadvantage is that if
+e.g. you use UTF-16-BE on a little endian machine you will always have
+to swap bytes on encoding and decoding. UTF-16 avoids this problem:
+Bytes will always be in natural endianness. When these bytes are read
+by a CPU with a different endianness, then bytes have to be swapped
+though. To be able to detect the endianness of a UTF-16 byte sequence,
+there's the so called BOM (the "Byte Order Mark"). This is the Unicode
+character U+FEFF. This character will be prepended to every UTF-16
+byte sequence. The byte swapped version of this character (0xFFFE) is
+an illegal character that may not appear in a Unicode text. So when
+the first character in an UTF-16 byte sequence appears to be a U+FFFE
+the bytes have to be swapped on decoding. Unfortunately upto Unicode
+4.0 the character U+FEFF had a second purpose as a "ZERO WIDTH
+NO-BREAK SPACE": A character that has no width and doesn't allow a
+word to be split. It can e.g. be used to give hints to a ligature
+algorithm. With Unicode 4.0 using U+FEFF as a ZERO WIDTH NO-BREAK
+SPACE has been deprecated (with U+2060 (WORD JOINER) assuming this
+role). Nevertheless Unicode software still must be able to handle
+U+FEFF in both roles: As a BOM it's a device to determine the storage
+layout of the encoded bytes, and vanishes once the byte sequence has
+been decoded into a Unicode string; as a ZERO WIDTH NO-BREAK SPACE
+it's a normal character that will be decoded like any other.
+
+There's another encoding that is able to encoding the full range of
+Unicode characters: UTF-8. UTF-8 is an 8bit encoding, which means
+there are no issues with byte order in UTF-8. Each byte in a UTF-8
+byte sequence consists of two parts: Marker bits (the most significant
+bits) and payload bits. The marker bits are a sequence of zero to six
+1 bits followed by a 0 bit. Unicode characters are encoded like this
+(with x being a payload bit, which when concatenated give the Unicode
+character):
+
+\begin{tableii}{l|l}{textrm}{}{Range}{Encoding}
+\lineii{U-00000000 ... U-0000007F}{0xxxxxxx}
+\lineii{U-00000080 ... U-000007FF}{110xxxxx 10xxxxxx}
+\lineii{U-00000800 ... U-0000FFFF}{1110xxxx 10xxxxxx 10xxxxxx}
+\lineii{U-00010000 ... U-001FFFFF}{11110xxx 10xxxxxx 10xxxxxx 10xxxxxx}
+\lineii{U-00200000 ... U-03FFFFFF}{111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx}
+\lineii{U-04000000 ... U-7FFFFFFF}{1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx}
+\end{tableii}
+
+The least significant bit of the Unicode character is the rightmost x
+bit.
+
+As UTF-8 is an 8bit encoding no BOM is required and any U+FEFF
+character in the decoded Unicode string (even if it's the first
+character) is treated as a ZERO WIDTH NO-BREAK SPACE.
+
+Without external information it's impossible to reliably determine
+which encoding was used for encoding a Unicode string. Each charmap
+encoding can decode any random byte sequence. However that's not
+possible with UTF-8, as UTF-8 byte sequences have a structure that
+doesn't allow arbitrary byte sequence. To increase the reliability
+with which an UTF-8 encoding can be detected, Microsoft invented a
+variant of UTF-8 (that Python 2.5 calls "utf-8-sig") for its Notepad
+program: Before any of the Unicode characters is written to the file,
+a UTF-8 encoded BOM (which looks like this as a byte sequence: 0xef,
+0xbb, 0xbf) is written. As it's rather improbably that any charmap
+encoded file starts with these byte values (which would e.g. map to
+
+   LATIN SMALL LETTER I WITH DIAERESIS
+   RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
+   INVERTED QUESTION MARK
+
+in iso-8859-1), this increases the probability that a utf-8-sig
+encoding can be correctly guessed from the byte sequence. So here the
+BOM is not used to be able to determine the byte order used for
+generating the byte sequence, but as a signature that helps in
+guessing the encoding. On encoding the utf-8-sig codec will write
+0xef, 0xbb, 0xbf as the first three bytes to the file. On decoding
+utf-8-sig will skip those three bytes if they appear as the first
+three bytes in the file.
+
+
 \subsection{Standard Encodings\label{standard-encodings}}
 
 Python comes with a number of codecs builtin, either implemented as C
@@ -890,6 +997,10 @@ exist:
         {U8, UTF, utf8}
         {all languages}
 
+\lineiii{utf_8_sig}
+        {}
+        {all languages}
+
 \end{longtableiii}
 
 A number of codecs are specific to Python, so their codec names have
@@ -1058,3 +1169,17 @@ Convert a label to \ASCII, as specified in \rfc{3490}.
 \begin{funcdesc}{ToUnicode}{label}
 Convert a label to Unicode, as specified in \rfc{3490}.
 \end{funcdesc}
+
+ \subsection{\module{encodings.utf_8_sig} ---
+             UTF-8 codec with BOM signature}
+\declaremodule{standard}{encodings.utf-8-sig}   % XXX utf_8_sig gives TeX errors
+\modulesynopsis{UTF-8 codec with BOM signature}
+\moduleauthor{Walter D\"orwald}
+
+\versionadded{2.5}
+
+This module implements a variant of the UTF-8 codec: On encoding a
+UTF-8 encoded BOM will be prepended to the UTF-8 encoded bytes. For
+the stateful encoder this is only done once (on the first write to the
+byte stream).  For decoding an optional UTF-8 encoded BOM at the start
+of the data will be skipped.