diff options
Diffstat (limited to 'Misc')
| -rw-r--r-- | Misc/unicode.txt | 885 |
1 files changed, 885 insertions, 0 deletions
diff --git a/Misc/unicode.txt b/Misc/unicode.txt new file mode 100644 index 0000000..b31beef --- /dev/null +++ b/Misc/unicode.txt @@ -0,0 +1,885 @@ +============================================================================= + Python Unicode Integration Proposal Version: 1.2 +----------------------------------------------------------------------------- + + +Introduction: +------------- + +The idea of this proposal is to add native Unicode 3.0 support to +Python in a way that makes use of Unicode strings as simple as +possible without introducing too many pitfalls along the way. + +Since this goal is not easy to achieve -- strings being one of the +most fundamental objects in Python --, we expect this proposal to +undergo some significant refinements. + +Note that the current version of this proposal is still a bit unsorted +due to the many different aspects of the Unicode-Python integration. + +The latest version of this document is always available at: + + http://starship.skyport.net/~lemburg/unicode-proposal.txt + +Older versions are available as: + + http://starship.skyport.net/~lemburg/unicode-proposal-X.X.txt + + +Conventions: +------------ + +· In examples we use u = Unicode object and s = Python string + +· 'XXX' markings indicate points of discussion (PODs) + + +General Remarks: +---------------- + +· Unicode encoding names should be lower case on output and + case-insensitive on input (they will be converted to lower case + by all APIs taking an encoding name as input). + + Encoding names should follow the name conventions as used by the + Unicode Consortium: spaces are converted to hyphens, e.g. 'utf 16' is + written as 'utf-16'. + + Codec modules should use the same names, but with hyphens converted + to underscores, e.g. utf_8, utf_16, iso_8859_1. + +· The <default encoding> should be the widely used 'utf-8' format. This + is very close to the standard 7-bit ASCII format and thus resembles the + standard used programming nowadays in most aspects. + + +Unicode Constructors: +--------------------- + +Python should provide a built-in constructor for Unicode strings which +is available through __builtins__: + + u = unicode(encoded_string[,encoding=<default encoding>][,errors="strict"]) + + u = u'<unicode-escape encoded Python string>' + + u = ur'<raw-unicode-escape encoded Python string>' + +With the 'unicode-escape' encoding being defined as: + +· all non-escape characters represent themselves as Unicode ordinal + (e.g. 'a' -> U+0061). + +· all existing defined Python escape sequences are interpreted as + Unicode ordinals; note that \xXXXX can represent all Unicode + ordinals, and \OOO (octal) can represent Unicode ordinals up to U+01FF. + +· a new escape sequence, \uXXXX, represents U+XXXX; it is a syntax + error to have fewer than 4 digits after \u. + +For an explanation of possible values for errors see the Codec section +below. + +Examples: + +u'abc' -> U+0061 U+0062 U+0063 +u'\u1234' -> U+1234 +u'abc\u1234\n' -> U+0061 U+0062 U+0063 U+1234 U+005c + +The 'raw-unicode-escape' encoding is defined as follows: + +· \uXXXX sequence represent the U+XXXX Unicode character if and + only if the number of leading backslashes is odd + +· all other characters represent themselves as Unicode ordinal + (e.g. 'b' -> U+0062) + + +Note that you should provide some hint to the encoding you used to +write your programs as pragma line in one the first few comment lines +of the source file (e.g. '# source file encoding: latin-1'). If you +only use 7-bit ASCII then everything is fine and no such notice is +needed, but if you include Latin-1 characters not defined in ASCII, it +may well be worthwhile including a hint since people in other +countries will want to be able to read you source strings too. + + +Unicode Type Object: +-------------------- + +Unicode objects should have the type UnicodeType with type name +'unicode', made available through the standard types module. + + +Unicode Output: +--------------- + +Unicode objects have a method .encode([encoding=<default encoding>]) +which returns a Python string encoding the Unicode string using the +given scheme (see Codecs). + + print u := print u.encode() # using the <default encoding> + + str(u) := u.encode() # using the <default encoding> + + repr(u) := "u%s" % repr(u.encode('unicode-escape')) + +Also see Internal Argument Parsing and Buffer Interface for details on +how other APIs written in C will treat Unicode objects. + + +Unicode Ordinals: +----------------- + +Since Unicode 3.0 has a 32-bit ordinal character set, the implementation +should provide 32-bit aware ordinal conversion APIs: + + ord(u[:1]) (this is the standard ord() extended to work with Unicode + objects) + --> Unicode ordinal number (32-bit) + + unichr(i) + --> Unicode object for character i (provided it is 32-bit); + ValueError otherwise + +Both APIs should go into __builtins__ just like their string +counterparts ord() and chr(). + +Note that Unicode provides space for private encodings. Usage of these +can cause different output representations on different machines. This +problem is not a Python or Unicode problem, but a machine setup and +maintenance one. + + +Comparison & Hash Value: +------------------------ + +Unicode objects should compare equal to other objects after these +other objects have been coerced to Unicode. For strings this means +that they are interpreted as Unicode string using the <default +encoding>. + +For the same reason, Unicode objects should return the same hash value +as their UTF-8 equivalent strings. + +Coercion: +--------- + +Using Python strings and Unicode objects to form new objects should +always coerce to the more precise format, i.e. Unicode objects. + + u + s := u + unicode(s) + + s + u := unicode(s) + u + +All string methods should delegate the call to an equivalent Unicode +object method call by converting all envolved strings to Unicode and +then applying the arguments to the Unicode method of the same name, +e.g. + + string.join((s,u),sep) := (s + sep) + u + + sep.join((s,u)) := (s + sep) + u + +For a discussion of %-formatting w/r to Unicode objects, see +Formatting Markers. + + +Exceptions: +----------- + +UnicodeError is defined in the exceptions module as subclass of +ValueError. It is available at the C level via PyExc_UnicodeError. +All exceptions related to Unicode encoding/decoding should be +subclasses of UnicodeError. + + +Codecs (Coder/Decoders) Lookup: +------------------------------- + +A Codec (see Codec Interface Definition) search registry should be +implemented by a module "codecs": + + codecs.register(search_function) + +Search functions are expected to take one argument, the encoding name +in all lower case letters, and return a tuple of functions (encoder, +decoder, stream_reader, stream_writer) taking the following arguments: + + encoder and decoder: + These must be functions or methods which have the same + interface as the .encode/.decode methods of Codec instances + (see Codec Interface). The functions/methods are expected to + work in a stateless mode. + + stream_reader and stream_writer: + These need to be factory functions with the following + interface: + + factory(stream,errors='strict') + + The factory functions must return objects providing + the interfaces defined by StreamWriter/StreamReader resp. + (see Codec Interface). Stream codecs can maintain state. + + Possible values for errors are defined in the Codec + section below. + +In case a search function cannot find a given encoding, it should +return None. + +Aliasing support for encodings is left to the search functions +to implement. + +The codecs module will maintain an encoding cache for performance +reasons. Encodings are first looked up in the cache. If not found, the +list of registered search functions is scanned. If no codecs tuple is +found, a LookupError is raised. Otherwise, the codecs tuple is stored +in the cache and returned to the caller. + +To query the Codec instance the following API should be used: + + codecs.lookup(encoding) + +This will either return the found codecs tuple or raise a LookupError. + + +Standard Codecs: +---------------- + +Standard codecs should live inside an encodings/ package directory in the +Standard Python Code Library. The __init__.py file of that directory should +include a Codec Lookup compatible search function implementing a lazy module +based codec lookup. + +Python should provide a few standard codecs for the most relevant +encodings, e.g. + + 'utf-8': 8-bit variable length encoding + 'utf-16': 16-bit variable length encoding (litte/big endian) + 'utf-16-le': utf-16 but explicitly little endian + 'utf-16-be': utf-16 but explicitly big endian + 'ascii': 7-bit ASCII codepage + 'iso-8859-1': ISO 8859-1 (Latin 1) codepage + 'unicode-escape': See Unicode Constructors for a definition + 'raw-unicode-escape': See Unicode Constructors for a definition + 'native': Dump of the Internal Format used by Python + +Common aliases should also be provided per default, e.g. 'latin-1' +for 'iso-8859-1'. + +Note: 'utf-16' should be implemented by using and requiring byte order +marks (BOM) for file input/output. + +All other encodings such as the CJK ones to support Asian scripts +should be implemented in seperate packages which do not get included +in the core Python distribution and are not a part of this proposal. + + +Codecs Interface Definition: +---------------------------- + +The following base class should be defined in the module +"codecs". They provide not only templates for use by encoding module +implementors, but also define the interface which is expected by the +Unicode implementation. + +Note that the Codec Interface defined here is well suitable for a +larger range of applications. The Unicode implementation expects +Unicode objects on input for .encode() and .write() and character +buffer compatible objects on input for .decode(). Output of .encode() +and .read() should be a Python string and .decode() must return an +Unicode object. + +First, we have the stateless encoders/decoders. These do not work in +chunks as the stream codecs (see below) do, because all components are +expected to be available in memory. + +class Codec: + + """ Defines the interface for stateless encoders/decoders. + + The .encode()/.decode() methods may implement different error + handling schemes by providing the errors argument. These + string values are defined: + + 'strict' - raise an error (or a subclass) + 'ignore' - ignore the character and continue with the next + 'replace' - replace with a suitable replacement character; + Python will use the official U+FFFD REPLACEMENT + CHARACTER for the builtin Unicode codecs. + + """ + def encode(self,input,errors='strict'): + + """ Encodes the object intput and returns a tuple (output + object, length consumed). + + errors defines the error handling to apply. It defaults to + 'strict' handling. + + The method may not store state in the Codec instance. Use + SteamCodec for codecs which have to keep state in order to + make encoding/decoding efficient. + + """ + ... + + def decode(self,input,errors='strict'): + + """ Decodes the object input and returns a tuple (output + object, length consumed). + + input must be an object which provides the bf_getreadbuf + buffer slot. Python strings, buffer objects and memory + mapped files are examples of objects providing this slot. + + errors defines the error handling to apply. It defaults to + 'strict' handling. + + The method may not store state in the Codec instance. Use + SteamCodec for codecs which have to keep state in order to + make encoding/decoding efficient. + + """ + ... + +StreamWriter and StreamReader define the interface for stateful +encoders/decoders which work on streams. These allow processing of the +data in chunks to efficiently use memory. If you have large strings in +memory, you may want to wrap them with cStringIO objects and then use +these codecs on them to be able to do chunk processing as well, +e.g. to provide progress information to the user. + +class StreamWriter(Codec): + + def __init__(self,stream,errors='strict'): + + """ Creates a StreamWriter instance. + + stream must be a file-like object open for writing + (binary) data. + + The StreamWriter may implement different error handling + schemes by providing the errors keyword argument. These + parameters are defined: + + 'strict' - raise a ValueError (or a subclass) + 'ignore' - ignore the character and continue with the next + 'replace'- replace with a suitable replacement character + + """ + self.stream = stream + self.errors = errors + + def write(self,object): + + """ Writes the object's contents encoded to self.stream. + """ + data, consumed = self.encode(object,self.errors) + self.stream.write(data) + + def reset(self): + + """ Flushes and resets the codec buffers used for keeping state. + + Calling this method should ensure that the data on the + output is put into a clean state, that allows appending + of new fresh data without having to rescan the whole + stream to recover state. + + """ + pass + + def __getattr__(self,name, + + getattr=getattr): + + """ Inherit all other methods from the underlying stream. + """ + return getattr(self.stream,name) + +class StreamReader(Codec): + + def __init__(self,stream,errors='strict'): + + """ Creates a StreamReader instance. + + stream must be a file-like object open for reading + (binary) data. + + The StreamReader may implement different error handling + schemes by providing the errors keyword argument. These + parameters are defined: + + 'strict' - raise a ValueError (or a subclass) + 'ignore' - ignore the character and continue with the next + 'replace'- replace with a suitable replacement character; + + """ + self.stream = stream + self.errors = errors + + def read(self,size=-1): + + """ Decodes data from the stream self.stream and returns the + resulting object. + + size indicates the approximate maximum number of bytes to + read from the stream for decoding purposes. The decoder + can modify this setting as appropriate. The default value + -1 indicates to read and decode as much as possible. size + is intended to prevent having to decode huge files in one + step. + + The method should use a greedy read strategy meaning that + it should read as much data as is allowed within the + definition of the encoding and the given size, e.g. if + optional encoding endings or state markers are available + on the stream, these should be read too. + + """ + # Unsliced reading: + if size < 0: + return self.decode(self.stream.read())[0] + + # Sliced reading: + read = self.stream.read + decode = self.decode + data = read(size) + i = 0 + while 1: + try: + object, decodedbytes = decode(data) + except ValueError,why: + # This method is slow but should work under pretty much + # all conditions; at most 10 tries are made + i = i + 1 + newdata = read(1) + if not newdata or i > 10: + raise + data = data + newdata + else: + return object + + def reset(self): + + """ Resets the codec buffers used for keeping state. + + Note that no stream repositioning should take place. + This method is primarely intended to be able to recover + from decoding errors. + + """ + pass + + def __getattr__(self,name, + + getattr=getattr): + + """ Inherit all other methods from the underlying stream. + """ + return getattr(self.stream,name) + +XXX What about .readline(), .readlines() ? These could be implemented + using .read() as generic functions instead of requiring their + implementation by all codecs. Also see Line Breaks. + +Stream codec implementors are free to combine the StreamWriter and +StreamReader interfaces into one class. Even combining all these with +the Codec class should be possible. + +Implementors are free to add additional methods to enhance the codec +functionality or provide extra state information needed for them to +work. The internal codec implementation will only use the above +interfaces, though. + +It is not required by the Unicode implementation to use these base +classes, only the interfaces must match; this allows writing Codecs as +extensions types. + +As guideline, large mapping tables should be implemented using static +C data in separate (shared) extension modules. That way multiple +processes can share the same data. + +A tool to auto-convert Unicode mapping files to mapping modules should be +provided to simplify support for additional mappings (see References). + + +Whitespace: +----------- + +The .split() method will have to know about what is considered +whitespace in Unicode. + + +Case Conversion: +---------------- + +Case conversion is rather complicated with Unicode data, since there +are many different conditions to respect. See + + http://www.unicode.org/unicode/reports/tr13/ + +for some guidelines on implementing case conversion. + +For Python, we should only implement the 1-1 conversions included in +Unicode. Locale dependent and other special case conversions (see the +Unicode standard file SpecialCasing.txt) should be left to user land +routines and not go into the core interpreter. + +The methods .capitalize() and .iscapitalized() should follow the case +mapping algorithm defined in the above technical report as closely as +possible. + + +Line Breaks: +------------ + +Line breaking should be done for all Unicode characters having the B +property as well as the combinations CRLF, CR, LF (interpreted in that +order) and other special line separators defined by the standard. + +The Unicode type should provide a .splitlines() method which returns a +list of lines according to the above specification. See Unicode +Methods. + + +Unicode Character Properties: +----------------------------- + +A separate module "unicodedata" should provide a compact interface to +all Unicode character properties defined in the standard's +UnicodeData.txt file. + +Among other things, these properties provide ways to recognize +numbers, digits, spaces, whitespace, etc. + +Since this module will have to provide access to all Unicode +characters, it will eventually have to contain the data from +UnicodeData.txt which takes up around 600kB. For this reason, the data +should be stored in static C data. This enables compilation as shared +module which the underlying OS can shared between processes (unlike +normal Python code modules). + +There should be a standard Python interface for accessing this information +so that other implementors can plug in their own possibly enhanced versions, +e.g. ones that do decompressing of the data on-the-fly. + + +Private Code Point Areas: +------------------------- + +Support for these is left to user land Codecs and not explicitly +intergrated into the core. Note that due to the Internal Format being +implemented, only the area between \uE000 and \uF8FF is useable for +private encodings. + + +Internal Format: +---------------- + +The internal format for Unicode objects should use a Python specific +fixed format <PythonUnicode> implemented as 'unsigned short' (or +another unsigned numeric type having 16 bits). Byte order is platform +dependent. + +This format will hold UTF-16 encodings of the corresponding Unicode +ordinals. The Python Unicode implementation will address these values +as if they were UCS-2 values. UCS-2 and UTF-16 are the same for all +currently defined Unicode character points. UTF-16 without surrogates +provides access to about 64k characters and covers all characters in +the Basic Multilingual Plane (BMP) of Unicode. + +It is the Codec's responsibility to ensure that the data they pass to +the Unicode object constructor repects this assumption. The +constructor does not check the data for Unicode compliance or use of +surrogates. + +Future implementations can extend the 32 bit restriction to the full +set of all UTF-16 addressable characters (around 1M characters). + +The Unicode API should provide inteface routines from <PythonUnicode> +to the compiler's wchar_t which can be 16 or 32 bit depending on the +compiler/libc/platform being used. + +Unicode objects should have a pointer to a cached Python string object +<defencstr> holding the object's value using the current <default +encoding>. This is needed for performance and internal parsing (see +Internal Argument Parsing) reasons. The buffer is filled when the +first conversion request to the <default encoding> is issued on the +object. + +Interning is not needed (for now), since Python identifiers are +defined as being ASCII only. + +codecs.BOM should return the byte order mark (BOM) for the format +used internally. The codecs module should provide the following +additional constants for convenience and reference (codecs.BOM will +either be BOM_BE or BOM_LE depending on the platform): + + BOM_BE: '\376\377' + (corresponds to Unicode U+0000FEFF in UTF-16 on big endian + platforms == ZERO WIDTH NO-BREAK SPACE) + + BOM_LE: '\377\376' + (corresponds to Unicode U+0000FFFE in UTF-16 on little endian + platforms == defined as being an illegal Unicode character) + + BOM4_BE: '\000\000\376\377' + (corresponds to Unicode U+0000FEFF in UCS-4) + + BOM4_LE: '\377\376\000\000' + (corresponds to Unicode U+0000FFFE in UCS-4) + +Note that Unicode sees big endian byte order as being "correct". The +swapped order is taken to be an indicator for a "wrong" format, hence +the illegal character definition. + +The configure script should provide aid in deciding whether Python can +use the native wchar_t type or not (it has to be a 16-bit unsigned +type). + + +Buffer Interface: +----------------- + +Implement the buffer interface using the <defencstr> Python string +object as basis for bf_getcharbuf (corresponds to the "t#" argument +parsing marker) and the internal buffer for bf_getreadbuf (corresponds +to the "s#" argument parsing marker). If bf_getcharbuf is requested +and the <defencstr> object does not yet exist, it is created first. + +This has the advantage of being able to write to output streams (which +typically use this interface) without additional specification of the +encoding to use. + +The internal format can also be accessed using the 'unicode-internal' +codec, e.g. via u.encode('unicode-internal'). + + +Pickle/Marshalling: +------------------- + +Should have native Unicode object support. The objects should be +encoded using platform independent encodings. + +Marshal should use UTF-8 and Pickle should either choose +Raw-Unicode-Escape (in text mode) or UTF-8 (in binary mode) as +encoding. Using UTF-8 instead of UTF-16 has the advantage of +eliminating the need to store a BOM mark. + + +Regular Expressions: +-------------------- + +Secret Labs AB is working on a Unicode-aware regular expression +machinery. It works on plain 8-bit, UCS-2, and (optionally) UCS-4 +internal character buffers. + +Also see + + http://www.unicode.org/unicode/reports/tr18/ + +for some remarks on how to treat Unicode REs. + + +Formatting Markers: +------------------- + +Format markers are used in Python format strings. If Python strings +are used as format strings, the following interpretations should be in +effect: + + '%s': '%s' does str(u) for Unicode objects embedded + in Python strings, so the output will be + u.encode(<default encoding>) + +In case the format string is an Unicode object, all parameters are coerced +to Unicode first and then put together and formatted according to the format +string. Numbers are first converted to strings and then to Unicode. + + '%s': Python strings are interpreted as Unicode + string using the <default encoding>. Unicode + objects are taken as is. + +All other string formatters should work accordingly. + +Example: + +u"%s %s" % (u"abc", "abc") == u"abc abc" + + +Internal Argument Parsing: +-------------------------- + +These markers are used by the PyArg_ParseTuple() APIs: + + 'U': Check for Unicode object and return a pointer to it + + 's': For Unicode objects: auto convert them to the <default encoding> + and return a pointer to the object's <defencstr> buffer. + + 's#': Access to the Unicode object via the bf_getreadbuf buffer interface + (see Buffer Interface); note that the length relates to the buffer + length, not the Unicode string length (this may be different + depending on the Internal Format). + + 't#': Access to the Unicode object via the bf_getcharbuf buffer interface + (see Buffer Interface); note that the length relates to the buffer + length, not necessarily to the Unicode string length (this may + be different depending on the <default encoding>). + + +File/Stream Output: +------------------- + +Since file.write(object) and most other stream writers use the "s#" +argument parsing marker for binary files and "t#" for text files, the +buffer interface implementation determines the encoding to use (see +Buffer Interface). + +For explicit handling of files using Unicode, the standard +stream codecs as available through the codecs module should +be used. + +XXX There should be a short-cut open(filename,mode,encoding) available which + also assures that mode contains the 'b' character when needed. + + +File/Stream Input: +------------------ + +Only the user knows what encoding the input data uses, so no special +magic is applied. The user will have to explicitly convert the string +data to Unicode objects as needed or use the file wrappers defined in +the codecs module (see File/Stream Output). + + +Unicode Methods & Attributes: +----------------------------- + +All Python string methods, plus: + + .encode([encoding=<default encoding>][,errors="strict"]) + --> see Unicode Output + + .splitlines([include_breaks=0]) + --> breaks the Unicode string into a list of (Unicode) lines; + returns the lines with line breaks included, if include_breaks + is true. See Line Breaks for a specification of how line breaking + is done. + + +Code Base: +---------- + +We should use Fredrik Lundh's Unicode object implementation as basis. +It already implements most of the string methods needed and provides a +well written code base which we can build upon. + +The object sharing implemented in Fredrik's implementation should +be dropped. + + +Test Cases: +----------- + +Test cases should follow those in Lib/test/test_string.py and include +additional checks for the Codec Registry and the Standard Codecs. + + +References: +----------- + +Unicode Consortium: + http://www.unicode.org/ + +Unicode FAQ: + http://www.unicode.org/unicode/faq/ + +Unicode 3.0: + http://www.unicode.org/unicode/standard/versions/Unicode3.0.html + +Unicode-TechReports: + http://www.unicode.org/unicode/reports/techreports.html + +Unicode-Mappings: + ftp://ftp.unicode.org/Public/MAPPINGS/ + +Introduction to Unicode (a little outdated by still nice to read): + http://www.nada.kth.se/i18n/ucs/unicode-iso10646-oview.html + +Encodings: + + Overview: + http://czyborra.com/utf/ + + UTC-2: + http://www.uazone.com/multiling/unicode/ucs2.html + + UTF-7: + Defined in RFC2152, e.g. + http://www.uazone.com/multiling/ml-docs/rfc2152.txt + + UTF-8: + Defined in RFC2279, e.g. + http://info.internet.isi.edu/in-notes/rfc/files/rfc2279.txt + + UTF-16: + http://www.uazone.com/multiling/unicode/wg2n1035.html + + +History of this Proposal: +------------------------- +1.2: +1.1: Added note about comparisons and hash values. Added note about + case mapping algorithms. Changed stream codecs .read() and + .write() method to match the standard file-like object methods + (bytes consumed information is no longer returned by the methods) +1.0: changed encode Codec method to be symmetric to the decode method + (they both return (object, data consumed) now and thus become + interchangeable); removed __init__ method of Codec class (the + methods are stateless) and moved the errors argument down to the + methods; made the Codec design more generic w/r to type of input + and output objects; changed StreamWriter.flush to StreamWriter.reset + in order to avoid overriding the stream's .flush() method; + renamed .breaklines() to .splitlines(); renamed the module unicodec + to codecs; modified the File I/O section to refer to the stream codecs. +0.9: changed errors keyword argument definition; added 'replace' error + handling; changed the codec APIs to accept buffer like objects on + input; some minor typo fixes; added Whitespace section and + included references for Unicode characters that have the whitespace + and the line break characteristic; added note that search functions + can expect lower-case encoding names; dropped slicing and offsets + in the codec APIs +0.8: added encodings package and raw unicode escape encoding; untabified + the proposal; added notes on Unicode format strings; added + .breaklines() method +0.7: added a whole new set of codec APIs; added a different encoder + lookup scheme; fixed some names +0.6: changed "s#" to "t#"; changed <defencbuf> to <defencstr> holding + a real Python string object; changed Buffer Interface to delegate + requests to <defencstr>'s buffer interface; removed the explicit + reference to the unicodec.codecs dictionary (the module can implement + this in way fit for the purpose); removed the settable default + encoding; move UnicodeError from unicodec to exceptions; "s#" + not returns the internal data; passed the UCS-2/UTF-16 checking + from the Unicode constructor to the Codecs +0.5: moved sys.bom to unicodec.BOM; added sections on case mapping, + private use encodings and Unicode character properties +0.4: added Codec interface, notes on %-formatting, changed some encoding + details, added comments on stream wrappers, fixed some discussion + points (most important: Internal Format), clarified the + 'unicode-escape' encoding, added encoding references +0.3: added references, comments on codec modules, the internal format, + bf_getcharbuffer and the RE engine; added 'unicode-escape' encoding + proposed by Tim Peters and fixed repr(u) accordingly +0.2: integrated Guido's suggestions, added stream codecs and file + wrapping +0.1: first version + + +----------------------------------------------------------------------------- +Written by Marc-Andre Lemburg, 1999-2000, mal@lemburg.com +----------------------------------------------------------------------------- |