diff options
| -rw-r--r-- | Misc/unicode.txt | 1116 |
1 files changed, 3 insertions, 1113 deletions
diff --git a/Misc/unicode.txt b/Misc/unicode.txt index b71e4ca..a252ebe 100644 --- a/Misc/unicode.txt +++ b/Misc/unicode.txt @@ -1,1115 +1,5 @@ -============================================================================= - Python Unicode Integration Proposal Version: 1.7 ------------------------------------------------------------------------------ +This document has been PEP-ified. Please see PEP 100 at: + http://www.python.org/peps/pep-0100.html -Introduction: -------------- - -The idea of this proposal is to add native Unicode 3.0 support to -Python in a way that makes use of Unicode strings as simple as -possible without introducing too many pitfalls along the way. - -Since this goal is not easy to achieve -- strings being one of the -most fundamental objects in Python --, we expect this proposal to -undergo some significant refinements. - -Note that the current version of this proposal is still a bit unsorted -due to the many different aspects of the Unicode-Python integration. - -The latest version of this document is always available at: - - http://starship.python.net/~lemburg/unicode-proposal.txt - -Older versions are available as: - - http://starship.python.net/~lemburg/unicode-proposal-X.X.txt - - -Conventions: ------------- - -· In examples we use u = Unicode object and s = Python string - -· 'XXX' markings indicate points of discussion (PODs) - - -General Remarks: ----------------- - -· Unicode encoding names should be lower case on output and - case-insensitive on input (they will be converted to lower case - by all APIs taking an encoding name as input). - -· Encoding names should follow the name conventions as used by the - Unicode Consortium: spaces are converted to hyphens, e.g. 'utf 16' is - written as 'utf-16'. - -· Codec modules should use the same names, but with hyphens converted - to underscores, e.g. utf_8, utf_16, iso_8859_1. - - -Unicode Default Encoding: -------------------------- - -The Unicode implementation has to make some assumption about the -encoding of 8-bit strings passed to it for coercion and about the -encoding to as default for conversion of Unicode to strings when no -specific encoding is given. This encoding is called <default encoding> -throughout this text. - -For this, the implementation maintains a global which can be set in -the site.py Python startup script. Subsequent changes are not -possible. The <default encoding> can be set and queried using the -two sys module APIs: - - sys.setdefaultencoding(encoding) - --> Sets the <default encoding> used by the Unicode implementation. - encoding has to be an encoding which is supported by the Python - installation, otherwise, a LookupError is raised. - - Note: This API is only available in site.py ! It is removed - from the sys module by site.py after usage. - - sys.getdefaultencoding() - --> Returns the current <default encoding>. - -If not otherwise defined or set, the <default encoding> defaults to -'ascii'. This encoding is also the startup default of Python (and in -effect before site.py is executed). - -Note that the default site.py startup module contains disabled -optional code which can set the <default encoding> according to the -encoding defined by the current locale. The locale module is used to -extract the encoding from the locale default settings defined by the -OS environment (see locale.py). If the encoding cannot be determined, -is unkown or unsupported, the code defaults to setting the <default -encoding> to 'ascii'. To enable this code, edit the site.py file or -place the appropriate code into the sitecustomize.py module of your -Python installation. - - -Unicode Constructors: ---------------------- - -Python should provide a built-in constructor for Unicode strings which -is available through __builtins__: - - u = unicode(encoded_string[,encoding=<default encoding>][,errors="strict"]) - - u = u'<unicode-escape encoded Python string>' - - u = ur'<raw-unicode-escape encoded Python string>' - -With the 'unicode-escape' encoding being defined as: - -· all non-escape characters represent themselves as Unicode ordinal - (e.g. 'a' -> U+0061). - -· all existing defined Python escape sequences are interpreted as - Unicode ordinals; note that \xXXXX can represent all Unicode - ordinals, and \OOO (octal) can represent Unicode ordinals up to U+01FF. - -· a new escape sequence, \uXXXX, represents U+XXXX; it is a syntax - error to have fewer than 4 digits after \u. - -For an explanation of possible values for errors see the Codec section -below. - -Examples: - -u'abc' -> U+0061 U+0062 U+0063 -u'\u1234' -> U+1234 -u'abc\u1234\n' -> U+0061 U+0062 U+0063 U+1234 U+005c - -The 'raw-unicode-escape' encoding is defined as follows: - -· \uXXXX sequence represent the U+XXXX Unicode character if and - only if the number of leading backslashes is odd - -· all other characters represent themselves as Unicode ordinal - (e.g. 'b' -> U+0062) - - -Note that you should provide some hint to the encoding you used to -write your programs as pragma line in one the first few comment lines -of the source file (e.g. '# source file encoding: latin-1'). If you -only use 7-bit ASCII then everything is fine and no such notice is -needed, but if you include Latin-1 characters not defined in ASCII, it -may well be worthwhile including a hint since people in other -countries will want to be able to read your source strings too. - - -Unicode Type Object: --------------------- - -Unicode objects should have the type UnicodeType with type name -'unicode', made available through the standard types module. - - -Unicode Output: ---------------- - -Unicode objects have a method .encode([encoding=<default encoding>]) -which returns a Python string encoding the Unicode string using the -given scheme (see Codecs). - - print u := print u.encode() # using the <default encoding> - - str(u) := u.encode() # using the <default encoding> - - repr(u) := "u%s" % repr(u.encode('unicode-escape')) - -Also see Internal Argument Parsing and Buffer Interface for details on -how other APIs written in C will treat Unicode objects. - - -Unicode Ordinals: ------------------ - -Since Unicode 3.0 has a 32-bit ordinal character set, the implementation -should provide 32-bit aware ordinal conversion APIs: - - ord(u[:1]) (this is the standard ord() extended to work with Unicode - objects) - --> Unicode ordinal number (32-bit) - - unichr(i) - --> Unicode object for character i (provided it is 32-bit); - ValueError otherwise - -Both APIs should go into __builtins__ just like their string -counterparts ord() and chr(). - -Note that Unicode provides space for private encodings. Usage of these -can cause different output representations on different machines. This -problem is not a Python or Unicode problem, but a machine setup and -maintenance one. - - -Comparison & Hash Value: ------------------------- - -Unicode objects should compare equal to other objects after these -other objects have been coerced to Unicode. For strings this means -that they are interpreted as Unicode string using the <default -encoding>. - -Unicode objects should return the same hash value as their ASCII -equivalent strings. Unicode strings holding non-ASCII values are not -guaranteed to return the same hash values as the default encoded -equivalent string representation. - -When compared using cmp() (or PyObject_Compare()) the implementation -should mask TypeErrors raised during the conversion to remain in synch -with the string behavior. All other errors such as ValueErrors raised -during coercion of strings to Unicode should not be masked and passed -through to the user. - -In containment tests ('a' in u'abc' and u'a' in 'abc') both sides -should be coerced to Unicode before applying the test. Errors occurring -during coercion (e.g. None in u'abc') should not be masked. - - -Coercion: ---------- - -Using Python strings and Unicode objects to form new objects should -always coerce to the more precise format, i.e. Unicode objects. - - u + s := u + unicode(s) - - s + u := unicode(s) + u - -All string methods should delegate the call to an equivalent Unicode -object method call by converting all involved strings to Unicode and -then applying the arguments to the Unicode method of the same name, -e.g. - - string.join((s,u),sep) := (s + sep) + u - - sep.join((s,u)) := (s + sep) + u - -For a discussion of %-formatting w/r to Unicode objects, see -Formatting Markers. - - -Exceptions: ------------ - -UnicodeError is defined in the exceptions module as a subclass of -ValueError. It is available at the C level via PyExc_UnicodeError. -All exceptions related to Unicode encoding/decoding should be -subclasses of UnicodeError. - - -Codecs (Coder/Decoders) Lookup: -------------------------------- - -A Codec (see Codec Interface Definition) search registry should be -implemented by a module "codecs": - - codecs.register(search_function) - -Search functions are expected to take one argument, the encoding name -in all lower case letters and with hyphens and spaces converted to -underscores, and return a tuple of functions (encoder, decoder, -stream_reader, stream_writer) taking the following arguments: - - encoder and decoder: - These must be functions or methods which have the same - interface as the .encode/.decode methods of Codec instances - (see Codec Interface). The functions/methods are expected to - work in a stateless mode. - - stream_reader and stream_writer: - These need to be factory functions with the following - interface: - - factory(stream,errors='strict') - - The factory functions must return objects providing - the interfaces defined by StreamWriter/StreamReader resp. - (see Codec Interface). Stream codecs can maintain state. - - Possible values for errors are defined in the Codec - section below. - -In case a search function cannot find a given encoding, it should -return None. - -Aliasing support for encodings is left to the search functions -to implement. - -The codecs module will maintain an encoding cache for performance -reasons. Encodings are first looked up in the cache. If not found, the -list of registered search functions is scanned. If no codecs tuple is -found, a LookupError is raised. Otherwise, the codecs tuple is stored -in the cache and returned to the caller. - -To query the Codec instance the following API should be used: - - codecs.lookup(encoding) - -This will either return the found codecs tuple or raise a LookupError. - - -Standard Codecs: ----------------- - -Standard codecs should live inside an encodings/ package directory in the -Standard Python Code Library. The __init__.py file of that directory should -include a Codec Lookup compatible search function implementing a lazy module -based codec lookup. - -Python should provide a few standard codecs for the most relevant -encodings, e.g. - - 'utf-8': 8-bit variable length encoding - 'utf-16': 16-bit variable length encoding (little/big endian) - 'utf-16-le': utf-16 but explicitly little endian - 'utf-16-be': utf-16 but explicitly big endian - 'ascii': 7-bit ASCII codepage - 'iso-8859-1': ISO 8859-1 (Latin 1) codepage - 'unicode-escape': See Unicode Constructors for a definition - 'raw-unicode-escape': See Unicode Constructors for a definition - 'native': Dump of the Internal Format used by Python - -Common aliases should also be provided per default, e.g. 'latin-1' -for 'iso-8859-1'. - -Note: 'utf-16' should be implemented by using and requiring byte order -marks (BOM) for file input/output. - -All other encodings such as the CJK ones to support Asian scripts -should be implemented in separate packages which do not get included -in the core Python distribution and are not a part of this proposal. - - -Codecs Interface Definition: ----------------------------- - -The following base class should be defined in the module -"codecs". They provide not only templates for use by encoding module -implementors, but also define the interface which is expected by the -Unicode implementation. - -Note that the Codec Interface defined here is well suitable for a -larger range of applications. The Unicode implementation expects -Unicode objects on input for .encode() and .write() and character -buffer compatible objects on input for .decode(). Output of .encode() -and .read() should be a Python string and .decode() must return an -Unicode object. - -First, we have the stateless encoders/decoders. These do not work in -chunks as the stream codecs (see below) do, because all components are -expected to be available in memory. - -class Codec: - - """ Defines the interface for stateless encoders/decoders. - - The .encode()/.decode() methods may implement different error - handling schemes by providing the errors argument. These - string values are defined: - - 'strict' - raise an error (or a subclass) - 'ignore' - ignore the character and continue with the next - 'replace' - replace with a suitable replacement character; - Python will use the official U+FFFD REPLACEMENT - CHARACTER for the builtin Unicode codecs. - - """ - def encode(self,input,errors='strict'): - - """ Encodes the object input and returns a tuple (output - object, length consumed). - - errors defines the error handling to apply. It defaults to - 'strict' handling. - - The method may not store state in the Codec instance. Use - StreamCodec for codecs which have to keep state in order to - make encoding/decoding efficient. - - """ - ... - - def decode(self,input,errors='strict'): - - """ Decodes the object input and returns a tuple (output - object, length consumed). - - input must be an object which provides the bf_getreadbuf - buffer slot. Python strings, buffer objects and memory - mapped files are examples of objects providing this slot. - - errors defines the error handling to apply. It defaults to - 'strict' handling. - - The method may not store state in the Codec instance. Use - StreamCodec for codecs which have to keep state in order to - make encoding/decoding efficient. - - """ - ... - -StreamWriter and StreamReader define the interface for stateful -encoders/decoders which work on streams. These allow processing of the -data in chunks to efficiently use memory. If you have large strings in -memory, you may want to wrap them with cStringIO objects and then use -these codecs on them to be able to do chunk processing as well, -e.g. to provide progress information to the user. - -class StreamWriter(Codec): - - def __init__(self,stream,errors='strict'): - - """ Creates a StreamWriter instance. - - stream must be a file-like object open for writing - (binary) data. - - The StreamWriter may implement different error handling - schemes by providing the errors keyword argument. These - parameters are defined: - - 'strict' - raise a ValueError (or a subclass) - 'ignore' - ignore the character and continue with the next - 'replace'- replace with a suitable replacement character - - """ - self.stream = stream - self.errors = errors - - def write(self,object): - - """ Writes the object's contents encoded to self.stream. - """ - data, consumed = self.encode(object,self.errors) - self.stream.write(data) - - def writelines(self, list): - - """ Writes the concatenated list of strings to the stream - using .write(). - """ - self.write(''.join(list)) - - def reset(self): - - """ Flushes and resets the codec buffers used for keeping state. - - Calling this method should ensure that the data on the - output is put into a clean state, that allows appending - of new fresh data without having to rescan the whole - stream to recover state. - - """ - pass - - def __getattr__(self,name, - - getattr=getattr): - - """ Inherit all other methods from the underlying stream. - """ - return getattr(self.stream,name) - -class StreamReader(Codec): - - def __init__(self,stream,errors='strict'): - - """ Creates a StreamReader instance. - - stream must be a file-like object open for reading - (binary) data. - - The StreamReader may implement different error handling - schemes by providing the errors keyword argument. These - parameters are defined: - - 'strict' - raise a ValueError (or a subclass) - 'ignore' - ignore the character and continue with the next - 'replace'- replace with a suitable replacement character; - - """ - self.stream = stream - self.errors = errors - - def read(self,size=-1): - - """ Decodes data from the stream self.stream and returns the - resulting object. - - size indicates the approximate maximum number of bytes to - read from the stream for decoding purposes. The decoder - can modify this setting as appropriate. The default value - -1 indicates to read and decode as much as possible. size - is intended to prevent having to decode huge files in one - step. - - The method should use a greedy read strategy meaning that - it should read as much data as is allowed within the - definition of the encoding and the given size, e.g. if - optional encoding endings or state markers are available - on the stream, these should be read too. - - """ - # Unsliced reading: - if size < 0: - return self.decode(self.stream.read())[0] - - # Sliced reading: - read = self.stream.read - decode = self.decode - data = read(size) - i = 0 - while 1: - try: - object, decodedbytes = decode(data) - except ValueError,why: - # This method is slow but should work under pretty much - # all conditions; at most 10 tries are made - i = i + 1 - newdata = read(1) - if not newdata or i > 10: - raise - data = data + newdata - else: - return object - - def readline(self, size=None): - - """ Read one line from the input stream and return the - decoded data. - - Note: Unlike the .readlines() method, this method inherits - the line breaking knowledge from the underlying stream's - .readline() method -- there is currently no support for - line breaking using the codec decoder due to lack of line - buffering. Subclasses should however, if possible, try to - implement this method using their own knowledge of line - breaking. - - size, if given, is passed as size argument to the stream's - .readline() method. - - """ - if size is None: - line = self.stream.readline() - else: - line = self.stream.readline(size) - return self.decode(line)[0] - - def readlines(self, sizehint=0): - - """ Read all lines available on the input stream - and return them as list of lines. - - Line breaks are implemented using the codec's decoder - method and are included in the list entries. - - sizehint, if given, is passed as size argument to the - stream's .read() method. - - """ - if sizehint is None: - data = self.stream.read() - else: - data = self.stream.read(sizehint) - return self.decode(data)[0].splitlines(1) - - def reset(self): - - """ Resets the codec buffers used for keeping state. - - Note that no stream repositioning should take place. - This method is primarily intended to be able to recover - from decoding errors. - - """ - pass - - def __getattr__(self,name, - - getattr=getattr): - - """ Inherit all other methods from the underlying stream. - """ - return getattr(self.stream,name) - - -Stream codec implementors are free to combine the StreamWriter and -StreamReader interfaces into one class. Even combining all these with -the Codec class should be possible. - -Implementors are free to add additional methods to enhance the codec -functionality or provide extra state information needed for them to -work. The internal codec implementation will only use the above -interfaces, though. - -It is not required by the Unicode implementation to use these base -classes, only the interfaces must match; this allows writing Codecs as -extension types. - -As guideline, large mapping tables should be implemented using static -C data in separate (shared) extension modules. That way multiple -processes can share the same data. - -A tool to auto-convert Unicode mapping files to mapping modules should be -provided to simplify support for additional mappings (see References). - - -Whitespace: ------------ - -The .split() method will have to know about what is considered -whitespace in Unicode. - - -Case Conversion: ----------------- - -Case conversion is rather complicated with Unicode data, since there -are many different conditions to respect. See - - http://www.unicode.org/unicode/reports/tr13/ - -for some guidelines on implementing case conversion. - -For Python, we should only implement the 1-1 conversions included in -Unicode. Locale dependent and other special case conversions (see the -Unicode standard file SpecialCasing.txt) should be left to user land -routines and not go into the core interpreter. - -The methods .capitalize() and .iscapitalized() should follow the case -mapping algorithm defined in the above technical report as closely as -possible. - - -Line Breaks: ------------- - -Line breaking should be done for all Unicode characters having the B -property as well as the combinations CRLF, CR, LF (interpreted in that -order) and other special line separators defined by the standard. - -The Unicode type should provide a .splitlines() method which returns a -list of lines according to the above specification. See Unicode -Methods. - - -Unicode Character Properties: ------------------------------ - -A separate module "unicodedata" should provide a compact interface to -all Unicode character properties defined in the standard's -UnicodeData.txt file. - -Among other things, these properties provide ways to recognize -numbers, digits, spaces, whitespace, etc. - -Since this module will have to provide access to all Unicode -characters, it will eventually have to contain the data from -UnicodeData.txt which takes up around 600kB. For this reason, the data -should be stored in static C data. This enables compilation as shared -module which the underlying OS can shared between processes (unlike -normal Python code modules). - -There should be a standard Python interface for accessing this information -so that other implementors can plug in their own possibly enhanced versions, -e.g. ones that do decompressing of the data on-the-fly. - - -Private Code Point Areas: -------------------------- - -Support for these is left to user land Codecs and not explicitly -integrated into the core. Note that due to the Internal Format being -implemented, only the area between \uE000 and \uF8FF is usable for -private encodings. - - -Internal Format: ----------------- - -The internal format for Unicode objects should use a Python specific -fixed format <PythonUnicode> implemented as 'unsigned short' (or -another unsigned numeric type having 16 bits). Byte order is platform -dependent. - -This format will hold UTF-16 encodings of the corresponding Unicode -ordinals. The Python Unicode implementation will address these values -as if they were UCS-2 values. UCS-2 and UTF-16 are the same for all -currently defined Unicode character points. UTF-16 without surrogates -provides access to about 64k characters and covers all characters in -the Basic Multilingual Plane (BMP) of Unicode. - -It is the Codec's responsibility to ensure that the data they pass to -the Unicode object constructor respects this assumption. The -constructor does not check the data for Unicode compliance or use of -surrogates. - -Future implementations can extend the 32 bit restriction to the full -set of all UTF-16 addressable characters (around 1M characters). - -The Unicode API should provide interface routines from <PythonUnicode> -to the compiler's wchar_t which can be 16 or 32 bit depending on the -compiler/libc/platform being used. - -Unicode objects should have a pointer to a cached Python string object -<defenc> holding the object's value using the <default encoding>. -This is needed for performance and internal parsing (see Internal -Argument Parsing) reasons. The buffer is filled when the first -conversion request to the <default encoding> is issued on the object. - -Interning is not needed (for now), since Python identifiers are -defined as being ASCII only. - -codecs.BOM should return the byte order mark (BOM) for the format -used internally. The codecs module should provide the following -additional constants for convenience and reference (codecs.BOM will -either be BOM_BE or BOM_LE depending on the platform): - - BOM_BE: '\376\377' - (corresponds to Unicode U+0000FEFF in UTF-16 on big endian - platforms == ZERO WIDTH NO-BREAK SPACE) - - BOM_LE: '\377\376' - (corresponds to Unicode U+0000FFFE in UTF-16 on little endian - platforms == defined as being an illegal Unicode character) - - BOM4_BE: '\000\000\376\377' - (corresponds to Unicode U+0000FEFF in UCS-4) - - BOM4_LE: '\377\376\000\000' - (corresponds to Unicode U+0000FFFE in UCS-4) - -Note that Unicode sees big endian byte order as being "correct". The -swapped order is taken to be an indicator for a "wrong" format, hence -the illegal character definition. - -The configure script should provide aid in deciding whether Python can -use the native wchar_t type or not (it has to be a 16-bit unsigned -type). - - -Buffer Interface: ------------------ - -Implement the buffer interface using the <defenc> Python string object -as basis for bf_getcharbuf and the internal buffer for -bf_getreadbuf. If bf_getcharbuf is requested and the <defenc> object -does not yet exist, it is created first. - -Note that as special case, the parser marker "s#" will not return raw -Unicode UTF-16 data (which the bf_getreadbuf returns), but instead -tries to encode the Unicode object using the default encoding and then -returns a pointer to the resulting string object (or raises an -exception in case the conversion fails). This was done in order to -prevent accidentely writing binary data to an output stream which the -other end might not recognize. - -This has the advantage of being able to write to output streams (which -typically use this interface) without additional specification of the -encoding to use. - -If you need to access the read buffer interface of Unicode objects, -use the PyObject_AsReadBuffer() interface. - -The internal format can also be accessed using the 'unicode-internal' -codec, e.g. via u.encode('unicode-internal'). - - -Pickle/Marshalling: -------------------- - -Should have native Unicode object support. The objects should be -encoded using platform independent encodings. - -Marshal should use UTF-8 and Pickle should either choose -Raw-Unicode-Escape (in text mode) or UTF-8 (in binary mode) as -encoding. Using UTF-8 instead of UTF-16 has the advantage of -eliminating the need to store a BOM mark. - - -Regular Expressions: --------------------- - -Secret Labs AB is working on a Unicode-aware regular expression -machinery. It works on plain 8-bit, UCS-2, and (optionally) UCS-4 -internal character buffers. - -Also see - - http://www.unicode.org/unicode/reports/tr18/ - -for some remarks on how to treat Unicode REs. - - -Formatting Markers: -------------------- - -Format markers are used in Python format strings. If Python strings -are used as format strings, the following interpretations should be in -effect: - - '%s': For Unicode objects this will cause coercion of the - whole format string to Unicode. Note that - you should use a Unicode format string to start - with for performance reasons. - -In case the format string is an Unicode object, all parameters are coerced -to Unicode first and then put together and formatted according to the format -string. Numbers are first converted to strings and then to Unicode. - - '%s': Python strings are interpreted as Unicode - string using the <default encoding>. Unicode - objects are taken as is. - -All other string formatters should work accordingly. - -Example: - -u"%s %s" % (u"abc", "abc") == u"abc abc" - - -Internal Argument Parsing: --------------------------- - -These markers are used by the PyArg_ParseTuple() APIs: - - "U": Check for Unicode object and return a pointer to it - - "s": For Unicode objects: return a pointer to the object's - <defenc> buffer (which uses the <default encoding>). - - "s#": Access to the default encoded version of the Unicode object - (see Buffer Interface); note that the length relates to the length - of the default encoded string rather than the Unicode object length. - - "t#": Same as "s#". - - "es": - Takes two parameters: encoding (const char *) and - buffer (char **). - - The input object is first coerced to Unicode in the usual way - and then encoded into a string using the given encoding. - - On output, a buffer of the needed size is allocated and - returned through *buffer as NULL-terminated string. - The encoded may not contain embedded NULL characters. - The caller is responsible for calling PyMem_Free() - to free the allocated *buffer after usage. - - "es#": - Takes three parameters: encoding (const char *), - buffer (char **) and buffer_len (int *). - - The input object is first coerced to Unicode in the usual way - and then encoded into a string using the given encoding. - - If *buffer is non-NULL, *buffer_len must be set to sizeof(buffer) - on input. Output is then copied to *buffer. - - If *buffer is NULL, a buffer of the needed size is - allocated and output copied into it. *buffer is then - updated to point to the allocated memory area. - The caller is responsible for calling PyMem_Free() - to free the allocated *buffer after usage. - - In both cases *buffer_len is updated to the number of - characters written (excluding the trailing NULL-byte). - The output buffer is assured to be NULL-terminated. - -Examples: - -Using "es#" with auto-allocation: - - static PyObject * - test_parser(PyObject *self, - PyObject *args) - { - PyObject *str; - const char *encoding = "latin-1"; - char *buffer = NULL; - int buffer_len = 0; - - if (!PyArg_ParseTuple(args, "es#:test_parser", - encoding, &buffer, &buffer_len)) - return NULL; - if (!buffer) { - PyErr_SetString(PyExc_SystemError, - "buffer is NULL"); - return NULL; - } - str = PyString_FromStringAndSize(buffer, buffer_len); - PyMem_Free(buffer); - return str; - } - -Using "es" with auto-allocation returning a NULL-terminated string: - - static PyObject * - test_parser(PyObject *self, - PyObject *args) - { - PyObject *str; - const char *encoding = "latin-1"; - char *buffer = NULL; - - if (!PyArg_ParseTuple(args, "es:test_parser", - encoding, &buffer)) - return NULL; - if (!buffer) { - PyErr_SetString(PyExc_SystemError, - "buffer is NULL"); - return NULL; - } - str = PyString_FromString(buffer); - PyMem_Free(buffer); - return str; - } - -Using "es#" with a pre-allocated buffer: - - static PyObject * - test_parser(PyObject *self, - PyObject *args) - { - PyObject *str; - const char *encoding = "latin-1"; - char _buffer[10]; - char *buffer = _buffer; - int buffer_len = sizeof(_buffer); - - if (!PyArg_ParseTuple(args, "es#:test_parser", - encoding, &buffer, &buffer_len)) - return NULL; - if (!buffer) { - PyErr_SetString(PyExc_SystemError, - "buffer is NULL"); - return NULL; - } - str = PyString_FromStringAndSize(buffer, buffer_len); - return str; - } - - -File/Stream Output: -------------------- - -Since file.write(object) and most other stream writers use the "s#" or -"t#" argument parsing marker for querying the data to write, the -default encoded string version of the Unicode object will be written -to the streams (see Buffer Interface). - -For explicit handling of files using Unicode, the standard stream -codecs as available through the codecs module should be used. - -The codecs module should provide a short-cut open(filename,mode,encoding) -available which also assures that mode contains the 'b' character when -needed. - - -File/Stream Input: ------------------- - -Only the user knows what encoding the input data uses, so no special -magic is applied. The user will have to explicitly convert the string -data to Unicode objects as needed or use the file wrappers defined in -the codecs module (see File/Stream Output). - - -Unicode Methods & Attributes: ------------------------------ - -All Python string methods, plus: - - .encode([encoding=<default encoding>][,errors="strict"]) - --> see Unicode Output - - .splitlines([include_breaks=0]) - --> breaks the Unicode string into a list of (Unicode) lines; - returns the lines with line breaks included, if include_breaks - is true. See Line Breaks for a specification of how line breaking - is done. - - -Code Base: ----------- - -We should use Fredrik Lundh's Unicode object implementation as basis. -It already implements most of the string methods needed and provides a -well written code base which we can build upon. - -The object sharing implemented in Fredrik's implementation should -be dropped. - - -Test Cases: ------------ - -Test cases should follow those in Lib/test/test_string.py and include -additional checks for the Codec Registry and the Standard Codecs. - - -References: ------------ - -Unicode Consortium: - http://www.unicode.org/ - -Unicode FAQ: - http://www.unicode.org/unicode/faq/ - -Unicode 3.0: - http://www.unicode.org/unicode/standard/versions/Unicode3.0.html - -Unicode-TechReports: - http://www.unicode.org/unicode/reports/techreports.html - -Unicode-Mappings: - ftp://ftp.unicode.org/Public/MAPPINGS/ - -Introduction to Unicode (a little outdated by still nice to read): - http://www.nada.kth.se/i18n/ucs/unicode-iso10646-oview.html - -For comparison: - Introducing Unicode to ECMAScript (aka JavaScript) -- - http://www-4.ibm.com/software/developer/library/internationalization-support.html - -IANA Character Set Names: - ftp://ftp.isi.edu/in-notes/iana/assignments/character-sets - -Discussion of UTF-8 and Unicode support for POSIX and Linux: - http://www.cl.cam.ac.uk/~mgk25/unicode.html - -Encodings: - - Overview: - http://czyborra.com/utf/ - - UTC-2: - http://www.uazone.com/multiling/unicode/ucs2.html - - UTF-7: - Defined in RFC2152, e.g. - http://www.uazone.com/multiling/ml-docs/rfc2152.txt - - UTF-8: - Defined in RFC2279, e.g. - http://info.internet.isi.edu/in-notes/rfc/files/rfc2279.txt - - UTF-16: - http://www.uazone.com/multiling/unicode/wg2n1035.html - - -History of this Proposal: -------------------------- -1.7: Added note about the changed behaviour of "s#". -1.6: Changed <defencstr> to <defenc> since this is the name used in the - implementation. Added notes about the usage of <defenc> in the - buffer protocol implementation. -1.5: Added notes about setting the <default encoding>. Fixed some - typos (thanks to Andrew Kuchling). Changed <defencstr> to <utf8str>. -1.4: Added note about mixed type comparisons and contains tests. - Changed treating of Unicode objects in format strings (if used - with '%s' % u they will now cause the format string to be - coerced to Unicode, thus producing a Unicode object on return). - Added link to IANA charset names (thanks to Lars Marius Garshol). - Added new codec methods .readline(), .readlines() and .writelines(). -1.3: Added new "es" and "es#" parser markers -1.2: Removed POD about codecs.open() -1.1: Added note about comparisons and hash values. Added note about - case mapping algorithms. Changed stream codecs .read() and - .write() method to match the standard file-like object methods - (bytes consumed information is no longer returned by the methods) -1.0: changed encode Codec method to be symmetric to the decode method - (they both return (object, data consumed) now and thus become - interchangeable); removed __init__ method of Codec class (the - methods are stateless) and moved the errors argument down to the - methods; made the Codec design more generic w/r to type of input - and output objects; changed StreamWriter.flush to StreamWriter.reset - in order to avoid overriding the stream's .flush() method; - renamed .breaklines() to .splitlines(); renamed the module unicodec - to codecs; modified the File I/O section to refer to the stream codecs. -0.9: changed errors keyword argument definition; added 'replace' error - handling; changed the codec APIs to accept buffer like objects on - input; some minor typo fixes; added Whitespace section and - included references for Unicode characters that have the whitespace - and the line break characteristic; added note that search functions - can expect lower-case encoding names; dropped slicing and offsets - in the codec APIs -0.8: added encodings package and raw unicode escape encoding; untabified - the proposal; added notes on Unicode format strings; added - .breaklines() method -0.7: added a whole new set of codec APIs; added a different encoder - lookup scheme; fixed some names -0.6: changed "s#" to "t#"; changed <defencbuf> to <defencstr> holding - a real Python string object; changed Buffer Interface to delegate - requests to <defencstr>'s buffer interface; removed the explicit - reference to the unicodec.codecs dictionary (the module can implement - this in way fit for the purpose); removed the settable default - encoding; move UnicodeError from unicodec to exceptions; "s#" - not returns the internal data; passed the UCS-2/UTF-16 checking - from the Unicode constructor to the Codecs -0.5: moved sys.bom to unicodec.BOM; added sections on case mapping, - private use encodings and Unicode character properties -0.4: added Codec interface, notes on %-formatting, changed some encoding - details, added comments on stream wrappers, fixed some discussion - points (most important: Internal Format), clarified the - 'unicode-escape' encoding, added encoding references -0.3: added references, comments on codec modules, the internal format, - bf_getcharbuffer and the RE engine; added 'unicode-escape' encoding - proposed by Tim Peters and fixed repr(u) accordingly -0.2: integrated Guido's suggestions, added stream codecs and file - wrapping -0.1: first version - - ------------------------------------------------------------------------------ -Written by Marc-Andre Lemburg, 1999-2000, mal@lemburg.com ------------------------------------------------------------------------------ +-Barry |