summaryrefslogtreecommitdiffstats
path: root/Objects/unicodeobject.c
Commit message (Collapse)AuthorAgeFilesLines
* Merge of descr-branch back into trunk.Tim Peters2001-08-021-9/+45
|
* Add _PyUnicode_AsDefaultEncodedString to unicodeobject.h.Jeremy Hylton2001-07-301-14/+0
| | | | | | | And remove all the extern decls in the middle of .c files. Apparently, it was excluded from the header file because it is intended for internal use by the interpreter. It's still intended for internal use and documented as such in the header file.
* Fix for bug #444493: u'\U00010001' segfaults with current CVS onMarc-André Lemburg2001-07-251-6/+21
| | | | wide builds.
* Make the unicode-escape and the UTF-16 codecs handle surrogatesMarc-André Lemburg2001-07-201-24/+46
| | | | | | | | correctly and thus roundtrip-safe. Some minor cleanups of the code. Added tests for the roundtrip-safety.
* #ifdef out generation of \U escapes unless Py_UNICODE_WIDE. ThisGuido van Rossum2001-07-201-0/+2
| | | | | | | | | | #caused warnings with the VMS C compiler. (SF bug #442998, in part.) On a narrow system the current code should never be executed since ch will always be < 0x10000. Marc-Andre: you may end up fixing this a different way, since I believe you have plans to generate \U for surrogate pairs. I'll leave that to you.
* use Py_UNICODE_WIDE instead of USE_UCS4_STORAGE and Py_UNICODE_SIZEFredrik Lundh2001-06-271-4/+4
| | | | tests.
* Encode surrogates in UTF-8 even for a wide Py_UNICODE.Martin v. Löwis2001-06-271-7/+12
| | | | | | | Implement sys.maxunicode. Explicitly wrap around upper/lower computations for wide Py_UNICODE. When decoding large characters with UTF-8, represent expected test results using the \U notation.
* When decoding UTF-16, don't assume that the buffer is in native endiannessMartin v. Löwis2001-06-261-4/+4
| | | | when checking surrogates.
* Support using UCS-4 as the Py_UNICODE type:Martin v. Löwis2001-06-261-30/+89
| | | | | | | | | | Add configure option --enable-unicode. Add config.h macros Py_USING_UNICODE, PY_UNICODE_TYPE, Py_UNICODE_SIZE, SIZEOF_WCHAR_T. Define Py_UCS2. Encode and decode large UTF-8 characters into single Py_UNICODE values for wide Unicode types; likewise for UTF-16. Remove test whether sizeof Py_UNICODE is two.
* experimental UCS-4 support: added USE_UCS4_STORAGE define toFredrik Lundh2001-06-261-0/+2
| | | | | | unicodeobject.h, which forces sizeof(Py_UNICODE) == sizeof(Py_UCS4). (this may be good enough for platforms that doesn't have a 16-bit type. the UTF-16 codecs don't work, though)
* experimental UCS-4 support: made compare a bit more robust, in caseFredrik Lundh2001-06-261-11/+14
| | | | | sizeof(Py_UNICODE) >= sizeof(long). also changed surrogate expansion to work if sizeof(Py_UNICODE) > 2.
* experimental UCS-4 support: don't assume that MS_WIN32 impliesFredrik Lundh2001-06-261-2/+2
| | | | HAVE_USABLE_WCHAR_T
* Fix a mis-indentation in _PyUnicode_New() that caused me to stare atGuido van Rossum2001-06-141-3/+3
| | | | some code for longer than needed.
* Fixes [ #430986 ] Buglet in PyUnicode_FromUnicode.Marc-André Lemburg2001-06-071-1/+1
|
* fix bogus indentationJeremy Hylton2001-05-291-1/+1
|
* This patch changes the behaviour of the UTF-16 codec family. Only theMarc-André Lemburg2001-05-211-17/+25
| | | | | | | | | UTF-16 codec will now interpret and remove a *leading* BOM mark. Sub- sequent BOM characters are no longer interpreted and removed. UTF-16-LE and -BE pass through all BOM mark characters. These changes should get the UTF-16 codec more in line with what the Unicode FAQ recommends w/r to BOM marks.
* Remove unused variableJeremy Hylton2001-05-081-1/+0
|
* Make unicode.join() work nice with iterators. This also required a changeTim Peters2001-05-051-11/+15
| | | | | | | | to string.join(), so that when the latter figures out in midstream that it really needs unicode.join() instead, unicode.join() can actually get all the sequence elements (i.e., there's no guarantee that the sequence passed to string.join() can be iterated over *again* by unicode.join(), so string.join() must not pass on the original sequence object anymore).
* A different approach to the problem reported inTim Peters2001-04-281-4/+15
| | | | | | | | | | | Patch #419651: Metrowerks on Mac adds 0x itself C std says %#x and %#X conversion of 0 do not add the 0x/0X base marker. Metrowerks apparently does. Mark Favas reported the same bug under a Compaq compiler on Tru64 Unix, but no other libc broken in this respect is known (known to be OK under MSVC and gcc). So just try the damn thing at runtime and see what the platform does. Note that we've always had bugs here, but never knew it before because a relevant test case didn't exist before 2.1.
* This patch originated from an idea by Martin v. Loewis who submitted aMarc-André Lemburg2001-04-231-51/+133
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | patch for sharing single character Unicode objects. Martin's patch had to be reworked in a number of ways to take Unicode resizing into consideration as well. Here's what the updated patch implements: * Single character Unicode strings in the Latin-1 range are shared (not only ASCII chars as in Martin's original patch). * The ASCII and Latin-1 codecs make use of this optimization, providing a noticable speedup for single character strings. Most Unicode methods can use the optimization as well (by virtue of using PyUnicode_FromUnicode()). * Some code cleanup was done (replacing memcpy with Py_UNICODE_COPY) * The PyUnicode_Resize() can now also handle the case of resizing unicode_empty which previously resulted in an error. * Modified the internal API _PyUnicode_Resize() and the public PyUnicode_Resize() API to handle references to shared objects correctly. The _PyUnicode_Resize() signature changed due to this. * Callers of PyUnicode_FromUnicode() may now only modify the Unicode object contents of the returned object in case they called the API with NULL as content template. Note that even though this patch passes the regression tests, there may still be subtle bugs in the sharing code.
* SF but #417587: compiler warnings compiling 2.1.Tim Peters2001-04-211-3/+0
| | | | Repaired *some* of the SGI compiler warnings Sjoerd Mullender reported.
* CVS patch 416248: 2.1c1 unicodeobject: unused vrbl cleanup, from Mark Favas.Tim Peters2001-04-191-2/+0
|
* Revert previous checkin, which caused test_unicodedata to fail.Jeremy Hylton2001-04-191-33/+0
|
* Patch #416953: Cache ASCII characters to speed up ASCII decoding.Martin v. Löwis2001-04-181-0/+33
|
* Bug 415514 reported that e.g.Tim Peters2001-04-121-13/+19
| | | | | | | | | | | | "%#x" % 0 blew up, at heart because C sprintf supplies a base marker if and only if the value is not 0. I then fixed that, by tolerating C's inconsistency when it does %#x, and taking away that *Python* produced 0x0 when formatting 0L (the "long" flavor of 0) under %#x itself. But after talking with Guido, we agreed it would be better to supply 0x for the short int case too, despite that it's inconsistent with C, because C is inconsistent with itself and with Python's hex(0) (plus, while "%#x" % 0 didn't work before, "%#x" % 0L *did*, and returned "0x0"). Similarly for %#X conversion.
* Fix for SF bug #415514: "%#x" % 0 caused assertion failure/abort.Tim Peters2001-04-121-11/+12
| | | | | | | | | | | | | http://sourceforge.net/tracker/index.php?func=detail&aid=415514&group_id=5470&atid=105470 For short ints, Python defers to the platform C library to figure out what %#x should do. The code asserted that the platform C returned a string beginning with "0x". However, that's not true when-- and only when --the *value* being formatted is 0. Changed the code to live with C's inconsistency here. In the meantime, the problem does not arise if you format a long 0 (0L) instead. However, that's because the code *we* wrote to do %#x conversions on longs produces a leading "0x" regardless of value. That's probably wrong too: we should drop leading "0x", for consistency with C, when (& only when) formatting 0L. So I changed the long formatting code to do that too.
* reorganized PyUnicode_DecodeUnicodeEscape a bit (in order to make itFredrik Lundh2001-02-181-110/+69
| | | | less likely that bug #132817 ever appears again)
* Fixed .capitalize() method of Unicode objects to work like theMarc-André Lemburg2001-01-291-4/+18
| | | | | | corresponding string method. Added tests for this too. Patch written by Marc-Andre Lemburg. Copyright assigned to Guido van Rossum.
* Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings.Ka-Ping Yee2001-01-241-6/+19
| | | | Switch from octal escapes to hex escapes for other nonprintable characters.
* Move uchhash functionality into unicodedata (after the recentFredrik Lundh2001-01-241-9/+11
| | | | | crop of changes, the files are small enough to do this). Also adds "name" and "lookup" functions to unicodedata.
* Better error message if ucnhash cannot be found (obscure attributeFredrik Lundh2001-01-201-3/+8
| | | | | | errors aren't that helpful), or doesn't contain what's expected from it. Also tweaked the test script so it compiles even if ucnhash is missing.
* refactored the unicodeobject/ucnhash interface, to hide theFredrik Lundh2001-01-191-103/+39
| | | | | | | implementation details inside the ucnhash module. also cleaned up the unicode copyright blurb a little; Secret Labs' internal revision history isn't that interesting...
* This patch adds a new builtin unistr() which behaves like str()Marc-André Lemburg2001-01-171-0/+1
| | | | | | | | | | except that it always returns Unicode objects. A new C API PyObject_Unicode() is also provided. This closes patch #101664. Written by Marc-Andre Lemburg. Copyright assigned to Guido van Rossum.
* Added checks to prevent PyUnicode_Count() from dumping coreMarc-André Lemburg2001-01-161-8/+19
| | | | | | | | | | | | in case the parameters are out of bounds and fixes error handling for .count(), .startswith() and .endswith() for the case of mixed string/Unicode objects. This patch adds Python style index semantics to PyUnicode_Count() indices (including the special handling of negative indices). The patch is an extended version of patch #103249 submitted by Michael Hudson (mwh) on SF. It also includes new test cases.
* This patch adds a new feature to the builtin charmap codec:Marc-André Lemburg2001-01-061-8/+48
| | | | | | | | | | | | | | | The mapping dictionaries can now contain 1-n mappings, meaning that character ordinals may be mapped to strings or Unicode object, e.g. 0x0078 ('x') -> u"abc", causing the ordinal to be replaced by the complete string or Unicode object instead of just one character. Another feature introduced by the patch is that of mapping oridnals to the emtpy string. This allows removing characters. The patch is different from patch #103100 in that it does not cause a performance hit for the normal use case of 1-1 mappings. Written by Marc-Andre Lemburg, copyright assigned to Guido van Rossum.
* This patch changes the default behaviour of the builtin charmapMarc-André Lemburg2001-01-031-13/+8
| | | | | | | | | | | | | | | | codec to not apply Latin-1 mappings for keys which are not found in the mapping dictionaries, but instead treat them as undefined mappings. The patch was originally written by Martin v. Loewis with some additional (cosmetic) changes and an updated test script by Marc-Andre Lemburg. The standard codecs were recreated from the most current files available at the Unicode.org site using the Tools/scripts/gencodec.py tool. This patch closes the bugs #116285 and #119960.
* Patch #102940: use only printable Unicode chars in reportingAndrew M. Kuchling2000-12-191-1/+2
| | | | | incorrect % characters; characters outside the printable range are replaced with '?'
* Fix off-by-one error in split_substring(). Fixes SF bug #122162.Guido van Rossum2000-12-191-1/+1
|
* [ Patch #102852 ] Make % error a bit more informative by indicates theAndrew M. Kuchling2000-12-151-2/+3
| | | | index at which an unknown %-escape was found
* Fox for SF bug #123859: %[duxXo] long formats inconsistent.Tim Peters2000-11-301-3/+1
|
* _PyUnicode_Fini(): Initialize the local freelist walking variable `u'Barry Warsaw2000-10-031-2/+3
| | | | | | after unicode_empty has been freed, otherwise it might not point to the real start of the unicode_freelist. Final closure for SF bug #110681, Jitterbug PR#398.
* In _PyUnicode_Fini(), decref unicode_empty before tearng down the freeGuido van Rossum2000-10-031-2/+2
| | | | list. Discovered by Barry, fix approved by MAL.
* Rationalize use of limits.h, moving the inclusion to Python.h.Fred Drake2000-09-261-6/+0
| | | | | | | | Add definitions of INT_MAX and LONG_MAX to pyport.h. Remove includes of limits.h and conditional definitions of INT_MAX and LONG_MAX elsewhere. This closes SourceForge patch #101659 and bug #115323.
* Derived from Martin's SF patch 110609: support unbounded ints in ↵Tim Peters2000-09-211-24/+68
| | | | | | | | | | | | | | | | %d,i,u,x,X,o formats. Note a curious extension to the std C rules: x, X and o formatting can never produce a sign character in C, so the '+' and ' ' flags are meaningless for them. But unbounded ints *can* produce a sign character under these conversions (no fixed- width bitstring is wide enough to hold all negative values in 2's-comp form). So these flags become meaningful in Python when formatting a Python long which is too big to fit in a C long. This required shuffling around existing code, which hacked x and X conversions to death when both the '#' and '0' flags were specified: the hacks weren't strong enough to deal with the simultaneous possibility of the ' ' or '+' flags too, since signs were always meaningless before for x and X conversions. Isomorphic shuffling was required in unicodeobject.c. Also added dozens of non-trivial new unbounded-int test cases to test_format.py.
* Fix for bug 113934. string*n and unicode*n did no overflow checking atTim Peters2000-09-091-2/+19
| | | | | | | all, either to see whether the # of chars fit in an int, or that the amount of memory needed fit in a size_t. Checking these is expensive, but the alternative is silently wrong answers (as in the bug report) or core dumps (which were easy to provoke using Unicode strings).
* changed \x to consume exactly two hex digits, also for unicodeFredrik Lundh2000-09-031-55/+66
| | | | | | strings. closes PEP-223. also added \U escape (eight hex digits).
* PyUnicode_AsUTF8String(): /F picks up what I missed: the local varBarry Warsaw2000-08-181-2/+0
| | | | `str' is no longer necessary. Gotta turn on -Wall!
* PyUnicode_AsUTF8String(): Don't need to explicitly incref str sinceBarry Warsaw2000-08-181-7/+3
| | | | | PyUnicode_EncodeUTF8() already returns the created object with the proper reference count. This fixes an Insure reported memory leak.
* Fixed a couple of instances where a 0-length string was beingMarc-André Lemburg2000-08-141-6/+13
| | | | | | | resized after creation. 0-length strings are usually shared and _PyString_Resize() fails on these shared strings. Fixes [ Bug #111667 ] unicode core dump.
* Clean up warning from Monterey compiler.Trent Mick2000-08-121-1/+1
| | | | | | Properly end a comment block. It was terminated fine later but by a subsequent block and. It was also in #if 0. This patch is so trivial I can't believe I am talking about it. :)
generated by cgit v0.12 at 2025-02-22 12:15:49 (GMT)