[Python-Dev] Unicode charmap decoders slow
walter at livinglogic.de
Tue Oct 4 23:48:08 CEST 2005
Am 04.10.2005 um 21:50 schrieb Martin v. Löwis:
> Walter Dörwald wrote:
>> For charmap decoding we might be able to use an array (e.g. a
>> tuple (or an array.array?) of codepoints instead of dictionary.
> This array would have to be sparse, of course.
For encoding yes, for decoding no.
> Using an array.array
> would be more efficient, I guess - but we would need a C API for
> (to validate the type code, and to get ob_item).
For decoding it should be sufficient to use a unicode string of
length 256. u"\ufffd" could be used for "maps to undefined". Or the
string might be shorter and byte values greater than the length of
the string are treated as "maps to undefined" too.
>> Or we could implement this array as a C array (i.e. gencodec.py
>> would generate C code).
> For decoding, we would not get any better than array.array, except for
> startup cost.
> For encoding, having a C trie might give considerable speedup. _codecs
> could offer an API to convert the current dictionaries into
> lookup-efficient structures, and the conversion would be done when
> importing the codec.
> For the trie, two levels (higher and lower byte) would probably be
> sufficient: I believe most encodings only use 2 "rows" (256 code
> point blocks), very few more than three.
This might work, although nobody has complained about charmap
encoding yet. Another option would be to generate a big switch
statement in C and let the compiler decide about the best data
More information about the Python-Dev