imaplib: is this really so unwieldy?
Terry Reedy
tjreedy at udel.edu
Tue May 25 18:23:34 EDT 2021
On 5/25/2021 1:25 PM, MRAB wrote:
> On 2021-05-25 16:41, Dennis Lee Bieber wrote:
>> In Python 3, strings are UNICODE, using 1, 2, or 4 bytes PER
>> CHARACTER
This is CPython 3.3+ specific. Before than, it depended on the OS. I
believe MicroPython uses utf-8 for strings.
>> (I don't recall if there is a 3-byte version).
There isn't. It would save space but cost time.
>> If your input bytes are all
>> 7-bit ASCII, then they map directly to a 1-byte per character string.
If your input bytes all have the upper bit 0 and they are interpreted as
encoding ascii characters then they map to overhead + 1 byte per char
>>> sys.getsizeof(b''.decode('ascii'))
49
>>> sys.getsizeof(b'a'.decode('ascii'))
50
>>> sys.getsizeof(11*b'a'.decode('ascii'))
60
>> If
>> they contain any 8-bit upper half character they may map into a 2-byte
>> per character string.
See below.
> In CPython 3.3+:
>
> U+0000..U+00FF are stored in 1 byte.
> U+0100..U+FFFF are stored in 2 bytes.
> U+010000..U+10FFFF are stored in 4 bytes.
In CPython's Flexible String Representation all characters in a string
are stored with the same number of bytes, depending on the largest
codepoint.
>>> sys.getsizeof('\U00011111')
80
>>> sys.getsizeof('\U00011111'*2)
84
>>> sys.getsizeof('a\U00011111')
84
>> Bytes in Python 3 are just a binary stream, which needs an
>> encoding to produce characters.
Or any other Python object.
>> Use the wrong encoding (say ISO-Latin-1) when the
>> data is really UTF-8 will result in garbage.
So does decoding bytes as text when the bytes encode something else,
such as an image ;-).
--
Terry Jan Reedy
More information about the Python-list
mailing list