[Python-Dev] PEP 393 review

["Martin v. Löwis"]

> With this PEP, the unicode object overhead grows to 10 pointer-sized
> words (including PyObject_HEAD), that's 80 bytes on a 64-bit machine.
> Does it have any adverse effects?

If I count correctly, it's only three *additional* words (compared to
3.2): four new ones, minus one that is removed. In addition, it drops
a memory block. Assuming a malloc overhead of two pointers per malloc
block, we get one additional pointer.

On a 32-bit machine with a 32-bit wchar_t, pure-ASCII strings of length
1 (+NUL) will take the same memory either way: 8 bytes for the
characters in 3.2, 2 bytes in 3.3 + extra pointer + padding. Strings
of 2 or more characters will take more space in 3.2.

On a 32-bit machine with a 16-bit wchar_t, pure-ASCII strings up
to 3 characters take the same space either way; space savings start at
four characters.

On a 64-bit machine with a 16-bit wchar_t, assuming a malloc minimum
block size of 16 bytes, pure-ASCII strings of up to 7 characters take
the same space. For 8 characters, 3.2 will need 32 bytes for the
characters, whereas 3.3 will only take 16 bytes (due to padding).

So: no, I can't see any adverse effects. Details depend on the
malloc implementation, though. A slight memory increase may occur
on compared to a narrow build may occur for strings that use
non-Latin-1, and a large increase for strings that use non-BMP
characters.

The real issue of memory consumption is the alternative representations,
if created. That applies for the default encoding in 3.2 as well as
the wchar_t and UTF-8 representations in 3.3.

> Are there any plans to make instantiation of small strings fast enough?
> Or is it already as fast as it should be?

I don't have any plans, and I don't see potential. Compared to 3.2, it
saves a malloc call, which may be quite an improvement. OTOH, it needs
to iterate over the characters twice, to find the largest character.

If you are referring to the reuse of Unicode objects: that's currently
not done, and is difficult to do in the 3.2 way due to the various sizes
of characters. One idea might be to only reuse UCS1 strings, and then
keep a freelist for these based on the string length.

> When interfacing with the Win32 "wide" APIs, what is the recommended
> way to get the required LPCWSTR?

As before: PyUnicode_AsUnicode.

> Will the format codes returning a Py_UNICODE pointer with
> PyArg_ParseTuple be deprecated?

Not for 3.3, no.

> Do you think the wstr representation could be removed in some future
> version of Python?

Yes. This probably has to wait for Python 4, though.

> Is PyUnicode_Ready() necessary for all unicode objects, or only those
> allocated through the legacy API?

Only for the latter (although it doesn't hurt to apply it to all
of them).

> “The Py_Unicode representation is not instantaneously available”: you
> mean the Py_UNICODE representation?

Thanks, fixed.

>> - conditions you would like to pose on the implementation before
>>   acceptance. I'll see which of these can be resolved, and list
>>   the ones that remain open.
> 
> That it doesn't significantly slow down benchmarks such as stringbench
> and iobench.

Can you please quantify "significantly"? Also, having a complete list
of benchmarks to perform prior to acceptance would be helpful.

Thanks,
Martin