22.03.19 09:45, Victor Stinner пише:

Internally, CPython has a _PyUnicodeWriter which is an efficient way to create a string but appending substrings or characters. _PyUnicodeWriter changes the internal storage format depending on characters code points (ascii or latin1: 1 byte/character, BMP: 2 b/c, full UCS: 4 b/c). I tried once to expose it in Python, but I wasn't convinced by performances. The overhead of method calls was quite significant, and I wasn't convinced by "writer += str" performance neither. Maybe I should try again. PyPy also has such object. It avoids the "str += str" hack in ceval.c to avoid very poor performance (_PyUnicodeWriter also uses overallocation which can be controlled with multiple parameters to reduce the number of realloc).

Another alternative would be have to add a "strarray" type similar to bytes/bytearray couple.

Another alternative of mutable string buffer and string builder is io.StringIO.

FYI, I have created issue on bugs.python.org about adding deprecation warning for array('u'). https://bugs.python.org/issue36299 I created PR to change Py_UNICODE to Py_UCS4, instead of deprecate it. https://github.com/python/cpython/pull/12497 Then, I found same change had made and reverted in the past. https://github.com/python/cpython/commit/62bb394729a167a46d950954c4aed5f3ba7... The issue for the revert is this. https://bugs.python.org/issue13072 -- Inada Naoki

On Fri, 22 Mar 2019 12:51:49 +0100 Stefan Behnel wrote:

Antoine Pitrou schrieb am 22.03.19 um 11:39:

...

On Fri, 22 Mar 2019 20:31:33 +1300 Greg Ewing wrote:

...

A poster on comp.lang.python is asking about array.array('u'). He wants an efficient mutable collection of unicode characters that can be initialised from a string.

TBH, I think anyone trying to use array.array should be directed to Numpy these days. The only reason for array.array being here is that it predates Numpy. Otherwise we'd never have added it.

Well, maybe it wouldn't get *added* these days anymore, with pip+PyPI nicely in place. But being there already, it makes for a nice and efficient "batteries included" list replacement for simple data that would otherwise waste a lot of object memory.

It's not really "batteries included". array.array() supports almost no useful operation. It's a bare-bones container for which you have to implement every useful feature by yourself. (yes, you can use generic mutable sequence algorithms such as heapq or random.shuffle; how often do you need to heapify or shuffle an array of unicode codepoints?) Also, when using a unicode array, there's no substantial win of memory compared to a single str object. You may be losing some actually, because of the flexible str representation. Regards Antoine.

22.03.19 13:33, Antoine Pitrou пише:

On Fri, 22 Mar 2019 13:27:08 +0200 Serhiy Storchaka wrote:

...

Making it always 32 bits would be compatibility breaking change. Currently array('u') represents the wchar_t string, and many API on Windows require it.

The question is: why would you use a array.array() with a Windows C API?

I do not. But maybe it is used. And changing the width of the 'u' code would break such use case.

On 22Mar2019 0433, Antoine Pitrou wrote:

The question is: why would you use a array.array() with a Windows C API?

I started replying to this with a whole lot of examples, and eventually convinced myself that you wouldn't (or shouldn't). That said, I see value in having a type for array/struct/memoryview that "is the same size as wchar_t", since that will avoid people having to guess (the docs for array in particular are deliberately vague about the actual size of the various types). This is not the same as "UCS-4" - that's a very Linux-centric point of view. Decoupling it from Py_UNICODE is fine though, since that type will have no meaning eventually. But the PyUnicode_*WideChar APIs are not going away, which means that wchar_t still exists and has to have a known size at compile time. Cheers, Steve

On 25Mar2019 0812, Martin (gzlist) wrote:

On Fri, 22 Mar 2019 at 16:12, Steve Dower wrote:

...

On 22Mar2019 0433, Antoine Pitrou wrote:

...

The question is: why would you use a array.array() with a Windows C API?

I started replying to this with a whole lot of examples, and eventually convinced myself that you wouldn't (or shouldn't).

That said, I see value in having a type for array/struct/memoryview that "is the same size as wchar_t", since that will avoid people having to guess (the docs for array in particular are deliberately vague about the actual size of the various types).

This is pretty much what ctypes provides for dealing with unicode?

https://docs.python.org/3/library/ctypes.html#ctypes.create_unicode_buffer

Seems a fine place to have things that help with win32 api interactions.

Sure, though there are other reasons to deal with "pure" data that would benefit from having the data type in array. I don't need to directly refer to an existing buffer in memory, just to easily create/reinterpret bytes (for which memoryview is often better, though it inherits its codes from struct, which has no 'u' code, which is probably why I end up using array instead ;) ) Also, I keep finding that every time I deploy Python these days, it's critical to remove ctypes to reduce the attack surface area (I'm getting it into more and more high value systems where the rules are more strict). So I'm a big fan of treating ctypes as optional. Cheers, Steve