[New-bugs-announce] [issue44170] ShareableList cannot safely handle multibyte utf-8 characters

Huw Jones report at bugs.python.org
Tue May 18 07:12:09 EDT 2021

New submission from Huw Jones <huw at huwcbjones.co.uk>:

I've experienced a UnicodeDecodeError when adding unicode strings that contain multibye utf-8 characters into a shareable list.
My observation is that ShareableList chunks the list of strings before sending it over the process boundary, however this chunking process is not multibyte aware and will chunk in the middle of multibyte characters.
On the other end, this results in the ShareableList throwing a UnicodeDecodeError when it fails to decode not-a-full multibyte utf-8 character.

>From running the attached MWE, I see that the string is sent in two chunks, the first being b'Boom \xf0\x9f\x92\xa5 \xf0\x9f\x92\xa5 \xf0' which clearly splits the 4 bytes of the 💥 character into the first byte and remaining 3 bytes.

components: Library (Lib)
files: shareablelist_mwe.py
messages: 393868
nosy: huwcbjones
priority: normal
severity: normal
status: open
title: ShareableList cannot safely handle multibyte utf-8 characters
type: crash
versions: Python 3.8
Added file: https://bugs.python.org/file50049/shareablelist_mwe.py

Python tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list