[issue2857] Add "java modified utf-8" codec

[Serhiy Storchaka]

Serhiy Storchaka <storchaka at gmail.com> added the comment:

As far as I understand, this codec can be implemented in Python. There is no need to modify the interpreter core.

def decode_cesu8(b):
    return re.sub('[\uD800-\uDBFF][\uDC00\DFFF]', lambda m: chr(0x10000 | ((ord(m.group()[0]) & 0x3FF) << 10) | (ord(m.group()[1]) & 0x3FF)), b.decode('utf-8', 'surrogatepass'))

def encode_cesu8(s):
    return re.sub('[\U00010000-\U0010FFFF]', lambda m: chr(0xD800 | ((ord(m.group()) >> 10) & 0x3FF)) + chr(0xDC00 | (ord(m.group() & 0x3FF)), s).encode('utf-8', 'surrogatepass')

def decode_mutf8(b):
    return decode_cesu8(b.replace(b'\xC0\x80', b'\x00'))

def encode_mutf8(s):
    return encode_cesu8(s).replace(b'\x00', b'\xC0\x80')

----------
nosy: +storchaka

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2857>
_______________________________________