[New-bugs-announce] [issue24870] surrogateescape is too slow

[INADA Naoki]

New submission from INADA Naoki:

surrogateescape is recommended way to mix binary data in string protocol.
But surrogateescape is too slow and it cause usability problem.

One actual problem is: https://github.com/PyMySQL/PyMySQL/issues/366

surrogateescape is slow because errorhandler is called with UnicodeError object.
bs.decode('utf-8', 'surrogateescape') may produce len(bs)/2 error objects internally when bs is random bytes.

surrogateescape is used with ASCII and UTF-8 encoding in ordinal.
Specialized implementation can make it faster.

I want to Python 3.4 and Python 3.5 solve this issue since it's critical problem
for some people.

----------
components: Unicode
messages: 248631
nosy: ezio.melotti, haypo, naoki
priority: normal
severity: normal
status: open
title: surrogateescape is too slow
type: performance
versions: Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue24870>
_______________________________________