[Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces
v+python at g.nevcal.com
Mon Apr 27 08:39:41 CEST 2009
On approximately 4/25/2009 5:35 AM, came the following characters from
the keyboard of Martin v. Löwis:
>> Because the encoding is not reliably reversible.
> Why do you say that? The encoding is completely reversible
> (unless we disagree on what "reversible" means).
>> I'm +1 on the concept, -1 on the PEP, due solely to the lack of a
>> reversible encoding.
> Then please provide an example for a setup where it is not reversible.
It is reversible if you know that it is decoded, and apply the encoding.
But if you don't know that has been encoded, then applying the reverse
transform can convert an undecoded str that matches the decoded str to
the form that it could have, but never did take.
The problem is that there is no guarantee that the str interface
provides only strictly conforming Unicode, so decoding bytes to
non-strictly conforming Unicode, can result in a data pun between
non-strictly conforming Unicode coming from the str interface vs bytes
being decoded to non-strictly conforming Unicode coming from the bytes
Any particular problem that always consistently uses one or the other
(bytes vs str) APIs under the covers might never be affected by such a
data pun, but programs that may use both types of interface could
potentially see a data pun.
If your PEP depends on consistent use of one or the other type of
interface, you should say so, and if the platform only provides that
type of interface, maybe all is well. Both types of interfaces are
available on Windows, perhaps POSIX only provides native bytes
interfaces, and if the PEP is the only way to provide str interfaces,
then perhaps consistency use is required.
There are still issues regarding how Windows and POSIX programs that are
sharing cross-mounted file systems might communicate file names between
each other, which is not at all clear from the PEP. If this is an
insoluble or un-addressed issue, it should be stated. (It is probably
insoluble, due to there being multiple ways that the cross-mounted file
systems might translate names; but if there are, can we learn something
from the rules the mounting systems use, to be compatible with (one of)
them, or not.
Together with your change to avoid using PUA characters, and the rule
suggested by MRAB in another branch of this thread, of treating
half-surrogates as invalid byte sequences may avoid the data puns I'm
It is not clear how half-surrogate characters would be displayed, when
the user prints or displays such a file name string. It would seem that
programs that display file names to users might still have issues with
such; an escaping mechanism that uses displayable characters would have
an advantage there.
Glenn -- http://nevcal.com/
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
More information about the Python-Dev