On 29.08.2014 13:22, Isaac Morland wrote:
On Fri, 29 Aug 2014, M.-A. Lemburg wrote:
On 29.08.2014 02:41, Stephen J. Turnbull wrote: Since Python allows working with lone surrogates in Unicode (they are valid code points) and we're using UTF-8 for marshal, we needed a way to make sure that Python 3 also optionally supports working with lone surrogates in such UTF-8 streams (nowadays called CESU-8: http://en.wikipedia.org/wiki/CESU-8).
If I want that wouldn't I specify "cesu-8" as the encoding?
i.e., instead of .decode ('utf-8') I would use .decode ('cesu-8'). Right now, trying this I get that cesu-8 is an unknown encoding but that could be changed without affecting the behaviour of the utf-8 codec.
Why write a new codec that's almost identical to the utf-8 codec, if you can get the same functionality by explicitly using a special error handler ?
From a maintenance POV that does not sound like a good approach.
It seems to me that .decode ('utf-8') should decode exactly and only valid utf-8, including the non-use of surrogate pairs as an intermediate encoding step.
It does in Python 3.