On Tue, 23 Apr 2013 11:16:20 +1200, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Victor Stinner wrote:

...

The last proposition is to add transform() and untransform() methods to bytes and str types. ... If I remember correctly, the missing point is how to define which types are supported by a codec

Also, for any given codec, which direction is "transform" and which is "untransform"?

You transform *into* the encoding, and untransform *out* of the encoding. Do you have an example where that would be ambiguous?

Also also, what's so special about base64 et al that they deserve an ultra-special way of invoking them, instead of having to import a class or function like you do for *every* *other* piece of library functionality?

You can ask the same question about all the other codecs. (And that question has indeed been asked in the past.) (One answer is that they used to work in Python2...but the longer we go without restoring the functionality to Python3, the weaker that particular argument becomes.) --David

On Apr 22, 2013, at 06:22 PM, Guido van Rossum wrote:

...

You can ask the same question about all the other codecs. (And that question has indeed been asked in the past.)

Except for rot13. :-)

The fact that you can do this instead *is* a bit odd. ;) from codecs import getencoder encoder = getencoder('rot-13') r13 = encoder('hello world')[0] -Barry

On Tue, Apr 23, 2013 at 8:22 AM, M.-A. Lemburg <mal@egenix.com> wrote:

Just as reminder: we have the general purpose encode()/decode() functions in the codecs module:

import codecs r13 = codecs.encode('hello world', 'rot-13')

These interface directly to the codec interfaces, without enforcing type restrictions. The codec defines the supported input and output types.

As an implementation mechanism I see nothing wrong with this. I hope the codecs module lets you introspect the input and output types of a codec given by name? -- --Guido van Rossum (python.org/~guido)

On Tue, Apr 23, 2013 at 9:04 AM, M.-A. Lemburg <mal@egenix.com> wrote:

On 23.04.2013 17:47, Guido van Rossum wrote:

...

On Tue, Apr 23, 2013 at 8:22 AM, M.-A. Lemburg <mal@egenix.com> wrote:

...

Just as reminder: we have the general purpose encode()/decode() functions in the codecs module:

import codecs r13 = codecs.encode('hello world', 'rot-13')

These interface directly to the codec interfaces, without enforcing type restrictions. The codec defines the supported input and output types.

As an implementation mechanism I see nothing wrong with this. I hope the codecs module lets you introspect the input and output types of a codec given by name?

At the moment there is no standard interface to access supported input and output types... but then: regular Python functions or methods also don't provide such functionality, so no surprise there ;-)

Not quite the same though. Each function has its own unique behavior. But codecs support a standard interface, *except* that the input and output types sometimes vary.

It's mostly a matter of specifying the supported type combinations in the codec documentation.

BTW: What would be a use case where you'd want to programmatically access such information before calling the codec ?

As you know, in Python 3, most code working with bytes doesn't also work with strings, and vice versa (except for a few cases where we've gone out of our way to write polymorphic code -- but users rarely do so, and any time you use a string or bytes literal you basically limit yourself to that type). Suppose I write a command-line utility that reads a file, runs it through a codec, and writes the result to another file. Suppose the name of the codec is a command-line argument (as well as the filenames). I need to know whether to open the files in text or binary mode based on the name of the codec. -- --Guido van Rossum (python.org/~guido)

On 4/24/2013 1:22 AM, M.-A. Lemburg wrote:

On 23.04.2013 19:24, Guido van Rossum wrote:

...

On Tue, Apr 23, 2013 at 9:04 AM, M.-A. Lemburg <mal@egenix.com> wrote:

...

...

...

Just as reminder: we have the general purpose encode()/decode() functions in the codecs module:

import codecs r13 = codecs.encode('hello world', 'rot-13')

These interface directly to the codec interfaces, without enforcing type restrictions. The codec defines the supported input and output types. As an implementation mechanism I see nothing wrong with this. I hope

On Tue, Apr 23, 2013 at 8:22 AM, M.-A. Lemburg <mal@egenix.com> wrote: the codecs module lets you introspect the input and output types of a codec given by name? At the moment there is no standard interface to access supported input and output types... but then: regular Python functions or methods also don't provide such functionality, so no surprise

On 23.04.2013 17:47, Guido van Rossum wrote: there ;-) Not quite the same though. Each function has its own unique behavior. But codecs support a standard interface, *except* that the input and output types sometimes vary. The codec system itself

...

...

It's mostly a matter of specifying the supported type combinations in the codec documentation.

BTW: What would be a use case where you'd want to programmatically access such information before calling the codec ? As you know, in Python 3, most code working with bytes doesn't also work with strings, and vice versa (except for a few cases where we've gone out of our way to write polymorphic code -- but users rarely do so, and any time you use a string or bytes literal you basically limit yourself to that type).

Suppose I write a command-line utility that reads a file, runs it through a codec, and writes the result to another file. Suppose the name of the codec is a command-line argument (as well as the filenames). I need to know whether to open the files in text or binary mode based on the name of the codec. Ok, so you need to know which codecs your tool can support and which of those need text input and which bytes input.

I've been thinking about this some more: I think that type information alone is not flexible enough to cover such use cases.

Maybe MIME type and encoding would be sufficient type information, but probably not str vs. bytes.

In your use case you'd want to only permit use of a certain set of codecs, not simply all of them, since some might not implement what you actually want to achieve with the tool, e.g. a user might have installed a codec set that adds support for reading and writing image data, but your intended use was to only support text data.

MIME type supports this sort of concept, with the two-level hierarchy of naming the type... text/xml text/plain image/jpeg

So what we need is a way to allow the codecs to say e.g. "I work on text", "I support encoding bytes and text", "I encode to bytes", "I'm reversible", "I transform input data", "I support bytes and text, and will create same type output", "I work on image data", "I work on X509 certificates", "I work on XML data", etc.

Guess what I think you are re-inventing here.... Nope, guess again.... Yep, MIME types _plus_ encodings.

In other words, we need a form of tagging system, with a set of standard tags that each codec can publish and which also allows non-standard tags (which can then at some point be made standard, if there's agreement on them).

Hmm. Sounds just like the registry for, um, you guessed it: MIME types.

Given a codec name you could then ask the codec registry for the codec tags and verify that the chosen codec handles text data, needs bytes or text encoding input and creates bytes as encoding output. If the registry returns codec tags that don't include the "I work on text" tag, the tool could then raise an error.

For just doing text encoding transformations, text/plain would work as a MIME type, and the encodings of interest for the encodings. Seems like "str" always means "Unicode" but the MIME type can vary; "bytes" might mean encoded text, and the MIME type can also vary. For non-textual transformations, "encoding" might mean Base 64, BinHex, or other such representations... but those can also be applied to text, so it might be a 3rd dimension, or it might just be a list of encodings rather than a single encoding. Compression could be another dimension, or perhaps another encoding. But really, then, a transformation needs to be a list of steps; a codec can sign up to perform one or more of the steps, a sequence of codecs would have to be found, capable of performing a subsequence of the steps, and then run in the appropriate order. This all sounds so general, that probably the Python compiler could be implemented as a codec :) Or any compiler. Probably a web server could be implemented as a codec too :) Well, maybe not, codecs have limited error handling and reporting abilities.

On 23.04.2013 23:37, Nick Coghlan wrote:

On 24 Apr 2013 01:25, "M.-A. Lemburg" <mal@egenix.com> wrote:

...

On 23.04.2013 17:15, Barry Warsaw wrote:

...

On Apr 22, 2013, at 06:22 PM, Guido van Rossum wrote:

...

...

You can ask the same question about all the other codecs. (And that question has indeed been asked in the past.)

Except for rot13. :-)

The fact that you can do this instead *is* a bit odd. ;)

from codecs import getencoder encoder = getencoder('rot-13') r13 = encoder('hello world')[0]

Just as reminder: we have the general purpose encode()/decode() functions in the codecs module:

import codecs r13 = codecs.encode('hello world', 'rot-13')

These interface directly to the codec interfaces, without enforcing type restrictions. The codec defines the supported input and output types.

If we already have those, why aren't they documented?

Good question. I added them in 2004 and probably just forgot to add the documentation: http://hg.python.org/cpython-fullhistory/rev/8ea2cb1ec598 I guess the doc-strings could be used as basis for the documentation. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 24 2013)

...

...

Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2013-04-17: Released eGenix mx Base 3.2.6 ... http://egenix.com/go43 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Tue, 23 Apr 2013 22:29:33 +0900, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:

R. David Murray writes:

...

You transform *into* the encoding, and untransform *out* of the encoding. Do you have an example where that would be ambiguous?

In the bytes-to-bytes case, any pair of character encodings (eg, UTF-8 and ISO-8859-15) would do. Or how about in text, ReST to HTML?

If I write: bytestring.transform('ISO-8859-15') that would indeed be ambiguous, but only because I haven't named the source encoding of the bytestring. So the above is obviously nonsense, and the easiest "fix" is to have the things that are currently bytes-to-text or text-to-bytes character set transformations *only* work with encode/decode, and not transform/untransform.

BASE64 itself is ambiguous. By RFC specification, BASE64 is a *textual* representation of arbitrary binary data. (Cf. URIs.) The natural interpretation of .encode('base64') in that context would be as a bytes-to-text encoder. However, this has several problems. In practice, we invariably use an ASCII octet stream to carry BASE64- encoded data. So web developers would almost certainly expect a bytes-to-bytes encoder. Such a bytes-to-bytes encoder can't be duck-typed. Double-encoding bugs wouldn't be detected until the stream arrives at the user. And the RFC-based signature of .encode('base64') as bytes-to-text is precisely opposite to that of .encode('utf-8') (text-to-bytes).

I believe that after much discussion we have settled on these transformations (in their respective modules) accepting either bytes or strings as input for decoding, only bytes as input for encoding, and *always* producing bytes as output. (Note that the base64 docs need some clarification about this.) Given this, the possible valid transformations would be: bytestring.transform('base64') bytesstring.untransform('base64') string.untransform('base64') and all would produce a byte string. That byte string would be in base64 for the first one, and a decoded binary string for the second two. Given our existing API, I don't think we want string.encode('base64') to work (taking an ascii-only unicode string and returning bytes), and we've already agreed that adding a 'decode' method to string is not going to happen. We could, however, and quite possibly should, disallow string.untransform('base64') even though the underly module supports it. Thus we would only have bytes-to-bytes transformations for 'base64' and its siblings, and you would write the unicode-ascii-to-bytes transformation as: string.encode('ascii').untransform('base64') which has some pedagogical value :). If you do transform('base64') on a bytestring already encoded as base64 you get a double encoding, yes. I don't see that it is our responsibility to try to protect you from this mistake. The module functions certainly don't. Given that, is there anything ambiguous about the proposed API? (Note: if you would like to argue that, eg, base64.b64encode or binascii.b2a_base64 should return a string, it is too late for that argument for backward compatibility reasons.)

It is certainly true that there are many unambiguous cases. In the case of a true text processing facility (eg, Emacs buffers or Python 3 str) where there is an unambiguous text type with a constant and opaque internal representation, it makes a lot of sense to treat the text type as special/central, and use the terminology "encode [from text]" and "decode [to text]". It's easy to remember, which one is special is obvious, and the difference in input and output types means that mistaken use of the API will be detected by duck-typing.

However, in the case of bytes-bytes or text-text transformations, it's not the presence of unambiguous cases that should drive API design IMO. It's the presence of the ambiguous cases that we should cater to. I don't see easy solutions to this issue.

When I asked about ambiguous cases, I was asking for cases where the meaning of "transform('somecodec')" was ambiguous. Sure, it is possible to feed the wrong input into that transformation, but I consider that a programming error, not an ambiguity in the API. After all, you have exactly the same problem if you use the module functions directly, which is currently the only option. --David

R. David Murray writes:

On Tue, 23 Apr 2013 22:29:33 +0900, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:

...

R. David Murray writes:

...

You transform *into* the encoding, and untransform *out* of the encoding. Do you have an example where that would be ambiguous?

In the bytes-to-bytes case, any pair of character encodings (eg, UTF-8 and ISO-8859-15) would do. Or how about in text, ReST to HTML?

If I write:

bytestring.transform('ISO-8859-15')

that would indeed be ambiguous, but only because I haven't named the source encoding of the bytestring. So the above is obviously nonsense, and the easiest "fix" is to have the things that are currently bytes-to-text or text-to-bytes character set transformations *only* work with encode/decode, and not transform/untransform.

I think you're completely missing my point here. The problem is that in the cases I mention, what is encoded data and what is decoded data can only be decided by asking the user.

I believe that after much discussion we have settled on these transformations (in their respective modules) accepting either bytes or strings as input for decoding, only bytes as input for encoding, and *always* producing bytes as output.

Which, of course, is quite broken from the point of view of the RFC! Of course, the RFC be damned[1], for the purposes of the Python stdlib, the specific codecs used for Content-Transfer-Encoding have a clear intuitive directionality, and their encoding methods should turn bytes into bytes (and str or bytes into bytes on decoding). Nevertheless, it's not TOOWTDI, it's a careful compromise.

Given this, the possible valid transformations would be:

bytestring.transform('base64') bytesstring.untransform('base64') string.untransform('base64')

Which is an obnoxious API, since (1) you've now made it impossible to use "transform" for bytestring.transform(from='utf-8', to='iso-8859-1') bytestring.transform(from='ulaw', to='mp3') textstring.transform(from='rest', to='html') without confusion, and (2) the whole world is going to wonder why you don't use .encode and .decode instead of .transform and .untransform. The idea in the examples is that we could generalize the codec registry to look up codecs by pairs of media-types. I'm not sure this makes sense ... much of the codec API presumes a stream, especially the incremental methods. But many MIME media types are streams only because they're serializations, incremental en/decoding is nonsense. So I suppose I would want to write bytestring.transform(from='octet-stream', to='BASE64') for this hypothetical API. (I suspect that in practice the 'application/octet-stream' media type would be spelled 'bytes', of course.) This kind of API could be used to improve the security of composition of transforms. In the case of BASE64, it would make sense to match anything at all as the other type (as long as it's represented in Python by a bytes object). So it would be possible to do object = bytestring.transform(from='BASE64', to='PNG') giving object a media_type attribute such that object.decode('iso-8859-1') would fail. (This would require changes to the charset codecs, to pay heed to the media_type attribute, so it's not immediately feasible.)

and all would produce a byte string. That byte string would be in base64 for the first one, and a decoded binary string for the second two.

Given our existing API, I don't think we want

string.encode('base64')

to work (taking an ascii-only unicode string and returning bytes),

No, we don't, but for reasons that have little to do with "ASCII-only". The problem with supporting that idiom is that *people can't read strs* [in the Python 3 internal representation] -- they can only read a str that has been encoded implicitly into the PYTHONIOENCODING or explicitly to an explicitly requested encoding. So the usage above is clearly ambiguous. Even if it is ASCII-only, in theory the user could want EBCDIC.

If you do transform('base64') on a bytestring already encoded as base64 you get a double encoding, yes. I don't see that it is our responsibility to try to protect you from this mistake. The module functions certainly don't.

Given that, is there anything ambiguous about the proposed API?

Not for BASE64. But what's so special about BASE64 that it deserves a new method name for the same old idiom, using a word that's an obvious candidate for naming a more general idiom?

(Note: if you would like to argue that, eg, base64.b64encode or binascii.b2a_base64 should return a string, it is too late for that argument for backward compatibility reasons.)

Even if it weren't too late, the byte-shoveling lobby is way too strong; that's not a winnable agument.

When I asked about ambiguous cases, I was asking for cases where the meaning of "transform('somecodec')" was ambiguous.

If "transform('somecodec')" isn't ambiguous, you really really want to spell it "encode" instead of "transform" IMO. Even though I don't see how to do that without generating more confusion than it's worth at the moment, I still harbor the hope that somebody will come up with a way to do it so everything still fits together. Footnotes: [1] I am *not* one to damn RFCs lightly!

R. David Murray writes:

...

I think you're completely missing my point here. The problem is that in the cases I mention, what is encoded data and what is decoded data can only be decided by asking the user.

I think I understood that. I don't understand why that's a problem.

It's a problem because in that case it's hard for users to remember the directionality of the codec based only on a single name; the API needs to indicate what is being transformed to what else.

I've been trying to explain what I thought the transform/untransform proposal was: a minimalist extension of the encode/decode semantic (under a different name) so that functionality that was lost from Python2 encode/decode could be restored to Python3 in a reasonably understandable way.

I think that the intention of the proposal is reasonably understandable, and reasonable. I just don't think the API proposed is understandable, and therefore it's not reasonable.<wink/>

People (at least you and Nick, and maybe Guido) seem to be more interested in a more general/powerful mechanism. I'm fine with that :)

I can't speak to the opinions of people who actually know about language design. For myself, I'm sympathetic to the proposal of a specific API limited to cases where the directionality is clear as a generality. I just don't think the "transform" proposal helps much, partly because the actual applications are few, and partly because "transform" is more ambiguous (to be unambiguous in English, you need both the direct object ("from media type") and the indirect object ("to media type") specified. It is quite possible to say "transform encoded text to raw text" or similar. At least for me, "encode transformed text to raw text" raises a WTFAssertion. I know that I've experienced worlds of pain in the character coding sphere from Emacs APIs and UIs that don't indicate directionality clearly. This is very delicate; GNU Emacs had an ugly bug that regressed multiple times over more than a decade merely because they exposed the internal representation of text to Lisp. XEmacs has never experienced that bug (to be precise, the presence of that kind of bug resulted in an immediate assertion, so it was eliminated early in development). Surprisingly to me, the fact that XEmacs uses the internal representation of *text* to also represent "byte streams" (with bytes of variable width!) has never caused me confusion. It does cause others confusion, though, so although the XEmacs model of text is easier to work with than Emacs's, I tend to think Python 3's (which never confounds text with bytes) is better. I suspect that delicacy extends to non-character transformations, so I am pretty demanding about proposals in this area. Specifically I insist on EIBTI and TOOWTDI.

Terry Jan Reedy writes:

.transform should be explicit and always take two args, no implicit defaults, the 'from form' and the 'to' form. They can labelled by position in the natural order (from, to)

Not natural to escaped-from-C programmers, though. I hesitate to say "make it keywords-only", but using keywords should be *strongly* encouraged.

str.transform would always be unicode to unicode and bytes.transform always bytes to bytes.

Which leaves the salient cases (MIME content transfer encodings) out in the cold, although I guess string.encode('ascii').transform(from='base64', to='bytes') isn't too horrible.

Greg Ewing writes:

Web developers might grumble about the need for an extra call, but they can no longer claim it would kill the performance of their web server.

Of course they can. There never was any performance measurement that supported that claim in the first place. I don't see how PEP 393 makes a difference to them. The real problem for them is that conceptually they think ASCII in byte form *is* text, and they want to do text processing on it. They'll use any flimsy excuse to avoid a transform to str, because it's just unbearably ugly given their givens. I have sympathy for their position, I just (even today) think it's the wrong thing for Python. However, I've long since been overruled, and I have no evidence to justify saying "I told you so".<wink/>

Tres Seaver writes:

On 04/23/2013 09:29 AM, Stephen J. Turnbull wrote:

...

By RFC specification, BASE64 is a *textual* representation of arbitrary binary data.

It isn't "text" in the sense Py3k means:

RFC 4648 repeatedly refers to *characters*, without specifying an encoding for them. In fact, if you copy accurately, you can write BASE64 on a napkin and that napkin will accurate transmit the data (assuming it doesn't run into sleet or gloom of night). What else is that but "text in the sense of Py3k"? My point is not that Python's base64 codec *should* be bytes-to-str and back. My point is that, both in the formal spec and in historical evolution, that is a plausible interpretation of ".encode('base64')" which happens to be the reverse of the normal codec convention, where ".encode(codec)" is a *string* method, and ".decode(codec)" is a *bytes* method. This is not harder to learn for people (for BASE64 encoding or for coded character sets), because in each case there's a natural sense of direction for *en*coding vs. *de*coding. But it does break duck- typing, as does the web developer bytes-to-bytes usage of BASE64. What I'm groping toward is an idea of a "variable method", so that we could use .encode and .decode where they are TOOWTDI for people even though a purely formal interpretation of duck-typing would say "but why is that blue whale quacking, waddling, and flying?" In other words (although I have no idea how best to implement it), I would like "somestring.encode('base64')" to fail with "I don't know how to do that" (an attribute lookup error?), the same way that "somebytes.encode('utf-8')" does in Python 3 today.

On Thu, 25 Apr 2013 04:19:36 +0200 Lennart Regebro <regebro@gmail.com> wrote:

On Thu, Apr 25, 2013 at 3:54 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

...

RFC 4648 repeatedly refers to *characters*, without specifying an encoding for them. [...]

Base64 is an encoding that transforms between 8-bit streams.

No, it isn't. What Stephen wrote above.

Either you get a "LookupError: unknown encoding: base64", which is what you get now, or you get an UnicodeEncodingError if the text is not ASCII. We don't want the latter, because it means that code that looks fine for the developer breaks in real life because the developer was American

That's bogus. By the same argument, we should suppress any encoding which isn't able to represent all possible unicode strings. That's almost all encodings provided by Python (including utf-8, if you consider lone surrogates). I'm sorry for Americans, but they *still* must know about character encodings, and be ready to handle UnicodeErrors, when using Python 3 for encoding/decoding bytestrings. There's no way around it. Regards Antoine.

Le Thu, 25 Apr 2013 08:38:12 +0200, Lennart Regebro <regebro@gmail.com> a écrit :

On Thu, Apr 25, 2013 at 7:43 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:

...

On Thu, 25 Apr 2013 04:19:36 +0200 Lennart Regebro <regebro@gmail.com> wrote:

...

On Thu, Apr 25, 2013 at 3:54 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

...

RFC 4648 repeatedly refers to *characters*, without specifying an encoding for them. [...]

Base64 is an encoding that transforms between 8-bit streams.

No, it isn't. What Stephen wrote above.

Yes it is. Base64 takes 8-bit bytes and transforms them into another 8-bit stream that can be safely transmitted over various channels that would mangle an unencoded 8-bit stream, such as email etc.

http://en.wikipedia.org/wiki/Base64

I don't see anything in that Wikipedia page that validates your opinion. The Wikipedia page does talk about *text* and *characters* for the result of base64 encoding. Besides, I would consider a RFC more authoritative than a Wikipedia definition.

...

By the same argument, we should suppress any encoding which isn't able to represent all possible unicode strings.

No, if you explicitly use such an encoding it is because you need to because you are transferring data to a system that needs the encoding in question. Unicode errors are unavoidable at that point, not an unexpected surprise because a conversion happened implicitly that you didn't know about.

I don't know what "implicit conversion" you are talking about. There's no "implicit conversion" in a scheme where the result of base64 encoding is a text string. Regards Antoine.

Le Thu, 25 Apr 2013 12:05:01 +0200, Lennart Regebro <regebro@gmail.com> a écrit :

...

The Wikipedia page does talk about *text* and *characters* for the result of base64 encoding.

So are saying that you want the Python implementation of base64 encoding to take 8-bit binary data in bytes format and return a Unicode string containing the Base64 encoded data?

I'm not wanting anything here, since that would clearly break backwards compatibility. But I think binascii should have gone that way in Python 3, indeed. binascii.b2a_hex(), for example, would be much more practical if it returned str, rather than bytes.

Python 3 still views text as Unicode only.

Python 3 doesn't *view* text as unicode, it *represents* it as unicode. That is, unicode is the character set that Python 3 is able to represent in the canonical text type, str. If you ever encounter a hypothetical text that uses characters outside of Unicode (obviously it will be encoded using a non-unicode encoding :-)), then you can't represent it as a str. And base64 is clearly representable as unicode, since it's representable using the ASCII character set (which is a subset of the unicode character set).

I can't think of any usecase where you would want base64 encoded data unless you intend to transmit it over an 8-bit channel,

I can think of many usecases where I want to *embed* base64-encoded data in a larger text *before* encoding that text and transmitting it over a 8-bit channel. (GPG signatures, binary data embedded in JSON objects, etc.) Regards Antoine.

On Apr 25, 2013, at 03:34 PM, Lennart Regebro wrote:

In the case of JSON objects, they are intended for data exchange, and hence in the end need to be byte strings.

Except that they're not. http://bugs.python.org/issue10976 -Barry

On 25/04/2013 14:34, Lennart Regebro wrote:

On Thu, Apr 25, 2013 at 2:57 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:

...

I can think of many usecases where I want to *embed* base64-encoded data in a larger text *before* encoding that text and transmitting it over a 8-bit channel.

That still doesn't mean that this should be the default behavior. Just because you *can* represent base64 as Unicode text doesn't mean that it should be.

...

(GPG signatures, binary data embedded in JSON objects, etc.)

Is the GPG signature calculated on the *Unicode* data? How is that done? Isn't it done on the encoded message? As I understand it a GPG signature is done on any sort of document. Either me or you have completely misunderstood how GPG works, I think. :-)

In the case of JSON objects, they are intended for data exchange, and hence in the end need to be byte strings. So if you have a byte string you want to base64 encode before transmitting it with json, you would just end up transforming it to a unicode string and then back. That doesn't seem useful.

The JSON specification says that it's text. Its string literals can contain Unicode codepoints. It needs to be encoded to bytes for transmission and storage, but JSON itself is not a bytestring format.

One use case where you clearly *do* want the base64 encoded data to be unicode strings is because you want to embed it in a text discussing base64 strings, for a blog or a book or something. That doesn't seem to be a very common usecase.

For the most part you base64 encode things because it's going to be transmitted, and hence the natural result of a base64 encoding should be data that is ready to be transmitted, hence byte strings, and not Unicode strings.

...

Python 3 doesn't *view* text as unicode, it *represents* it as unicode.

I don't agree that there is a significant difference between those wordings in this context. The end result is the same: Things intended to be handled/seen as textual should be unicode strings, things intended for data exchange should be byte strings. Something that is base64 encoded is primarily intended for data exchange. A base64 encoding should therefore return byte strings, especially since most API's that perform this transmission will take byte strings as input. If you want to include this in textual data, for whatever reason, like printing it in a book, then the conversion is trivial, but that is clearly the less common use case, and should therefore not be the default behavior.

base64 is a way of encoding binary data as text. The problem is that traditionally text has been encoded with one byte per character, except in those locales where there were too many characters in the character set for that to be possible. In Python 3 we're trying to stop mixing binary data (bytestrings) with text (Unicode strings).

On Thu, 25 Apr 2013, Lennart Regebro wrote:

On Thu, Apr 25, 2013 at 4:22 PM, MRAB <python@mrabarnett.plus.com> wrote:

...

The JSON specification says that it's text. Its string literals can contain Unicode codepoints. It needs to be encoded to bytes for transmission and storage, but JSON itself is not a bytestring format.

OK, fair enough.

...

base64 is a way of encoding binary data as text.

It's a way of encoding binary data using ASCII. There is a subtle but important difference.

It is a way of encoding arrays of 8-bit bytes as arrays of characters that are part of the printable, non-whitespace subset of the ASCII repertoire. Since the ASCII repertoire is now simply the first 128 code points in the Unicode repertoire, it is equally correct to say that base64 is a way of encoding binary data as Unicode text.

...

In Python 3 we're trying to stop mixing binary data (bytestrings) with text (Unicode strings).

Yup. And that's why a byte64 encoding shouldn't return Unicode strings.

That is exactly why it should return Unicode strings. What bytes should get sent if base64 is used to send a byte array over an EBCDIC link? [*] Having said that, there may be other reasons for base64 encoding to return bytes - I can conceive of arguments involving efficiency, or practicality, or the most common use cases. So I can't say for sure what base64 encoding actually ought to return in Python. But the purist stance should be that base64 encoding should return text, i.e. a string, i.e. unicode. [*] I apologize to anybody who just ate. Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist

On 25/04/2013 15:22, MRAB wrote:

On 25/04/2013 14:34, Lennart Regebro wrote:

...

On Thu, Apr 25, 2013 at 2:57 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:

...

I can think of many usecases where I want to *embed* base64-encoded data in a larger text *before* encoding that text and transmitting it over a 8-bit channel.

That still doesn't mean that this should be the default behavior. Just because you *can* represent base64 as Unicode text doesn't mean that it should be.

[snip]

...

One use case where you clearly *do* want the base64 encoded data to be unicode strings is because you want to embed it in a text discussing base64 strings, for a blog or a book or something. That doesn't seem to be a very common usecase.

For the most part you base64 encode things because it's going to be transmitted, and hence the natural result of a base64 encoding should be data that is ready to be transmitted, hence byte strings, and not Unicode strings.

...

Python 3 doesn't *view* text as unicode, it *represents* it as unicode.

I don't agree that there is a significant difference between those wordings in this context. The end result is the same: Things intended to be handled/seen as textual should be unicode strings, things intended for data exchange should be byte strings. Something that is base64 encoded is primarily intended for data exchange. A base64 encoding should therefore return byte strings, especially since most API's that perform this transmission will take byte strings as input. If you want to include this in textual data, for whatever reason, like printing it in a book, then the conversion is trivial, but that is clearly the less common use case, and should therefore not be the default behavior.

base64 is a way of encoding binary data as text. The problem is that traditionally text has been encoded with one byte per character, except in those locales where there were too many characters in the character set for that to be possible.

In Python 3 we're trying to stop mixing binary data (bytestrings) with text (Unicode strings).

RFC 4648 says """Base encoding of data is used in many situations to store or transfer data in environments that, perhaps for legacy reasons, are restricted to US-ASCII [1] data.""". To me, "US-ASCII" is an encoding, so it appears to be talking about encoding binary data (bytestrings) to ASCII-encoded text (bytestrings).

Le Thu, 25 Apr 2013 15:34:45 +0200, Lennart Regebro <regebro@gmail.com> a écrit :

I don't agree that there is a significant difference between those wordings in this context. The end result is the same: Things intended to be handled/seen as textual should be unicode strings, things intended for data exchange should be byte strings.

I don't think this distinction is meaningful at all. In the end, everything is a byte string on a classical computer (including unicode strings displayed on your monitor, obviously). If you think the technicalities of an operation should never be hidden or abstracted away, then you're better off with C than Python ;-) Regards Antoine.

On 2013-04-25, at 11:25 , Antoine Pitrou wrote:

Besides, I would consider a RFC more authoritative than a Wikipedia definition.

Base encoding of data is used in many situations to store or transfer data in environments that, perhaps for legacy reasons, are restricted to US-ASCII [1] data.

so the output is US-ASCII data, a byte stream. Stephen is correct that you could decide you don't care about those semantics, and implement base64 encoding as a bytes -> str decoding then requiring a re-encoding (to ascii) before wire transmission. The clarity of the interface (or lack thereof) would probably make users want to send a strongly worded letter to whoever implemented it though, I don't think `data.decode('base64').encode('ascii')` would fit the "obviousness" or "readability" expectations of most users.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/25/2013 01:43 AM, Antoine Pitrou wrote:

On Thu, 25 Apr 2013 04:19:36 +0200 Lennart Regebro <regebro@gmail.com> wrote:

...

On Thu, Apr 25, 2013 at 3:54 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

...

RFC 4648 repeatedly refers to *characters*, without specifying an encoding for them. [...]

Base64 is an encoding that transforms between 8-bit streams.

No, it isn't. What Stephen wrote above.

Stephen was incorrect: the base64 standard is about encoding a binary stream (8-bit bites) onto another binary stream (6-bit bytes), but one which can be safely transmitted over a 7-bit-only medium. Text in Py3ks sense is irrelevant.

...

Either you get a "LookupError: unknown encoding: base64", which is what you get now, or you get an UnicodeEncodingError if the text is not ASCII. We don't want the latter, because it means that code that looks fine for the developer breaks in real life because the developer was American

That's bogus. By the same argument, we should suppress any encoding which isn't able to represent all possible unicode strings. That's almost all encodings provided by Python (including utf-8, if you consider lone surrogates).

I'm sorry for Americans, but they *still* must know about character encodings, and be ready to handle UnicodeErrors, when using Python 3 for encoding/decoding bytestrings. There's no way around it.

WHat does that snark have to do with this discussion? base64 has no more to do with character set encodings than it does the moon. It would be a "transform" (bytes -> bytes), not an "encoding". Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF5Nc4ACgkQ+gerLs4ltQ7f9ACgx19dzyLXCDzkLkWITSU+7WyD XEMAn38mZgK8F1/FGWJc+ANOJz2tfHI/ =qpSL -----END PGP SIGNATURE-----

Lennart Regebro writes:

Base64 is an encoding that transforms between 8-bit streams. Let it be that. Don't try to shoehorn it into a completely different kind of encoding.

By "completely different kind of encoding" do you mean "codec"? I think that would be an unfortunate result. These operations on streams are theoretically nicely composable. It would be nice if practice reflected that by having a uniform API for all of these operations (charset translation, encoded text to internal, content transfer encoding, compression ...). I think it would be useful, too, though I can't prove that. Anyway, this discussion belongs on python-ideas at this point. Or would, if I had an idea about implementation. I'll take it there when I do have something to say about implementation.

On Thu, Apr 25, 2013 at 4:57 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

Lennart Regebro writes:

...

Base64 is an encoding that transforms between 8-bit streams. Let it be that. Don't try to shoehorn it into a completely different kind of encoding.

By "completely different kind of encoding" do you mean "codec"?

I think that would be an unfortunate result. These operations on streams are theoretically nicely composable. It would be nice if practice reflected that by having a uniform API for all of these operations (charset translation, encoded text to internal, content transfer encoding, compression ...). I think it would be useful, too, though I can't prove that.

Anyway, this discussion belongs on python-ideas at this point. Or would, if I had an idea about implementation. I'll take it there when I do have something to say about implementation.

Bringing the mailing list thread up to date with the state of the relevant tracker issues: I created http://bugs.python.org/issue17827 to cover adding the missing documentation for "codecs.encode" and "codecs.decode" as the officially supported solutions for easy use of the codec infrastructure *without* the additional text model specific input and output type restrictions imposed by the str.encode, bytes.decode and bytearray.decode methods. I created http://bugs.python.org/issue17828 to cover emitting more meaningful exceptions when a codec throws TypeError or ValueError, as well as when the additional type checking fails for str.encode, bytes.decode and bytearray.decode. I created http://bugs.python.org/issue17839 to cover the fact that part of the problem here is that the base64 module currently only accepts bytes and bytearray as inputs, rather than anything that supports the PEP 3118 buffer interface. http://bugs.python.org/issue7475 (linked earlier in the thread) is now strictly about restoring the shorthand aliases for "base64_codec", "bz2_codec" et al that were removed in http://bugs.python.org/issue10807. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Apr 22, 2013, at 10:04 PM, Steven D'Aprano <steve@pearwood.info> wrote:

On 23/04/13 09:16, Greg Ewing wrote:

...

Victor Stinner wrote:

...

The last proposition is to add transform() and untransform() methods to bytes and str types. ... If I remember correctly, the missing point is how to define which types are supported by a codec

Also, for any given codec, which direction is "transform" and which is "untransform"?

Also also, what's so special about base64 et al that they deserve an ultra-special way of invoking them, instead of having to import a class or function like you do for *every* *other* piece of library functionality?

As others have pointed out in the past, repeatedly, the codec system is completely general and can transform bytes->bytes and text->text just as easily as bytes<->text. Or indeed any bijection, as the docs for 2.7 point out. The question isn't "What's so special about base64?" The questions should be:

- What's so special about exotic legacy transformations like ISO-8859-10 and MacRoman that they deserve a string method for invoking them?

- Why have common transformations like base64, which worked in 2.x, been relegated to second-class status in 3.x?

- If it is no burden to have to import a module and call an external function for some transformations, why have encode and decode methods at all?

If you haven't read this, you should:

http://lucumr.pocoo.org/2012/8/11/codec-confusion/

I may be dull, but it wasn't until I started using Python 3 that it really clicked in my head what encode/decode did exactly. In Python2 I just sort of sprinkled one or the other when there was errors until the pain stopped. I mostly attribute this to str.decode and bytes.encode not existing.

-- Steven _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Mon, Apr 22, 2013 at 7:04 PM, Steven D'Aprano <steve@pearwood.info> wrote:

As others have pointed out in the past, repeatedly, the codec system is completely general and can transform bytes->bytes and text->text just as easily as bytes<->text. Or indeed any bijection, as the docs for 2.7 point out. The question isn't "What's so special about base64?" The questions should be:

- What's so special about exotic legacy transformations like ISO-8859-10 and MacRoman that they deserve a string method for invoking them?

- Why have common transformations like base64, which worked in 2.x, been relegated to second-class status in 3.x?

- If it is no burden to have to import a module and call an external function for some transformations, why have encode and decode methods at all?

There are good answers to all of these, and your rhetoric is not appreciated. The special status is for the translation between bytes and Unicode characters (code points). There are many contexts where a byte stream is labeled (either separately or in-line) as being encoded using some specific encoding. -- --Guido van Rossum (python.org/~guido)

Using decode() and encode() would break that predictability. But someone suggested the use of transform() and untransform() instead. That would clarify that the transformation is bytes > bytes and Unicode string > Unicode string. On 23 Apr 2013 05:50, "Lennart Regebro" <regebro@gmail.com> wrote:

On Tue, Apr 23, 2013 at 4:04 AM, Steven D'Aprano <steve@pearwood.info> wrote:

...

As others have pointed out in the past, repeatedly, the codec system is completely general and can transform bytes->bytes and text->text just as easily as bytes<->text.

Yes, but the encode()/decode() methods are not, and the fact that you now know what goes in and what comes out means that people get much fewer Decode/EncodeErrors. Which is a good thing.

//Lennart _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fabiosantosart%40gmail.com

On Mon, Apr 22, 2013 at 7:39 AM, Calvin Spealman <ironfroggy@gmail.com> wrote:

if two lines is cumbersome, you're in for a cumbersome life a programmer.

Other encodings are either missing completely from the stdlib, or have corrupted behavior. For example, string_escape is gone, and unicode_escape doesn't make any sense anymore -- python code is text, not bytes, so why does 'abc'.encode('unicode_escape') return bytes? I don't think this change was thought through completely before it was implemented. I agree base64 is a bad place to pick at the encode/decode changes, though. :( -- Devin

Devin Jeanpierre writes:

why does 'abc'.encode('unicode_escape') return bytes?

Duck-typing: encode always turns unicode into bytes.

On Tue, 23 Apr 2013 11:16:20 +1200, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Victor Stinner wrote:

...

The last proposition is to add transform() and untransform() methods to bytes and str types. ... If I remember correctly, the missing point is how to define which types are supported by a codec

Also, for any given codec, which direction is "transform" and which is "untransform"?

You transform *into* the encoding, and untransform *out* of the encoding. Do you have an example where that would be ambiguous?

Also also, what's so special about base64 et al that they deserve an ultra-special way of invoking them, instead of having to import a class or function like you do for *every* *other* piece of library functionality?

You can ask the same question about all the other codecs. (And that question has indeed been asked in the past.) (One answer is that they used to work in Python2...but the longer we go without restoring the functionality to Python3, the weaker that particular argument becomes.) --David

On Apr 22, 2013, at 06:22 PM, Guido van Rossum wrote:

...

You can ask the same question about all the other codecs. (And that question has indeed been asked in the past.)

Except for rot13. :-)

The fact that you can do this instead *is* a bit odd. ;) from codecs import getencoder encoder = getencoder('rot-13') r13 = encoder('hello world')[0] -Barry

On Tue, Apr 23, 2013 at 8:22 AM, M.-A. Lemburg <mal@egenix.com> wrote:

Just as reminder: we have the general purpose encode()/decode() functions in the codecs module:

import codecs r13 = codecs.encode('hello world', 'rot-13')

These interface directly to the codec interfaces, without enforcing type restrictions. The codec defines the supported input and output types.

As an implementation mechanism I see nothing wrong with this. I hope the codecs module lets you introspect the input and output types of a codec given by name? -- --Guido van Rossum (python.org/~guido)

On Tue, Apr 23, 2013 at 9:04 AM, M.-A. Lemburg <mal@egenix.com> wrote:

On 23.04.2013 17:47, Guido van Rossum wrote:

...

On Tue, Apr 23, 2013 at 8:22 AM, M.-A. Lemburg <mal@egenix.com> wrote:

...

Just as reminder: we have the general purpose encode()/decode() functions in the codecs module:

import codecs r13 = codecs.encode('hello world', 'rot-13')

These interface directly to the codec interfaces, without enforcing type restrictions. The codec defines the supported input and output types.

As an implementation mechanism I see nothing wrong with this. I hope the codecs module lets you introspect the input and output types of a codec given by name?

At the moment there is no standard interface to access supported input and output types... but then: regular Python functions or methods also don't provide such functionality, so no surprise there ;-)

Not quite the same though. Each function has its own unique behavior. But codecs support a standard interface, *except* that the input and output types sometimes vary.

It's mostly a matter of specifying the supported type combinations in the codec documentation.

BTW: What would be a use case where you'd want to programmatically access such information before calling the codec ?

As you know, in Python 3, most code working with bytes doesn't also work with strings, and vice versa (except for a few cases where we've gone out of our way to write polymorphic code -- but users rarely do so, and any time you use a string or bytes literal you basically limit yourself to that type). Suppose I write a command-line utility that reads a file, runs it through a codec, and writes the result to another file. Suppose the name of the codec is a command-line argument (as well as the filenames). I need to know whether to open the files in text or binary mode based on the name of the codec. -- --Guido van Rossum (python.org/~guido)

On 4/24/2013 1:22 AM, M.-A. Lemburg wrote:

On 23.04.2013 19:24, Guido van Rossum wrote:

...

On Tue, Apr 23, 2013 at 9:04 AM, M.-A. Lemburg <mal@egenix.com> wrote:

...

...

...

Just as reminder: we have the general purpose encode()/decode() functions in the codecs module:

import codecs r13 = codecs.encode('hello world', 'rot-13')

These interface directly to the codec interfaces, without enforcing type restrictions. The codec defines the supported input and output types. As an implementation mechanism I see nothing wrong with this. I hope

On Tue, Apr 23, 2013 at 8:22 AM, M.-A. Lemburg <mal@egenix.com> wrote: the codecs module lets you introspect the input and output types of a codec given by name? At the moment there is no standard interface to access supported input and output types... but then: regular Python functions or methods also don't provide such functionality, so no surprise

On 23.04.2013 17:47, Guido van Rossum wrote: there ;-) Not quite the same though. Each function has its own unique behavior. But codecs support a standard interface, *except* that the input and output types sometimes vary. The codec system itself

...

...

It's mostly a matter of specifying the supported type combinations in the codec documentation.

BTW: What would be a use case where you'd want to programmatically access such information before calling the codec ? As you know, in Python 3, most code working with bytes doesn't also work with strings, and vice versa (except for a few cases where we've gone out of our way to write polymorphic code -- but users rarely do so, and any time you use a string or bytes literal you basically limit yourself to that type).

Suppose I write a command-line utility that reads a file, runs it through a codec, and writes the result to another file. Suppose the name of the codec is a command-line argument (as well as the filenames). I need to know whether to open the files in text or binary mode based on the name of the codec. Ok, so you need to know which codecs your tool can support and which of those need text input and which bytes input.

I've been thinking about this some more: I think that type information alone is not flexible enough to cover such use cases.

Maybe MIME type and encoding would be sufficient type information, but probably not str vs. bytes.

In your use case you'd want to only permit use of a certain set of codecs, not simply all of them, since some might not implement what you actually want to achieve with the tool, e.g. a user might have installed a codec set that adds support for reading and writing image data, but your intended use was to only support text data.

MIME type supports this sort of concept, with the two-level hierarchy of naming the type... text/xml text/plain image/jpeg

So what we need is a way to allow the codecs to say e.g. "I work on text", "I support encoding bytes and text", "I encode to bytes", "I'm reversible", "I transform input data", "I support bytes and text, and will create same type output", "I work on image data", "I work on X509 certificates", "I work on XML data", etc.

Guess what I think you are re-inventing here.... Nope, guess again.... Yep, MIME types _plus_ encodings.

In other words, we need a form of tagging system, with a set of standard tags that each codec can publish and which also allows non-standard tags (which can then at some point be made standard, if there's agreement on them).

Hmm. Sounds just like the registry for, um, you guessed it: MIME types.

Given a codec name you could then ask the codec registry for the codec tags and verify that the chosen codec handles text data, needs bytes or text encoding input and creates bytes as encoding output. If the registry returns codec tags that don't include the "I work on text" tag, the tool could then raise an error.

For just doing text encoding transformations, text/plain would work as a MIME type, and the encodings of interest for the encodings. Seems like "str" always means "Unicode" but the MIME type can vary; "bytes" might mean encoded text, and the MIME type can also vary. For non-textual transformations, "encoding" might mean Base 64, BinHex, or other such representations... but those can also be applied to text, so it might be a 3rd dimension, or it might just be a list of encodings rather than a single encoding. Compression could be another dimension, or perhaps another encoding. But really, then, a transformation needs to be a list of steps; a codec can sign up to perform one or more of the steps, a sequence of codecs would have to be found, capable of performing a subsequence of the steps, and then run in the appropriate order. This all sounds so general, that probably the Python compiler could be implemented as a codec :) Or any compiler. Probably a web server could be implemented as a codec too :) Well, maybe not, codecs have limited error handling and reporting abilities.

On 23.04.2013 23:37, Nick Coghlan wrote:

On 24 Apr 2013 01:25, "M.-A. Lemburg" <mal@egenix.com> wrote:

...

On 23.04.2013 17:15, Barry Warsaw wrote:

...

On Apr 22, 2013, at 06:22 PM, Guido van Rossum wrote:

...

...

You can ask the same question about all the other codecs. (And that question has indeed been asked in the past.)

Except for rot13. :-)

The fact that you can do this instead *is* a bit odd. ;)

from codecs import getencoder encoder = getencoder('rot-13') r13 = encoder('hello world')[0]

Just as reminder: we have the general purpose encode()/decode() functions in the codecs module:

import codecs r13 = codecs.encode('hello world', 'rot-13')

These interface directly to the codec interfaces, without enforcing type restrictions. The codec defines the supported input and output types.

If we already have those, why aren't they documented?

Good question. I added them in 2004 and probably just forgot to add the documentation: http://hg.python.org/cpython-fullhistory/rev/8ea2cb1ec598 I guess the doc-strings could be used as basis for the documentation. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 24 2013)

...

...

Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2013-04-17: Released eGenix mx Base 3.2.6 ... http://egenix.com/go43 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Tue, 23 Apr 2013 22:29:33 +0900, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:

R. David Murray writes:

...

You transform *into* the encoding, and untransform *out* of the encoding. Do you have an example where that would be ambiguous?

In the bytes-to-bytes case, any pair of character encodings (eg, UTF-8 and ISO-8859-15) would do. Or how about in text, ReST to HTML?

If I write: bytestring.transform('ISO-8859-15') that would indeed be ambiguous, but only because I haven't named the source encoding of the bytestring. So the above is obviously nonsense, and the easiest "fix" is to have the things that are currently bytes-to-text or text-to-bytes character set transformations *only* work with encode/decode, and not transform/untransform.

BASE64 itself is ambiguous. By RFC specification, BASE64 is a *textual* representation of arbitrary binary data. (Cf. URIs.) The natural interpretation of .encode('base64') in that context would be as a bytes-to-text encoder. However, this has several problems. In practice, we invariably use an ASCII octet stream to carry BASE64- encoded data. So web developers would almost certainly expect a bytes-to-bytes encoder. Such a bytes-to-bytes encoder can't be duck-typed. Double-encoding bugs wouldn't be detected until the stream arrives at the user. And the RFC-based signature of .encode('base64') as bytes-to-text is precisely opposite to that of .encode('utf-8') (text-to-bytes).

I believe that after much discussion we have settled on these transformations (in their respective modules) accepting either bytes or strings as input for decoding, only bytes as input for encoding, and *always* producing bytes as output. (Note that the base64 docs need some clarification about this.) Given this, the possible valid transformations would be: bytestring.transform('base64') bytesstring.untransform('base64') string.untransform('base64') and all would produce a byte string. That byte string would be in base64 for the first one, and a decoded binary string for the second two. Given our existing API, I don't think we want string.encode('base64') to work (taking an ascii-only unicode string and returning bytes), and we've already agreed that adding a 'decode' method to string is not going to happen. We could, however, and quite possibly should, disallow string.untransform('base64') even though the underly module supports it. Thus we would only have bytes-to-bytes transformations for 'base64' and its siblings, and you would write the unicode-ascii-to-bytes transformation as: string.encode('ascii').untransform('base64') which has some pedagogical value :). If you do transform('base64') on a bytestring already encoded as base64 you get a double encoding, yes. I don't see that it is our responsibility to try to protect you from this mistake. The module functions certainly don't. Given that, is there anything ambiguous about the proposed API? (Note: if you would like to argue that, eg, base64.b64encode or binascii.b2a_base64 should return a string, it is too late for that argument for backward compatibility reasons.)

It is certainly true that there are many unambiguous cases. In the case of a true text processing facility (eg, Emacs buffers or Python 3 str) where there is an unambiguous text type with a constant and opaque internal representation, it makes a lot of sense to treat the text type as special/central, and use the terminology "encode [from text]" and "decode [to text]". It's easy to remember, which one is special is obvious, and the difference in input and output types means that mistaken use of the API will be detected by duck-typing.

However, in the case of bytes-bytes or text-text transformations, it's not the presence of unambiguous cases that should drive API design IMO. It's the presence of the ambiguous cases that we should cater to. I don't see easy solutions to this issue.

When I asked about ambiguous cases, I was asking for cases where the meaning of "transform('somecodec')" was ambiguous. Sure, it is possible to feed the wrong input into that transformation, but I consider that a programming error, not an ambiguity in the API. After all, you have exactly the same problem if you use the module functions directly, which is currently the only option. --David

R. David Murray writes:

On Tue, 23 Apr 2013 22:29:33 +0900, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:

...

R. David Murray writes:

...

You transform *into* the encoding, and untransform *out* of the encoding. Do you have an example where that would be ambiguous?

In the bytes-to-bytes case, any pair of character encodings (eg, UTF-8 and ISO-8859-15) would do. Or how about in text, ReST to HTML?

If I write:

bytestring.transform('ISO-8859-15')

that would indeed be ambiguous, but only because I haven't named the source encoding of the bytestring. So the above is obviously nonsense, and the easiest "fix" is to have the things that are currently bytes-to-text or text-to-bytes character set transformations *only* work with encode/decode, and not transform/untransform.

I think you're completely missing my point here. The problem is that in the cases I mention, what is encoded data and what is decoded data can only be decided by asking the user.

I believe that after much discussion we have settled on these transformations (in their respective modules) accepting either bytes or strings as input for decoding, only bytes as input for encoding, and *always* producing bytes as output.

Which, of course, is quite broken from the point of view of the RFC! Of course, the RFC be damned[1], for the purposes of the Python stdlib, the specific codecs used for Content-Transfer-Encoding have a clear intuitive directionality, and their encoding methods should turn bytes into bytes (and str or bytes into bytes on decoding). Nevertheless, it's not TOOWTDI, it's a careful compromise.

Given this, the possible valid transformations would be:

bytestring.transform('base64') bytesstring.untransform('base64') string.untransform('base64')

Which is an obnoxious API, since (1) you've now made it impossible to use "transform" for bytestring.transform(from='utf-8', to='iso-8859-1') bytestring.transform(from='ulaw', to='mp3') textstring.transform(from='rest', to='html') without confusion, and (2) the whole world is going to wonder why you don't use .encode and .decode instead of .transform and .untransform. The idea in the examples is that we could generalize the codec registry to look up codecs by pairs of media-types. I'm not sure this makes sense ... much of the codec API presumes a stream, especially the incremental methods. But many MIME media types are streams only because they're serializations, incremental en/decoding is nonsense. So I suppose I would want to write bytestring.transform(from='octet-stream', to='BASE64') for this hypothetical API. (I suspect that in practice the 'application/octet-stream' media type would be spelled 'bytes', of course.) This kind of API could be used to improve the security of composition of transforms. In the case of BASE64, it would make sense to match anything at all as the other type (as long as it's represented in Python by a bytes object). So it would be possible to do object = bytestring.transform(from='BASE64', to='PNG') giving object a media_type attribute such that object.decode('iso-8859-1') would fail. (This would require changes to the charset codecs, to pay heed to the media_type attribute, so it's not immediately feasible.)

and all would produce a byte string. That byte string would be in base64 for the first one, and a decoded binary string for the second two.

Given our existing API, I don't think we want

string.encode('base64')

to work (taking an ascii-only unicode string and returning bytes),

No, we don't, but for reasons that have little to do with "ASCII-only". The problem with supporting that idiom is that *people can't read strs* [in the Python 3 internal representation] -- they can only read a str that has been encoded implicitly into the PYTHONIOENCODING or explicitly to an explicitly requested encoding. So the usage above is clearly ambiguous. Even if it is ASCII-only, in theory the user could want EBCDIC.

If you do transform('base64') on a bytestring already encoded as base64 you get a double encoding, yes. I don't see that it is our responsibility to try to protect you from this mistake. The module functions certainly don't.

Given that, is there anything ambiguous about the proposed API?

Not for BASE64. But what's so special about BASE64 that it deserves a new method name for the same old idiom, using a word that's an obvious candidate for naming a more general idiom?

(Note: if you would like to argue that, eg, base64.b64encode or binascii.b2a_base64 should return a string, it is too late for that argument for backward compatibility reasons.)

Even if it weren't too late, the byte-shoveling lobby is way too strong; that's not a winnable agument.

When I asked about ambiguous cases, I was asking for cases where the meaning of "transform('somecodec')" was ambiguous.

If "transform('somecodec')" isn't ambiguous, you really really want to spell it "encode" instead of "transform" IMO. Even though I don't see how to do that without generating more confusion than it's worth at the moment, I still harbor the hope that somebody will come up with a way to do it so everything still fits together. Footnotes: [1] I am *not* one to damn RFCs lightly!

R. David Murray writes:

...

I think you're completely missing my point here. The problem is that in the cases I mention, what is encoded data and what is decoded data can only be decided by asking the user.

I think I understood that. I don't understand why that's a problem.

It's a problem because in that case it's hard for users to remember the directionality of the codec based only on a single name; the API needs to indicate what is being transformed to what else.

I've been trying to explain what I thought the transform/untransform proposal was: a minimalist extension of the encode/decode semantic (under a different name) so that functionality that was lost from Python2 encode/decode could be restored to Python3 in a reasonably understandable way.

I think that the intention of the proposal is reasonably understandable, and reasonable. I just don't think the API proposed is understandable, and therefore it's not reasonable.<wink/>

People (at least you and Nick, and maybe Guido) seem to be more interested in a more general/powerful mechanism. I'm fine with that :)

I can't speak to the opinions of people who actually know about language design. For myself, I'm sympathetic to the proposal of a specific API limited to cases where the directionality is clear as a generality. I just don't think the "transform" proposal helps much, partly because the actual applications are few, and partly because "transform" is more ambiguous (to be unambiguous in English, you need both the direct object ("from media type") and the indirect object ("to media type") specified. It is quite possible to say "transform encoded text to raw text" or similar. At least for me, "encode transformed text to raw text" raises a WTFAssertion. I know that I've experienced worlds of pain in the character coding sphere from Emacs APIs and UIs that don't indicate directionality clearly. This is very delicate; GNU Emacs had an ugly bug that regressed multiple times over more than a decade merely because they exposed the internal representation of text to Lisp. XEmacs has never experienced that bug (to be precise, the presence of that kind of bug resulted in an immediate assertion, so it was eliminated early in development). Surprisingly to me, the fact that XEmacs uses the internal representation of *text* to also represent "byte streams" (with bytes of variable width!) has never caused me confusion. It does cause others confusion, though, so although the XEmacs model of text is easier to work with than Emacs's, I tend to think Python 3's (which never confounds text with bytes) is better. I suspect that delicacy extends to non-character transformations, so I am pretty demanding about proposals in this area. Specifically I insist on EIBTI and TOOWTDI.

Terry Jan Reedy writes:

.transform should be explicit and always take two args, no implicit defaults, the 'from form' and the 'to' form. They can labelled by position in the natural order (from, to)

Not natural to escaped-from-C programmers, though. I hesitate to say "make it keywords-only", but using keywords should be *strongly* encouraged.

str.transform would always be unicode to unicode and bytes.transform always bytes to bytes.

Which leaves the salient cases (MIME content transfer encodings) out in the cold, although I guess string.encode('ascii').transform(from='base64', to='bytes') isn't too horrible.

Greg Ewing writes:

Web developers might grumble about the need for an extra call, but they can no longer claim it would kill the performance of their web server.

Of course they can. There never was any performance measurement that supported that claim in the first place. I don't see how PEP 393 makes a difference to them. The real problem for them is that conceptually they think ASCII in byte form *is* text, and they want to do text processing on it. They'll use any flimsy excuse to avoid a transform to str, because it's just unbearably ugly given their givens. I have sympathy for their position, I just (even today) think it's the wrong thing for Python. However, I've long since been overruled, and I have no evidence to justify saying "I told you so".<wink/>

Tres Seaver writes:

On 04/23/2013 09:29 AM, Stephen J. Turnbull wrote:

...

By RFC specification, BASE64 is a *textual* representation of arbitrary binary data.

It isn't "text" in the sense Py3k means:

RFC 4648 repeatedly refers to *characters*, without specifying an encoding for them. In fact, if you copy accurately, you can write BASE64 on a napkin and that napkin will accurate transmit the data (assuming it doesn't run into sleet or gloom of night). What else is that but "text in the sense of Py3k"? My point is not that Python's base64 codec *should* be bytes-to-str and back. My point is that, both in the formal spec and in historical evolution, that is a plausible interpretation of ".encode('base64')" which happens to be the reverse of the normal codec convention, where ".encode(codec)" is a *string* method, and ".decode(codec)" is a *bytes* method. This is not harder to learn for people (for BASE64 encoding or for coded character sets), because in each case there's a natural sense of direction for *en*coding vs. *de*coding. But it does break duck- typing, as does the web developer bytes-to-bytes usage of BASE64. What I'm groping toward is an idea of a "variable method", so that we could use .encode and .decode where they are TOOWTDI for people even though a purely formal interpretation of duck-typing would say "but why is that blue whale quacking, waddling, and flying?" In other words (although I have no idea how best to implement it), I would like "somestring.encode('base64')" to fail with "I don't know how to do that" (an attribute lookup error?), the same way that "somebytes.encode('utf-8')" does in Python 3 today.

On Thu, 25 Apr 2013 04:19:36 +0200 Lennart Regebro <regebro@gmail.com> wrote:

On Thu, Apr 25, 2013 at 3:54 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

...

RFC 4648 repeatedly refers to *characters*, without specifying an encoding for them. [...]

Base64 is an encoding that transforms between 8-bit streams.

No, it isn't. What Stephen wrote above.

Either you get a "LookupError: unknown encoding: base64", which is what you get now, or you get an UnicodeEncodingError if the text is not ASCII. We don't want the latter, because it means that code that looks fine for the developer breaks in real life because the developer was American

That's bogus. By the same argument, we should suppress any encoding which isn't able to represent all possible unicode strings. That's almost all encodings provided by Python (including utf-8, if you consider lone surrogates). I'm sorry for Americans, but they *still* must know about character encodings, and be ready to handle UnicodeErrors, when using Python 3 for encoding/decoding bytestrings. There's no way around it. Regards Antoine.

Le Thu, 25 Apr 2013 08:38:12 +0200, Lennart Regebro <regebro@gmail.com> a écrit :

On Thu, Apr 25, 2013 at 7:43 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:

...

On Thu, 25 Apr 2013 04:19:36 +0200 Lennart Regebro <regebro@gmail.com> wrote:

...

On Thu, Apr 25, 2013 at 3:54 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

...

RFC 4648 repeatedly refers to *characters*, without specifying an encoding for them. [...]

Base64 is an encoding that transforms between 8-bit streams.

No, it isn't. What Stephen wrote above.

Yes it is. Base64 takes 8-bit bytes and transforms them into another 8-bit stream that can be safely transmitted over various channels that would mangle an unencoded 8-bit stream, such as email etc.

http://en.wikipedia.org/wiki/Base64

I don't see anything in that Wikipedia page that validates your opinion. The Wikipedia page does talk about *text* and *characters* for the result of base64 encoding. Besides, I would consider a RFC more authoritative than a Wikipedia definition.

...

By the same argument, we should suppress any encoding which isn't able to represent all possible unicode strings.

No, if you explicitly use such an encoding it is because you need to because you are transferring data to a system that needs the encoding in question. Unicode errors are unavoidable at that point, not an unexpected surprise because a conversion happened implicitly that you didn't know about.

I don't know what "implicit conversion" you are talking about. There's no "implicit conversion" in a scheme where the result of base64 encoding is a text string. Regards Antoine.

Le Thu, 25 Apr 2013 12:05:01 +0200, Lennart Regebro <regebro@gmail.com> a écrit :

...

The Wikipedia page does talk about *text* and *characters* for the result of base64 encoding.

So are saying that you want the Python implementation of base64 encoding to take 8-bit binary data in bytes format and return a Unicode string containing the Base64 encoded data?

I'm not wanting anything here, since that would clearly break backwards compatibility. But I think binascii should have gone that way in Python 3, indeed. binascii.b2a_hex(), for example, would be much more practical if it returned str, rather than bytes.

Python 3 still views text as Unicode only.

Python 3 doesn't *view* text as unicode, it *represents* it as unicode. That is, unicode is the character set that Python 3 is able to represent in the canonical text type, str. If you ever encounter a hypothetical text that uses characters outside of Unicode (obviously it will be encoded using a non-unicode encoding :-)), then you can't represent it as a str. And base64 is clearly representable as unicode, since it's representable using the ASCII character set (which is a subset of the unicode character set).

I can't think of any usecase where you would want base64 encoded data unless you intend to transmit it over an 8-bit channel,

I can think of many usecases where I want to *embed* base64-encoded data in a larger text *before* encoding that text and transmitting it over a 8-bit channel. (GPG signatures, binary data embedded in JSON objects, etc.) Regards Antoine.

On Apr 25, 2013, at 03:34 PM, Lennart Regebro wrote:

In the case of JSON objects, they are intended for data exchange, and hence in the end need to be byte strings.

Except that they're not. http://bugs.python.org/issue10976 -Barry

On 25/04/2013 14:34, Lennart Regebro wrote:

On Thu, Apr 25, 2013 at 2:57 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:

...

I can think of many usecases where I want to *embed* base64-encoded data in a larger text *before* encoding that text and transmitting it over a 8-bit channel.

That still doesn't mean that this should be the default behavior. Just because you *can* represent base64 as Unicode text doesn't mean that it should be.

...

(GPG signatures, binary data embedded in JSON objects, etc.)

Is the GPG signature calculated on the *Unicode* data? How is that done? Isn't it done on the encoded message? As I understand it a GPG signature is done on any sort of document. Either me or you have completely misunderstood how GPG works, I think. :-)

In the case of JSON objects, they are intended for data exchange, and hence in the end need to be byte strings. So if you have a byte string you want to base64 encode before transmitting it with json, you would just end up transforming it to a unicode string and then back. That doesn't seem useful.

The JSON specification says that it's text. Its string literals can contain Unicode codepoints. It needs to be encoded to bytes for transmission and storage, but JSON itself is not a bytestring format.

One use case where you clearly *do* want the base64 encoded data to be unicode strings is because you want to embed it in a text discussing base64 strings, for a blog or a book or something. That doesn't seem to be a very common usecase.

For the most part you base64 encode things because it's going to be transmitted, and hence the natural result of a base64 encoding should be data that is ready to be transmitted, hence byte strings, and not Unicode strings.

...

Python 3 doesn't *view* text as unicode, it *represents* it as unicode.

I don't agree that there is a significant difference between those wordings in this context. The end result is the same: Things intended to be handled/seen as textual should be unicode strings, things intended for data exchange should be byte strings. Something that is base64 encoded is primarily intended for data exchange. A base64 encoding should therefore return byte strings, especially since most API's that perform this transmission will take byte strings as input. If you want to include this in textual data, for whatever reason, like printing it in a book, then the conversion is trivial, but that is clearly the less common use case, and should therefore not be the default behavior.

base64 is a way of encoding binary data as text. The problem is that traditionally text has been encoded with one byte per character, except in those locales where there were too many characters in the character set for that to be possible. In Python 3 we're trying to stop mixing binary data (bytestrings) with text (Unicode strings).

On Thu, 25 Apr 2013, Lennart Regebro wrote:

On Thu, Apr 25, 2013 at 4:22 PM, MRAB <python@mrabarnett.plus.com> wrote:

...

The JSON specification says that it's text. Its string literals can contain Unicode codepoints. It needs to be encoded to bytes for transmission and storage, but JSON itself is not a bytestring format.

OK, fair enough.

...

base64 is a way of encoding binary data as text.

It's a way of encoding binary data using ASCII. There is a subtle but important difference.

It is a way of encoding arrays of 8-bit bytes as arrays of characters that are part of the printable, non-whitespace subset of the ASCII repertoire. Since the ASCII repertoire is now simply the first 128 code points in the Unicode repertoire, it is equally correct to say that base64 is a way of encoding binary data as Unicode text.

...

In Python 3 we're trying to stop mixing binary data (bytestrings) with text (Unicode strings).

Yup. And that's why a byte64 encoding shouldn't return Unicode strings.

That is exactly why it should return Unicode strings. What bytes should get sent if base64 is used to send a byte array over an EBCDIC link? [*] Having said that, there may be other reasons for base64 encoding to return bytes - I can conceive of arguments involving efficiency, or practicality, or the most common use cases. So I can't say for sure what base64 encoding actually ought to return in Python. But the purist stance should be that base64 encoding should return text, i.e. a string, i.e. unicode. [*] I apologize to anybody who just ate. Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist

On 25/04/2013 15:22, MRAB wrote:

On 25/04/2013 14:34, Lennart Regebro wrote:

...

On Thu, Apr 25, 2013 at 2:57 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:

...

I can think of many usecases where I want to *embed* base64-encoded data in a larger text *before* encoding that text and transmitting it over a 8-bit channel.

That still doesn't mean that this should be the default behavior. Just because you *can* represent base64 as Unicode text doesn't mean that it should be.

[snip]

...

One use case where you clearly *do* want the base64 encoded data to be unicode strings is because you want to embed it in a text discussing base64 strings, for a blog or a book or something. That doesn't seem to be a very common usecase.

For the most part you base64 encode things because it's going to be transmitted, and hence the natural result of a base64 encoding should be data that is ready to be transmitted, hence byte strings, and not Unicode strings.

...

Python 3 doesn't *view* text as unicode, it *represents* it as unicode.

I don't agree that there is a significant difference between those wordings in this context. The end result is the same: Things intended to be handled/seen as textual should be unicode strings, things intended for data exchange should be byte strings. Something that is base64 encoded is primarily intended for data exchange. A base64 encoding should therefore return byte strings, especially since most API's that perform this transmission will take byte strings as input. If you want to include this in textual data, for whatever reason, like printing it in a book, then the conversion is trivial, but that is clearly the less common use case, and should therefore not be the default behavior.

base64 is a way of encoding binary data as text. The problem is that traditionally text has been encoded with one byte per character, except in those locales where there were too many characters in the character set for that to be possible.

In Python 3 we're trying to stop mixing binary data (bytestrings) with text (Unicode strings).

RFC 4648 says """Base encoding of data is used in many situations to store or transfer data in environments that, perhaps for legacy reasons, are restricted to US-ASCII [1] data.""". To me, "US-ASCII" is an encoding, so it appears to be talking about encoding binary data (bytestrings) to ASCII-encoded text (bytestrings).

Le Thu, 25 Apr 2013 15:34:45 +0200, Lennart Regebro <regebro@gmail.com> a écrit :

I don't agree that there is a significant difference between those wordings in this context. The end result is the same: Things intended to be handled/seen as textual should be unicode strings, things intended for data exchange should be byte strings.

I don't think this distinction is meaningful at all. In the end, everything is a byte string on a classical computer (including unicode strings displayed on your monitor, obviously). If you think the technicalities of an operation should never be hidden or abstracted away, then you're better off with C than Python ;-) Regards Antoine.

On 2013-04-25, at 11:25 , Antoine Pitrou wrote:

Besides, I would consider a RFC more authoritative than a Wikipedia definition.

Base encoding of data is used in many situations to store or transfer data in environments that, perhaps for legacy reasons, are restricted to US-ASCII [1] data.

so the output is US-ASCII data, a byte stream. Stephen is correct that you could decide you don't care about those semantics, and implement base64 encoding as a bytes -> str decoding then requiring a re-encoding (to ascii) before wire transmission. The clarity of the interface (or lack thereof) would probably make users want to send a strongly worded letter to whoever implemented it though, I don't think `data.decode('base64').encode('ascii')` would fit the "obviousness" or "readability" expectations of most users.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/25/2013 01:43 AM, Antoine Pitrou wrote:

On Thu, 25 Apr 2013 04:19:36 +0200 Lennart Regebro <regebro@gmail.com> wrote:

...

On Thu, Apr 25, 2013 at 3:54 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

...

RFC 4648 repeatedly refers to *characters*, without specifying an encoding for them. [...]

Base64 is an encoding that transforms between 8-bit streams.

No, it isn't. What Stephen wrote above.

Stephen was incorrect: the base64 standard is about encoding a binary stream (8-bit bites) onto another binary stream (6-bit bytes), but one which can be safely transmitted over a 7-bit-only medium. Text in Py3ks sense is irrelevant.

...

Either you get a "LookupError: unknown encoding: base64", which is what you get now, or you get an UnicodeEncodingError if the text is not ASCII. We don't want the latter, because it means that code that looks fine for the developer breaks in real life because the developer was American

That's bogus. By the same argument, we should suppress any encoding which isn't able to represent all possible unicode strings. That's almost all encodings provided by Python (including utf-8, if you consider lone surrogates).

I'm sorry for Americans, but they *still* must know about character encodings, and be ready to handle UnicodeErrors, when using Python 3 for encoding/decoding bytestrings. There's no way around it.

WHat does that snark have to do with this discussion? base64 has no more to do with character set encodings than it does the moon. It would be a "transform" (bytes -> bytes), not an "encoding". Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF5Nc4ACgkQ+gerLs4ltQ7f9ACgx19dzyLXCDzkLkWITSU+7WyD XEMAn38mZgK8F1/FGWJc+ANOJz2tfHI/ =qpSL -----END PGP SIGNATURE-----

Lennart Regebro writes:

Base64 is an encoding that transforms between 8-bit streams. Let it be that. Don't try to shoehorn it into a completely different kind of encoding.

By "completely different kind of encoding" do you mean "codec"? I think that would be an unfortunate result. These operations on streams are theoretically nicely composable. It would be nice if practice reflected that by having a uniform API for all of these operations (charset translation, encoded text to internal, content transfer encoding, compression ...). I think it would be useful, too, though I can't prove that. Anyway, this discussion belongs on python-ideas at this point. Or would, if I had an idea about implementation. I'll take it there when I do have something to say about implementation.

On Thu, Apr 25, 2013 at 4:57 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

Lennart Regebro writes:

...

Base64 is an encoding that transforms between 8-bit streams. Let it be that. Don't try to shoehorn it into a completely different kind of encoding.

By "completely different kind of encoding" do you mean "codec"?

I think that would be an unfortunate result. These operations on streams are theoretically nicely composable. It would be nice if practice reflected that by having a uniform API for all of these operations (charset translation, encoded text to internal, content transfer encoding, compression ...). I think it would be useful, too, though I can't prove that.

Anyway, this discussion belongs on python-ideas at this point. Or would, if I had an idea about implementation. I'll take it there when I do have something to say about implementation.

Bringing the mailing list thread up to date with the state of the relevant tracker issues: I created http://bugs.python.org/issue17827 to cover adding the missing documentation for "codecs.encode" and "codecs.decode" as the officially supported solutions for easy use of the codec infrastructure *without* the additional text model specific input and output type restrictions imposed by the str.encode, bytes.decode and bytearray.decode methods. I created http://bugs.python.org/issue17828 to cover emitting more meaningful exceptions when a codec throws TypeError or ValueError, as well as when the additional type checking fails for str.encode, bytes.decode and bytearray.decode. I created http://bugs.python.org/issue17839 to cover the fact that part of the problem here is that the base64 module currently only accepts bytes and bytearray as inputs, rather than anything that supports the PEP 3118 buffer interface. http://bugs.python.org/issue7475 (linked earlier in the thread) is now strictly about restoring the shorthand aliases for "base64_codec", "bz2_codec" et al that were removed in http://bugs.python.org/issue10807. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Apr 22, 2013, at 10:04 PM, Steven D'Aprano <steve@pearwood.info> wrote:

On 23/04/13 09:16, Greg Ewing wrote:

...

Victor Stinner wrote:

...

The last proposition is to add transform() and untransform() methods to bytes and str types. ... If I remember correctly, the missing point is how to define which types are supported by a codec

Also, for any given codec, which direction is "transform" and which is "untransform"?

Also also, what's so special about base64 et al that they deserve an ultra-special way of invoking them, instead of having to import a class or function like you do for *every* *other* piece of library functionality?

As others have pointed out in the past, repeatedly, the codec system is completely general and can transform bytes->bytes and text->text just as easily as bytes<->text. Or indeed any bijection, as the docs for 2.7 point out. The question isn't "What's so special about base64?" The questions should be:

- What's so special about exotic legacy transformations like ISO-8859-10 and MacRoman that they deserve a string method for invoking them?

- Why have common transformations like base64, which worked in 2.x, been relegated to second-class status in 3.x?

- If it is no burden to have to import a module and call an external function for some transformations, why have encode and decode methods at all?

If you haven't read this, you should:

http://lucumr.pocoo.org/2012/8/11/codec-confusion/

I may be dull, but it wasn't until I started using Python 3 that it really clicked in my head what encode/decode did exactly. In Python2 I just sort of sprinkled one or the other when there was errors until the pain stopped. I mostly attribute this to str.decode and bytes.encode not existing.

-- Steven _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Mon, Apr 22, 2013 at 7:04 PM, Steven D'Aprano <steve@pearwood.info> wrote:

As others have pointed out in the past, repeatedly, the codec system is completely general and can transform bytes->bytes and text->text just as easily as bytes<->text. Or indeed any bijection, as the docs for 2.7 point out. The question isn't "What's so special about base64?" The questions should be:

- What's so special about exotic legacy transformations like ISO-8859-10 and MacRoman that they deserve a string method for invoking them?

- Why have common transformations like base64, which worked in 2.x, been relegated to second-class status in 3.x?

- If it is no burden to have to import a module and call an external function for some transformations, why have encode and decode methods at all?

There are good answers to all of these, and your rhetoric is not appreciated. The special status is for the translation between bytes and Unicode characters (code points). There are many contexts where a byte stream is labeled (either separately or in-line) as being encoded using some specific encoding. -- --Guido van Rossum (python.org/~guido)

Using decode() and encode() would break that predictability. But someone suggested the use of transform() and untransform() instead. That would clarify that the transformation is bytes > bytes and Unicode string > Unicode string. On 23 Apr 2013 05:50, "Lennart Regebro" <regebro@gmail.com> wrote:

On Tue, Apr 23, 2013 at 4:04 AM, Steven D'Aprano <steve@pearwood.info> wrote:

...

As others have pointed out in the past, repeatedly, the codec system is completely general and can transform bytes->bytes and text->text just as easily as bytes<->text.

Yes, but the encode()/decode() methods are not, and the fact that you now know what goes in and what comes out means that people get much fewer Decode/EncodeErrors. Which is a good thing.

//Lennart _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fabiosantosart%40gmail.com

On Mon, Apr 22, 2013 at 7:39 AM, Calvin Spealman <ironfroggy@gmail.com> wrote:

if two lines is cumbersome, you're in for a cumbersome life a programmer.

Other encodings are either missing completely from the stdlib, or have corrupted behavior. For example, string_escape is gone, and unicode_escape doesn't make any sense anymore -- python code is text, not bytes, so why does 'abc'.encode('unicode_escape') return bytes? I don't think this change was thought through completely before it was implemented. I agree base64 is a bad place to pick at the encode/decode changes, though. :( -- Devin

Devin Jeanpierre writes:

why does 'abc'.encode('unicode_escape') return bytes?

Duck-typing: encode always turns unicode into bytes.