Why does base64 return bytes?
Normally I'd take a question like this to Python-List, but this question has turned out to be quite diversive, with people having strong opinions but no definitive answer. So I thought I'd ask here and hope that some of the core devs would have an idea. Why does base64 encoding in Python return bytes? base64.b64encode take bytes as input and returns bytes. Some people are arguing that this is wrong behaviour, as RFC 3548 specifies that Base64 should transform bytes to characters: https://tools.ietf.org/html/rfc3548.html albeit US-ASCII characters. E.g.: The encoding process represents 24-bit groups of input bits as output strings of 4 encoded characters. [...] Each 6-bit group is used as an index into an array of 64 printable characters. The character referenced by the index is placed in the output string. Are they misinterpreting the standard? Has Python got it wrong? Is there a good reason for returning bytes? I see that other languages choose different strategies. Microsoft's languages C#, F# and VB (plus their C++ compiler) take an array of bytes as input, and outputs a UTF-16 string: https://msdn.microsoft.com/en-us/library/dhx0d524%28v=vs.110%29.aspx Java's base64 encoder takes and returns bytes: https://docs.oracle.com/javase/8/docs/api/java/util/Base64.Encoder.html and Javascript's Base64 encoder takes input as UTF-16 encoded text and returns the same: https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encodin... I'm not necessarily arguing that Python's strategy is the wrong one, but I am interested in what (if any) reasons are behind it. Thanks in advance, Steve
On 14 June 2016 at 12:19, Steven D'Aprano <steve@pearwood.info> wrote:
Is there a good reason for returning bytes?
What about: it returns 0-255 numeric values for each position in a stream, with no clue whatsoever to how those values map to text characters beyond the 32-128 range? Maybe base64.decode could take a "encoding" optional parameter - or there could be a separate 'decote_to_text" method that would explicitly take a text codec name. Otherwise, no, you simply can't take a bunch of bytes and say they represent text. João (see ^- the "ã" ?)
On Jun 14, 2016 8:32 AM, "Joao S. O. Bueno" <jsbueno@python.org.br> wrote:
On 14 June 2016 at 12:19, Steven D'Aprano <steve@pearwood.info> wrote:
Is there a good reason for returning bytes?
What about: it returns 0-255 numeric values for each position in a
stream, with
no clue whatsoever to how those values map to text characters beyond the 32-128 range?
Maybe base64.decode could take a "encoding" optional parameter - or there could be a separate 'decote_to_text" method that would explicitly take a text codec name. Otherwise, no, you simply can't take a bunch of bytes and say they represent text.
Although it's not explicit, the question seems to be about the output of encoding (and for symmetry, the input of decoding). In both of those cases, valid output will consist only of ascii characters. The input to encoding would have to remain bytes (that's the main purpose of base64... to turn bytes into an ascii string). -Toshio
On 14 June 2016 at 13:32, Toshio Kuratomi <a.badger@gmail.com> wrote:
On Jun 14, 2016 8:32 AM, "Joao S. O. Bueno" <jsbueno@python.org.br> wrote:
On 14 June 2016 at 12:19, Steven D'Aprano <steve@pearwood.info> wrote:
Is there a good reason for returning bytes?
What about: it returns 0-255 numeric values for each position in a stream, with no clue whatsoever to how those values map to text characters beyond the 32-128 range?
Maybe base64.decode could take a "encoding" optional parameter - or there could be a separate 'decote_to_text" method that would explicitly take a text codec name. Otherwise, no, you simply can't take a bunch of bytes and say they represent text.
Although it's not explicit, the question seems to be about the output of encoding (and for symmetry, the input of decoding). In both of those cases, valid output will consist only of ascii characters.
The input to encoding would have to remain bytes (that's the main purpose of base64... to turn bytes into an ascii string).
Sorry, it is 2016, and I don't think at this point anyone can consider an ASCII string as a representative pattern of textual data in any field of application. Bytes are not text. Bytes with an associated, meaningful, encoding are text. I thought this had been through when Python 3 was out. Unless you are working with COBOL generated data (and intending to keep the file format) , it does not make sense in any real-world field. (supposing your Cobol data is ASCII and nort EBCDIC).
-Toshio
On Tue, Jun 14, 2016, at 13:05, Joao S. O. Bueno wrote:
Sorry, it is 2016, and I don't think at this point anyone can consider an ASCII string as a representative pattern of textual data in any field of application. Bytes are not text. Bytes with an associated, meaningful, encoding are text. I thought this had been through when Python 3 was out.
Of all the things that anyone has said in this thread, this makes the *least* contextual sense. The input to base64 encoding, which is what is under discussion, is not text in any way. It is images, it is zip files, it is executables, it could be the output of os.urandom (at least, provided it doesn't block ;) for all anyone cares. The *output* is only an ascii string in the sense that it is a text string consisting of characters within (a carefully chosen subset of) ASCII's repertoire, but the output wasn't what he was claiming should be bytes in the sentence you replied to. Is your objection to the phrase "ascii string"?
On 14 June 2016 at 14:45, Random832 <random832@fastmail.com> wrote:
On Tue, Jun 14, 2016, at 13:05, Joao S. O. Bueno wrote:
Sorry, it is 2016, and I don't think at this point anyone can consider an ASCII string as a representative pattern of textual data in any field of application. Bytes are not text. Bytes with an associated, meaningful, encoding are text. I thought this had been through when Python 3 was out.
Of all the things that anyone has said in this thread, this makes the *least* contextual sense. The input to base64 encoding, which is what is under discussion, is not text in any way. It is images, it is zip files, it is executables, it could be the output of os.urandom (at least, provided it doesn't block ;) for all anyone cares.
The *output* is only an ascii string in the sense that it is a text string consisting of characters within (a carefully chosen subset of) ASCII's repertoire, but the output wasn't what he was claiming should be bytes in the sentence you replied to. Is your objection to the phrase "ascii string"? Sorry - everything I wrote, I was thinking about _decoding_ base 64. As for the result of an encoded base64, yes, of course it fits into ASCII.
The arguments about compactness and what is most likely to happen next applies (transmission trhough a binary network protocol), but the strong objection I had was just because I thought it was a suggestion of decoding base 64 automatically to text without providing a text encoding.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br
Joao S. O. Bueno wrote:
The arguments about compactness and what is most likely to happen next applies (transmission trhough a binary network protocol),
I'm not convinced that this is what is most likely to happen next *in a Python program*. How many people implement their own binary network protocols in Python? It seems to me most people will be using a protocol library written by someone else. -- Greg
On Tue, 14 Jun 2016 14:05:19 -0300, "Joao S. O. Bueno" <jsbueno@python.org.br> wrote:
On 14 June 2016 at 13:32, Toshio Kuratomi <a.badger@gmail.com> wrote:
On Jun 14, 2016 8:32 AM, "Joao S. O. Bueno" <jsbueno@python.org.br> wrote:
On 14 June 2016 at 12:19, Steven D'Aprano <steve@pearwood.info> wrote:
Is there a good reason for returning bytes?
What about: it returns 0-255 numeric values for each position in a stream, with no clue whatsoever to how those values map to text characters beyond the 32-128 range?
Maybe base64.decode could take a "encoding" optional parameter - or there could be a separate 'decote_to_text" method that would explicitly take a text codec name. Otherwise, no, you simply can't take a bunch of bytes and say they represent text.
Although it's not explicit, the question seems to be about the output of encoding (and for symmetry, the input of decoding). In both of those cases, valid output will consist only of ascii characters.
The input to encoding would have to remain bytes (that's the main purpose of base64... to turn bytes into an ascii string).
Sorry, it is 2016, and I don't think at this point anyone can consider an ASCII string as a representative pattern of textual data in any field of application. Bytes are not text. Bytes with an associated, meaningful, encoding are text. I thought this had been through when Python 3 was out.
Unless you are working with COBOL generated data (and intending to keep the file format) , it does not make sense in any real-world field. (supposing your Cobol data is ASCII and nort EBCDIC).
The fundamental purpose of the base64 encoding is to take a series of arbitrary bytes and reversibly turn them into another series of bytes in which the eighth bit is not significant. Its utility is for transmitting eight bit bytes over a channel that is not eight bit clean. Before unicode, that meant bytes. Now that we have unicode in use in lots of places, you can think of unicode as a communications channel that is not eight bit clean. So, we might want to use base64 encoding to transmit arbitrary bytes over a unicode channel. This gives a legitimate reason to want unicode output from a base64 encoder. However, it is equally legitimate in the Python context to say you should be explicit about your intentions by decoding the bytes output of the base64 encoder using the ASCII codec. This was indeed discussed at length. For a while we didn't even allow unicode input on either side, but we relaxed that. My understanding of Python's current stance on functions that handle both bytes and string is that *either* the function accepts both types and outputs the *same* type as the input, *or* it accepts both types but always outputs *one* type or the other. You can't have unicode output if you give unicode input to the base64 decoder in the general case. So decode, at least, has to always give bytes output. Likewise, there is small to zero utility for using unicode input to the base64 encoder, since the unicode would have to be ASCII only and there'd be no point in doing the encoding. So, the only thing that makes sense is to follow the "one output type" rule here. Now, you can argue whether or not it would make sense for the encoder to always produce unicode. However, you then immediately run into the backward compatibility issue: the primary use case of the base64 encoding is to produce *wire ready* bytes. This is what the email package uses it for, for example. So for backward compatibility reasons, which are consonant with its primary use case, it makes more sense for the encoder to produce bytes than string. If you need to transmit bytes over a unicode channel, you can decode it from ASCII. That is, unicode is the *exceptional* use case here, not the rule. That might in fact be changing, but for backward compatibility reasons, Python won't change. And that should answer Steve's original question :) --David
R. David Murray wrote:
The fundamental purpose of the base64 encoding is to take a series of arbitrary bytes and reversibly turn them into another series of bytes in which the eighth bit is not significant.
No, it's not. If that were its only purpose, it would be called base128, and the RFC would describe it purely in terms of bit patterns and not mention characters or character sets at all. The RFC does *not* do that. It describes the output in terms of characters, and does not specify any bit patterns for the output. The intention is clearly to represent binary data as *text*. -- Greg
Greg Ewing writes:
The RFC does *not* do that. It describes the output in terms of characters, and does not specify any bit patterns for the output.
The RFC is unclear on this point, but I read it as specifying the ASCII coded character set, not the ASCII repertoire of (abstract) characters. Therefore, it specifies an invertible mapping from a particular set of integers to characters.
The intention is clearly to represent binary data as *text*.
It's more subtle than that. *RFCs do not deal with text.* Text is an internal concept of (some) programming environments. RFCs may deal with *encoded text*, and RFC 4648 indeed specifically mentions "encoded characters" as the output of the BASE64 algorithm.[1] The intention then is to represent binary data with *binary data that may be conveniently interpreted as text* (ie, without reencoding), eg, by a terminal or a printer.[2] It is also desirable that it be likely to pass unscathed through channels that are not necessarily even 7-bit clean (file system directories and JIS X 0201, for example) which *inadvertantly* treat it as text. Both requirements are conveniently fulfilled by using appropriate ASCII subsets, and encoding on the wire using the usual bit patterns. However, I suppose you could also use EBCDIC or UTF-16, as long as you have agreed with the receiver to do so. So I would say that Python can do what it wants with the type that base64.b64encode returns as far as the RFC is concerned; that's an internal aspect of Python. It's purely a matter of our convenience (as programmer *in* Python) whether we return str or bytes. My own experience is biased toward email and web (not to be confused with SMTP and HTTP), and so my experience is that most composers (1) automatically handle text encodings for the users, and then the content transfer encoding as necessary for the underlying protocol, and (2) handle attachments by placing a reference in the composed content, which is replaced by the object just before transmission (and any desired content transfer encoding is applied at that time, at the option of the composing agent, which rarely needs to bother the user with such trivia). Bytes seem more convenient to me, and give an on- the-wire representation consistent with that of Python 2 str. Footnotes: [1] Admittedly, RFC 3986 (URIs) does stretch the notion of "encoded text" to the breaking point by including marks on paper. [2] Thus, BASE64-encoding resources provides a more efficient, alternative datagram protocol for the physical links used by RFC 1149 networks.
On Tue, Jun 14, 2016, at 22:58, Stephen J. Turnbull wrote:
The RFC is unclear on this point, but I read it as specifying the ASCII coded character set, not the ASCII repertoire of (abstract) characters. Therefore, it specifies an invertible mapping from a particular set of integers to characters.
There are multiple descriptions of base 64 that specifically mention using it with EBCDIC and with local character sets of unspecified nature.
The intention is clearly to represent binary data as *text*.
It's more subtle than that. *RFCs do not deal with text.* Text is an internal concept of (some) programming environments.
It's also a human concept. Plenty of RFCs deal with human concept rather than purely programming topics.
Stephen J. Turnbull wrote:
The RFC is unclear on this point, but I read it as specifying the ASCII coded character set, not the ASCII repertoire of (abstract) characters.
Well, I think you've misread it. Or at least there is a more general reading possible that is entirely consistent with the stated purpose and doesn't assume any particular output encoding.
It's more subtle than that. *RFCs do not deal with text.*
That may be true of most RFCs, but I think this particular one really *is* talking about text, even if the authors didn't realise it at the time.
It is also desirable that it be likely to pass unscathed through channels that ... *inadvertantly* treat it as text. Both requirements are conveniently fulfilled by using appropriate ASCII subsets, and encoding on the wire using the usual bit patterns.
But only if the part that is (deliberately or inadvertently) treating it as text is using ASCII as its encoding. So, by your reading of the RFC, base64 is *only* intended for channels that use ASCII encoding. Whereas if you drop the assumption of ASCII and use whatever encoding the channel uses for text, then it works for all channels. RFC 4648 doesn't mention it, but an earlier RFC on base64 explicitly said that characters were chosen that also exist in EBCDIC, so it seems they were intending that base64 should work on EBCDIC-bases systems as well as ASCII-based ones.
It's purely a matter of our convenience (as programmer *in* Python) whether we return str or bytes.
Yes, and it seems to me the decision has been made by people with their noses stuck in low-level protocol implementations. Whenever *I've* needed to base64 encode something, I've wanted the output as text, because that's what I needed to feed into the next stage of the process. Maybe there should be two versions of the base64 codec, one producing bytes and one producing text? -- Greg
On Wed, 15 Jun 2016 11:51:05 +1200, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
R. David Murray wrote:
The fundamental purpose of the base64 encoding is to take a series of arbitrary bytes and reversibly turn them into another series of bytes in which the eighth bit is not significant.
No, it's not. If that were its only purpose, it would be called base128, and the RFC would describe it purely in terms of bit patterns and not mention characters or character sets at all.
Sorry, you are correct. IMO it is to encode it to a representation that consists of a limited subset of printable (makes marks on paper or screen) characters (which is an imprecise term); ie: data that will not be interpreted as having control information by most programs processing the data stream as either human-readable or raw bytes. The rest of the argument still applies, specifically the part about wire encoding to seven bit bytes being the currently-most-used[*] and backward-compatible use case. And I say this despite the fact that the email package currently handles everything as surrogate-escaped text and so does in fact decode the output of base64.encode to ASCII and only later re-encodes it. That's a design issue in the email package deriving from the fact that bytes and string used to be the same thing in python2. It might some day get corrected, but probably won't be, and it is a legacy of *not* making the distinction between bytes and string. --David [*] Yes this is changing, I already said that :)
On 6/14/2016 12:32 PM, Toshio Kuratomi wrote:
The input to encoding would have to remain bytes (that's the main purpose of base64... to turn bytes into an ascii string).
The purpose is to turn arbitrary binary data (commonly images) into 'safe bytes' that will not get mangled on transmission (7 bit channels were once common) and that will not mangle a display of data transmitted or received. Ignoring the EBCDIC world, which Python mostly does, the set of 'safe bytes' is the set that encodes printable ascii characters. Those bytes pass through 7 bit channels and display on ascii-based terminals. -- Terry Jan Reedy
To port OpenStack to Python 3, I wrote 4 (2x2) helper functions which accept bytes *and* Unicode as input. xxx_as_bytes() functions return bytes, xxx_as_text() return Unicode: http://docs.openstack.org/developer/oslo.serialization/api.html Victor Le 14 juin 2016 5:21 PM, "Steven D'Aprano" <steve@pearwood.info> a écrit :
Normally I'd take a question like this to Python-List, but this question has turned out to be quite diversive, with people having strong opinions but no definitive answer. So I thought I'd ask here and hope that some of the core devs would have an idea.
Why does base64 encoding in Python return bytes?
base64.b64encode take bytes as input and returns bytes. Some people are arguing that this is wrong behaviour, as RFC 3548 specifies that Base64 should transform bytes to characters:
https://tools.ietf.org/html/rfc3548.html
albeit US-ASCII characters. E.g.:
The encoding process represents 24-bit groups of input bits as output strings of 4 encoded characters. [...] Each 6-bit group is used as an index into an array of 64 printable characters. The character referenced by the index is placed in the output string.
Are they misinterpreting the standard? Has Python got it wrong? Is there a good reason for returning bytes?
I see that other languages choose different strategies. Microsoft's languages C#, F# and VB (plus their C++ compiler) take an array of bytes as input, and outputs a UTF-16 string:
https://msdn.microsoft.com/en-us/library/dhx0d524%28v=vs.110%29.aspx
Java's base64 encoder takes and returns bytes:
https://docs.oracle.com/javase/8/docs/api/java/util/Base64.Encoder.html
and Javascript's Base64 encoder takes input as UTF-16 encoded text and returns the same:
https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encodin...
I'm not necessarily arguing that Python's strategy is the wrong one, but I am interested in what (if any) reasons are behind it.
Thanks in advance,
Steve _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co...
On 14 June 2016 at 16:19, Steven D'Aprano <steve@pearwood.info> wrote:
Why does base64 encoding in Python return bytes?
I seem to recall there was a debate about this around the time of the Python 3 move. (IIRC, it was related to the fact that there used to be a base64 "codec", that wasn't available in Python 3 because it wasn't clear whether it converted bytes to text or bytes). I don't remember any of the details, let alone if a conclusion was reached, but a search of the archives may find something. Paul
On 14/06/2016 16:51, Paul Moore wrote:
On 14 June 2016 at 16:19, Steven D'Aprano <steve@pearwood.info> wrote:
Why does base64 encoding in Python return bytes?
I seem to recall there was a debate about this around the time of the Python 3 move. (IIRC, it was related to the fact that there used to be a base64 "codec", that wasn't available in Python 3 because it wasn't clear whether it converted bytes to text or bytes). I don't remember any of the details, let alone if a conclusion was reached, but a search of the archives may find something.
Paul
As I've the time to play detective I'd suggest https://mail.python.org/pipermail/python-3000/2007-July/008975.html -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence
On 6/14/2016 12:29 PM, Mark Lawrence via Python-Dev wrote:
As I've the time to play detective I'd suggest https://mail.python.org/pipermail/python-3000/2007-July/008975.html
Thank you for finding that. I reread it and still believe that bytes was the right choice. Base64 is an generic edge encoding for binary data. It fits in with the the standard paradigm as a edge encoding. Receive encoded bytes. Decode bytes to python objects Manipulate python objects Encode python objects to bytes Send bytes. Receive and send can be from and to either local files or sockets usually connected to remote systems. Transmissions can have blocks with different encodings. In the latter case, the bytes need to be parsed into blocks with different encodings. In the (fairly common) special case that a transmission consists entirely of text in *1* encoding (ignoring any transmission wrappers), decode and encode can be incorporated into a text-mode file object. If a transmission consists entirely or partly of binary, one can open in binary mode and .write one or more blocks of encoded bytes, possible with encoding data. -- Terry Jan Reedy
On Tue, Jun 14, 2016 at 8:42 PM, Terry Reedy <tjreedy@udel.edu> wrote:
Thank you for finding that. I reread it and still believe that bytes was the right choice. Base64 is an generic edge encoding for binary data. It fits in with the the standard paradigm as a edge encoding.
I'd like to me-too Terry's sentiment, but also expand on it a bit. Base64 encoding is used to convert bytes into a limited set of symbols for inclusion in a stream of data. Whether bytes or unicode characters are appropriate depends on whether the stream being constructed is a byte stream or a unicode character stream. Many people do deal with byte streams in Python and we have large sub-communities for who this use case is important (e.g. Twisted, Asyncio, anyone using the socket module). It is also no longer 1980 though, and there are many protocols layered on top of unicode character streams rather than bytes. Ideally I'd like us to support both options (like we've been increasingly doing for reading from other external sources such as file systems or environment variables). If we only support one, I would prefer it to be bytes since (bytes -> bytes -> unicode) seems like less overhead and slightly conceptually clearer than (bytes -> unicode -> bytes), but I consider this a personal preference rather than any sort of one-true-way. Schiavo Simon
Simon Cross wrote:
If we only support one, I would prefer it to be bytes since (bytes -> bytes -> unicode) seems like less overhead and slightly conceptually clearer than (bytes -> unicode -> bytes),
Whereas bytes -> unicode, followed if needed by unicode -> bytes, seems conceptually clearer to me. IOW, base64 is conceptually a bytes-to-text transformation, and the usual way to represent text in Python 3 is unicode. -- Greg
On Wed, 15 Jun 2016, Greg Ewing wrote:
Simon Cross wrote:
If we only support one, I would prefer it to be bytes since (bytes -> bytes -> unicode) seems like less overhead and slightly conceptually clearer than (bytes -> unicode -> bytes),
Whereas bytes -> unicode, followed if needed by unicode -> bytes, seems conceptually clearer to me. IOW, base64 is conceptually a bytes-to-text transformation, and the usual way to represent text in Python 3 is unicode.
And in CPython, do I understand correctly that the output text would be represented using one byte per character? If so, would there be a way of encoding that into UTF-8 that re-used the raw memory that backs the Unicode object? And, therefore, avoids almost all the inefficiency of going via Unicode? If so, this would be a win - proper use of Unicode to represent a text string, combined with instantaneous conversion into a bytes object for the purpose of writing to the OS. Isaac Morland CSCF Web Guru DC 2619, x36650 WWW Software Specialist
It would be a codec. base64_text in the codecs module. Probably 1 line different than the existing codec. Very easy to use and maintain. Less surprising and less error prone for everyone who thinks base64 should convert between bytes to text. Sounds like an obvious win to me. On Wed, Jun 15, 2016 at 11:08 AM Isaac Morland <ijmorlan@uwaterloo.ca> wrote:
On Wed, 15 Jun 2016, Greg Ewing wrote:
Simon Cross wrote:
If we only support one, I would prefer it to be bytes since (bytes -> bytes -> unicode) seems like less overhead and slightly conceptually clearer than (bytes -> unicode -> bytes),
Whereas bytes -> unicode, followed if needed by unicode -> bytes, seems conceptually clearer to me. IOW, base64 is conceptually a bytes-to-text transformation, and the usual way to represent text in Python 3 is unicode.
And in CPython, do I understand correctly that the output text would be represented using one byte per character? If so, would there be a way of encoding that into UTF-8 that re-used the raw memory that backs the Unicode object? And, therefore, avoids almost all the inefficiency of going via Unicode? If so, this would be a win - proper use of Unicode to represent a text string, combined with instantaneous conversion into a bytes object for the purpose of writing to the OS.
Isaac Morland CSCF Web Guru DC 2619, x36650 WWW Software Specialist _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
On Tue, Jun 14, 2016 at 05:29:12PM +0100, Mark Lawrence via Python-Dev wrote:
As I've the time to play detective I'd suggest https://mail.python.org/pipermail/python-3000/2007-July/008975.html
Thanks Mark, that's great! -- Steve
Hello, On Tue, 14 Jun 2016 16:51:44 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
On 14 June 2016 at 16:19, Steven D'Aprano <steve@pearwood.info> wrote:
Why does base64 encoding in Python return bytes?
I seem to recall there was a debate about this around the time of the Python 3 move. (IIRC, it was related to the fact that there used to be a base64 "codec", that wasn't available in Python 3 because it wasn't clear whether it converted bytes to text or bytes). I don't remember any of the details, let alone if a conclusion was reached, but a search of the archives may find something.
Well, it's easy to remember the conclusion - it was decided to return bytes. The reason also wouldn't be hard to imagine - regardless of the fact that base64 uses ASCII codes for digits and letters, it's still essentially a binary data. And the most natural step for it is to send it down the socket (socket.send() accepts bytes), etc. I'd find it a bit more surprising that binascii.hexlify() returns bytes, but I personally got used to it, and consider it a consistency thing on binascii module. Generally, with Python3 by default using (inefficient) Unicode for strings, any efficient data processing would use bytes, and then one appreciates the fact that data encoding/decoding routines also return bytes, avoiding implicit expensive conversion to strings. -- Best regards, Paul mailto:pmiscml@gmail.com
On Tue, Jun 14, 2016, at 13:19, Paul Sokolovsky wrote:
Well, it's easy to remember the conclusion - it was decided to return bytes. The reason also wouldn't be hard to imagine - regardless of the fact that base64 uses ASCII codes for digits and letters, it's still essentially a binary data.
Only in the sense that all text is binary data. There's nothing in the definition of base64 specifying ASCII codes. It specifies *characters* that all happen to be in ASCII's character repertoire.
And the most natural step for it is to send it down the socket (socket.send() accepts bytes), etc.
How is that more natural than to send it to a text buffer that is ultimately encoded (maybe not even in an ASCII-compatible encoding... though probably) and sent down a socket or written to a file by a layer that is outside your control? Yes, everything eventually ends up as bytes. That doesn't mean that we should obsessively convert things to bytes as early as possible. I mean if we were gonna do that why bother even having a unicode string type at all?
I'd find it a bit more surprising that binascii.hexlify() returns bytes, but I personally got used to it, and consider it a consistency thing on binascii module.
Generally, with Python3 by default using (inefficient) Unicode for strings,
Why is it inefficient?
any efficient data processing would use bytes, and then one appreciates the fact that data encoding/decoding routines also return bytes, avoiding implicit expensive conversion to strings.
IMO this is more a philosophical problem than a programming problem. base64 has a dual-nature. It is both text and bytes. At least it should fit in a 1-byte-per-character efficient Python 3 unicode string also.
Hello, On Tue, 14 Jun 2016 18:13:11 +0000 Daniel Holth <dholth@gmail.com> wrote:
IMO this is more a philosophical problem than a programming problem. base64 has a dual-nature. It is both text and bytes. At least it should fit in a 1-byte-per-character efficient Python 3 unicode string also.
You probably mean "CPython3 1-byte-per-character "efficient" string". But CPython3 is merely one of half-dozen Python3 language implementations. Yup, a special one, but hopefully it's special in a respect that it doesn't abuse its powers to make language API *changes* based on its own implementation details. API changes, because API *decisions* have been done long ago already. -- Best regards, Paul mailto:pmiscml@gmail.com
Hello, On Tue, 14 Jun 2016 14:02:02 -0400 Random832 <random832@fastmail.com> wrote:
On Tue, Jun 14, 2016, at 13:19, Paul Sokolovsky wrote:
Well, it's easy to remember the conclusion - it was decided to return bytes. The reason also wouldn't be hard to imagine - regardless of the fact that base64 uses ASCII codes for digits and letters, it's still essentially a binary data.
Only in the sense that all text is binary data. There's nothing in the definition of base64 specifying ASCII codes. It specifies *characters* that all happen to be in ASCII's character repertoire.
And the most natural step for it is to send it down the socket (socket.send() accepts bytes), etc.
How is that more natural than to send it to a text buffer that is
It's more natural because it's more efficient. It's more natural in the same sense that the most natural way to get from point A to point B is a straight line.
ultimately encoded (maybe not even in an ASCII-compatible encoding... though probably) and sent down a socket or written to a file by a layer that is outside your control? Yes, everything eventually ends up as bytes. That doesn't mean that we should obsessively convert things to bytes as early as possible.
It's vice-versa - there's no need to obsessively convert simple, primary type of bytes (everything in computers are bytes) to more complex things like Unicode strings.
I mean if we were gonna do that why bother even having a unicode string type at all?
You're trying to raise the topic which is a subject of gigantic flame wars on python-list for years. Here's my summary: not using unicode string type *at all* is better than not using bytes type at all. So, feel free to use unicode string *only* when it's needed, which is *only* when you accept input from or produce output for *human* (like real human, walking down a street to do grocery shopping). In all other cases, data should stay bytes (mind - stay, as it's bytes in the beginning, and it requires extra effort to convert it to a strings).
I'd find it a bit more surprising that binascii.hexlify() returns bytes, but I personally got used to it, and consider it a consistency thing on binascii module.
Generally, with Python3 by default using (inefficient) Unicode for strings,
Why is it inefficient?
Because bytes is the most efficient basic representation of data. Everything which tries to convert it to something is less efficient in general. Less efficient == inefficient. -- Best regards, Paul mailto:pmiscml@gmail.com
On 6/14/2016 11:19 AM, Steven D'Aprano wrote:
Normally I'd take a question like this to Python-List, but this question has turned out to be quite diversive, with people having strong opinions but no definitive answer. So I thought I'd ask here and hope that some of the core devs would have an idea.
Why does base64 encoding in Python return bytes?
Ultimately, because we never decided to change this in 3.0.
base64.b64encode take bytes as input and returns bytes. Some people are arguing that this is wrong behaviour, as RFC 3548 specifies that Base64 should transform bytes to characters:
https://tools.ietf.org/html/rfc3548.html
albeit US-ASCII characters. E.g.:
The encoding process represents 24-bit groups of input bits as output strings of 4 encoded characters.
One could argue that 'encoded character' means 'bytes' in Python, but I don't know what the standard writer meant, as unicode characters always have some internal encoding.
[...] Each 6-bit group is used as an index into an array of 64 printable characters. The character referenced by the index is placed in the output string.
-- Terry Jan Reedy
Steven D'Aprano writes:
base64.b64encode take bytes as input and returns bytes. Some people are arguing that this is wrong behaviour, as RFC 3548
That RFC is obsolete: the replacement is RFC 4648. However, the text is essentially unchanged.
specifies that Base64 should transform bytes to characters:
Without defining "character" except as a "subset" of ASCII. That omission is evidently deliberate. Unfortunately the RFC is unclear whether a subset of the ASCII repertoire of (abstract) characters is meant, or a subset of the ASCII codes. I believe the latter is meant, but either way, it does refer to *encoded* characters as the output of the encoding process:
The encoding process represents 24-bit groups of input bits as output strings of 4 encoded characters.
and I see no reason to deny that the bytes output by base64.b64encode are the octets representing the ASCII codes for the characters of the BASE64 alphabet.
Are they misinterpreting the standard?
I think they are. As I understand it, the intention of the standard in using "character" to denote the code unit is similar to that of RFC 3986: BASE encodings are intended to be printable and recognizable to humans. If you're using a non-ASCII-superset encoding such as EBCDIC for text I/O, then you should translate from ASCII to that encoding for display, and in the (unlikely) case that a human types BASE encoding from the terminal, the reverse transformation is necessary.
Has Python got it wrong?
I can't see anything in the RFC that suggests that. And, in the end, an RFC is not concerned with Python's internal fiddling, but rather with what goes out over the wire. All of the implementations you mention will eventually send to the wire octets that are interpreted as ASCII-encoded characters according to their integer values.
Is there a good reason for returning bytes?
I suppose practicality over purity: BASE encodings are normally used on the wire, and so programs need to encode text to appropriately encoded octets *before* BASE encoding, and then normally immediately put the BASE-encoded content on the wire. Why round-trip from UTF-8 bytes to a str in BASE64 representation, and then do the (trivial) conversion back to bytes? OK, it's not that expensive, but still...
Stephen J. Turnbull wrote:
it does refer to *encoded* characters as the output of the encoding process:
The encoding process represents 24-bit groups of input bits as output strings of 4 encoded characters.
The "encoding" being referred to there is the encoding from input bytes to output characters, not an encoding of the output characters as bytes. Nowhere in RFC 4648 does it refer to the output as being made up of "bytes" or "octets". It's always described in terms of "characters".
As I understand it, the intention of the standard in using "character" to denote the code unit is similar to that of RFC 3986: BASE encodings are intended to be printable and recognizable to humans.
Hmmm... so why then does it say, in section 4: The Base 64 encoding is designed to represent arbitrary sequences of octets in a form that ... need not be human readable.
If you're using a non-ASCII-superset encoding such as EBCDIC for text I/O, then you should translate from ASCII to that encoding for display,
What about the channel you're sending the encoded data over? Suppose I'm on Windows and I'm embedding the base64 encoded data in a text message that I'm sending through a mail client that accepts text in utf-16. I hope you would agree that, in that situation, encoding the base64 output in ASCII and giving those bytes directly to the mail client would be very much the wrong thing to do? -- Greg
participants (15)
-
Daniel Holth -
Greg Ewing -
Isaac Morland -
Joao S. O. Bueno -
Mark Lawrence -
Paul Moore -
Paul Sokolovsky -
R. David Murray -
Random832 -
Simon Cross -
Stephen J. Turnbull -
Steven D'Aprano -
Terry Reedy -
Toshio Kuratomi -
Victor Stinner