.from and .to instead of .encode and .decode
Hi, While looking at the code like: 'os': sysinfo['os'].decode('utf-8'), 'hostname': sysinfo['hostname'].decode('utf-8'), I can't really read if the result will be unicode or binary string in utf-8. It would be more convenient for readability to have these instead: bytes.from(encoding) -> unicode unicode.to(encoding) -> bytes .encode/.decode are confusing, because it Python 2 it was: str.encode(encoding) -> str str.decode(encoding) -> str with no encoding info attached. -- anatoly t.
On Fri, Mar 6, 2015 at 12:40 AM, anatoly techtonik <techtonik@gmail.com> wrote:
Hi,
While looking at the code like:
'os': sysinfo['os'].decode('utf-8'), 'hostname': sysinfo['hostname'].decode('utf-8'),
I can't really read if the result will be unicode or binary string in utf-8.
If it says "decode", the result is a Unicode string. If it says "encode", the result is bytes. I'm not sure what is difficult here.
.encode/.decode are confusing, because it Python 2 it was:
str.encode(encoding) -> str str.decode(encoding) -> str
with no encoding info attached.
$ python Python 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
"test".decode("utf-8") u'test' u"test".encode("utf-8") 'test'
Looks to me like str.decode() -> unicode, and unicode.encode() -> str, at least for UTF-8 and other encodings that apply to Unicode. Yes, there are some oddities in Py2:
"74657374".decode("hex") 'test'
in which str.decode returns a str, but AFAIK all of those are buried away in Python 3:
codecs.decode(b"74657374","hex") b'test'
So where's the confusion? ChrisA
On Sat, Mar 7, 2015 at 8:41 AM, Chris Angelico <rosuav@gmail.com> wrote:
If it says "decode", the result is a Unicode string. If it says "encode", the result is bytes. I'm not sure what is difficult here.
Yep. When I teach, I use this mnemonic, which I can now quote from my book [1] ;-) [TIP] ==== If you need a memory aid to distinguish `.decode()` from `.encode()`, convince yourself that a Unicode `str` contains "human" text, while byte sequences can be cryptic machine core dumps. Therefore, it makes sense that we *decode* `bytes` to `str` to get human readable text, and we *encode* text to `bytes` for storage or transmission. ==== [1] http://shop.oreilly.com/product/0636920032519.do Cheers, Luciano
.encode/.decode are confusing, because it Python 2 it was:
str.encode(encoding) -> str str.decode(encoding) -> str
with no encoding info attached.
$ python Python 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
"test".decode("utf-8") u'test' u"test".encode("utf-8") 'test'
Looks to me like str.decode() -> unicode, and unicode.encode() -> str, at least for UTF-8 and other encodings that apply to Unicode. Yes, there are some oddities in Py2:
"74657374".decode("hex") 'test'
in which str.decode returns a str, but AFAIK all of those are buried away in Python 3:
codecs.decode(b"74657374","hex") b'test'
So where's the confusion?
ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr
On Sat, Mar 7, 2015 at 2:42 PM, Luciano Ramalho <luciano@ramalho.org> wrote:
On Sat, Mar 7, 2015 at 8:41 AM, Chris Angelico <rosuav@gmail.com> wrote:
If it says "decode", the result is a Unicode string. If it says "encode", the result is bytes. I'm not sure what is difficult here.
Yep. When I teach, I use this mnemonic, which I can now quote from my book [1] ;-)
[TIP] ==== If you need a memory aid to distinguish `.decode()` from `.encode()`, convince yourself that a Unicode `str` contains "human" text, while byte sequences can be cryptic machine core dumps. Therefore, it makes sense that we *decode* `bytes` to `str` to get human readable text, and we *encode* text to `bytes` for storage or transmission. ====
This is a great advice (and yes, I also often get confused by the two). -- Giampaolo - http://grodola.blogspot.com
On 3/8/2015 9:07 AM, Giampaolo Rodola' wrote:
On Sat, Mar 7, 2015 at 2:42 PM, Luciano Ramalho
[TIP] ==== If you need a memory aid to distinguish `.decode()` from `.encode()`, convince yourself that a Unicode `str` contains "human" text, while byte sequences can be cryptic machine core dumps. Therefore, it makes sense that we *decode* `bytes` to `str` to get human readable text, and we *encode* text to `bytes` for storage or transmission. ====
This is a great advice (and yes, I also often get confused by the two).
Unless a bytes object simply represents a sequence of counts in range(256), which in unusual, then the bytes are an encoding of some other information, which in Python is embodied in an object. Struct.pack (en)codes python ints to bytes. Here the coding issue is big versus little endian. Struct.unpack decodes bytes back to int objects. PIL can decode (unpack and deserialize) bytes in various image encodings to an image object and can likewise encode an image object to bytes in various formats. The common theme is that non-bytes objects are encoded to bytes for transmission and storage and decoded back to non-bytes objects. -- Terry Jan Reedy
On Sun, Mar 8, 2015 at 4:07 PM, Giampaolo Rodola' <g.rodola@gmail.com> wrote:
On Sat, Mar 7, 2015 at 2:42 PM, Luciano Ramalho <luciano@ramalho.org> wrote:
On Sat, Mar 7, 2015 at 8:41 AM, Chris Angelico <rosuav@gmail.com> wrote:
If it says "decode", the result is a Unicode string. If it says "encode", the result is bytes. I'm not sure what is difficult here.
Yep. When I teach, I use this mnemonic, which I can now quote from my book [1] ;-)
[TIP] ==== If you need a memory aid to distinguish `.decode()` from `.encode()`, convince yourself that a Unicode `str` contains "human" text, while byte sequences can be cryptic machine core dumps. Therefore, it makes sense that we *decode* `bytes` to `str` to get human readable text, and we *encode* text to `bytes` for storage or transmission. ====
This is a great advice (and yes, I also often get confused by the two).
I use the analogy of "encoding abstract numbers into specific bytes" myself. It is still hard to review the source, which may use it wrong. db.get(query).from('utf-8') gives me a clear indication that a person expects that db contents is in 'utf-8'. db.get(query).decode('utf-8') gives me a feeling that query result should be decoded from whatever format it is in to 'utf-8'. So if Python supported .from and .to, the part about encoding wouldn't feel like math.
On Mon, Mar 16, 2015 at 6:09 AM, anatoly techtonik <techtonik@gmail.com> wrote:
db.get(query).from('utf-8') gives me a clear indication that a person expects that db contents is in 'utf-8'.
db.get(query).decode('utf-8') gives me a feeling that query result should be decoded from whatever format it is in to 'utf-8'.
In teaching this, yes, newbies do get confused about what "decode" vs. "encode" mean. The fact that py2 strings have BOTH methods does confuse the heck out of it! -- but that problem is solved in py3. And in this case "from" might be a bit more obvious to a newbie. (though only might -- hard to tell without usability testing...) But "decode" and "encode" are widely used terms that clearly specify what's going on -- I think that even with teh "from", it's clear to you because you already understand a bit about the concept of unicode and the distiction between a string object and encoded bytes. And once you get that, then you kind know what "encode" and "decode" mean. Hopefully, folks will get that "decoded to utf-8" is meaningless... I suppose we could have used "decode_from" and "encode_to" in the first place to be really clear, but I don't think it's worth making that change now. -CHB
So if Python supported .from and .to, the part about encoding wouldn't feel like math. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Tue, Mar 17, 2015 at 3:41 AM, Chris Barker <chris.barker@noaa.gov> wrote:
I suppose we could have used "decode_from" and "encode_to" in the first place to be really clear, but I don't think it's worth making that change now.
I don't think it'd help much. It'd just be noise words, at best. What you're doing is encoding or decoding using a particular scheme. What scheme? That's the parameter. You're encoding to bytes and decoding to text; and in order to perform that transformation, you use a standard technique by the name of UTF-8. The result of "spam".encode("UTF-8") is not a UTF-8, it's a stream of bytes. So there'd be just as much confusion the other way around. *shrug* ChrisA
On Mon, Mar 16, 2015 at 10:09 AM, anatoly techtonik <techtonik@gmail.com> wrote:
I use the analogy of "encoding abstract numbers into specific bytes" myself.
Your version of the analogy doesn't sound as helpful to me; "abastract numbers" sounds even more cryptic than bytes. My mnemonics are "decode from cryptic bytes to human text" and "encode from human text to cryptic bytes". A more objective problem is: "from" is a keyword, so it can't be a method name. Best, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg
On Tue, Mar 17, 2015 at 7:10 PM, Luciano Ramalho <luciano@ramalho.org> wrote:
On Mon, Mar 16, 2015 at 10:09 AM, anatoly techtonik <techtonik@gmail.com> wrote:
I use the analogy of "encoding abstract numbers into specific bytes" myself.
Your version of the analogy doesn't sound as helpful to me; "abastract numbers" sounds even more cryptic than bytes.
My mnemonics are "decode from cryptic bytes to human text" and "encode from human text to cryptic bytes".
Maybe I am thinking too low level, but there is no such thing as "human text" in computer memory. But I agree that using the words like "abstraction" is not a good idea, because Python is used by children of age 7, and things should be simpler.
A more objective problem is: "from" is a keyword, so it can't be a method name.
I don't see why "from" can not be used as a method name in Python. It can not be used as function name, because it will override the keyword, but method names are always namespaced with self. or class prefixes. -- anatoly t.
On Wed, Mar 18, 2015 at 2:54 AM, anatoly techtonik <techtonik@gmail.com> wrote:
My mnemonics are "decode from cryptic bytes to human text" and "encode from human text to cryptic bytes".
Maybe I am thinking too low level, but there is no such thing as "human text" in computer memory. But I agree that using the words like "abstraction" is not a good idea, because Python is used by children of age 7, and things should be simpler.
It's a didactic mnemonic ;-). Although str objects hold codepoints, they *are* used to represent human text. On the other hand, we convert str to bytes when we need to talk to the machine: writing to disk, sending over the network...
A more objective problem is: "from" is a keyword, so it can't be a method name.
I don't see why "from" can not be used as a method name in Python. It can not be used as function name, because it will override the keyword, but method names are always namespaced with self. or class prefixes.
You can't write it's definition, "def from..." and you can't even write "x.from" without a SyntaxError (instead of NameError or AttributeError). Try it in the Python console. The parser would need to be changed. Best, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg
On Wed, Mar 18, 2015 at 2:24 PM, Luciano Ramalho <luciano@ramalho.org> wrote:
A more objective problem is: "from" is a keyword, so it can't be a method name.
I don't see why "from" can not be used as a method name in Python. It can not be used as function name, because it will override the keyword, but method names are always namespaced with self. or class prefixes.
You can't write it's definition, "def from..." and you can't even write "x.from" without a SyntaxError (instead of NameError or AttributeError). Try it in the Python console.
The parser would need to be changed.
And that won't break anything. -- anatoly t.
On 05.03.15 15:40, anatoly techtonik wrote:
While looking at the code like:
'os': sysinfo['os'].decode('utf-8'), 'hostname': sysinfo['hostname'].decode('utf-8'),
I can't really read if the result will be unicode or binary string in utf-8. It would be more convenient for readability to have these instead:
bytes.from(encoding) -> unicode unicode.to(encoding) -> bytes
Don't write sysinfo['os'].decode('utf-8') if it confuses you. Write str(sysinfo['os'], 'utf-8').
On Mar 5, 2015, at 5:40 AM, anatoly techtonik <techtonik@gmail.com> wrote:
Hi,
While looking at the code like:
'os': sysinfo['os'].decode('utf-8'), 'hostname': sysinfo['hostname'].decode('utf-8'),
I can't really read if the result will be unicode or binary string in utf-8. It would be more convenient for readability to have these instead:
bytes.from(encoding) -> unicode unicode.to(encoding) -> bytes
.encode/.decode are confusing, because it Python 2 it was:
str.encode(encoding) -> str str.decode(encoding) -> str
Except for a handful of confusing cases like hex and rot13, str.decode always returned unicode, and unicode.encode always returned str. However, str.encode _did_ return str--which it did by first decoding the str with sys.getdefaultencoding() so it had something to encode, which often failed, giving those wonderful error messages about being unable to decode ASCII when you asked it to encode UTF-8. It's little wonder that so many novices resorted to throwing in extra random calls to encode and decode and the str constructor until it seemed to work for one example, and then threw their hands up in despair when it didn't work for the next example input. Python 3 fixed that, simply by getting rid of the Unicode decode and bytes encode methods (which were occasionally useful for codecs like hex and rot13, but far more often just confusing). So now you can only encode Unicode to bytes, and decode bytes to Unicode, and you're the first person I've seen who found the Python 3 API at all confusing. I suspect that's because you didn't realize you were confused by Python 2, and carried that confusion over to 3, and somehow didn't run into any problems for the first few years?
with no encoding info attached. -- anatoly t. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
participants (8)
-
anatoly techtonik -
Andrew Barnert -
Chris Angelico -
Chris Barker -
Giampaolo Rodola' -
Luciano Ramalho -
Serhiy Storchaka -
Terry Reedy