Allow more flexibility for describing bytes objects.
Refer to PEP 3137 and PEP 358. Bytes objects are for conveying binary data (or encoded strings). Such binary data is customary specified in hex-dump or base64 format in source files. It would be nice to introduce a way in python to do that 'natively' (at lexical analysis time) using x-strings and y-strings (in lieu of f-strings). x-strings: x"0123456789abcdef" (x-prefixed quoted string of even-numbered amount of characters of the set [0-9a-fA-F], whitespace ignored) equivalent to: b"\x01\x23\x45\x67\x89\xab\xcd\xef" (but more readable) y-strings: y"ASNFZ4mrze8=" (y-prefixed quoted string of valid base64, whitespace ignored) equivalent to: b"\x01\x23\x45\x67\x89\xab\xcd\xef" (but shorter) This is not a replacement of the hex/base64 encoding, binascii packages etc. It just gives the programmer more freedom to specify literal bytes objects in the source code.
Well - one can already do: x = bytes.fromhex x("01 23 45 67 89") As for base64, I really, really, don't see a use case where allowing base64 native strings as part oif the syntax could be useful - less so when decoding is one call away. Base64 was already created to allow one to convey arbitrary data as strings - let them be strings, As for the "x" mode: I think the burden - including cognitive when teaching Python - to include new prefix strings easily overcome any gains there might be - again, more so once that a call like `bytes.frohex` is readily available. js -><- On Wed, Nov 23, 2022 at 9:33 AM Ronald Hoogenboom via Python-ideas < python-ideas@python.org> wrote:
Refer to PEP 3137 and PEP 358.
Bytes objects are for conveying binary data (or encoded strings). Such binary data is customary specified in hex-dump or base64 format in source files.
It would be nice to introduce a way in python to do that ‘natively’ (at lexical analysis time) using x-strings and y-strings (in lieu of f-strings).
x-strings:
x”0123456789abcdef” (x-prefixed quoted string of even-numbered amount of characters of the set [0-9a-fA-F], whitespace ignored)
equivalent to:
b”\x01\x23\x45\x67\x89\xab\xcd\xef” (but more readable)
y-strings:
y”ASNFZ4mrze8=” (y-prefixed quoted string of valid base64, whitespace ignored)
equivalent to:
b”\x01\x23\x45\x67\x89\xab\xcd\xef” (but shorter)
This is not a replacement of the hex/base64 encoding, binascii packages etc. It just gives the programmer more freedom to specify literal bytes objects in the source code.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/VGXARY... Code of Conduct: http://python.org/psf/codeofconduct/
On 23 Nov 2022, at 12:32, Ronald Hoogenboom via Python-ideas <python-ideas@python.org> wrote:
Refer to PEP 3137 and PEP 358.
Bytes objects are for conveying binary data (or encoded strings). Such binary data is customary specified in hex-dump or base64 format in source files. It would be nice to introduce a way in python to do that ‘natively’ (at lexical analysis time) using x-strings and y-strings (in lieu of f-strings).
x-strings:
x”0123456789abcdef” (x-prefixed quoted string of even-numbered amount of characters of the set [0-9a-fA-F], whitespace ignored) equivalent to: b”\x01\x23\x45\x67\x89\xab\xcd\xef” (but more readable)
y-strings:
y”ASNFZ4mrze8=” (y-prefixed quoted string of valid base64, whitespace ignored) equivalent to: b”\x01\x23\x45\x67\x89\xab\xcd\xef” (but shorter)
This is not a replacement of the hex/base64 encoding, binascii packages etc. It just gives the programmer more freedom to specify literal bytes objects in the source code.
How often would this be useful? If its rare and only nice to have I would doubt it would be added to python. I have written a lot of low level protocol and IOCTL calls that cannot think of a time this would have helped me. struct is often a mean to create blocks of binary data. Long strings of binary data would be a maintenance issue. What do all the bits mean? Barry
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org <mailto:python-ideas@python.org> To unsubscribe send an email to python-ideas-leave@python.org <mailto:python-ideas-leave@python.org> https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/VGXARY... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, 24 Nov 2022 at 06:46, Barry Scott <barry@barrys-emacs.org> wrote:
I have written a lot of low level protocol and IOCTL calls that cannot think of a time this would have helped me. struct is often a mean to create blocks of binary data.
Long strings of binary data would be a maintenance issue. What do all the bits mean?
Sounds like a system of compile-time evaluation - as a way to extend constant folding to more forms - might be the way to do this, then. I don't think the compiler can optimize this: x = bytes.fromhex("0123456789abcdef") and while it's theoretically possible to optimize this, it falls down badly on the "why was this written like this?" test: x = b"".fromhex("0123456789abcdef") But suppose there were some way to say "this is intended to be a compile-time constant, please evaluate it and retain the result as a literal". Obviously this is only valid if the result is a type that can be saved, but it would allow some quite nice optimizations that might even include struct calls. On the flip side, the performance gain probably wouldn't be all that much compared to doing the work at import time. For literals used in tight loops, there'd still be SOME benefit (if the work is done at import time, it would have to be stored as a global, and that means the inner loop is looking up a global rather than loading a constant), but it would have to be a fairly hot piece of code to be worth the effort. ChrisA
On 11/23/2022 3:02 PM, Chris Angelico wrote:
On Thu, 24 Nov 2022 at 06:46, Barry Scott<barry@barrys-emacs.org> wrote:
I have written a lot of low level protocol and IOCTL calls that cannot think of a time this would have helped me. struct is often a mean to create blocks of binary data.
Long strings of binary data would be a maintenance issue. What do all the bits mean? Sounds like a system of compile-time evaluation - as a way to extend constant folding to more forms - might be the way to do this, then. I don't think the compiler can optimize this:
x = bytes.fromhex("0123456789abcdef")
and while it's theoretically possible to optimize this, it falls down badly on the "why was this written like this?" test:
x = b"".fromhex("0123456789abcdef")
But suppose there were some way to say "this is intended to be a compile-time constant, please evaluate it and retain the result as a literal". Obviously this is only valid if the result is a type that can be saved, but it would allow some quite nice optimizations that might even include struct calls.
PEP 638 – Syntactic Macros would be perfect for this. It gives this example for f-strings: "The f-string |f"..."| could be implemented as macro as |f!("...")|. Not quite as nice to read, but would still be useful for experimenting with."
On the flip side, the performance gain probably wouldn't be all that much compared to doing the work at import time. For literals used in tight loops, there'd still be SOME benefit (if the work is done at import time, it would have to be stored as a global, and that means the inner loop is looking up a global rather than loading a constant), but it would have to be a fairly hot piece of code to be worth the effort.
That would be the beauty of 638: if it's important to one piece of code, it could be just enabled there. Of course the implementing macro would need to be either part of your application or become a dependency. But I'm under no delusions that this PEP will ever get accepted, unfortunately. Eric
participants (5)
-
Barry Scott -
Chris Angelico -
Eric V. Smith -
Joao S. O. Bueno -
Ronald Hoogenboom