PEP 637 - problems not solved by existing language
SUMMARY This post asks for some examples that will improve PEP 637, by providing examples where the existing language specification is not adequate (but PEP 637 is). The rest of this post should tell you why I want these examples, and what they should look like. DISCLAIMER I'm an editor of PEP 637, writing in my personal capacity. And I'm also the author of package kwkey mentioned below. SUMMARY PEP 637 adds to Python's syntax and semantics. It adds statements such as
PEP 1, which defines the PEP process, states that any PEP that changes the existing language specification should clearly explain "why the existing language specification is inadequate to address the problem that the PEP solves". https://www.python.org/dev/peps/pep-0001/#what-belongs-in-a-successful-pep As part of addressing this question of existing Python being adequate, I wrote a new package kwey. This package provides, I believe, a class B such that >>> B(1, 2, a=3, b=4)[x] = val is equivalent to >>> x[1, 2, a=3, b=4] = val once PEP 637 is implemented. (And if not, please report a bug, and if you can provide a patch.) https://pypi.org/project/kwkey/ GOAL The current version of PEP 637 doesn't mention kwkey. I'd particularly like us to look for problems where using kwkey as above is not adequate but PEP 637 is adequate. Aside. Please treat this as a semantic problem, or in other words assume that >>> x[SOMETHING] >>> f(SOMETHING) impose the same constraint on a well-formed expression SOMETHING. Here's why. PEP 637 allows syntax such as >>> x[1:2, 3:4, a=5:6, b=7:8] and even after PEP 637 (as it currently is) >>> f(1:2, 3:4, a=5:6, b=7:8) is forbidden syntax. But PEP 637 is much more than a syntax extension. Recall that we already have literal expressions such as >>> [1, 2, 3] # List literal >>> (1, 2, 3) # Tuple literal >>> {1, 2, 3} # Set literal in Python. If current Python turns out to have adequate semantics, then perhaps adding >>> (1:2:3) # Slice literal might by itself be enough. This is why I want particularly semantics examples. They're worth more. CONCLUSION I'd like to see semantic examples of problems that can be adequately solved by PEP 637, but not by using kwkey. The PEP process says, in so many words, that such examples should be provided. -- Jonathan
Jonathan, It sounds like you want to propose a competing PEP that provides your kwkey as a built-in as an alternative to PEP 637. I encourage you to do so, although I have no interest in acting as sponsor, and I fear that your window of opportunity to write a competing PEP is small. Alternatively, you could write a PEP that proposes a new kwkey object as complementary to, not competing with, PEP 637. That would provide a "kwkey" builtin similar to slice(), and people can explicitly opt-in to using it: # plan old PEP 637 semantics obj[spam, eggs, cheese='cheddar', aardvark=True] # explicit use of kwkey obj[kwkey(spam, eggs, cheese='cheddar', aardvark=True)] Another possibility is that you are not advocating for subscript keywords, and you prefer the status quo. (Your kwkey project counts as the status quo. It involves no new syntax or language features. Anyone can write their own kwkey class, or install a third-party library to use it.) Perhaps you agreed to co-author PEP 637 to have it rejected once and for all, not accepted, and you think that the current PEP is not neutral enough and advocates too strongly for the feature. PEPs can, and often are, written to be rejected. The PEP then becomes the canonical record of why the feature has been rejected. It is unfortunate if you and Stefano had radically opposite reasons for writing the PEP, but I think that's something the two of you should sort out and find some middle ground. Unfortunately Jonathan, I cannot tell from your post what your purpose is. I cannot tell if you oppose or support PEP 637, whether you want to propose a competing PEP, a complementary PEP, or just want to get credit for your kwkey project by having it mentioned in the PEP, or what exactly you want. I will comment further on the details in your post in another message. -- Steve
On Tue, Oct 06, 2020 at 04:04:37PM +0100, Jonathan Fine wrote:
I believe that the PEP meets that requirement by showing the inelegant workarounds used by popular projects in the absence subscript keywords. PEP 1 doesn't require a PEP to show that the existing language is *incapable* of solving the problem. Python is Turing Complete, so it can solve any computable problem. There are Turing Complete languages with as few as two symbols: https://esolangs.org/wiki/Jot so in a sense everything is syntactic sugar. The PEP, in my opinion, satisfactorily demonstrates that the existing methods of getting the semantics of subscript keywords are awkward and ugly.
Having to write something like `B(1, 2, a=3, b=4)[x]` would be an example of the awkwardness and inelegance that PEP 637 is designed to solve. If your kwkey package was a widely-known, commonly used third-party library, it would be a fantastic example of the sort of work-around that people fall back on in the absense of keyword subscripts. But I don't think that kwkey is widely used. It is a contrived library created in response to this issue being raised on the Python-Ideas mailing list, and as such, it would be a very weak example. A single widely-used example, such as from pandas or xarray, is worth a dozen contrived examples. So I'm not really sure what purpose you see your kwkey library taking in this PEP. As I mentioned in a previous post, I cannot tell whether you see it as competing with this PEP, complementary to the PEP, or perhaps as justifying the PEP?
But they don't. `f()` is legal; `f[]` is not, and will remain not legal. In addition, the precedences are different. In `f(a,b)` the use of the comma to separate arguments take precedence over the use of the comma to create tuples. But in `f[a,b]` it is the other way around.
Adding slice literals is a separate, unrelated issue. Allowing slice syntax `start:stop:step` outside of subscripts is independent and orthogonal to this PEP. Please don't burden this PEP with it.
Point of terminology: these are not literals (although informally people often call them that, including me on occassion). They are *displays*. One difference between a display and a literal can be seen from the byte-code generated: >>> import dis >>> dis.dis(compile('[1,2,3]', '', 'single')) 1 0 LOAD_CONST 0 (1) 2 LOAD_CONST 1 (2) 4 LOAD_CONST 2 (3) 6 BUILD_LIST 3 8 PRINT_EXPR 10 LOAD_CONST 3 (None) 12 RETURN_VALUE The constants 1, 2 and 3 are literals and the compiler can insert them directly into the code. The list [1, 2, 3] is not, the compiler has to insert code to create the list at runtime.
Adding slice literals isn't enough. I believe that the PEP already discusses that we cannot write: f(spam=1) = 2 del f(spam=1) and this is a fundamental difference between a subscript and a function call. Of course Python is Turing Complete and we can always perform the same operation by creating proxies and wrappers and other levels of indirection. As they say, every problem can be solved by adding sufficient levels of indirection, except the problem of having too many levels of indirection.
The most obvious problem that kwkey does not solve is the problem, how do I write a subscript with keywords without having to fall back on an inelegant and awkward work-around like kwkey? -- Steve
On Wed, Oct 7, 2020 at 1:46 PM Steven D'Aprano <steve@pearwood.info> wrote:
Be careful with this, as it's only a one-way proof. Any literal can and will be inserted directly as a constant, but there are some non-literals that can be as well:
Complex numbers and frozen sets don't technically have literals, but thanks to compiler magic, they can appear to. (Also, raw string literals are a thing, but in the compiled code, what you get is just a string object, further confusing this type of proof.) Although when it comes to f-strings, I honestly don't know what counts as a "literal". Is a single f-string a literal? What if two of them abut - are they one literal or two? The compiler builds the code to do the formatting and joining as though the parts of the string were combined (even if they're not all f-strings). But hey, it wouldn't be python-ideas without pedantry :) ChrisA
On Wed, Oct 07, 2020 at 01:59:24PM +1100, Chris Angelico wrote:
Sure, I believe that is a function of the keyhole optimizer: it can recognise certain non-literals, such as complex numbers and certain sets and tuples, and pre-generate them at compile time. That's a nice little optimization, especially swapping a mutable set for a frozen set because the compiler sees that the set is only ever read from, never mutated. (I think that was one of Serhiy's optimizations? Whoever it was, thank you.)
Raw strings are just an alternative syntax for strings. Just as we can write the same int as `0x4e` or `78`, or the same float as `12e-2` or `0.12`, or the same string as `"""abc"""` or `'\x61\x62\x63'`. They're not different "things", just different ways of writing the same thing.
Although when it comes to f-strings, I honestly don't know what counts as a "literal". Is a single f-string a literal?
An f-string is executable code. py> string = f"π/2 = {print('Calculating...') or __import__('math').atan(float('INF'))}" Calculating... py> string 'π/2 = 1.5707963267948966' Think of it as somewhat analogous to a list comprehension, in that it is an expression that contains executable code that runs at runtime, but it returns a string instead of a list, and rather than a for loop, you can include an arbitrary number of executable expressions. So not much like a list comprehension but a lot like calling eval on string literals :-)
But hey, it wouldn't be python-ideas without pedantry :)
Technically, there are lots of other mailing lists that are equally pedantic *wink* -- Steve
On Wed, Oct 7, 2020 at 9:20 PM Steven D'Aprano <steve@pearwood.info> wrote:
Right. They're the same object but different literal forms. That's the point. You can't use the disassembly to discover literal forms.
And yet the PEP defining them very definitely calls them literals. So what IS a literal, exactly?
Indeed. :) ChrisA
Jonathan, It sounds like you want to propose a competing PEP that provides your kwkey as a built-in as an alternative to PEP 637. I encourage you to do so, although I have no interest in acting as sponsor, and I fear that your window of opportunity to write a competing PEP is small. Alternatively, you could write a PEP that proposes a new kwkey object as complementary to, not competing with, PEP 637. That would provide a "kwkey" builtin similar to slice(), and people can explicitly opt-in to using it: # plan old PEP 637 semantics obj[spam, eggs, cheese='cheddar', aardvark=True] # explicit use of kwkey obj[kwkey(spam, eggs, cheese='cheddar', aardvark=True)] Another possibility is that you are not advocating for subscript keywords, and you prefer the status quo. (Your kwkey project counts as the status quo. It involves no new syntax or language features. Anyone can write their own kwkey class, or install a third-party library to use it.) Perhaps you agreed to co-author PEP 637 to have it rejected once and for all, not accepted, and you think that the current PEP is not neutral enough and advocates too strongly for the feature. PEPs can, and often are, written to be rejected. The PEP then becomes the canonical record of why the feature has been rejected. It is unfortunate if you and Stefano had radically opposite reasons for writing the PEP, but I think that's something the two of you should sort out and find some middle ground. Unfortunately Jonathan, I cannot tell from your post what your purpose is. I cannot tell if you oppose or support PEP 637, whether you want to propose a competing PEP, a complementary PEP, or just want to get credit for your kwkey project by having it mentioned in the PEP, or what exactly you want. I will comment further on the details in your post in another message. -- Steve
On Tue, Oct 06, 2020 at 04:04:37PM +0100, Jonathan Fine wrote:
I believe that the PEP meets that requirement by showing the inelegant workarounds used by popular projects in the absence subscript keywords. PEP 1 doesn't require a PEP to show that the existing language is *incapable* of solving the problem. Python is Turing Complete, so it can solve any computable problem. There are Turing Complete languages with as few as two symbols: https://esolangs.org/wiki/Jot so in a sense everything is syntactic sugar. The PEP, in my opinion, satisfactorily demonstrates that the existing methods of getting the semantics of subscript keywords are awkward and ugly.
Having to write something like `B(1, 2, a=3, b=4)[x]` would be an example of the awkwardness and inelegance that PEP 637 is designed to solve. If your kwkey package was a widely-known, commonly used third-party library, it would be a fantastic example of the sort of work-around that people fall back on in the absense of keyword subscripts. But I don't think that kwkey is widely used. It is a contrived library created in response to this issue being raised on the Python-Ideas mailing list, and as such, it would be a very weak example. A single widely-used example, such as from pandas or xarray, is worth a dozen contrived examples. So I'm not really sure what purpose you see your kwkey library taking in this PEP. As I mentioned in a previous post, I cannot tell whether you see it as competing with this PEP, complementary to the PEP, or perhaps as justifying the PEP?
But they don't. `f()` is legal; `f[]` is not, and will remain not legal. In addition, the precedences are different. In `f(a,b)` the use of the comma to separate arguments take precedence over the use of the comma to create tuples. But in `f[a,b]` it is the other way around.
Adding slice literals is a separate, unrelated issue. Allowing slice syntax `start:stop:step` outside of subscripts is independent and orthogonal to this PEP. Please don't burden this PEP with it.
Point of terminology: these are not literals (although informally people often call them that, including me on occassion). They are *displays*. One difference between a display and a literal can be seen from the byte-code generated: >>> import dis >>> dis.dis(compile('[1,2,3]', '', 'single')) 1 0 LOAD_CONST 0 (1) 2 LOAD_CONST 1 (2) 4 LOAD_CONST 2 (3) 6 BUILD_LIST 3 8 PRINT_EXPR 10 LOAD_CONST 3 (None) 12 RETURN_VALUE The constants 1, 2 and 3 are literals and the compiler can insert them directly into the code. The list [1, 2, 3] is not, the compiler has to insert code to create the list at runtime.
Adding slice literals isn't enough. I believe that the PEP already discusses that we cannot write: f(spam=1) = 2 del f(spam=1) and this is a fundamental difference between a subscript and a function call. Of course Python is Turing Complete and we can always perform the same operation by creating proxies and wrappers and other levels of indirection. As they say, every problem can be solved by adding sufficient levels of indirection, except the problem of having too many levels of indirection.
The most obvious problem that kwkey does not solve is the problem, how do I write a subscript with keywords without having to fall back on an inelegant and awkward work-around like kwkey? -- Steve
On Wed, Oct 7, 2020 at 1:46 PM Steven D'Aprano <steve@pearwood.info> wrote:
Be careful with this, as it's only a one-way proof. Any literal can and will be inserted directly as a constant, but there are some non-literals that can be as well:
Complex numbers and frozen sets don't technically have literals, but thanks to compiler magic, they can appear to. (Also, raw string literals are a thing, but in the compiled code, what you get is just a string object, further confusing this type of proof.) Although when it comes to f-strings, I honestly don't know what counts as a "literal". Is a single f-string a literal? What if two of them abut - are they one literal or two? The compiler builds the code to do the formatting and joining as though the parts of the string were combined (even if they're not all f-strings). But hey, it wouldn't be python-ideas without pedantry :) ChrisA
On Wed, Oct 07, 2020 at 01:59:24PM +1100, Chris Angelico wrote:
Sure, I believe that is a function of the keyhole optimizer: it can recognise certain non-literals, such as complex numbers and certain sets and tuples, and pre-generate them at compile time. That's a nice little optimization, especially swapping a mutable set for a frozen set because the compiler sees that the set is only ever read from, never mutated. (I think that was one of Serhiy's optimizations? Whoever it was, thank you.)
Raw strings are just an alternative syntax for strings. Just as we can write the same int as `0x4e` or `78`, or the same float as `12e-2` or `0.12`, or the same string as `"""abc"""` or `'\x61\x62\x63'`. They're not different "things", just different ways of writing the same thing.
Although when it comes to f-strings, I honestly don't know what counts as a "literal". Is a single f-string a literal?
An f-string is executable code. py> string = f"π/2 = {print('Calculating...') or __import__('math').atan(float('INF'))}" Calculating... py> string 'π/2 = 1.5707963267948966' Think of it as somewhat analogous to a list comprehension, in that it is an expression that contains executable code that runs at runtime, but it returns a string instead of a list, and rather than a for loop, you can include an arbitrary number of executable expressions. So not much like a list comprehension but a lot like calling eval on string literals :-)
But hey, it wouldn't be python-ideas without pedantry :)
Technically, there are lots of other mailing lists that are equally pedantic *wink* -- Steve
On Wed, Oct 7, 2020 at 9:20 PM Steven D'Aprano <steve@pearwood.info> wrote:
Right. They're the same object but different literal forms. That's the point. You can't use the disassembly to discover literal forms.
And yet the PEP defining them very definitely calls them literals. So what IS a literal, exactly?
Indeed. :) ChrisA
participants (3)
-
Chris Angelico -
Jonathan Fine -
Steven D'Aprano