Allow kwargs in __{get|set|del|}item__
While there is no restriction on passing dicts to getitem. Doing so tends to be a bit ugly. I have two main use cases in mind for this syntax. The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ``` Beyond relational queries my own personal use case is a shorthand for dataclasses / protocols. ``` foo: ProtoRecord[x=int, y=int] = DataRecord[x=int, y=int](0, 1) ``` where `DataRecord[field0=T0, ..., fieldk=Tk]` generates ``` @dataclass class Record: field0: T0 ... fieldk: Tk ``` and `ProtoRecord[field0=T0, ..., fieldk=Tk]` generates a similar protocol. Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index. Similar to how the all the positional arguments are gather into a single tuple. ``` class Foo: def __getitem__(self, idx): print(idx) f = Foo() f[x=1, y=2] # {'x': 1, 'y': 2} ``` This would make any legacy code using normal dicts as keys (I don't know how prevalent that is) automatically work with the new syntax. There doesn't necessarily need to be support for mixing of tuple based indexing and keyword indexing. i.e. ``` obj[0, x=1] # SyntaxError ``` I don't really know anything about parsers but I think the grammar could be extended without issue with the following rule: ``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' test ``` if `NAME '=' test` would result in ambiguity similar to argument it could be `test '=' test` with a block in ast.c - Caleb Donovick
On Fri, Oct 4, 2019 at 3:59 PM Caleb Donovick <donovick@cs.stanford.edu> wrote:
While there is no restriction on passing dicts to getitem. Doing so tends to be a bit ugly. I have two main use cases in mind for this syntax.
The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ```
Beyond relational queries my own personal use case is a shorthand for dataclasses / protocols. ``` foo: ProtoRecord[x=int, y=int] = DataRecord[x=int, y=int](0, 1) ``` where `DataRecord[field0=T0, ..., fieldk=Tk]` generates ``` @dataclass class Record: field0: T0 ... fieldk: Tk ``` and `ProtoRecord[field0=T0, ..., fieldk=Tk]` generates a similar protocol.
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index. Similar to how the all the positional arguments are gather into a single tuple. ``` class Foo: def __getitem__(self, idx): print(idx)
f = Foo() f[x=1, y=2] # {'x': 1, 'y': 2} ``` This would make any legacy code using normal dicts as keys (I don't know how prevalent that is) automatically work with the new syntax.
There doesn't necessarily need to be support for mixing of tuple based indexing and keyword indexing. i.e. ``` obj[0, x=1] # SyntaxError ```
I don't really know anything about parsers but I think the grammar could be extended without issue with the following rule: ``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' test ``` if `NAME '=' test` would result in ambiguity similar to argument it could be `test '=' test` with a block in ast.c
- Caleb Donovick
I thought about this in terms of labelled arrays like pandas and xarray, but I haven't talked to them about whether they would actually want to use such a feature so I hesitated to post it here. One possible approach I thought of for backwards-compatibility would be to be to create a new "kwslice" object, that would be API-incompatible and not derived from "slice". Anything that isn't explicitly programmed to deal with it would simply choke on the unknown object. The downside is the error message would be somewhat cryptic.
On 04/10/2019 20:34, Caleb Donovick wrote:
While there is no restriction on passing dicts to getitem. Doing so tends to be a bit ugly. I have two main use cases in mind for this syntax.
The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ```
OK, I'm not sure what you're trying to do here, which all on its own says that what you're doing isn't self-explanatory. Would I be right in thinking you want a shorthand for: where_x_1 = [k for k,v in db if v == 1] If so, I'd rather the comprehension, thanks. It at least says what it does.
Beyond relational queries my own personal use case is a shorthand for dataclasses / protocols. ``` foo: ProtoRecord[x=int, y=int] = DataRecord[x=int, y=int](0, 1) ``` where `DataRecord[field0=T0, ..., fieldk=Tk]` generates ``` @dataclass class Record: field0: T0 ... fieldk: Tk ``` and `ProtoRecord[field0=T0, ..., fieldk=Tk]` generates a similar protocol.
Explicit is better than implicit.
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index.
Um. Stop me if I'm wrong, but isn't that exactly a change to the interface of getitem? Sorry, I don't like it. -- Rhodri James *-* Kynesim Ltd
On Mon, Oct 7, 2019, at 09:22, Rhodri James wrote:
On 04/10/2019 20:34, Caleb Donovick wrote:
The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ```
OK, I'm not sure what you're trying to do here, which all on its own says that what you're doing isn't self-explanatory. Would I be right in thinking you want a shorthand for:
where_x_1 = [k for k,v in db if v == 1]
If so, I'd rather the comprehension, thanks. It at least says what it does.
It'd be [v for v in db if v['x'] == 1], anyway, but the point is the list comprehension can't be analyzed by frameworks like Pandas (to execute more efficiently) or SqlAlchemy (to turn it into a sql query that gets executed server-side). And for that matter can't return any type other than a list [or generator, if you used the generator expression syntax instead] I'm not convinced either (this is effectively a special syntax that only works for equality, whereas you can't do db[x>=0]), but using comprehension syntax for cases like these would require a much more sweeping change like AST literals (something that, for example, C# has, but which can't possibly work as smoothly as it does there without static typing)
On 07/10/2019 14:52, Random832 wrote:
On Mon, Oct 7, 2019, at 09:22, Rhodri James wrote:
On 04/10/2019 20:34, Caleb Donovick wrote:
The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ```
OK, I'm not sure what you're trying to do here, which all on its own says that what you're doing isn't self-explanatory. Would I be right in thinking you want a shorthand for:
where_x_1 = [k for k,v in db if v == 1]
If so, I'd rather the comprehension, thanks. It at least says what it does.
It'd be [v for v in db if v['x'] == 1], anyway,
Sorry. I regard this as more proof that what's being asked for isn't in the least bit obvious!
but the point is the list comprehension can't be analyzed by frameworks like Pandas (to execute more efficiently) or SqlAlchemy (to turn it into a sql query that gets executed server-side). And for that matter can't return any type other than a list [or generator, if you used the generator expression syntax instead]
Or a dict, or a set... or whatever the right tool for the specific job actually turns out to be. I still think it's completely wrong-headed syntax trying to do something that would be much better done another way, in particular explicitly. Caleb is right that "db[x=1]" looks prettier than all his alternatives, but that's a really low bar. Is there something inherently wrong with a functional interface? -- Rhodri James *-* Kynesim Ltd
On Mon, Oct 7, 2019, at 09:22, Rhodri James wrote:
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index.
Um. Stop me if I'm wrong, but isn't that exactly a change to the interface of getitem?
Missed this on my previous reply - I think his point is that that the __getitem__ function *itself* would continue to take a singular positional argument (and so the C API for tp_getitem etc wouldn't have to change), it would merely be an additional way to construct that argument, just as x[a, b] passes a tuple (a, b) rather than two positional arguments. Presumably, x[a, b, c=d, e=f] would pass a tuple (a, b, {c: d, e: f}).
On 07/10/2019 14:56, Random832 wrote:
On Mon, Oct 7, 2019, at 09:22, Rhodri James wrote:
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index.
Um. Stop me if I'm wrong, but isn't that exactly a change to the interface of getitem?
Missed this on my previous reply - I think his point is that that the __getitem__ function *itself* would continue to take a singular positional argument (and so the C API for tp_getitem etc wouldn't have to change), it would merely be an additional way to construct that argument, just as x[a, b] passes a tuple (a, b) rather than two positional arguments. Presumably, x[a, b, c=d, e=f] would pass a tuple (a, b, {c: d, e: f}).
It would still break a lot of user __getitem__ implementations IHMO. -- Rhodri James *-* Kynesim Ltd
On Mon, Oct 7, 2019, at 10:02, Rhodri James wrote:
It would still break a lot of user __getitem__ implementations IHMO.
It is already possible to pass any object. The fact that passing an object that an object is not designed to accept won't get good results doesn't mean the ability to do so "breaks" anything. And all existing syntax would continue to result in the same object being passed to __getitem__.
On 07/10/2019 15:43, Random832 wrote:
On Mon, Oct 7, 2019, at 10:02, Rhodri James wrote:
It would still break a lot of user __getitem__ implementations IHMO.
It is already possible to pass any object. The fact that passing an object that an object is not designed to accept won't get good results doesn't mean the ability to do so "breaks" anything. And all existing syntax would continue to result in the same object being passed to __getitem__.
You're right. Sorry, I have __getattr__ on the brain right now and totally misread what was written. -- Rhodri James *-* Kynesim Ltd
On Mon, Oct 07, 2019 at 02:22:22PM +0100, Rhodri James wrote:
On 04/10/2019 20:34, Caleb Donovick wrote:
``` where_x_1 = db[x=1] ```
which would be equivalent to the existing syntax where_x_1 = db[{'x': 1}] [Rhodi]
OK, I'm not sure what you're trying to do here, which all on its own says that what you're doing isn't self-explanatory.
I think it is pretty self-explanatory. Caleb wants to allow passing arguments to ``__getitem__`` by keyword, not just position. Subscripting obj[x] is, effectively, an alternate form of function call syntax with a number of differences and restrictions compared to obj(x): - calls the ``__getitem__`` dunder instead of ``__call__`` dunder - by convention, is supposed to be used for item indexing, key lookups, and type annotations, rather than arbitrary uses - the syntax is different, and highly restricted, compared to regular function calls: 1. there is no zero-argument form 2. the one argument form can accept any object (except a slice) 3. but only by position ``obj[x]`` not by keyword ``obj[spam=x]`` 4. the two and three argument forms use colons as seperators, rather than commas: ``obj[a:b:c]`` Caleb wants to remove the restriction 3.
Would I be right in thinking you want a shorthand for:
where_x_1 = [k for k,v in db if v == 1]
If so, I'd rather the comprehension, thanks. It at least says what it does.
I know what the list comp does because if I stare at it for a minute or two, and mentally run through the code, I can work out that it's equivalent to iterating over the db (which gives key,value pairs), and returning the key if the value equals one. I'd rather hide the implementation details behind a descriptive method name, or a documented query syntax. Just as we prefer to say: chunk = sequence[1:-2] rather than chunk = [sequence[i] for i in range(n:=len(sequence)) if 1 <= i < n - 2] I'm sure the list comp is more explicit and "says what it does" but I'm even more sure that you could count the number of experienced Python developers who prefer that over a slice on the fingers of one hand. -- Steven
On Mon, Oct 7, 2019 at 8:34 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Mon, Oct 07, 2019 at 02:22:22PM +0100, Rhodri James wrote:
On 04/10/2019 20:34, Caleb Donovick wrote:
``` where_x_1 = db[x=1] ```
which would be equivalent to the existing syntax
where_x_1 = db[{'x': 1}]
[Rhodi]
OK, I'm not sure what you're trying to do here, which all on its own says that what you're doing isn't self-explanatory.
I think it is pretty self-explanatory. Caleb wants to allow passing arguments to ``__getitem__`` by keyword, not just position.
Subscripting obj[x] is, effectively, an alternate form of function call syntax with a number of differences and restrictions compared to obj(x):
- calls the ``__getitem__`` dunder instead of ``__call__`` dunder
- by convention, is supposed to be used for item indexing, key lookups, and type annotations, rather than arbitrary uses
- the syntax is different, and highly restricted, compared to regular function calls:
1. there is no zero-argument form
d[...]
So, in short, your idea is to allow "=" signs inside `[]` get notation to be translated to dicts on the call, in the same way comma separated values are translated to tuples? I see no backwards syntax incompatibility in that, and the tuple-translation is indeed quite a helper in many cases. The use cases for this even in existing popular libraries, such as Pandas, is very clear. I am full +1 for this proposal - yes, it can help. I hope people can argue here and see the benefits without a lot of bike-shedding - and if it turns out good, I offer to help formalizing a proposal for that. On Fri, 4 Oct 2019 at 16:58, Caleb Donovick <donovick@cs.stanford.edu> wrote:
While there is no restriction on passing dicts to getitem. Doing so tends to be a bit ugly. I have two main use cases in mind for this syntax.
The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ```
Beyond relational queries my own personal use case is a shorthand for dataclasses / protocols. ``` foo: ProtoRecord[x=int, y=int] = DataRecord[x=int, y=int](0, 1) ``` where `DataRecord[field0=T0, ..., fieldk=Tk]` generates ``` @dataclass class Record: field0: T0 ... fieldk: Tk ``` and `ProtoRecord[field0=T0, ..., fieldk=Tk]` generates a similar protocol.
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index. Similar to how the all the positional arguments are gather into a single tuple. ``` class Foo: def __getitem__(self, idx): print(idx)
f = Foo() f[x=1, y=2] # {'x': 1, 'y': 2} ``` This would make any legacy code using normal dicts as keys (I don't know how prevalent that is) automatically work with the new syntax.
There doesn't necessarily need to be support for mixing of tuple based indexing and keyword indexing. i.e. ``` obj[0, x=1] # SyntaxError ```
I don't really know anything about parsers but I think the grammar could be extended without issue with the following rule: ``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' test ``` if `NAME '=' test` would result in ambiguity similar to argument it could be `test '=' test` with a block in ast.c
- Caleb Donovick _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EUGDRT... Code of Conduct: http://python.org/psf/codeofconduct/
On 07Oct2019 10:56, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
So, in short, your idea is to allow "=" signs inside `[]` get notation to be translated to dicts on the call,
Subjectively that seems like a tiny tiny win. I'm quite -1 on this idea; language spec bloat to neglible gain.
in the same way comma separated values are translated to tuples?
Chris pointed out to me recently that tuples don't need commas, the commas alone suffice. You see brackets _around_ tuples a lot because of precedence. Tuple assignment: x, y = y, x Tuple iteration (tuple assignment in a loop): for x, y in some_dict.items(): Store a tuple in a variable: t = x, y Call a function with an int, a tuple and an int: r = f(1, (2, 3), 4) Here we put some brakcets in to bind 2,3 as a tuple rather than as parameters, _no_ different to: r = 4 * (5 + 7) here we use brakcets to bind 5+7 together in preference to 4*5.
I see no backwards syntax incompatibility in that, and the tuple-translation is indeed quite a helper in many cases.
It isn't a translator, it is merely a situation where a tuple does not need surrounding brackets to avoid precedence giving a different result. Cheers, Cameron Simpson <cs@cskk.id.au>
On 08Oct2019 09:19, Cameron Simpson <cs@cskk.id.au> wrote:
On 07Oct2019 10:56, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
So, in short, your idea is to allow "=" signs inside `[]` get notation to be translated to dicts on the call,
Subjectively that seems like a tiny tiny win. I'm quite -1 on this idea; language spec bloat to neglible gain.
in the same way comma separated values are translated to tuples?
Chris pointed out to me recently that tuples don't need commas, the commas alone suffice. You see brackets _around_ tuples a lot because of precedence.
I wrote "tuples don't need commas". That should read "tuples don't need brackets". The brackets are not part of the tuple syntax. Ouch, Cameron Simpson <cs@cskk.id.au>
On 2019-10-07 23:22, Cameron Simpson wrote:
On 08Oct2019 09:19, Cameron Simpson <cs@cskk.id.au> wrote:
On 07Oct2019 10:56, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
So, in short, your idea is to allow "=" signs inside `[]` get notation to be translated to dicts on the call,
Subjectively that seems like a tiny tiny win. I'm quite -1 on this idea; language spec bloat to neglible gain.
in the same way comma separated values are translated to tuples?
Chris pointed out to me recently that tuples don't need commas, the commas alone suffice. You see brackets _around_ tuples a lot because of precedence.
I wrote "tuples don't need commas". That should read "tuples don't need brackets". The brackets are not part of the tuple syntax.
You're forgetting about the empty tuple, which has no comma, but does need parentheses.
On Mon, Oct 7, 2019, at 18:19, Cameron Simpson wrote:
On 07Oct2019 10:56, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
So, in short, your idea is to allow "=" signs inside `[]` get notation to be translated to dicts on the call,
Subjectively that seems like a tiny tiny win. I'm quite -1 on this idea; language spec bloat to neglible gain.
in the same way comma separated values are translated to tuples?
Chris pointed out to me recently that tuples don't need commas, the commas alone suffice. You see brackets _around_ tuples a lot because of precedence.
To be clear, since slices can be a member of the tuple, the subscripting syntax does have to handle commas specially - it doesn't simply fall out of the regular tuple syntax as you may be suggesting. You also, can't, for example, use "*sequence" inside the subscripting brackets, whereas you can for normal tuples even without parentheses.
class C: ... def __getitem__(self, i): return i ... C()[1:2, 3:4] (slice(1, 2, None), slice(3, 4, None)) C()[1, 2, *'ab'] File "<stdin>", line 1 C()[1, 2, *'ab'] ^ SyntaxError: invalid syntax x = 1, 2, *'ab' x (1, 2, 'a', 'b')
On 08Oct2019 09:19, Cameron Simpson <cs@cskk.id.au> wrote:
Chris pointed out to me recently that tuples don't need commas,
Cough, "brackets", cough.
the commas alone suffice. You see brackets _around_ tuples a lot because of precedence.
Reviewing my mail folder, it was actually Steven D'Aprano who pointed this out to me. Cheers, Cameron Simpson <cs@cskk.id.au>
On Mon, 7 Oct 2019 at 19:19, Cameron Simpson <cs@cskk.id.au> wrote:
On 07Oct2019 10:56, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
So, in short, your idea is to allow "=" signs inside `[]` get notation to be translated to dicts on the call,
Subjectively that seems like a tiny tiny win. I'm quite -1 on this idea; language spec bloat to neglible gain.
in the same way comma separated values are translated to tuples?
Chris pointed out to me recently that tuples don't need commas, the commas alone suffice. You see brackets _around_ tuples a lot because of precedence.
But in the case of index-contents, there _is_ a translation, as the slice syntax is allowed in any comma separated value inside the `[ ] `, and parsed to a slice object in the argument received by __getitem__. So, mixing single values and "named indexes", and slice-parsing apart, just parsing the "="s in df[axis="y"] to df.__getitem__({"axis": "y"}) seems to be quite usefull - not only to facilitate slicing and selection in dataframe, or multidimensional data structures, but also for ORMs, allowing a simple syntax to have filter expression as data. I agree that to maximise usability, ideally the syntax should get to the point of allowing combining "positional" and "named" index parts, and I think the object that would be passed to __getitem__ in this case have to be discussed further. But the dictionary idea seems straightforward, and won't complicate the language specs, as you say - it would just remoce redundant mark signals in image[{"x": 10, "y": 30] when on types "image[x=10, y=30]", just as you demonstrate tuples benefit from this in the continuation of your message. Now, on another line of though, I think a valid objection is that indeed, when calling a function we get this and the whole "*" and "**" mechanisms on the calling side and callee side - and rebuilding all expressiveness of function signatures in index would be too much. So, maybe, people thinking they need this feature could simply put together a ".get" method to accepted the named indexes, and all other features possible with function signatures.
Tuple assignment:
x, y = y, x
Tuple iteration (tuple assignment in a loop):
for x, y in some_dict.items():
Store a tuple in a variable:
t = x, y
Call a function with an int, a tuple and an int:
r = f(1, (2, 3), 4)
Here we put some brakcets in to bind 2,3 as a tuple rather than as parameters, _no_ different to:
r = 4 * (5 + 7)
here we use brakcets to bind 5+7 together in preference to 4*5.
I see no backwards syntax incompatibility in that, and the tuple-translation is indeed quite a helper in many cases.
It isn't a translator, it is merely a situation where a tuple does not need surrounding brackets to avoid precedence giving a different result.
Cheers, Cameron Simpson <cs@cskk.id.au>
On Tue, Oct 08, 2019 at 09:19:07AM +1100, Cameron Simpson wrote:
On 07Oct2019 10:56, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
So, in short, your idea is to allow "=" signs inside `[]` get notation to be translated to dicts on the call,
Subjectively that seems like a tiny tiny win. I'm quite -1 on this idea; language spec bloat to neglible gain.
As per Caleb's initial post, this is how Pandas currently does it: db[db['x'] == 1] Replacing that with db[x=1] seems like a HUGE win to me. Even db[{'x': 1}] is pretty clunky. -- Steven
It's really not a worthwhile win. It captures a tiny fraction of Pandas style filtering while complicating the syntax of Python. Here's another Pandas filter: db[db.x < 1] No help there with the next syntax. Here's another: db[(db.x == 1) | (db.y == 2)] A much better idea doesn't require any changes in Python, just a clever class method. Pandas did this for a while, but deprecated it because... reasons. Still, the OP is free to create his version: db['x=1'] Or db['x<1'] db['x=1 or y=2'] You can bikeshed the spelling of those predicates, but it doesn't matter, they are just strings that you can see however you decide is best. On Mon, Oct 7, 2019, 8:38 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Tue, Oct 08, 2019 at 09:19:07AM +1100, Cameron Simpson wrote:
On 07Oct2019 10:56, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
So, in short, your idea is to allow "=" signs inside `[]` get notation to be translated to dicts on the call,
Subjectively that seems like a tiny tiny win. I'm quite -1 on this idea; language spec bloat to neglible gain.
As per Caleb's initial post, this is how Pandas currently does it:
db[db['x'] == 1]
Replacing that with db[x=1] seems like a HUGE win to me.
Even db[{'x': 1}] is pretty clunky.
-- Steven _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RQH4VJ... Code of Conduct: http://python.org/psf/codeofconduct/
It captures a tiny fraction of Pandas style filtering while complicating the syntax of Python
Sure maybe I we can't represent all filters super concisely but at least inequalities, or any filter on single axis, would not be hard. E.g. db[x=LT(1)] == db[db.x < 1] Granted I don’t really see a way to express logical connectives between filters in a beautiful way -- beyond doing something like db[filter=OR(x=1, y=2)] which really isn't any better than db.filter(OR(x=1, y=2))
db['x=1']
Ah yes cause parsing strings is a reasonable replacement for language support. I have no idea why Pandas dropped support for this but I have to imagine it's because it's horribly ugly, prone to bugs and difficult to metaprogram. Semantically meaningful strings are terrible. Everytime I write a string literal for any reason other than I a want human to read that string I die a little inside. Which is part of the reason I want db[x=1] instead of db[{'x':1}]. And yes everything is a string under the hood in python but that doesn't make semantic strings less terrible. Really under the hood (in assembly) everything is gotos but that doesn't make their use better either. /rant On Mon, Oct 7, 2019 at 10:07 PM David Mertz <mertz@gnosis.cx> wrote:
It's really not a worthwhile win. It captures a tiny fraction of Pandas style filtering while complicating the syntax of Python. Here's another Pandas filter:
db[db.x < 1]
No help there with the next syntax. Here's another:
db[(db.x == 1) | (db.y == 2)]
A much better idea doesn't require any changes in Python, just a clever class method. Pandas did this for a while, but deprecated it because... reasons. Still, the OP is free to create his version:
db['x=1']
Or
db['x<1'] db['x=1 or y=2']
You can bikeshed the spelling of those predicates, but it doesn't matter, they are just strings that you can see however you decide is best.
On Mon, Oct 7, 2019, 8:38 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Tue, Oct 08, 2019 at 09:19:07AM +1100, Cameron Simpson wrote:
On 07Oct2019 10:56, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
So, in short, your idea is to allow "=" signs inside `[]` get notation to be translated to dicts on the call,
Subjectively that seems like a tiny tiny win. I'm quite -1 on this idea; language spec bloat to neglible gain.
As per Caleb's initial post, this is how Pandas currently does it:
db[db['x'] == 1]
Replacing that with db[x=1] seems like a HUGE win to me.
Even db[{'x': 1}] is pretty clunky.
-- Steven _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RQH4VJ... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5O7BLO... Code of Conduct: http://python.org/psf/codeofconduct/
Caleb Donovick wrote:
It captures a tiny fraction of Pandas style filtering while complicating the syntax of Python Sure maybe I we can't represent all filters super concisely but at least inequalities, or any filter on single axis, would not be hard. E.g. db[x=LT(1)] == db[db.x < 1]
Django for example allows filtering with keyword args including inequalities in this way: `x__lt=1` (which translates to "x < 1"). Whether that's more readable or not is another question but at least with "keyword args" in `[]` every package could come up with its own specifications on how this is used.
db['x=1'] Ah yes cause parsing strings is a reasonable replacement for language support. I have no idea why Pandas dropped support for this but I have to imagine it's because it's horribly ugly, prone to bugs and difficult to
Granted I don’t really see a way to express logical connectives between filters in a beautiful way -- beyond doing something like db[filter=OR(x=1, y=2)] which really isn't any better than db.filter(OR(x=1, y=2)) metaprogram.
Pandas didn't drop the support for query strings, they can be used via the `df.query` method. For example: `df.query('x == 1 and y == 2')`. That's equally explicit but creating query strings (dynamically) has of course its downsides. Given that `[]` is used for element access, extending it to further use cases is indeed appealing. On the other hand one could always argue that a functional interface equally does the job, e.g. pandas could provide a function accepting keyword args: `df.select(x=1, y=2)`. Or the Django style: `Q(x=1) | Q(y__lt=2)`. Semantically meaningful strings are terrible. Everytime I
write a string literal for any reason other than I a want human to read that string I die a little inside. Which is part of the reason I want db[x=1] instead of db[{'x':1}]. And yes everything is a string under the hood in python but that doesn't make semantic strings less terrible. Really under the hood (in assembly) everything is gotos but that doesn't make their use better either. /rant On Mon, Oct 7, 2019 at 10:07 PM David Mertz mertz@gnosis.cx wrote:
It's really not a worthwhile win. It captures a tiny fraction of Pandas style filtering while complicating the syntax of Python. Here's another Pandas filter: db[db.x < 1]
No help there with the next syntax. Here's another: db[(db.x == 1) | (db.y == 2)]
A much better idea doesn't require any changes in Python, just a clever class method. Pandas did this for a while, but deprecated it because... reasons. Still, the OP is free to create his version: db['x=1']
Or db['x<1'] db['x=1 or y=2']
You can bikeshed the spelling of those predicates, but it doesn't matter, they are just strings that you can see however you decide is best. On Mon, Oct 7, 2019, 8:38 PM Steven D'Aprano steve@pearwood.info wrote: On Tue, Oct 08, 2019 at 09:19:07AM +1100, Cameron Simpson wrote: On 07Oct2019 10:56, Joao S. O. Bueno jsbueno@python.org.br wrote: So, in short, your idea is to allow "=" signs inside [] get notation to be translated to dicts on the call, Subjectively that seems like a tiny tiny win. I'm quite -1 on this idea; language spec bloat to neglible gain. As per Caleb's initial post, this is how Pandas currently does it: db[db['x'] == 1]
Replacing that with db[x=1] seems like a HUGE win to me. Even db[{'x': 1}] is pretty clunky. -- Steven
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RQH4VJ... Code of Conduct: http://python.org/psf/codeofconduct/
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5O7BLO... Code of Conduct: http://python.org/psf/codeofconduct/
On Mon, Oct 7, 2019 at 8:37 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Tue, Oct 08, 2019 at 09:19:07AM +1100, Cameron Simpson wrote:
On 07Oct2019 10:56, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
So, in short, your idea is to allow "=" signs inside `[]` get notation to be translated to dicts on the call,
Subjectively that seems like a tiny tiny win. I'm quite -1 on this idea; language spec bloat to neglible gain.
As per Caleb's initial post, this is how Pandas currently does it:
db[db['x'] == 1]
Replacing that with db[x=1] seems like a HUGE win to me.
Even db[{'x': 1}] is pretty clunky.
At least in my opinion, this is even more important for xarray than pandas, since xarray has named dimensions. Positional indexing really doesn't make all that much sense with xarray, being able to use keyword-based indexing would drastically simplify things.
On 2019-10-07 20:35, Steven D'Aprano wrote:
As per Caleb's initial post, this is how Pandas currently does it:
db[db['x'] == 1]
Replacing that with db[x=1] seems like a HUGE win to me.
Even db[{'x': 1}] is pretty clunky.
For Pandas, db['x'] which creates an expression involving `x`, and can be used in more than one place. It is small matter to start your method (or class, or module) with some declarations: x = db['x'] which is boilerplate, but also easily ignored. Then you can write db[x==1] Which is just as clean, and more specific, than whatever `db[x=1]` means. I do not believe this syntax will help Pandas because equality is just one-of-many useful operators. This does not help me if I want to say db[x>1] or say db[x==1 || t > 2] I am not sure how many expressions are actually found in realworld code. Examples are written out, but most code is not examples. Most code, that I have seen, manipulates expressions that come from elsewhere. Here is an Elasticsearch expression: e = {"term": {"x": 1}} which is not seen in the code; assignment to `e` is done elsewhere, maybe from a config file, maybe from another application. The code only sees `e`. Therefore an example like db[db['x'] == 1] looks like db[e] in realworld code.
In fact, that would be a cool feature for ORMs. IMHO instead of ugly call chain with filters, slicing is a better option (on `__class_getattr__`). As said there are some disadvantages but i think this proposal deserves a PEP. On Fri, Oct 4, 2019, 10:59 PM Caleb Donovick <donovick@cs.stanford.edu> wrote:
While there is no restriction on passing dicts to getitem. Doing so tends to be a bit ugly. I have two main use cases in mind for this syntax.
The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ```
Beyond relational queries my own personal use case is a shorthand for dataclasses / protocols. ``` foo: ProtoRecord[x=int, y=int] = DataRecord[x=int, y=int](0, 1) ``` where `DataRecord[field0=T0, ..., fieldk=Tk]` generates ``` @dataclass class Record: field0: T0 ... fieldk: Tk ``` and `ProtoRecord[field0=T0, ..., fieldk=Tk]` generates a similar protocol.
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index. Similar to how the all the positional arguments are gather into a single tuple. ``` class Foo: def __getitem__(self, idx): print(idx)
f = Foo() f[x=1, y=2] # {'x': 1, 'y': 2} ``` This would make any legacy code using normal dicts as keys (I don't know how prevalent that is) automatically work with the new syntax.
There doesn't necessarily need to be support for mixing of tuple based indexing and keyword indexing. i.e. ``` obj[0, x=1] # SyntaxError ```
I don't really know anything about parsers but I think the grammar could be extended without issue with the following rule: ``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' test ``` if `NAME '=' test` would result in ambiguity similar to argument it could be `test '=' test` with a block in ast.c
- Caleb Donovick _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EUGDRT... Code of Conduct: http://python.org/psf/codeofconduct/
On 7 Oct 2019, at 16:36, Batuhan Taskaya <isidentical@gmail.com> wrote:
In fact, that would be a cool feature for ORMs. IMHO instead of ugly call chain with filters, slicing is a better option (on `__class_getattr__`). As said there are some disadvantages but i think this proposal deserves a PEP.
I really don't see why. Is () so much different than [] to you? Because that seems like the only difference here.
On Fri, Oct 4, 2019, 10:59 PM Caleb Donovick <donovick@cs.stanford.edu> wrote: While there is no restriction on passing dicts to getitem. Doing so tends to be a bit ugly. I have two main use cases in mind for this syntax.
The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ```
Beyond relational queries my own personal use case is a shorthand for dataclasses / protocols. ``` foo: ProtoRecord[x=int, y=int] = DataRecord[x=int, y=int](0, 1) ``` where `DataRecord[field0=T0, ..., fieldk=Tk]` generates ``` @dataclass class Record: field0: T0 ... fieldk: Tk ``` and `ProtoRecord[field0=T0, ..., fieldk=Tk]` generates a similar protocol.
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index. Similar to how the all the positional arguments are gather into a single tuple. ``` class Foo: def __getitem__(self, idx): print(idx)
f = Foo() f[x=1, y=2] # {'x': 1, 'y': 2} ``` This would make any legacy code using normal dicts as keys (I don't know how prevalent that is) automatically work with the new syntax.
There doesn't necessarily need to be support for mixing of tuple based indexing and keyword indexing. i.e. ``` obj[0, x=1] # SyntaxError ```
I don't really know anything about parsers but I think the grammar could be extended without issue with the following rule: ``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' test ``` if `NAME '=' test` would result in ambiguity similar to argument it could be `test '=' test` with a block in ast.c
- Caleb Donovick _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EUGDRT... Code of Conduct: http://python.org/psf/codeofconduct/
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LG6OKC... Code of Conduct: http://python.org/psf/codeofconduct/
On Mon, Oct 07, 2019 at 05:18:39PM +0200, Anders Hovmöller wrote:
On 7 Oct 2019, at 16:36, Batuhan Taskaya <isidentical@gmail.com> wrote:
In fact, that would be a cool feature for ORMs. IMHO instead of ugly call chain with filters, slicing is a better option (on `__class_getattr__`). As said there are some disadvantages but i think this proposal deserves a PEP.
I really don't see why. Is () so much different than [] to you? Because that seems like the only difference here.
To me, "()" says "arbitrary function call potentially with side- effects". "[]" says "lookup". We use list[idx] and dict[key], not list(idx) and dict(key). It is big enough that the distinction between () and [] calling conventions has been built into the language since before Python 1.0. And it is big enough that when we were looking for syntax to use for type annotations, we settled on Union[Spam, Eggs] rather than round brackets Union(Spam, Eggs). So yes, the difference between [] and () is a big difference. -- Steven
On Mon, Oct 7, 2019 at 11:19 AM Anders Hovmöller <boxed@killingar.net> wrote:
On 7 Oct 2019, at 16:36, Batuhan Taskaya <isidentical@gmail.com> wrote:
In fact, that would be a cool feature for ORMs. IMHO instead of ugly call chain with filters, slicing is a better option (on `__class_getattr__`). As said there are some disadvantages but i think this proposal deserves a PEP.
I really don't see why. Is () so much different than [] to you? Because that seems like the only difference here.
You can't assign to function calls. So: x(b=1) = 5 won't work, but x[b=1] = 5 could work
On 8 Oct 2019, at 18:27, Todd <toddrjen@gmail.com> wrote:
On Mon, Oct 7, 2019 at 11:19 AM Anders Hovmöller <boxed@killingar.net <mailto:boxed@killingar.net>> wrote:
On 7 Oct 2019, at 16:36, Batuhan Taskaya <isidentical@gmail.com <mailto:isidentical@gmail.com>> wrote:
In fact, that would be a cool feature for ORMs. IMHO instead of ugly call chain with filters, slicing is a better option (on `__class_getattr__`). As said there are some disadvantages but i think this proposal deserves a PEP.
I really don't see why. Is () so much different than [] to you? Because that seems like the only difference here.
You can't assign to function calls. So:
x(b=1) = 5
won't work, but
x[b=1] = 5
could work
Ah. That's a strong argument I can get behind!
On Oct 4, 2019, at 12:34, Caleb Donovick <donovick@cs.stanford.edu> wrote:
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index. Similar to how the all the positional arguments are gather into a single tuple. ``` class Foo: def __getitem__(self, idx): print(idx)
f = Foo() f[x=1, y=2] # {'x': 1, 'y': 2} ``` This would make any legacy code using normal dicts as keys (I don't know how prevalent that is) automatically work with the new syntax.
There doesn't necessarily need to be support for mixing of tuple based indexing and keyword indexing. i.e. ``` obj[0, x=1] # SyntaxError
I think it might be better if it actually passed them as keyword arguments. Then, using the syntax with existing objects would raise a TypeError saying that the type doesn’t take keyword arguments, instead of saying that dict can’t be converted to an int or something else less clear. Also, I think there are user implementations that accept any iterable, whether to treat it as a tuple or as an array-like, and a dict is an iterable of its keys, so it might do the wrong thing rather than raising at all. It would be a bit weird that what look like positional arguments are actually a tuple, but what look like keyword arguments are keyword arguments, but I don’t think that’s too bad. Maybe it would help if you showed a realistic __getitem__ that did the existing type-switching to handle slice, ellipsis, thing with __index__, or tuple of the above, and then also did the switching on dict or also took **kwargs. I suspect that either way, it would turn out that most of the complexity is due to the existing complicated interface of __getitem__ and keywords don’t make it much different. But I’m not sure that would be true. And I think many of the objections are from people who suspect the opposite.
Are you aware of PEP 472 https://www.python.org/dev/peps/pep-0472 ? Maybe you have something different in mind, but for me your idea looks pretty the same. While the PEP 472 is in Rejected, Abandoned section, I do not remember any serious criticism of this idea. It’s just that the authors of that proposal lost interest and it did not receive further progress. And in this regard, over time, it was abandoned. with kind regards, -gdg пт, 4 окт. 2019 г. в 23:01, Caleb Donovick <donovick@cs.stanford.edu>:
While there is no restriction on passing dicts to getitem. Doing so tends to be a bit ugly. I have two main use cases in mind for this syntax.
The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ```
Beyond relational queries my own personal use case is a shorthand for dataclasses / protocols. ``` foo: ProtoRecord[x=int, y=int] = DataRecord[x=int, y=int](0, 1) ``` where `DataRecord[field0=T0, ..., fieldk=Tk]` generates ``` @dataclass class Record: field0: T0 ... fieldk: Tk ``` and `ProtoRecord[field0=T0, ..., fieldk=Tk]` generates a similar protocol.
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index. Similar to how the all the positional arguments are gather into a single tuple. ``` class Foo: def __getitem__(self, idx): print(idx)
f = Foo() f[x=1, y=2] # {'x': 1, 'y': 2} ``` This would make any legacy code using normal dicts as keys (I don't know how prevalent that is) automatically work with the new syntax.
There doesn't necessarily need to be support for mixing of tuple based indexing and keyword indexing. i.e. ``` obj[0, x=1] # SyntaxError ```
I don't really know anything about parsers but I think the grammar could be extended without issue with the following rule: ``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' test ``` if `NAME '=' test` would result in ambiguity similar to argument it could be `test '=' test` with a block in ast.c
- Caleb Donovick _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EUGDRT... Code of Conduct: http://python.org/psf/codeofconduct/
Are you aware of PEP 472 https://www.python.org/dev/peps/pep-0472 ?
That is indeed the same idea, though perhaps the details are a bit different. This example from the PEP: gridValues[x=3, y=5, z=8] Makes me wonder: Should that yield the same results as: gridValues[3,5,8] Much like positional and keyword arguments work on function calls? I suppose that would be up to the implementation, as __getitem__ doesn’t currently provide much help with parsing out what’s in there, other than making slice objects. But if something like this did go forward, it would be nice to provide utilities, maybe built in, that would parse and sort out the “arguments”, similar to function calls. -CHB Maybe you have something different in mind, but for me your idea looks
pretty the same. While the PEP 472 is in Rejected, Abandoned section, I do not remember any serious criticism of this idea. It’s just that the authors of that proposal lost interest and it did not receive further progress. And in this regard, over time, it was abandoned.
with kind regards, -gdg
пт, 4 окт. 2019 г. в 23:01, Caleb Donovick <donovick@cs.stanford.edu>:
While there is no restriction on passing dicts to getitem. Doing so tends to be a bit ugly. I have two main use cases in mind for this syntax.
The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ```
Beyond relational queries my own personal use case is a shorthand for dataclasses / protocols. ``` foo: ProtoRecord[x=int, y=int] = DataRecord[x=int, y=int](0, 1) ``` where `DataRecord[field0=T0, ..., fieldk=Tk]` generates ``` @dataclass class Record: field0: T0 ... fieldk: Tk ``` and `ProtoRecord[field0=T0, ..., fieldk=Tk]` generates a similar protocol.
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index. Similar to how the all the positional arguments are gather into a single tuple. ``` class Foo: def __getitem__(self, idx): print(idx)
f = Foo() f[x=1, y=2] # {'x': 1, 'y': 2} ``` This would make any legacy code using normal dicts as keys (I don't know how prevalent that is) automatically work with the new syntax.
There doesn't necessarily need to be support for mixing of tuple based indexing and keyword indexing. i.e. ``` obj[0, x=1] # SyntaxError ```
I don't really know anything about parsers but I think the grammar could be extended without issue with the following rule: ``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' test ``` if `NAME '=' test` would result in ambiguity similar to argument it could be `test '=' test` with a block in ast.c
- Caleb Donovick _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EUGDRT... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XPQC6A... Code of Conduct: http://python.org/psf/codeofconduct/
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
I think it might be better if it actually passed them as keyword arguments.
If only keyword arguments are passed what happens to the positional index? Is it the empty tuple? Currently subscript with no index (`dict()[]`) is a syntax error should it continue to be?
Also, I think there are user implementations that accept any iterable, whether to treat it as a tuple or as an array-like, and a dict is an iterable of its keys, so it might do the wrong thing rather than raising at all.
I had not thought about this. I have a lot of code in the form: ``` if not isinstance(key, iterable): key = key, # do stuff assuming key is an iterable ``` Which would have very strange behavior if a dict was passed. On Mon, Oct 7, 2019 at 2:44 PM Christopher Barker <pythonchb@gmail.com> wrote:
Are you aware of PEP 472 https://www.python.org/dev/peps/pep-0472 ?
That is indeed the same idea, though perhaps the details are a bit different.
This example from the PEP:
gridValues[x=3, y=5, z=8]
Makes me wonder:
Should that yield the same results as:
gridValues[3,5,8]
Much like positional and keyword arguments work on function calls?
I suppose that would be up to the implementation, as __getitem__ doesn’t currently provide much help with parsing out what’s in there, other than making slice objects.
But if something like this did go forward, it would be nice to provide utilities, maybe built in, that would parse and sort out the “arguments”, similar to function calls.
-CHB
Maybe you have something different in mind, but for me your idea looks
pretty the same. While the PEP 472 is in Rejected, Abandoned section, I do not remember any serious criticism of this idea. It’s just that the authors of that proposal lost interest and it did not receive further progress. And in this regard, over time, it was abandoned.
with kind regards, -gdg
пт, 4 окт. 2019 г. в 23:01, Caleb Donovick <donovick@cs.stanford.edu>:
While there is no restriction on passing dicts to getitem. Doing so tends to be a bit ugly. I have two main use cases in mind for this syntax.
The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ```
Beyond relational queries my own personal use case is a shorthand for dataclasses / protocols. ``` foo: ProtoRecord[x=int, y=int] = DataRecord[x=int, y=int](0, 1) ``` where `DataRecord[field0=T0, ..., fieldk=Tk]` generates ``` @dataclass class Record: field0: T0 ... fieldk: Tk ``` and `ProtoRecord[field0=T0, ..., fieldk=Tk]` generates a similar protocol.
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index. Similar to how the all the positional arguments are gather into a single tuple. ``` class Foo: def __getitem__(self, idx): print(idx)
f = Foo() f[x=1, y=2] # {'x': 1, 'y': 2} ``` This would make any legacy code using normal dicts as keys (I don't know how prevalent that is) automatically work with the new syntax.
There doesn't necessarily need to be support for mixing of tuple based indexing and keyword indexing. i.e. ``` obj[0, x=1] # SyntaxError ```
I don't really know anything about parsers but I think the grammar could be extended without issue with the following rule: ``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' test ``` if `NAME '=' test` would result in ambiguity similar to argument it could be `test '=' test` with a block in ast.c
- Caleb Donovick _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EUGDRT... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XPQC6A... Code of Conduct: http://python.org/psf/codeofconduct/
-- Christopher Barker, PhD
Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Are you aware of PEP 472 https://www.python.org/dev/peps/pep-0472 ? Maybe you have something different in mind, but for me your idea looks pretty the same. While the PEP 472 is in Rejected, Abandoned section, I do not remember any serious criticism of this idea. It’s just that the authors of that proposal lost interest and it did not receive further progress. And in this regard, over time, it was abandoned.
I was not aware of it. I had exactly this in mind. Specifically the strict dictionary strategy https://www.python.org/dev/peps/pep-0472/#strategy-strict-dictionary On Mon, Oct 7, 2019 at 5:56 PM Caleb Donovick <donovick@cs.stanford.edu> wrote:
I think it might be better if it actually passed them as keyword arguments.
If only keyword arguments are passed what happens to the positional index? Is it the empty tuple?
Currently subscript with no index (`dict()[]`) is a syntax error should it continue to be?
Also, I think there are user implementations that accept any iterable, whether to treat it as a tuple or as an array-like, and a dict is an iterable of its keys, so it might do the wrong thing rather than raising at all.
I had not thought about this. I have a lot of code in the form: ``` if not isinstance(key, iterable): key = key, # do stuff assuming key is an iterable ``` Which would have very strange behavior if a dict was passed.
On Mon, Oct 7, 2019 at 2:44 PM Christopher Barker <pythonchb@gmail.com> wrote:
Are you aware of PEP 472 https://www.python.org/dev/peps/pep-0472 ?
That is indeed the same idea, though perhaps the details are a bit different.
This example from the PEP:
gridValues[x=3, y=5, z=8]
Makes me wonder:
Should that yield the same results as:
gridValues[3,5,8]
Much like positional and keyword arguments work on function calls?
I suppose that would be up to the implementation, as __getitem__ doesn’t currently provide much help with parsing out what’s in there, other than making slice objects.
But if something like this did go forward, it would be nice to provide utilities, maybe built in, that would parse and sort out the “arguments”, similar to function calls.
-CHB
Maybe you have something different in mind, but for me your idea looks
pretty the same. While the PEP 472 is in Rejected, Abandoned section, I do not remember any serious criticism of this idea. It’s just that the authors of that proposal lost interest and it did not receive further progress. And in this regard, over time, it was abandoned.
with kind regards, -gdg
пт, 4 окт. 2019 г. в 23:01, Caleb Donovick <donovick@cs.stanford.edu>:
While there is no restriction on passing dicts to getitem. Doing so tends to be a bit ugly. I have two main use cases in mind for this syntax.
The first and perhaps the most obvious, is doing relational queries. ``` where_x_1 = db[x=1] ``` is more beautiful than ``` where_x_1 = db[dict(x=1)] where_x_1 = db[{'x': 1}] # or by abusing slices where_x_1 = db['x':1] # or in the style of Pandas where_x_1 = db[db['x'] == 1] ```
Beyond relational queries my own personal use case is a shorthand for dataclasses / protocols. ``` foo: ProtoRecord[x=int, y=int] = DataRecord[x=int, y=int](0, 1) ``` where `DataRecord[field0=T0, ..., fieldk=Tk]` generates ``` @dataclass class Record: field0: T0 ... fieldk: Tk ``` and `ProtoRecord[field0=T0, ..., fieldk=Tk]` generates a similar protocol.
Allowing key value pairs in geitem need not change the interface of getitem. All the key value pairs could be collected as a dict and passed to getitem as the index. Similar to how the all the positional arguments are gather into a single tuple. ``` class Foo: def __getitem__(self, idx): print(idx)
f = Foo() f[x=1, y=2] # {'x': 1, 'y': 2} ``` This would make any legacy code using normal dicts as keys (I don't know how prevalent that is) automatically work with the new syntax.
There doesn't necessarily need to be support for mixing of tuple based indexing and keyword indexing. i.e. ``` obj[0, x=1] # SyntaxError ```
I don't really know anything about parsers but I think the grammar could be extended without issue with the following rule: ``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' test ``` if `NAME '=' test` would result in ambiguity similar to argument it could be `test '=' test` with a block in ast.c
- Caleb Donovick _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EUGDRT... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XPQC6A... Code of Conduct: http://python.org/psf/codeofconduct/
-- Christopher Barker, PhD
Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Oct 7, 2019, at 14:56, Caleb Donovick <donovick@cs.stanford.edu> wrote:
I think it might be better if it actually passed them as keyword arguments.
If only keyword arguments are passed what happens to the positional index? Is it the empty tuple?
That seems like the obvious, and also most useful, answer.
Currently subscript with no index (`dict()[]`) is a syntax error should it continue to be?
Why not? If you want to look up the empty tuple today, you can, you just have to be explicit: `d[()]`. This rarely comes up, but when it does, this reads pretty obvious (especially given that when people are using tuples as dict keys, they usually wrap them in parens anyway, even if it’s not necessary, because it looks clearer). I don’t see any reason to change that. Now, assuming you want to allow ** in getitem lookups, this would add another way to write the same thing: `d[**{}]`. I suppose this should be legal (since it could easily come up with a dynamic `d[**kw]` when there happen to be no keywords), but I don’t think anyone will ever be tempted to write it, and I don’t think anyone would be confused for more than a few seconds if they did, and I don’t think that allowing it requires us to start allowing `d[]`. That’s a restriction on the syntax for readability, not a limitation on the semantics, and the same readability issue still applies.
Also, I think there are user implementations that accept any iterable, whether to treat it as a tuple or as an array-like, and a dict is an iterable of its keys, so it might do the wrong thing rather than raising at all.
I had not thought about this. I have a lot of code in the form: ``` if not isinstance(key, iterable): key = key, # do stuff assuming key is an iterable ``` Which would have very strange behavior if a dict was passed.
Exactly. And I think what you want to happen here is a `TypeError: Spam.__getitem__ takes no keyword arguments` or similar, not treating the keywords as an iterable and ignoring their values. That’s why I think passing kwargs separately rather than just making key a dict might make more sense. But then I haven’t put a huge amount of thought into this. Given that there’s an existing abandoned PEP, I’ll bet this was already hashed our in more detail, and you probably want to go back to that, come up with answers to the objections and open questions, and start over
Why not?
What if I want a getitem that only has keyword arguments? I have to take the empty tuple as a positional argument, instead of just ensuring that the key is a dict.
Now, assuming you want to allow ** in getitem lookups
I don't. *args are not allowed in subscripts either. However, that would make the syntax pretty much useless (in the keyword version) as indices could not be passed to a function. While I agree, not using a keywords create a certain class of bugs, those bugs already exists. At least in the sense that if I pass a dictionary as index things are going to get weird. However, I'd rather not have all the special casing that comes with keyword arguments. On Mon, Oct 7, 2019 at 7:57 PM Andrew Barnert <abarnert@yahoo.com> wrote:
On Oct 7, 2019, at 14:56, Caleb Donovick <donovick@cs.stanford.edu> wrote:
I think it might be better if it actually passed them as keyword
arguments.
If only keyword arguments are passed what happens to the positional
index? Is it the empty tuple?
That seems like the obvious, and also most useful, answer.
Currently subscript with no index (`dict()[]`) is a syntax error should it continue to be?
Why not? If you want to look up the empty tuple today, you can, you just have to be explicit: `d[()]`. This rarely comes up, but when it does, this reads pretty obvious (especially given that when people are using tuples as dict keys, they usually wrap them in parens anyway, even if it’s not necessary, because it looks clearer). I don’t see any reason to change that.
Now, assuming you want to allow ** in getitem lookups, this would add another way to write the same thing: `d[**{}]`. I suppose this should be legal (since it could easily come up with a dynamic `d[**kw]` when there happen to be no keywords), but I don’t think anyone will ever be tempted to write it, and I don’t think anyone would be confused for more than a few seconds if they did, and I don’t think that allowing it requires us to start allowing `d[]`. That’s a restriction on the syntax for readability, not a limitation on the semantics, and the same readability issue still applies.
Also, I think there are user implementations that accept any iterable, whether to treat it as a tuple or as an array-like, and a dict is an iterable of its keys, so it might do the wrong thing rather than raising at all.
I had not thought about this. I have a lot of code in the form: ``` if not isinstance(key, iterable): key = key, # do stuff assuming key is an iterable ``` Which would have very strange behavior if a dict was passed.
Exactly. And I think what you want to happen here is a `TypeError: Spam.__getitem__ takes no keyword arguments` or similar, not treating the keywords as an iterable and ignoring their values. That’s why I think passing kwargs separately rather than just making key a dict might make more sense.
But then I haven’t put a huge amount of thought into this. Given that there’s an existing abandoned PEP, I’ll bet this was already hashed our in more detail, and you probably want to go back to that, come up with answers to the objections and open questions, and start over
On Tue, Oct 8, 2019 at 12:47 PM Caleb Donovick <donovick@cs.stanford.edu> wrote:
Why not?
What if I want a getitem that only has keyword arguments? I have to take the empty tuple as a positional argument, instead of just ensuring that the key is a dict.
But what if you wanted to take both positional AND keyword?You can't ensure that the key is both a dict and a tuple. Cleaner to continue passing the tuple exactly as normal. ChrisA
But what if you wanted to take both positional AND keyword?
I was suggesting that that wouldn't be allowed. So subscript either has a single argument, a tuple of arguments, or a dictionary of arguments. Allowing both has some advantages but is less cleanly integratible. -- Caleb Donovick On Tue, Oct 8, 2019 at 12:16 AM Chris Angelico <rosuav@gmail.com> wrote:
On Tue, Oct 8, 2019 at 12:47 PM Caleb Donovick <donovick@cs.stanford.edu> wrote:
Why not?
What if I want a getitem that only has keyword arguments? I have to take
the empty tuple as a positional argument, instead of just ensuring that the key is a dict.
But what if you wanted to take both positional AND keyword?You can't ensure that the key is both a dict and a tuple.
Cleaner to continue passing the tuple exactly as normal.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/C4FHJR... Code of Conduct: http://python.org/psf/codeofconduct/
On Oct 7, 2019, at 21:21, Caleb Donovick <donovick@cs.stanford.edu> wrote:
But what if you wanted to take both positional AND keyword?
I was suggesting that that wouldn't be allowed. So subscript either has a single argument, a tuple of arguments, or a dictionary of arguments. Allowing both has some advantages but is less cleanly integratible.
The problem is that half the examples people conjure up involve both: using the keywords as options, while using the positional arguments for the actual indices. Calling the proposal “kwargs in getitem” encourages that thinking, because that’s the prototypical reason for kwargs in function calls. If there were non-toy examples, so people didn’t have to imagine how it would be used for themselves, that might be helpful. Why is `d[x=1]` useful but `d[1, flag=True]` not needed? If you showed real Pandas queries or whatever instead of toy ones, it might be clearer. And you could challenge people who want flags to give real examples that you could then dissect. Also, an actual __get/set/delitem__ implementation using the proposed feature would help. You can argue that dealing with the dicts-are-iterable issue won’t be a problem in practice, but just showing methods that deal with it and remain perfectly readable makes the case a lot better.
On Tue, Oct 8, 2019 at 12:22 PM Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:
On Oct 7, 2019, at 21:21, Caleb Donovick <donovick@cs.stanford.edu> wrote:
But what if you wanted to take both positional AND keyword?
I was suggesting that that wouldn't be allowed. So subscript either has
a single argument, a tuple of arguments, or a dictionary of arguments. Allowing both has some advantages but is less cleanly integratible.
The problem is that half the examples people conjure up involve both: using the keywords as options, while using the positional arguments for the actual indices. Calling the proposal “kwargs in getitem” encourages that thinking, because that’s the prototypical reason for kwargs in function calls.
If there were non-toy examples, so people didn’t have to imagine how it would be used for themselves, that might be helpful.
Here is an example modified from the xarray documentation, where you want to assign to a subset of your array: da.isel(space=0, time=slice(None, 2))[...] = spam With this syntax this could be changed to: da[space=0, time=:2] = spam
On 8 Oct 2019, at 18:35, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019 at 12:22 PM Andrew Barnert via Python-ideas <python-ideas@python.org <mailto:python-ideas@python.org>> wrote: On Oct 7, 2019, at 21:21, Caleb Donovick <donovick@cs.stanford.edu <mailto:donovick@cs.stanford.edu>> wrote:
But what if you wanted to take both positional AND keyword?
I was suggesting that that wouldn't be allowed. So subscript either has a single argument, a tuple of arguments, or a dictionary of arguments. Allowing both has some advantages but is less cleanly integratible.
The problem is that half the examples people conjure up involve both: using the keywords as options, while using the positional arguments for the actual indices. Calling the proposal “kwargs in getitem” encourages that thinking, because that’s the prototypical reason for kwargs in function calls.
If there were non-toy examples, so people didn’t have to imagine how it would be used for themselves, that might be helpful.
Here is an example modified from the xarray documentation, where you want to assign to a subset of your array:
da.isel(space=0, time=slice(None, 2))[...] = spam
With this syntax this could be changed to:
da[space=0, time=:2] = spam
I must have missed something... when did the proposal we're discussing start allowing : there? / Anders
On Tue, Oct 8, 2019, 12:46 Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 18:35, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019 at 12:22 PM Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:
On Oct 7, 2019, at 21:21, Caleb Donovick <donovick@cs.stanford.edu> wrote:
But what if you wanted to take both positional AND keyword?
I was suggesting that that wouldn't be allowed. So subscript either
has a single argument, a tuple of arguments, or a dictionary of arguments. Allowing both has some advantages but is less cleanly integratible.
The problem is that half the examples people conjure up involve both: using the keywords as options, while using the positional arguments for the actual indices. Calling the proposal “kwargs in getitem” encourages that thinking, because that’s the prototypical reason for kwargs in function calls.
If there were non-toy examples, so people didn’t have to imagine how it would be used for themselves, that might be helpful.
Here is an example modified from the xarray documentation, where you want to assign to a subset of your array:
da.isel(space=0, time=slice(None, 2))[...] = spam
With this syntax this could be changed to:
da[space=0, time=:2] = spam
I must have missed something... when did the proposal we're discussing start allowing : there?
/ Anders
Why wouldn't it?
On 8 Oct 2019, at 18:59, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019, 12:46 Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 18:35, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019 at 12:22 PM Andrew Barnert via Python-ideas <python-ideas@python.org> wrote: On Oct 7, 2019, at 21:21, Caleb Donovick <donovick@cs.stanford.edu> wrote:
But what if you wanted to take both positional AND keyword?
I was suggesting that that wouldn't be allowed. So subscript either has a single argument, a tuple of arguments, or a dictionary of arguments. Allowing both has some advantages but is less cleanly integratible.
The problem is that half the examples people conjure up involve both: using the keywords as options, while using the positional arguments for the actual indices. Calling the proposal “kwargs in getitem” encourages that thinking, because that’s the prototypical reason for kwargs in function calls.
If there were non-toy examples, so people didn’t have to imagine how it would be used for themselves, that might be helpful.
Here is an example modified from the xarray documentation, where you want to assign to a subset of your array:
da.isel(space=0, time=slice(None, 2))[...] = spam
With this syntax this could be changed to:
da[space=0, time=:2] = spam
I must have missed something... when did the proposal we're discussing start allowing : there?
/ Anders
Why wouldn't it?
Because
dict(foo=:1) File "<string>", line 1 dict(foo=:1) ^ SyntaxError: invalid syntax
Because
dict(foo=:1) File "<string>", line 1 dict(foo=:1) ^ SyntaxError: invalid syntax
I don't see how that's an argument, we are talking about a syntax extension. Slice builder syntax is only every allowed in a subscript. Edit my original grammar change proposal to: ``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' subscript ``` Now slices are allowed in keyword arguments. -- Caleb Donovick On Tue, Oct 8, 2019 at 1:09 PM Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 18:59, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019, 12:46 Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 18:35, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019 at 12:22 PM Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:
On Oct 7, 2019, at 21:21, Caleb Donovick <donovick@cs.stanford.edu> wrote:
But what if you wanted to take both positional AND keyword?
I was suggesting that that wouldn't be allowed. So subscript either
has a single argument, a tuple of arguments, or a dictionary of arguments. Allowing both has some advantages but is less cleanly integratible.
The problem is that half the examples people conjure up involve both: using the keywords as options, while using the positional arguments for the actual indices. Calling the proposal “kwargs in getitem” encourages that thinking, because that’s the prototypical reason for kwargs in function calls.
If there were non-toy examples, so people didn’t have to imagine how it would be used for themselves, that might be helpful.
Here is an example modified from the xarray documentation, where you want to assign to a subset of your array:
da.isel(space=0, time=slice(None, 2))[...] = spam
With this syntax this could be changed to:
da[space=0, time=:2] = spam
I must have missed something... when did the proposal we're discussing start allowing : there?
/ Anders
Why wouldn't it?
Because
dict(foo=:1) File "<string>", line 1 dict(foo=:1) ^ SyntaxError: invalid syntax
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ZJDP2H... Code of Conduct: http://python.org/psf/codeofconduct/
On 8 Oct 2019, at 19:19, Caleb Donovick <donovick@cs.stanford.edu> wrote:
Because
dict(foo=:1) File "<string>", line 1 dict(foo=:1) ^ SyntaxError: invalid syntax
I don't see how that's an argument, we are talking about a syntax extension. Slice builder syntax is only every allowed in a subscript. Edit my original grammar change proposal to:
``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' subscript ```
Now slices are allowed in keyword arguments.
I wasn't making an argument, I was wondering what exactly we are even discussing. It seems like people are inventing new syntax willy nilly in this thread and I am getting very confused :) / Anders
-- Caleb Donovick
On Tue, Oct 8, 2019 at 1:09 PM Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 18:59, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019, 12:46 Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 18:35, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019 at 12:22 PM Andrew Barnert via Python-ideas <python-ideas@python.org> wrote: On Oct 7, 2019, at 21:21, Caleb Donovick <donovick@cs.stanford.edu> wrote: > > > But what if you wanted to take both positional AND keyword? > > I was suggesting that that wouldn't be allowed. So subscript either has a single argument, a tuple of arguments, or a dictionary of arguments. Allowing both has some advantages but is less cleanly integratible.
The problem is that half the examples people conjure up involve both: using the keywords as options, while using the positional arguments for the actual indices. Calling the proposal “kwargs in getitem” encourages that thinking, because that’s the prototypical reason for kwargs in function calls.
If there were non-toy examples, so people didn’t have to imagine how it would be used for themselves, that might be helpful.
Here is an example modified from the xarray documentation, where you want to assign to a subset of your array:
da.isel(space=0, time=slice(None, 2))[...] = spam
With this syntax this could be changed to:
da[space=0, time=:2] = spam
I must have missed something... when did the proposal we're discussing start allowing : there?
/ Anders
Why wouldn't it?
Because
dict(foo=:1) File "<string>", line 1 dict(foo=:1) ^ SyntaxError: invalid syntax
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ZJDP2H... Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, Oct 8, 2019 at 1:30 PM Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 19:19, Caleb Donovick <donovick@cs.stanford.edu> wrote:
Because
dict(foo=:1) File "<string>", line 1 dict(foo=:1) ^ SyntaxError: invalid syntax
I don't see how that's an argument, we are talking about a syntax extension. Slice builder syntax is only every allowed in a subscript. Edit my original grammar change proposal to:
``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' subscript ```
Now slices are allowed in keyword arguments.
I wasn't making an argument, I was wondering what exactly we are even discussing. It seems like people are inventing new syntax willy nilly in this thread and I am getting very confused :)
/ Anders
I thought we were talking about allowing __getitem__ to support keywords. I assumed the keywords would use the same syntax as positional values, and converting colons to slice objects is part of that syntax. So this isn't new syntax, it is just making the positional and keyword syntaxes the same.
On 8 Oct 2019, at 20:07, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019 at 1:30 PM Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 19:19, Caleb Donovick <donovick@cs.stanford.edu> wrote:
Because
> dict(foo=:1) File "<string>", line 1 dict(foo=:1) ^ SyntaxError: invalid syntax
I don't see how that's an argument, we are talking about a syntax extension. Slice builder syntax is only every allowed in a subscript. Edit my original grammar change proposal to:
``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' subscript ```
Now slices are allowed in keyword arguments.
I wasn't making an argument, I was wondering what exactly we are even discussing. It seems like people are inventing new syntax willy nilly in this thread and I am getting very confused :)
/ Anders
I thought we were talking about allowing __getitem__ to support keywords. I assumed the keywords would use the same syntax as positional values, and converting colons to slice objects is part of that syntax. So this isn't new syntax, it is just making the positional and keyword syntaxes the same.
I don't see it. Can you give examples of all the variations of slicing and their keyword equivalent so I understand what you mean? I'll write out the slicing variants and you can fill in how it would look with keyword arguments: x[:] x[a:] x[-a:] x[a:b] x[-a:b] x[a:-b] x[-a:-b] x[:b] x[:-b]
On Tue, Oct 8, 2019 at 2:18 PM Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 20:07, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019 at 1:30 PM Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 19:19, Caleb Donovick <donovick@cs.stanford.edu> wrote:
Because
dict(foo=:1) File "<string>", line 1 dict(foo=:1) ^ SyntaxError: invalid syntax
I don't see how that's an argument, we are talking about a syntax extension. Slice builder syntax is only every allowed in a subscript. Edit my original grammar change proposal to:
``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' subscript ```
Now slices are allowed in keyword arguments.
I wasn't making an argument, I was wondering what exactly we are even discussing. It seems like people are inventing new syntax willy nilly in this thread and I am getting very confused :)
/ Anders
I thought we were talking about allowing __getitem__ to support keywords. I assumed the keywords would use the same syntax as positional values, and converting colons to slice objects is part of that syntax. So this isn't new syntax, it is just making the positional and keyword syntaxes the same.
I don't see it. Can you give examples of all the variations of slicing and their keyword equivalent so I understand what you mean? I'll write out the slicing variants and you can fill in how it would look with keyword arguments:
x[:] x[a:] x[-a:] x[a:b] x[-a:b] x[a:-b] x[-a:-b] x[:b] x[:-b]
The colon operation would be converted to slices identically to how it is with positional arguments, it is just that those slices would assigned to values in a dict (or some other mapping) instead of values in a tuple. Otherwise it would work exactly the same. x[:] is x[slice(None, None)] x[foo=:] is x[foo=slice(None, None)] x[a:] is x[slice(a, None)] x[foo=a:] is x[foo=slice(a, None)] x[-a:] is x[slice(-a, None)] x[foo=-a:] is x[foo=slice(-a, None)] x[a:b] is x[slice(a, b)] x[foo=a:b] is x[foo=slice(a, b)] x[-a:b] is x[slice(-a, b)] x[foo=-a:b] is x[foo=slice(-a, b)] x[a:-b] is x[slice(a, -b)] x[foo=a:-b] is x[foo=slice(a, -b)] x[-a:-b] is x[slice(-a, -b)] x[foo=-a:-b] is x[foo=slice(-a, -b)] x[:b] is x[slice(None, b)] x[foo=:b] is x[foo=slice(None, b)] x[:-b] is x[slice(None, -b)] x[foo=:-b] is x[foo=slice(None, -b)] If you look at multiple indices, x[a, -b:c] is x[a, slice(-b, c)] x[foo=a, bar=-b:c] is x[foo=a, bar=slice(-b, c)]
On 8 Oct 2019, at 20:49, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019 at 2:18 PM Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 20:07, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019 at 1:30 PM Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 19:19, Caleb Donovick <donovick@cs.stanford.edu> wrote:
Because
>>> dict(foo=:1) File "<string>", line 1 dict(foo=:1) ^ SyntaxError: invalid syntax
I don't see how that's an argument, we are talking about a syntax extension. Slice builder syntax is only every allowed in a subscript. Edit my original grammar change proposal to:
``` subscriptlist: ... | kwargsubscript (',' kwargsubscript )* [','] kwargsubscript: NAME '=' subscript ```
Now slices are allowed in keyword arguments.
I wasn't making an argument, I was wondering what exactly we are even discussing. It seems like people are inventing new syntax willy nilly in this thread and I am getting very confused :)
/ Anders
I thought we were talking about allowing __getitem__ to support keywords. I assumed the keywords would use the same syntax as positional values, and converting colons to slice objects is part of that syntax. So this isn't new syntax, it is just making the positional and keyword syntaxes the same.
I don't see it. Can you give examples of all the variations of slicing and their keyword equivalent so I understand what you mean? I'll write out the slicing variants and you can fill in how it would look with keyword arguments:
x[:] x[a:] x[-a:] x[a:b] x[-a:b] x[a:-b] x[-a:-b] x[:b] x[:-b]
The colon operation would be converted to slices identically to how it is with positional arguments, it is just that those slices would assigned to values in a dict (or some other mapping) instead of values in a tuple. Otherwise it would work exactly the same.
x[:] is x[slice(None, None)] x[foo=:] is x[foo=slice(None, None)]
x[a:] is x[slice(a, None)] x[foo=a:] is x[foo=slice(a, None)]
x[-a:] is x[slice(-a, None)] x[foo=-a:] is x[foo=slice(-a, None)]
x[a:b] is x[slice(a, b)] x[foo=a:b] is x[foo=slice(a, b)]
x[-a:b] is x[slice(-a, b)] x[foo=-a:b] is x[foo=slice(-a, b)]
x[a:-b] is x[slice(a, -b)] x[foo=a:-b] is x[foo=slice(a, -b)]
x[-a:-b] is x[slice(-a, -b)] x[foo=-a:-b] is x[foo=slice(-a, -b)]
x[:b] is x[slice(None, b)] x[foo=:b] is x[foo=slice(None, b)]
x[:-b] is x[slice(None, -b)] x[foo=:-b] is x[foo=slice(None, -b)]
If you look at multiple indices,
x[a, -b:c] is x[a, slice(-b, c)] x[foo=a, bar=-b:c] is x[foo=a, bar=slice(-b, c)
Aaaah. Now I see it. Thanks for clarifying that for me.
On Wed, Oct 9, 2019 at 10:11 AM David Mertz <mertz@gnosis.cx> wrote:
x[foo=:] is x[foo=slice(None, None)]
We can combine this with the walrus operator for extra clarity:
db[o0=o0:=o0==Oo:o0==oO]
:-)
Perl isn't dead, it's just resting.
Are you sure it isn't dead? I think the correct pronunciation of the text inside the square brackets is a ghostly wail, "OOOoooooOOOooOOOOOOooooooo"... ChrisA
On Tue, Oct 8, 2019, at 14:18, Anders Hovmöller wrote:
I don't see it. Can you give examples of all the variations of slicing and their keyword equivalent so I understand what you mean? I'll write out the slicing variants and you can fill in how it would look with keyword arguments:
x[:] x[a:] x[-a:] x[a:b] x[-a:b] x[a:-b] x[-a:-b] x[:b] x[:-b]
x[kw=:] x[kw=a:] x[kw=-a:] x[kw=a:b] x[kw=-a:b] x[kw=a:-b] x[kw=-a:-b] x[kw=:b] x[kw=:-b] I don't see what's confusing. All that's needed is for the slice syntax ['colon operator', if you like] to have higher precedence than the keyword syntax, as it already has higher precedence than the comma. As I said originally, I'm +0 on the whole feature but I think weird restrictions like "slice syntax only works for positional arguments" or "can't have both positional and keyword args" will be surprising to most people.
On 10/8/2019 4:53 PM, Random832 wrote:
On Tue, Oct 8, 2019, at 14:18, Anders Hovmöller wrote:
I don't see it. Can you give examples of all the variations of slicing and their keyword equivalent so I understand what you mean? I'll write out the slicing variants and you can fill in how it would look with keyword arguments:
x[:] x[a:] x[-a:] x[a:b] x[-a:b] x[a:-b] x[-a:-b] x[:b] x[:-b]
x[kw=:] x[kw=a:] x[kw=-a:] x[kw=a:b] x[kw=-a:b] x[kw=a:-b] x[kw=-a:-b] x[kw=:b] x[kw=:-b]
I don't see what's confusing. All that's needed is for the slice syntax ['colon operator', if you like] to have higher precedence than the keyword syntax, as it already has higher precedence than the comma.
I look forward to the walrus operator being thrown in, too! Eric
On 10/8/2019 5:53 PM, Eric V. Smith wrote:
On 10/8/2019 4:53 PM, Random832 wrote:
On Tue, Oct 8, 2019, at 14:18, Anders Hovmöller wrote:
I don't see it. Can you give examples of all the variations of slicing and their keyword equivalent so I understand what you mean? I'll write out the slicing variants and you can fill in how it would look with keyword arguments:
x[:] x[a:] x[-a:] x[a:b] x[-a:b] x[a:-b] x[-a:-b] x[:b] x[:-b]
x[kw=:] x[kw=a:] x[kw=-a:] x[kw=a:b] x[kw=-a:b] x[kw=a:-b] x[kw=-a:-b] x[kw=:b] x[kw=:-b]
I don't see what's confusing. All that's needed is for the slice syntax ['colon operator', if you like] to have higher precedence than the keyword syntax, as it already has higher precedence than the comma.
I look forward to the walrus operator being thrown in, too!
I realize you'd have to use parens for the walrus operator to be used inside []. Eric
On Tue, Oct 8, 2019 at 1:05 PM Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 18:59, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019, 12:46 Anders Hovmöller <boxed@killingar.net> wrote:
On 8 Oct 2019, at 18:35, Todd <toddrjen@gmail.com> wrote:
On Tue, Oct 8, 2019 at 12:22 PM Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:
On Oct 7, 2019, at 21:21, Caleb Donovick <donovick@cs.stanford.edu> wrote:
But what if you wanted to take both positional AND keyword?
I was suggesting that that wouldn't be allowed. So subscript either
has a single argument, a tuple of arguments, or a dictionary of arguments. Allowing both has some advantages but is less cleanly integratible.
The problem is that half the examples people conjure up involve both: using the keywords as options, while using the positional arguments for the actual indices. Calling the proposal “kwargs in getitem” encourages that thinking, because that’s the prototypical reason for kwargs in function calls.
If there were non-toy examples, so people didn’t have to imagine how it would be used for themselves, that might be helpful.
Here is an example modified from the xarray documentation, where you want to assign to a subset of your array:
da.isel(space=0, time=slice(None, 2))[...] = spam
With this syntax this could be changed to:
da[space=0, time=:2] = spam
I must have missed something... when did the proposal we're discussing start allowing : there?
/ Anders
Why wouldn't it?
Because
dict(foo=:1) File "<string>", line 1 dict(foo=:1) ^ SyntaxError: invalid syntax
tuple(:1) is also invalid syntax, hence the need for a "slice" type. x[foo=:2] would be equivalent to x[{'foo': slice(None, 2)}], as it is in my example. This is the same way it is handled with positional indexing.
On Tue, Oct 08, 2019 at 12:55:40PM -0400, Todd wrote:
da.isel(space=0, time=slice(None, 2))[...] = spam
With this syntax this could be changed to:
da[space=0, time=:2] = spam
Anders:
I must have missed something... when did the proposal we're discussing start allowing : there?
Todd:
Why wouldn't it?
Wrong question. New features are not "default accept", we accept them unless there is a strong reason to reject them. They are "default reject", we reject them unless there is a good reason to accept them. So the right question is, why should we accept colons there? This is a proposal for adding keyword args to subscripting, not colon-syntax for slice objects inside arbitrary expressions. If you wish to propose that, you can propose it separately. But for the record, here is one reason why we probably should not allow `time=:2` as syntactic sugar for `time=slice(None, 2)`. It is confusing as hell. When I saw this da[space=0, time=:2] I read it as a slice: da[ slice( (space=0, time=), 2, None) ] and thought "That must be a typo, because the time keyword doesn't have a value." And combining slice syntax with keywords in the same call is a recipe for over-complicated, confusing subscripts, which is why Caleb initially suggested you can use one, or the other, but not both. -- Steven
On Tue, Oct 8, 2019 at 2:03 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Tue, Oct 08, 2019 at 12:55:40PM -0400, Todd wrote:
da.isel(space=0, time=slice(None, 2))[...] = spam
With this syntax this could be changed to:
da[space=0, time=:2] = spam
Anders:
I must have missed something... when did the proposal we're discussing start allowing : there?
Todd:
Why wouldn't it?
Wrong question. New features are not "default accept", we accept them unless there is a strong reason to reject them. They are "default reject", we reject them unless there is a good reason to accept them. So the right question is, why should we accept colons there?
My point, which I could have made clearer, is that converting colons to slices is part of how __getitem__ works. So __getitem__ would not really be supporting keyword arguments in the same way it supports positional arguments unless it was supporting the same syntax. This would be adding additional restrictions on the keyword syntax relative to the positional one rather than making the two equivalent. I assumed, I guess wrongly, that the general assumption would be that a new extension to the syntax would behave consistently with whatever it is extending unless there was some specific reason for it not to.
This is a proposal for adding keyword args to subscripting, not colon-syntax for slice objects inside arbitrary expressions. If you wish to propose that, you can propose it separately.
No, if you read the OP, this is a proposal for adding keyword arguments to __getitem__. At least in my mind, __getitem__ would not have full support for keyword arguments if it didn't support the same syntax as positional arguments, since everywhere else in Python you can do the same things with either as far as I am aware. As far as I can tell no one ever mentioned limiting just to subscripts, so I assumed such limitations were not important. You apparently assumed differently. And I don't think splitting that off into a separate thread would be productive. The issue is tightly linked in with this one, since the decision would have to be made in any implementation whether the positional or keyword syntaxes would be equivalent, or if additional restrictions are to be imposed on a keyword-based syntax. That decision would make an enormous impact on how useful this new syntax would be.
But for the record, here is one reason why we probably should not allow `time=:2` as syntactic sugar for `time=slice(None, 2)`.
It is confusing as hell. When I saw this
da[space=0, time=:2]
I read it as a slice:
da[ slice( (space=0, time=), 2, None) ]
and thought "That must be a typo, because the time keyword doesn't have a value."
I am really sorry, but I am really having a hard time understanding your reasoning here. Can you explaining how you came up with that result? I am not sure why you would think keyword arguments would work so completely differently than positional ones in this context. Why should that parse to a single slice when da[0, :2] parses to a slice and an index?
And combining slice syntax with keywords in the same call is a recipe for over-complicated, confusing subscripts, which is why Caleb initially suggested you can use one, or the other, but not both.
I have looked through the whole thread and I can't see anyone, especially not Caleb, saying that. Caleb said we shouldn't mix positional with keyword arguments, is that what you are thinking of?
On Tue, Oct 8, 2019, at 13:57, Steven D'Aprano wrote:
It is confusing as hell. When I saw this
da[space=0, time=:2]
I read it as a slice:
da[ slice( (space=0, time=), 2, None) ]
and thought "That must be a typo, because the time keyword doesn't have a value."
Do you read [0,:2] as slice((0,), 2) as well? Is [0:1, 2:3] slice(0, (1, 2), 3) or (slice(0, 1), slice(2, 3))? [it is the latter.] Slices are already allowed within subscripting, and already have higher precedence than comma.
And combining slice syntax with keywords in the same call is a recipe for over-complicated, confusing subscripts, which is why Caleb initially suggested you can use one, or the other, but not both.
I'm not sure the topic of slice syntax had been mentioned yet... as I recall it, what Caleb suggested was that you couldn't use positional arguments and keyword arguments in the same call. Incidentally, I don't agree with the reasoning there - principle of least surprise suggests making the rules as close to function call syntax as possible, with the exception that values can be slices.
On 08/10/2019 17:35, Todd wrote:
On Tue, Oct 8, 2019 at 12:22 PM Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:
On Oct 7, 2019, at 21:21, Caleb Donovick <donovick@cs.stanford.edu> wrote:
But what if you wanted to take both positional AND keyword?
I was suggesting that that wouldn't be allowed. So subscript either has
a single argument, a tuple of arguments, or a dictionary of arguments. Allowing both has some advantages but is less cleanly integratible.
The problem is that half the examples people conjure up involve both: using the keywords as options, while using the positional arguments for the actual indices. Calling the proposal “kwargs in getitem” encourages that thinking, because that’s the prototypical reason for kwargs in function calls.
If there were non-toy examples, so people didn’t have to imagine how it would be used for themselves, that might be helpful.
Here is an example modified from the xarray documentation, where you want to assign to a subset of your array:
da.isel(space=0, time=slice(None, 2))[...] = spam
With this syntax this could be changed to:
da[space=0, time=:2] = spam
Can I just say how glad I am that I never have to deal with pandas? ;-/ -- Rhodri James *-* Kynesim Ltd
participants (18)
-
Anders Hovmöller -
Andrew Barnert -
Batuhan Taskaya -
Caleb Donovick -
Cameron Simpson -
Chris Angelico -
Christopher Barker -
David Mertz -
Dominik Vilsmeier -
Eric V. Smith -
Joao S. O. Bueno -
Kirill Balunov -
Kyle Lahnakoski -
MRAB -
Random832 -
Rhodri James -
Steven D'Aprano -
Todd