A better (simpler) approach to PEP 505
The need addressed by PEP 505 is real; it's also MUCH more niche and uncommon than something that would merit new syntax. Moreover, the actual legitimate purpose served by the PEP 505 syntax is easily served by existing Python simply by using a wrapper class. Here is a way of solving the "deep attribute access to messy data" problem that is: (1) Much more explicit (2) Requires no change in syntax (3) Will not be a bug magnet (4) Inasmuch as there are semantic traps, they are announced by the use of a class whose documentation would be pointed to for readers The API that could be useful might be something like this: In [1]: from none_aware import NoneAware In [2]: from types import SimpleNamespace In [3]: foo = SimpleNamespace() In [4]: foo.bar = SimpleNamespace() In [5]: foo.bar.baz = SimpleNamespace() In [6]: foo.bar.baz.blat = 42 In [7]: NoneAware(foo).bar.blim Out[7]: <none_aware.NoneAware at 0x11156a748> In [8]: NoneAware(foo).bar.blim.unbox() In [9]: NoneAware(foo).bar.baz.blat.unbox() Out[9]: 42 In [10]: NoneAware(foo).bar.baz.blat Out[10]: <none_aware.NoneAware at 0x11157d908> In [11]: NoneAware(foo).bar.baz.flam.unbox() In [12]: NoneAware(foo).bar.baz.flam Out[12]: <none_aware.NoneAware at 0x1115832b0> The particular names I use are nothing special, and better ones might be found. I just called the class NoneAware and the "escape" method `.unbox()` because that seemed intuitive at first brush. I don't disagree that needing to call .unbox() at the end of the chained attribute access is a little bit ugly. But it's a lot less ugly than large family of new operators. And honestly, it's a nice way of being explicit about the fact that we're entering then leaving a special world where attribute accesses don't fail. I haven't implemented the equivalent dictionary lookups in the below. That would be straightforward, and I'm sure my 5 minute throwaway code could be improved in other ways also. But something better than this in the standard library would address ALL the actual needs described in PEP 505. Even the pattern Steve Dower is especially fond of like: favorite = cfg?.user?.profile?.food ?? "Spam" (i.e. a configuration may be incomplete at any level, if levels are missing default favorite food is Spam). We could simply spell that: favorite = NoneAware(cfg, "Spam").user.profile.food.unbox() I think that's 14 characters more in this example, but still compact. We could get that down to 2 characters if we used one-letter names for the class and method. I suppose down to zero characters if .unbox() was a property. So completely toy implementation: class NoneAware(object): def __init__(self, thing, sentinal=None): self.thing = thing self.sentinal = sentinal def __getattr__(self, attr): try: return NoneAware(getattr(self.thing, attr)) except AttributeError: return NoneAware(self.sentinal) def unbox(self): return self.thing -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 23 July 2018 at 16:12, David Mertz <mertz@gnosis.cx> wrote:
[...]
Thank you. That's the sort of "what would this look like if implemented as a library rather than language syntax" solution that I had in mind with my earlier post. I would be very interested to hear discussion of the pros and cons of adding new syntax to the language as per PEP 505 in comparison to a solution like this (ultimately more fleshed out and "production quality") rather than comparisons PEP 505 to raw "roll your own" Python code. For me: * Library solution works with all versions of Python * The need for unbox is a little ugly, but arguably less so than ?. (conceded that's a subjective view) * Mixing ?. and . is terser than unboxing and reboxing - but are there any real examples where that's needed? * Having the default at the beginning rather than at the end doesn't follow natural reading order (but again that's pretty subjective) There's little here that isn't subjective, so I expect a lot of heated debate that ultimately convinces no-one. But the "you can use this solution now" aspect of the library solution seems pretty compelling to me - at least in terms of "let's publish the library solution and then based on experience with it, review PEP 505". Paul
On Mon, Jul 23, 2018 at 11:34 AM Robert Vanden Eynde <robertve92@gmail.com> wrote:
The default could be at the end with an argument to unboxing :
favorite = NoneAware(cfg).user.profile.food.unbox("Spam")
That's what PyMaybe does: https://pymaybe.readthedocs.io/en/latest/readme.html#examples-use-cases PyMaybe is a great implementation of the idea, but it's obviously limited in what it can do. It doesn't have short-circuiting (but it uses lambda to approximate it). It swallows a multitude of attribute and look up errors. It doesn't support coalescing. And it can't be used with any library that isn't PyMaybe-aware. For people who want to explore deeply nested objects/dictionaries (perhaps from unserialized JSON), there's also a pure Python library for that: http://objectpath.org/
Hi David and Paul Thank you both for your contributions, and for starting a new thread. David, you wrote
The need addressed by PEP 505 is real;
I completely agree. My view is that with PEP 505 as it it, the problem is better than the solution. But all the same, something can be done to improve matters.
I'm with you here also. First explore solving the problem using existing syntax. If nothing else, it helps us understand the problem. There's one area where I find attraction in a new syntax. The ?? operator is well described, I think, as a None-aware 'or'. But not in the name '??'. We all understand val1 = EXP1 or EXP2 So how about val2 = EXP1 or-if-None EXP2 with 'or-if-None' being a new language keyword!
These are all good goals. And (2) makes exploration much easier.
The API that could be useful might be something like this:
I have some different ideas, which I'll post later (to this thread, if you don't mind). Paul, you wrote
I like that approach. I find exploration using Python modules to be more open (democratic?) than syntax changes that require building a new executable. Once again, thank you both for starting at the problem and pointing in a different direction. -- Jonathan
On Mon, Jul 23, 2018 at 11:26 AM Paul Moore <p.f.moore@gmail.com> wrote:
On the last point, it would be easy enough to change the API to make it `NoneAware(obj).a.b.c.unbox(sentinel)` if that was thought a better API than `NoneAware(obj, sentinel).a.b.c.unbox()`. Oh... and the production code should DEFINITELY spell 'sentinel' correctly if that's a part of the API. :-) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
Le 23/07/2018 à 18:00, David Mertz a écrit :
One of the good things about this proposal, is that it may be expanded to other sentinels: ret = NoneAware(obj, sentinel=0).a.unbox(42) # would be equivalent to ret = obj.a if obj.a is not 0 else 42 # or, if tmp_a had side effects tmp_a = obj.a ret = tmp_a if tmp_a is not 0 else 42 One thing that this proposal doesn't cover easily (although I'd refactor the code before using this syntax, as excepted in some very specific cases like when using ORMs, I don't like chaining methods and properties like this): favorite = cfg?.user.profile?.food ?? "Spam" # every user should have a profile favorite = NoneAware(cfg).user.profile.food.unbox("Spam") # We're silencing an exception that I didn't plan to silence Another thing is that both parts are executed. If instead of "Spam" we had any variable with a side effect, it would have been executed even if cfg.user.profile.food existed. We're missing the lazy evaluation of this part here. That being said, I deeply believe the use case for that are not spread enough to justify such a change. And I can see from here code where every '.' is prepended by a '?' just in case (the same way we encounter frequently exceptions catching for Exception).
Your wrapper class can also, I think: Swallow exceptions, especially AttributeError Specify the sentinel, if it is not None Work for getitem Mixing is indeed more difficult, and there is probably performance impact. Still, it's general and does not need to touch the implementation of the class you want to descend... I like it better than the operators Another idea, may be stupid, maybe not: what if None fails with a subinstance of AttributeError, and a subinstance of KeyError, or IndexError. Then a try/except catching only those exceptions would also work quite nicely...
On Mon, 23 Jul 2018 11:12:45 -0400 David Mertz <mertz@gnosis.cx> wrote:
You could use .__call__() instead of .unbox(). Also you can make unboxing unnecessary in most cases by having your class proxy most operations, like weakref.proxy does. The "wrapt" library may help with that: http://wrapt.readthedocs.io/en/latest/wrappers.html Regards Antoine.
On Mon, Jul 23, 2018 at 12:47 PM Antoine Pitrou <solipsis@pitrou.net> wrote:
favorite = cfg?.user?.profile?.food ?? "Spam" favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
I'm not sure I entirely understand how that class proxy could be made transparent. Maybe you can make a toy implementation to show me? How can we make this work? favorite = NoneAware(cfg, "Spam").user.profile.food + "and more spam" I just noticed in my scratch directory that I had implemented almost the same thing a year ago when this discussion last came up. It's simple enough that the code was almost the same back then. I had included `.__getitem__()` in that version, but had not considered sentinels. However, one thing I *did* implement was `.__call__()`, but that reminds me of why your idea cannot work. I had it either call or fall back to more boxing. The end point of the chained attributes (or dict lookups) can perfectly well be a callable. E.g. favorite = NoneAware(cfg, "Spam").user.profile.get_food() Well, that's not going to work in my current toy code either since it would need to be: favorite = NoneAware(cfg, "Spam").user.profile.get_food.unbox()() But I could implement special stuff for "if the tail is a callable, unbox it automatically", I suppose. However, it would be fundamentally ambiguous, I think, whether those final parens should mean "unbox" or "call". Or I guess if calling *always* meant "unbox" we could have: favorite = NoneAware(cfg, "Spam").user.profile.get_food()() That feels weird to me though. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
Btw. I *do* realize that the semantics of my suggested NoneAware class is different from the coalescing syntax. I just look at whether attribute access succeeds or fails in that code, not whether the starting value is specifically None (nor any particular sentinel). I believe that that behavior better fits the ACTUAL need underlying these ideas (i.e. potentially deeply nested but messy data; from JSON or similar sources). If we wanted to match the semantics of PEP 505 more exactly, it would be easy enough to compare a current level of the hierarchy to None specifically rather than check more generically "is this a thing that has that attribute?" However, if the PEP 505 semantics really are the ones most desirable (i.e. including raising AttributeError on things that are not-None-but-don't-have-attribute), that's perfectly straightforward to implement using existing syntax and an API very close to what I suggest. On Mon, Jul 23, 2018 at 11:12 AM David Mertz <mertz@gnosis.cx> wrote:
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Mon, Jul 23, 2018 at 11:12:45AM -0400, David Mertz wrote:
That's one opinion.
(4) Inasmuch as there are semantic traps, they are announced by the use of a class whose documentation would be pointed to for readers
Operators have documentation too. If you think people won't read the documentation for the operator, what makes you think they're read the documentation for the class?
There's the first bug right there. foo.bar.blim ought to raise AttributeError, since there is no blim attribute defined on foo.bar. Reminder: the proposal is for a null-coalescing operator, not an AttributeError suppressing operator. The PEP is explicit that catching AttributeError is rejected: https://www.python.org/dev/peps/pep-0505/#id21
How many bugs will be caused by people forgetting to unbox when they're done?
That ought to be AttributeError again, since there is no "flam" attribute defined on foo.bar.baz.
But we aren't entering such a world, at least not in PEP 505. Attribute access can fail. spam.eggs = 42 spam?.eggs?.upper is still going to raise AttributeError, because eggs is not None, it is an int, and ints don't have an attribute "upper".
How does your class implement short-circuit behaviour?
So we don't like operators like ?? because its too cryptic, but you're happy to have one-character class and property names. favorite = N(cfg, "Spam").user.profile.food.u What happens if you have an attribute that happens to be called "unbox" in your attribute look-up chain? result = NoneAware(something, "default").spam.foo.unbox.eggs.unbox() -- Steve
The proto here swallow and short circuit on attribute error. Changing to do it on Noneness is easy, and you can choose between the two behavior: it's a strength compared to the operator approach. It's working in current python, another strength. I think short-circuit behavior is similar: once stopping condition has been met, the gard object cascade till the end of the nested attribute lookup. Like None do for the operator. More complex stuff when None-coalescing attribute access is mixed with normal access is better done with operators. Maybe complex operations fine of the middle of the access chain would also be easier with the operators. How many times would you encounter that in real cases? I bet on very few. In those few case, would a one line ?. chain expression be better than a multiple line logic with temporaries? I think no. That's my feeling, but in the absence of concrete examples, feelings are ok.
On Mon, Jul 23, 2018 at 10:53:11AM -0700, Grégory Lielens wrote:
Being able to choose between one desirable behaviour and one undesirable, rejected behaviour is *not* a strength. PEP 505 has a section explaining why catching AttributeError is undesirable. I find the reasons it gives are compelling. Can you explain why you reject the PEP's arguments against catching AttributeError? -- Steve
On Tuesday, July 24, 2018 at 2:39:19 AM UTC+2, Steven D'Aprano wrote:
Yes, and people have done it multiple time already. I will try to make it as explicit as possible, with a terminology that some may find useful... when u have a object hierarchy linked through attributes, which -seems- the use pattern of ?. , the hierarchy is either - 1) fully regular (each attribute contains the same type of objects (same type=with the same attributes), then normal . access is what you want - 2) regular-or-None (contain same type of objects, or None). That's what ?. is for, and ?. is only for that. - 3) its partially regular (does not contain the same type of objects, some objects have more attributes than other, e.g. partial objects). - 4) its irregular (objects may be of completely different type (int, str,... and do not specailly share attributes). ?. is intended for 2). 2) and 3) can be dealed with swallowing AttributeError 4) can also be traversed swallowing AttributeError, but it's probably a very bad idea in almost all cases. I've encountered all cases, from time to time, but none is really typical of most of my code. Which is more common? not sure, I would say 3). What is sure is that 2) do not stand as hugely more common than the others, so introducing a special syntax for 2) do not please me, so +0... Now as I find ?. unpleasant to parse (personal opinion), and having grown to share the general operaror / line noise allergy mainstream in the python community, i am a firm -1 on this: too small a niche, do not bring much compared to current approach, and visual line noise. firm -1 on ?? too, it's less polemic, less visually displeasing, but even less usefull I think: it does not gain that much compared to a more general ternary. As the pymaybe/this thread cover 2) and 3), and do it without any change to python, I find it better. No change > lib change > backward compatible object change - like dict becoming ordered, or None triggering NoneAttributeError(AttributeError) > syntax change (expecially a bunch of linenoisy operators). Now I will stop for a while here, arguments have become circular, proponents and opponents have exposed their view, but mostly proponents have provided very few additional information for a while, especially about real use cases, despite a few requests. We still do not know what motivated the proposal..We had to guess (JSON). Even the PEP do not say, it provide a set of real world example from the standard library, which is admirable but different. And personaly, while am very thankful for providing those concrete examples, I find them unconvicing: the code before is usually fine and improved little by the new operators. Sometimes it's worse: only slightly shorter and less clear.
On Mon, Jul 23, 2018 at 1:28 PM Steven D'Aprano <steve@pearwood.info> wrote:
There's the first bug right there. foo.bar.blim ought to raise AttributeError, since there is no blim attribute defined on foo.bar.
Note my observation that this is deliberately different semantics. I want to solve the underlying need, not simply emulate the less useful semantics of PEP 505. But those same semantics COULD be perfectly well emulated with a similar class.
Far fewer and far more easily caught. When the object you get wherever the real failure happens is ` <none_aware.NoneAware at 0x11156a748>` or the like, the source of the problem become dead obvious. Having `None` somewhere unexpected is much more mysterious and difficult to diagnose. That said, NoneAware is the wrong name for the class I wrote. Maybe GreedyAccess would be a better name. Perhaps I'll make slightly less toy implementations of an actual NoneCoalesce and GreedyAccess class and post in this thread and/or on PyPI. It does look like PyMaybe does much of what I'm thinking of. I didn't think my toy was the first or best implementation.
But we aren't entering such a world, at least not in PEP 505. Attribute access can fail.
spam.eggs = 42
spam?.eggs?.upper
is still going to raise AttributeError, because eggs is not None, it is an int, and ints don't have an attribute "upper".
True enough. But it's extremely difficult to imagine a real-world case where those particular semantics are actually particularly useful. I guess my revulsion at the syntax blinded me to the fact that the semantics are actually pretty bad on their own.
How does your class implement short-circuit behaviour?
You mean if we want something like this? favorite = GreedyAccess(cfg, sentinel=ExpensiveLookup()).user.profile.food.unbox() It doesn't. Something else (like PyMaybe) could defer the computation using a lambda or other means. My toy code doesn't do that. We could also avoid the visible lambda by doing an 'iscallable(sentinel)' and only calling it if it turns out to be needed, but that might be too magical. E.g.: # If we fail, favorite = ExpensiveLookup(), by magic favorite = GreedyAccess(cfg, sentinel=ExpensiveLookup).user.profile.food.unbox() So we don't like operators like ?? because its too cryptic, but you're
happy to have one-character class and property names.
No, I'm not happy with that. I was just pointing out that code golf is possible for someone who really wants it. I'm happy to spend a few characters for readability. But one COULD write: from nested import GreedyAccess as G G._ = G.unbox
What happens if you have an attribute that happens to be called "unbox" in your attribute look-up chain?
Don't use this class and/or modify the API slightly? This is getting trite. My 5 minute code isn't a final proposal, just a way of seeing a napkin-sketch of a better approach. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Mon, Jul 23, 2018 at 01:55:48PM -0400, David Mertz wrote:
I know that "laziness and hubris" are virtues in a programmer, but the PEP authors describe use-cases where *testing for None* is precisely the semantics wanted. What are your use-cases for greedily swallowing AttributeError? Before telling the PEP authors that they have misunderstood their own uses-cases and that their need isn't what they thought (coalescing None) but something radically different which they have already explicitly rejected (suppressing AttributeError), you better have some damn compelling use-cases. "Laziness and hubris" are supposed to be virtues in programmers, but when you insist that people have their own use-cases wrong, and that the designers of C#, Dart and Swift made a serious mistake introducing this feature, you ought to back it up with more than just an assertion. [...]
It does look like PyMaybe does much of what I'm thinking of. I didn't think my toy was the first or best implementation.
Are you aware that the PEP includes pymaybe as a rejected idea?
You can't imagine why it would be useful to NOT suppress the exception from a coding error like trying to uppercase an int. -- Steve
On 23/07/18 16:12, David Mertz wrote:
Here is a way of solving the "deep attribute access to messy data" problem
Before we go too far, and /pace/ Steve's work, how real is this problem? I get that there is a use in accessing JSON trees, but this feels awfully like a rather specific issue about avoiding cleaning data before using it. As such, should be really be encouraging it? (For the avoidance of doubt, this refers to the proposed ?. and ?[] operators.
Worthy goals.
...and here's the first semantic trap. I was expecting to get None back there, and even your pre-announcement in goal (4) didn't stop me puzzling over it for a couple of minutes. The only reason I wasn't expecting an AttributeError was that I was confident you wouldn't propose this without doing something magic to effectively override the meaning of "." A naive user might be very surprised.
[snip]
You and I have very different definitions of "nice" ;-) I have to say it's questionable which of "?." and "NoneAware()...unbox()" are uglier. Then again, I go back to my original comment: is this really a problem we want to solve? How are you supposed to do method calling, the equivalent of "foo?.bar()" ? "NoneAware(foo).bar.unbox()()" looks downright weird. Is there more magic in NoneAware to cover this case? (Not that I think we should be encouraging people to do this, but...) -- Rhodri James *-* Kynesim Ltd
On Mon, Jul 23, 2018 at 2:12 PM Rhodri James <rhodri@kynesim.co.uk> wrote:
Is there more magic? I don't know, and I don't really care that much. That's the point. This is just a plain old Python class that will work back to probably Python 1.4 or so. If you want to write a version that has just the right amount of magic, you are free to. That said, I *DID* give it the wrong name in my first post of this thread. GreedyAccess is more accurate, and a NoneCoalesce class would behave somewhat differently. In my opinion, the need at issue is worthwhile, but niche. Using a special class to deal with such a case is absolutely the right level of abstraction. Syntax is not! So sure, figure out how to tweak the API to be most useful, find a better name for the class, etc. I wouldn't think it terrible if a class like this (but better) found a home in the standard library, but it doesn't deserve more prominence than that. Not even builtins, and *definitely* not syntax. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 23/07/18 19:21, David Mertz wrote:
I care only in as much as you were proposing an incomplete solution (in my eyes at least). I'm pretty convinced by now I would never use it.
I think part of my point was that a special class is not as readable or trap-free as writing the conditions out explicitly, given the unexpected "boxed" results, which makes me question whether the issue *is* worthwhile.
I think I might find it terrible. -- Rhodri James *-* Kynesim Ltd
Le 23/07/2018 à 17:12, David Mertz a écrit :
https://github.com/ekampf/pymaybe It's interest is much more limited than in Haskell because in Python, calls are not lazy, objects are dynamic and function calls are expensive. Take the following: foo().bar[0] Doing it safely would be: val = foo() try: val = val.bar except AttributeError: val = None else: try: val = val[0] except KeyError: val = None Clearly we see that the operation goes up to the end if everything goes right. Otherwise, it jumps to the default value. With the operator proposal: val = foo()?.bar?[0] The "?" operator is like "or" / "and", and it will shotcircuit as well. With the try / else proposal: val = try foo().bar[0] else None This is supposed to transform this ast to into the first try/except form. So it shortcircuit as well. But with your proposal, it becomes: val = maybe(foo()).bar[0].or_else(None) Which means foo() # call maybe(foo()) # call maybe(foo()).__getattr__('bar') # call maybe(foo()).__getattr__('bar').__getitem__(0) # call maybe(foo()).__getattr__('bar').__getitem__(0).or_else(None) # call There is no shortcircuit, you get 5 calls in all cases, plus the maybe object proxing the call to the underlying value every time. So that's 7 calls, added to the try/excepts you still have behind the scene. Plus, you don't get the same calls depending of the values, but the switch is implicit and behind the maybe object: no linter can save you from a typo. So yes, it works right now. But it's a suboptimal solution.
Short circuit is indeed the main difference. Makes me re-think about the None trigger subclasses of invalid operations exceptions. Like None.a trigger a NoneAttributeError(AttributeError), and so on... I think it solves many issues without much problems nor syntax changes, but probably I miss something... Sure, it's a big change, but maybe less so than 3 non-classic operators working only on None...
I agree a class can provide a very good alternative to PEP505. I built one a while ago, and still use it https://github.com/klahnakoski/mo-dots for most my data transformation code. The library only boxes dicts and lists, which removes the need for .unbox() in many of my use cases. My point is, if a class is chosen instead of PEP505, I doubt the unboxing will be a big issue. On 2018-07-23 11:12, David Mertz wrote:
Here is a way of solving the "deep attribute access to messy data" problem that is:
David Mertz schrieb am 23.07.2018 um 16:12:
The discussion so far made it clear to me that a) there is a use case for this feature, although I never needed it myself b) throwing new syntax at it is not the right solution Especially since there seem to be slightly diverging ideas about the exact details in behaviour. Since this can be done in form of a library, people should just drop it into a couple of competing libraries and let users choose what they like better in their specific situation. And since we already have a PEP now, let's continue to use it as a basis for discussion about how these libraries should best behave in general and what mistakes they should avoid. Stefan
On Tue, Jul 24, 2018 at 1:57 AM Stefan Behnel <stefan_ml@behnel.de> wrote:
+1 There is still no proof that such a programming pattern would be widely used or is desirable in practice. A library on PYPI could help clarifying that (and the PEP should mention it). -- Giampaolo - http://grodola.blogspot.com
On 23 July 2018 at 16:12, David Mertz <mertz@gnosis.cx> wrote:
[...]
Thank you. That's the sort of "what would this look like if implemented as a library rather than language syntax" solution that I had in mind with my earlier post. I would be very interested to hear discussion of the pros and cons of adding new syntax to the language as per PEP 505 in comparison to a solution like this (ultimately more fleshed out and "production quality") rather than comparisons PEP 505 to raw "roll your own" Python code. For me: * Library solution works with all versions of Python * The need for unbox is a little ugly, but arguably less so than ?. (conceded that's a subjective view) * Mixing ?. and . is terser than unboxing and reboxing - but are there any real examples where that's needed? * Having the default at the beginning rather than at the end doesn't follow natural reading order (but again that's pretty subjective) There's little here that isn't subjective, so I expect a lot of heated debate that ultimately convinces no-one. But the "you can use this solution now" aspect of the library solution seems pretty compelling to me - at least in terms of "let's publish the library solution and then based on experience with it, review PEP 505". Paul
On Mon, Jul 23, 2018 at 11:34 AM Robert Vanden Eynde <robertve92@gmail.com> wrote:
The default could be at the end with an argument to unboxing :
favorite = NoneAware(cfg).user.profile.food.unbox("Spam")
That's what PyMaybe does: https://pymaybe.readthedocs.io/en/latest/readme.html#examples-use-cases PyMaybe is a great implementation of the idea, but it's obviously limited in what it can do. It doesn't have short-circuiting (but it uses lambda to approximate it). It swallows a multitude of attribute and look up errors. It doesn't support coalescing. And it can't be used with any library that isn't PyMaybe-aware. For people who want to explore deeply nested objects/dictionaries (perhaps from unserialized JSON), there's also a pure Python library for that: http://objectpath.org/
Hi David and Paul Thank you both for your contributions, and for starting a new thread. David, you wrote
The need addressed by PEP 505 is real;
I completely agree. My view is that with PEP 505 as it it, the problem is better than the solution. But all the same, something can be done to improve matters.
I'm with you here also. First explore solving the problem using existing syntax. If nothing else, it helps us understand the problem. There's one area where I find attraction in a new syntax. The ?? operator is well described, I think, as a None-aware 'or'. But not in the name '??'. We all understand val1 = EXP1 or EXP2 So how about val2 = EXP1 or-if-None EXP2 with 'or-if-None' being a new language keyword!
These are all good goals. And (2) makes exploration much easier.
The API that could be useful might be something like this:
I have some different ideas, which I'll post later (to this thread, if you don't mind). Paul, you wrote
I like that approach. I find exploration using Python modules to be more open (democratic?) than syntax changes that require building a new executable. Once again, thank you both for starting at the problem and pointing in a different direction. -- Jonathan
On Mon, Jul 23, 2018 at 11:26 AM Paul Moore <p.f.moore@gmail.com> wrote:
On the last point, it would be easy enough to change the API to make it `NoneAware(obj).a.b.c.unbox(sentinel)` if that was thought a better API than `NoneAware(obj, sentinel).a.b.c.unbox()`. Oh... and the production code should DEFINITELY spell 'sentinel' correctly if that's a part of the API. :-) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
Le 23/07/2018 à 18:00, David Mertz a écrit :
One of the good things about this proposal, is that it may be expanded to other sentinels: ret = NoneAware(obj, sentinel=0).a.unbox(42) # would be equivalent to ret = obj.a if obj.a is not 0 else 42 # or, if tmp_a had side effects tmp_a = obj.a ret = tmp_a if tmp_a is not 0 else 42 One thing that this proposal doesn't cover easily (although I'd refactor the code before using this syntax, as excepted in some very specific cases like when using ORMs, I don't like chaining methods and properties like this): favorite = cfg?.user.profile?.food ?? "Spam" # every user should have a profile favorite = NoneAware(cfg).user.profile.food.unbox("Spam") # We're silencing an exception that I didn't plan to silence Another thing is that both parts are executed. If instead of "Spam" we had any variable with a side effect, it would have been executed even if cfg.user.profile.food existed. We're missing the lazy evaluation of this part here. That being said, I deeply believe the use case for that are not spread enough to justify such a change. And I can see from here code where every '.' is prepended by a '?' just in case (the same way we encounter frequently exceptions catching for Exception).
Your wrapper class can also, I think: Swallow exceptions, especially AttributeError Specify the sentinel, if it is not None Work for getitem Mixing is indeed more difficult, and there is probably performance impact. Still, it's general and does not need to touch the implementation of the class you want to descend... I like it better than the operators Another idea, may be stupid, maybe not: what if None fails with a subinstance of AttributeError, and a subinstance of KeyError, or IndexError. Then a try/except catching only those exceptions would also work quite nicely...
On Mon, 23 Jul 2018 11:12:45 -0400 David Mertz <mertz@gnosis.cx> wrote:
You could use .__call__() instead of .unbox(). Also you can make unboxing unnecessary in most cases by having your class proxy most operations, like weakref.proxy does. The "wrapt" library may help with that: http://wrapt.readthedocs.io/en/latest/wrappers.html Regards Antoine.
On Mon, Jul 23, 2018 at 12:47 PM Antoine Pitrou <solipsis@pitrou.net> wrote:
favorite = cfg?.user?.profile?.food ?? "Spam" favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
I'm not sure I entirely understand how that class proxy could be made transparent. Maybe you can make a toy implementation to show me? How can we make this work? favorite = NoneAware(cfg, "Spam").user.profile.food + "and more spam" I just noticed in my scratch directory that I had implemented almost the same thing a year ago when this discussion last came up. It's simple enough that the code was almost the same back then. I had included `.__getitem__()` in that version, but had not considered sentinels. However, one thing I *did* implement was `.__call__()`, but that reminds me of why your idea cannot work. I had it either call or fall back to more boxing. The end point of the chained attributes (or dict lookups) can perfectly well be a callable. E.g. favorite = NoneAware(cfg, "Spam").user.profile.get_food() Well, that's not going to work in my current toy code either since it would need to be: favorite = NoneAware(cfg, "Spam").user.profile.get_food.unbox()() But I could implement special stuff for "if the tail is a callable, unbox it automatically", I suppose. However, it would be fundamentally ambiguous, I think, whether those final parens should mean "unbox" or "call". Or I guess if calling *always* meant "unbox" we could have: favorite = NoneAware(cfg, "Spam").user.profile.get_food()() That feels weird to me though. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
Btw. I *do* realize that the semantics of my suggested NoneAware class is different from the coalescing syntax. I just look at whether attribute access succeeds or fails in that code, not whether the starting value is specifically None (nor any particular sentinel). I believe that that behavior better fits the ACTUAL need underlying these ideas (i.e. potentially deeply nested but messy data; from JSON or similar sources). If we wanted to match the semantics of PEP 505 more exactly, it would be easy enough to compare a current level of the hierarchy to None specifically rather than check more generically "is this a thing that has that attribute?" However, if the PEP 505 semantics really are the ones most desirable (i.e. including raising AttributeError on things that are not-None-but-don't-have-attribute), that's perfectly straightforward to implement using existing syntax and an API very close to what I suggest. On Mon, Jul 23, 2018 at 11:12 AM David Mertz <mertz@gnosis.cx> wrote:
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Mon, Jul 23, 2018 at 11:12:45AM -0400, David Mertz wrote:
That's one opinion.
(4) Inasmuch as there are semantic traps, they are announced by the use of a class whose documentation would be pointed to for readers
Operators have documentation too. If you think people won't read the documentation for the operator, what makes you think they're read the documentation for the class?
There's the first bug right there. foo.bar.blim ought to raise AttributeError, since there is no blim attribute defined on foo.bar. Reminder: the proposal is for a null-coalescing operator, not an AttributeError suppressing operator. The PEP is explicit that catching AttributeError is rejected: https://www.python.org/dev/peps/pep-0505/#id21
How many bugs will be caused by people forgetting to unbox when they're done?
That ought to be AttributeError again, since there is no "flam" attribute defined on foo.bar.baz.
But we aren't entering such a world, at least not in PEP 505. Attribute access can fail. spam.eggs = 42 spam?.eggs?.upper is still going to raise AttributeError, because eggs is not None, it is an int, and ints don't have an attribute "upper".
How does your class implement short-circuit behaviour?
So we don't like operators like ?? because its too cryptic, but you're happy to have one-character class and property names. favorite = N(cfg, "Spam").user.profile.food.u What happens if you have an attribute that happens to be called "unbox" in your attribute look-up chain? result = NoneAware(something, "default").spam.foo.unbox.eggs.unbox() -- Steve
The proto here swallow and short circuit on attribute error. Changing to do it on Noneness is easy, and you can choose between the two behavior: it's a strength compared to the operator approach. It's working in current python, another strength. I think short-circuit behavior is similar: once stopping condition has been met, the gard object cascade till the end of the nested attribute lookup. Like None do for the operator. More complex stuff when None-coalescing attribute access is mixed with normal access is better done with operators. Maybe complex operations fine of the middle of the access chain would also be easier with the operators. How many times would you encounter that in real cases? I bet on very few. In those few case, would a one line ?. chain expression be better than a multiple line logic with temporaries? I think no. That's my feeling, but in the absence of concrete examples, feelings are ok.
On Mon, Jul 23, 2018 at 10:53:11AM -0700, Grégory Lielens wrote:
Being able to choose between one desirable behaviour and one undesirable, rejected behaviour is *not* a strength. PEP 505 has a section explaining why catching AttributeError is undesirable. I find the reasons it gives are compelling. Can you explain why you reject the PEP's arguments against catching AttributeError? -- Steve
On Tuesday, July 24, 2018 at 2:39:19 AM UTC+2, Steven D'Aprano wrote:
Yes, and people have done it multiple time already. I will try to make it as explicit as possible, with a terminology that some may find useful... when u have a object hierarchy linked through attributes, which -seems- the use pattern of ?. , the hierarchy is either - 1) fully regular (each attribute contains the same type of objects (same type=with the same attributes), then normal . access is what you want - 2) regular-or-None (contain same type of objects, or None). That's what ?. is for, and ?. is only for that. - 3) its partially regular (does not contain the same type of objects, some objects have more attributes than other, e.g. partial objects). - 4) its irregular (objects may be of completely different type (int, str,... and do not specailly share attributes). ?. is intended for 2). 2) and 3) can be dealed with swallowing AttributeError 4) can also be traversed swallowing AttributeError, but it's probably a very bad idea in almost all cases. I've encountered all cases, from time to time, but none is really typical of most of my code. Which is more common? not sure, I would say 3). What is sure is that 2) do not stand as hugely more common than the others, so introducing a special syntax for 2) do not please me, so +0... Now as I find ?. unpleasant to parse (personal opinion), and having grown to share the general operaror / line noise allergy mainstream in the python community, i am a firm -1 on this: too small a niche, do not bring much compared to current approach, and visual line noise. firm -1 on ?? too, it's less polemic, less visually displeasing, but even less usefull I think: it does not gain that much compared to a more general ternary. As the pymaybe/this thread cover 2) and 3), and do it without any change to python, I find it better. No change > lib change > backward compatible object change - like dict becoming ordered, or None triggering NoneAttributeError(AttributeError) > syntax change (expecially a bunch of linenoisy operators). Now I will stop for a while here, arguments have become circular, proponents and opponents have exposed their view, but mostly proponents have provided very few additional information for a while, especially about real use cases, despite a few requests. We still do not know what motivated the proposal..We had to guess (JSON). Even the PEP do not say, it provide a set of real world example from the standard library, which is admirable but different. And personaly, while am very thankful for providing those concrete examples, I find them unconvicing: the code before is usually fine and improved little by the new operators. Sometimes it's worse: only slightly shorter and less clear.
On Mon, Jul 23, 2018 at 1:28 PM Steven D'Aprano <steve@pearwood.info> wrote:
There's the first bug right there. foo.bar.blim ought to raise AttributeError, since there is no blim attribute defined on foo.bar.
Note my observation that this is deliberately different semantics. I want to solve the underlying need, not simply emulate the less useful semantics of PEP 505. But those same semantics COULD be perfectly well emulated with a similar class.
Far fewer and far more easily caught. When the object you get wherever the real failure happens is ` <none_aware.NoneAware at 0x11156a748>` or the like, the source of the problem become dead obvious. Having `None` somewhere unexpected is much more mysterious and difficult to diagnose. That said, NoneAware is the wrong name for the class I wrote. Maybe GreedyAccess would be a better name. Perhaps I'll make slightly less toy implementations of an actual NoneCoalesce and GreedyAccess class and post in this thread and/or on PyPI. It does look like PyMaybe does much of what I'm thinking of. I didn't think my toy was the first or best implementation.
But we aren't entering such a world, at least not in PEP 505. Attribute access can fail.
spam.eggs = 42
spam?.eggs?.upper
is still going to raise AttributeError, because eggs is not None, it is an int, and ints don't have an attribute "upper".
True enough. But it's extremely difficult to imagine a real-world case where those particular semantics are actually particularly useful. I guess my revulsion at the syntax blinded me to the fact that the semantics are actually pretty bad on their own.
How does your class implement short-circuit behaviour?
You mean if we want something like this? favorite = GreedyAccess(cfg, sentinel=ExpensiveLookup()).user.profile.food.unbox() It doesn't. Something else (like PyMaybe) could defer the computation using a lambda or other means. My toy code doesn't do that. We could also avoid the visible lambda by doing an 'iscallable(sentinel)' and only calling it if it turns out to be needed, but that might be too magical. E.g.: # If we fail, favorite = ExpensiveLookup(), by magic favorite = GreedyAccess(cfg, sentinel=ExpensiveLookup).user.profile.food.unbox() So we don't like operators like ?? because its too cryptic, but you're
happy to have one-character class and property names.
No, I'm not happy with that. I was just pointing out that code golf is possible for someone who really wants it. I'm happy to spend a few characters for readability. But one COULD write: from nested import GreedyAccess as G G._ = G.unbox
What happens if you have an attribute that happens to be called "unbox" in your attribute look-up chain?
Don't use this class and/or modify the API slightly? This is getting trite. My 5 minute code isn't a final proposal, just a way of seeing a napkin-sketch of a better approach. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Mon, Jul 23, 2018 at 01:55:48PM -0400, David Mertz wrote:
I know that "laziness and hubris" are virtues in a programmer, but the PEP authors describe use-cases where *testing for None* is precisely the semantics wanted. What are your use-cases for greedily swallowing AttributeError? Before telling the PEP authors that they have misunderstood their own uses-cases and that their need isn't what they thought (coalescing None) but something radically different which they have already explicitly rejected (suppressing AttributeError), you better have some damn compelling use-cases. "Laziness and hubris" are supposed to be virtues in programmers, but when you insist that people have their own use-cases wrong, and that the designers of C#, Dart and Swift made a serious mistake introducing this feature, you ought to back it up with more than just an assertion. [...]
It does look like PyMaybe does much of what I'm thinking of. I didn't think my toy was the first or best implementation.
Are you aware that the PEP includes pymaybe as a rejected idea?
You can't imagine why it would be useful to NOT suppress the exception from a coding error like trying to uppercase an int. -- Steve
On 23/07/18 16:12, David Mertz wrote:
Here is a way of solving the "deep attribute access to messy data" problem
Before we go too far, and /pace/ Steve's work, how real is this problem? I get that there is a use in accessing JSON trees, but this feels awfully like a rather specific issue about avoiding cleaning data before using it. As such, should be really be encouraging it? (For the avoidance of doubt, this refers to the proposed ?. and ?[] operators.
Worthy goals.
...and here's the first semantic trap. I was expecting to get None back there, and even your pre-announcement in goal (4) didn't stop me puzzling over it for a couple of minutes. The only reason I wasn't expecting an AttributeError was that I was confident you wouldn't propose this without doing something magic to effectively override the meaning of "." A naive user might be very surprised.
[snip]
You and I have very different definitions of "nice" ;-) I have to say it's questionable which of "?." and "NoneAware()...unbox()" are uglier. Then again, I go back to my original comment: is this really a problem we want to solve? How are you supposed to do method calling, the equivalent of "foo?.bar()" ? "NoneAware(foo).bar.unbox()()" looks downright weird. Is there more magic in NoneAware to cover this case? (Not that I think we should be encouraging people to do this, but...) -- Rhodri James *-* Kynesim Ltd
On Mon, Jul 23, 2018 at 2:12 PM Rhodri James <rhodri@kynesim.co.uk> wrote:
Is there more magic? I don't know, and I don't really care that much. That's the point. This is just a plain old Python class that will work back to probably Python 1.4 or so. If you want to write a version that has just the right amount of magic, you are free to. That said, I *DID* give it the wrong name in my first post of this thread. GreedyAccess is more accurate, and a NoneCoalesce class would behave somewhat differently. In my opinion, the need at issue is worthwhile, but niche. Using a special class to deal with such a case is absolutely the right level of abstraction. Syntax is not! So sure, figure out how to tweak the API to be most useful, find a better name for the class, etc. I wouldn't think it terrible if a class like this (but better) found a home in the standard library, but it doesn't deserve more prominence than that. Not even builtins, and *definitely* not syntax. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 23/07/18 19:21, David Mertz wrote:
I care only in as much as you were proposing an incomplete solution (in my eyes at least). I'm pretty convinced by now I would never use it.
I think part of my point was that a special class is not as readable or trap-free as writing the conditions out explicitly, given the unexpected "boxed" results, which makes me question whether the issue *is* worthwhile.
I think I might find it terrible. -- Rhodri James *-* Kynesim Ltd
Le 23/07/2018 à 17:12, David Mertz a écrit :
https://github.com/ekampf/pymaybe It's interest is much more limited than in Haskell because in Python, calls are not lazy, objects are dynamic and function calls are expensive. Take the following: foo().bar[0] Doing it safely would be: val = foo() try: val = val.bar except AttributeError: val = None else: try: val = val[0] except KeyError: val = None Clearly we see that the operation goes up to the end if everything goes right. Otherwise, it jumps to the default value. With the operator proposal: val = foo()?.bar?[0] The "?" operator is like "or" / "and", and it will shotcircuit as well. With the try / else proposal: val = try foo().bar[0] else None This is supposed to transform this ast to into the first try/except form. So it shortcircuit as well. But with your proposal, it becomes: val = maybe(foo()).bar[0].or_else(None) Which means foo() # call maybe(foo()) # call maybe(foo()).__getattr__('bar') # call maybe(foo()).__getattr__('bar').__getitem__(0) # call maybe(foo()).__getattr__('bar').__getitem__(0).or_else(None) # call There is no shortcircuit, you get 5 calls in all cases, plus the maybe object proxing the call to the underlying value every time. So that's 7 calls, added to the try/excepts you still have behind the scene. Plus, you don't get the same calls depending of the values, but the switch is implicit and behind the maybe object: no linter can save you from a typo. So yes, it works right now. But it's a suboptimal solution.
Short circuit is indeed the main difference. Makes me re-think about the None trigger subclasses of invalid operations exceptions. Like None.a trigger a NoneAttributeError(AttributeError), and so on... I think it solves many issues without much problems nor syntax changes, but probably I miss something... Sure, it's a big change, but maybe less so than 3 non-classic operators working only on None...
I agree a class can provide a very good alternative to PEP505. I built one a while ago, and still use it https://github.com/klahnakoski/mo-dots for most my data transformation code. The library only boxes dicts and lists, which removes the need for .unbox() in many of my use cases. My point is, if a class is chosen instead of PEP505, I doubt the unboxing will be a big issue. On 2018-07-23 11:12, David Mertz wrote:
Here is a way of solving the "deep attribute access to messy data" problem that is:
David Mertz schrieb am 23.07.2018 um 16:12:
The discussion so far made it clear to me that a) there is a use case for this feature, although I never needed it myself b) throwing new syntax at it is not the right solution Especially since there seem to be slightly diverging ideas about the exact details in behaviour. Since this can be done in form of a library, people should just drop it into a couple of competing libraries and let users choose what they like better in their specific situation. And since we already have a PEP now, let's continue to use it as a basis for discussion about how these libraries should best behave in general and what mistakes they should avoid. Stefan
On Tue, Jul 24, 2018 at 1:57 AM Stefan Behnel <stefan_ml@behnel.de> wrote:
+1 There is still no proof that such a programming pattern would be widely used or is desirable in practice. A library on PYPI could help clarifying that (and the PEP should mention it). -- Giampaolo - http://grodola.blogspot.com
participants (14)
-
Antoine Pitrou -
Brice Parent -
David Mertz -
Giampaolo Rodola' -
Grégory Lielens -
Jonathan Fine -
Kyle Lahnakoski -
Mark E. Haase -
Michel Desmoulin -
Paul Moore -
Rhodri James -
Robert Vanden Eynde -
Stefan Behnel -
Steven D'Aprano