A better (simpler) approach to PEP 505
The need addressed by PEP 505 is real; it's also MUCH more niche and uncommon than something that would merit new syntax. Moreover, the actual legitimate purpose served by the PEP 505 syntax is easily served by existing Python simply by using a wrapper class.
Here is a way of solving the "deep attribute access to messy data" problem that is:
(1) Much more explicit (2) Requires no change in syntax (3) Will not be a bug magnet (4) Inasmuch as there are semantic traps, they are announced by the use of a class whose documentation would be pointed to for readers
The API that could be useful might be something like this:
In [1]: from none_aware import NoneAware In [2]: from types import SimpleNamespace In [3]: foo = SimpleNamespace() In [4]: foo.bar = SimpleNamespace() In [5]: foo.bar.baz = SimpleNamespace() In [6]: foo.bar.baz.blat = 42 In [7]: NoneAware(foo).bar.blim Out[7]: <none_aware.NoneAware at 0x11156a748> In [8]: NoneAware(foo).bar.blim.unbox() In [9]: NoneAware(foo).bar.baz.blat.unbox() Out[9]: 42 In [10]: NoneAware(foo).bar.baz.blat Out[10]: <none_aware.NoneAware at 0x11157d908> In [11]: NoneAware(foo).bar.baz.flam.unbox() In [12]: NoneAware(foo).bar.baz.flam Out[12]: <none_aware.NoneAware at 0x1115832b0>
The particular names I use are nothing special, and better ones might be found. I just called the class NoneAware and the "escape" method `.unbox()` because that seemed intuitive at first brush.
I don't disagree that needing to call .unbox() at the end of the chained attribute access is a little bit ugly. But it's a lot less ugly than large family of new operators. And honestly, it's a nice way of being explicit about the fact that we're entering then leaving a special world where attribute accesses don't fail.
I haven't implemented the equivalent dictionary lookups in the below. That would be straightforward, and I'm sure my 5 minute throwaway code could be improved in other ways also. But something better than this in the standard library would address ALL the actual needs described in PEP 505. Even the pattern Steve Dower is especially fond of like:
favorite = cfg?.user?.profile?.food ?? "Spam"
(i.e. a configuration may be incomplete at any level, if levels are missing default favorite food is Spam). We could simply spell that:
favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
I think that's 14 characters more in this example, but still compact. We could get that down to 2 characters if we used one-letter names for the class and method. I suppose down to zero characters if .unbox() was a property.
So completely toy implementation:
class NoneAware(object): def __init__(self, thing, sentinal=None): self.thing = thing self.sentinal = sentinal
def __getattr__(self, attr): try: return NoneAware(getattr(self.thing, attr)) except AttributeError: return NoneAware(self.sentinal)
def unbox(self): return self.thing
On 23 July 2018 at 16:12, David Mertz mertz@gnosis.cx wrote:
The need addressed by PEP 505 is real; it's also MUCH more niche and uncommon than something that would merit new syntax. Moreover, the actual legitimate purpose served by the PEP 505 syntax is easily served by existing Python simply by using a wrapper class.
Here is a way of solving the "deep attribute access to messy data" problem that is:
(1) Much more explicit (2) Requires no change in syntax (3) Will not be a bug magnet (4) Inasmuch as there are semantic traps, they are announced by the use of a class whose documentation would be pointed to for readers
[...]
I haven't implemented the equivalent dictionary lookups in the below. That would be straightforward, and I'm sure my 5 minute throwaway code could be improved in other ways also. But something better than this in the standard library would address ALL the actual needs described in PEP 505. Even the pattern Steve Dower is especially fond of like:
favorite = cfg?.user?.profile?.food ?? "Spam"
(i.e. a configuration may be incomplete at any level, if levels are missing default favorite food is Spam). We could simply spell that:
favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
Thank you. That's the sort of "what would this look like if implemented as a library rather than language syntax" solution that I had in mind with my earlier post.
I would be very interested to hear discussion of the pros and cons of adding new syntax to the language as per PEP 505 in comparison to a solution like this (ultimately more fleshed out and "production quality") rather than comparisons PEP 505 to raw "roll your own" Python code.
For me:
* Library solution works with all versions of Python * The need for unbox is a little ugly, but arguably less so than ?. (conceded that's a subjective view) * Mixing ?. and . is terser than unboxing and reboxing - but are there any real examples where that's needed? * Having the default at the beginning rather than at the end doesn't follow natural reading order (but again that's pretty subjective)
There's little here that isn't subjective, so I expect a lot of heated debate that ultimately convinces no-one. But the "you can use this solution now" aspect of the library solution seems pretty compelling to me - at least in terms of "let's publish the library solution and then based on experience with it, review PEP 505".
Paul
The default could be at the end with an argument to unboxing :
favorite = NoneAware(cfg).user.profile.food.unbox("Spam")
Le lun. 23 juil. 2018 à 17:26, Paul Moore p.f.moore@gmail.com a écrit :
On 23 July 2018 at 16:12, David Mertz mertz@gnosis.cx wrote:
The need addressed by PEP 505 is real; it's also MUCH more niche and uncommon than something that would merit new syntax. Moreover, the
actual
legitimate purpose served by the PEP 505 syntax is easily served by
existing
Python simply by using a wrapper class.
Here is a way of solving the "deep attribute access to messy data"
problem
that is:
(1) Much more explicit (2) Requires no change in syntax (3) Will not be a bug magnet (4) Inasmuch as there are semantic traps, they are announced by the use
of a
class whose documentation would be pointed to for readers
[...]
I haven't implemented the equivalent dictionary lookups in the below.
That
would be straightforward, and I'm sure my 5 minute throwaway code could
be
improved in other ways also. But something better than this in the
standard
library would address ALL the actual needs described in PEP 505. Even
the
pattern Steve Dower is especially fond of like:
favorite = cfg?.user?.profile?.food ?? "Spam"
(i.e. a configuration may be incomplete at any level, if levels are
missing
default favorite food is Spam). We could simply spell that:
favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
Thank you. That's the sort of "what would this look like if implemented as a library rather than language syntax" solution that I had in mind with my earlier post.
I would be very interested to hear discussion of the pros and cons of adding new syntax to the language as per PEP 505 in comparison to a solution like this (ultimately more fleshed out and "production quality") rather than comparisons PEP 505 to raw "roll your own" Python code.
For me:
- Library solution works with all versions of Python
- The need for unbox is a little ugly, but arguably less so than ?.
(conceded that's a subjective view)
- Mixing ?. and . is terser than unboxing and reboxing - but are there
any real examples where that's needed?
- Having the default at the beginning rather than at the end doesn't
follow natural reading order (but again that's pretty subjective)
There's little here that isn't subjective, so I expect a lot of heated debate that ultimately convinces no-one. But the "you can use this solution now" aspect of the library solution seems pretty compelling to me - at least in terms of "let's publish the library solution and then based on experience with it, review PEP 505".
Paul _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Mon, Jul 23, 2018 at 11:34 AM Robert Vanden Eynde robertve92@gmail.com wrote:
The default could be at the end with an argument to unboxing :
favorite = NoneAware(cfg).user.profile.food.unbox("Spam")
That's what PyMaybe does:
https://pymaybe.readthedocs.io/en/latest/readme.html#examples-use-cases
PyMaybe is a great implementation of the idea, but it's obviously limited in what it can do. It doesn't have short-circuiting (but it uses lambda to approximate it). It swallows a multitude of attribute and look up errors. It doesn't support coalescing. And it can't be used with any library that isn't PyMaybe-aware.
For people who want to explore deeply nested objects/dictionaries (perhaps from unserialized JSON), there's also a pure Python library for that:
Hi David and Paul
Thank you both for your contributions, and for starting a new thread.
David, you wrote
The need addressed by PEP 505 is real;
I completely agree. My view is that with PEP 505 as it it, the problem is better than the solution. But all the same, something can be done to improve matters.
Moreover, the actual legitimate purpose served by the PEP 505 syntax is easily served by existing Python simply by using a wrapper class.
I'm with you here also. First explore solving the problem using existing syntax. If nothing else, it helps us understand the problem.
There's one area where I find attraction in a new syntax. The ?? operator is well described, I think, as a None-aware 'or'. But not in the name '??'. We all understand val1 = EXP1 or EXP2 So how about val2 = EXP1 or-if-None EXP2 with 'or-if-None' being a new language keyword!
(1) Much more explicit (2) Requires no change in syntax (3) Will not be a bug magnet (4) [Fewer, clear] semantic traps
These are all good goals. And (2) makes exploration much easier.
The API that could be useful might be something like this:
I have some different ideas, which I'll post later (to this thread, if you don't mind).
Paul, you wrote
I would be very interested to hear discussion of the pros and cons of adding new syntax to the language as per PEP 505 in comparison to a solution like this (ultimately more fleshed out and "production quality") rather than comparisons PEP 505 to raw "roll your own" Python code.
I like that approach. I find exploration using Python modules to be more open (democratic?) than syntax changes that require building a new executable.
Once again, thank you both for starting at the problem and pointing in a different direction.
On Mon, Jul 23, 2018 at 11:26 AM Paul Moore p.f.moore@gmail.com wrote:
- Library solution works with all versions of Python
- The need for unbox is a little ugly, but arguably less so than ?.
(conceded that's a subjective view)
- Mixing ?. and . is terser than unboxing and reboxing - but are there
any real examples where that's needed?
- Having the default at the beginning rather than at the end doesn't
follow natural reading order (but again that's pretty subjective)
On the last point, it would be easy enough to change the API to make it `NoneAware(obj).a.b.c.unbox(sentinel)` if that was thought a better API than `NoneAware(obj, sentinel).a.b.c.unbox()`.
Oh... and the production code should DEFINITELY spell 'sentinel' correctly if that's a part of the API. :-)
Le 23/07/2018 à 18:00, David Mertz a écrit :
On Mon, Jul 23, 2018 at 11:26 AM Paul Moore <p.f.moore@gmail.com mailto:p.f.moore@gmail.com> wrote:
* Library solution works with all versions of Python * The need for unbox is a little ugly, but arguably less so than ?. (conceded that's a subjective view) * Mixing ?. and . is terser than unboxing and reboxing - but are there any real examples where that's needed? * Having the default at the beginning rather than at the end doesn't follow natural reading order (but again that's pretty subjective)On the last point, it would be easy enough to change the API to make it `NoneAware(obj).a.b.c.unbox(sentinel)` if that was thought a better API than `NoneAware(obj, sentinel).a.b.c.unbox()`. Oh... and the production code should DEFINITELY spell 'sentinel' correctly if that's a part of the API. :-)
One of the good things about this proposal, is that it may be expanded to other sentinels:
ret = NoneAware(obj, sentinel=0).a.unbox(42)
# would be equivalent to ret = obj.a if obj.a is not 0 else 42
# or, if tmp_a had side effects tmp_a = obj.a ret = tmp_a if tmp_a is not 0 else 42
One thing that this proposal doesn't cover easily (although I'd refactor the code before using this syntax, as excepted in some very specific cases like when using ORMs, I don't like chaining methods and properties like this):
favorite = cfg?.user.profile?.food ?? "Spam" # every user should have a profile
favorite = NoneAware(cfg).user.profile.food.unbox("Spam") # We're silencing an exception that I didn't plan to silence
Another thing is that both parts are executed. If instead of "Spam" we had any variable with a side effect, it would have been executed even if cfg.user.profile.food existed. We're missing the lazy evaluation of this part here.
That being said, I deeply believe the use case for that are not spread enough to justify such a change. And I can see from here code where every '.' is prepended by a '?' just in case (the same way we encounter frequently exceptions catching for Exception).
Your wrapper class can also, I think: Swallow exceptions, especially AttributeError Specify the sentinel, if it is not None Work for getitem
Mixing is indeed more difficult, and there is probably performance impact. Still, it's general and does not need to touch the implementation of the class you want to descend...
I like it better than the operators
Another idea, may be stupid, maybe not: what if None fails with a subinstance of AttributeError, and a subinstance of KeyError, or IndexError. Then a try/except catching only those exceptions would also work quite nicely...
On Mon, 23 Jul 2018 11:12:45 -0400 David Mertz mertz@gnosis.cx wrote:
The particular names I use are nothing special, and better ones might be found. I just called the class NoneAware and the "escape" method `.unbox()` because that seemed intuitive at first brush.
I don't disagree that needing to call .unbox() at the end of the chained attribute access is a little bit ugly. But it's a lot less ugly than large family of new operators. And honestly, it's a nice way of being explicit about the fact that we're entering then leaving a special world where attribute accesses don't fail.
I haven't implemented the equivalent dictionary lookups in the below. That would be straightforward, and I'm sure my 5 minute throwaway code could be improved in other ways also. But something better than this in the standard library would address ALL the actual needs described in PEP 505. Even the pattern Steve Dower is especially fond of like:
favorite = cfg?.user?.profile?.food ?? "Spam"
(i.e. a configuration may be incomplete at any level, if levels are missing default favorite food is Spam). We could simply spell that:
favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
I think that's 14 characters more in this example, but still compact. We could get that down to 2 characters if we used one-letter names for the class and method. I suppose down to zero characters if .unbox() was a property.
You could use .__call__() instead of .unbox(). Also you can make unboxing unnecessary in most cases by having your class proxy most operations, like weakref.proxy does. The "wrapt" library may help with that: http://wrapt.readthedocs.io/en/latest/wrappers.html
Regards
Antoine.
On Mon, Jul 23, 2018 at 12:47 PM Antoine Pitrou solipsis@pitrou.net wrote:
favorite = cfg?.user?.profile?.food ?? "Spam" favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
You could use .__call__() instead of .unbox(). Also you can make unboxing unnecessary in most cases by having your class proxy most operations, like weakref.proxy does. The "wrapt" library may help with that: http://wrapt.readthedocs.io/en/latest/wrappers.html
I'm not sure I entirely understand how that class proxy could be made transparent. Maybe you can make a toy implementation to show me? How can we make this work?
favorite = NoneAware(cfg, "Spam").user.profile.food + "and more spam"
I just noticed in my scratch directory that I had implemented almost the same thing a year ago when this discussion last came up. It's simple enough that the code was almost the same back then. I had included `.__getitem__()` in that version, but had not considered sentinels.
However, one thing I *did* implement was `.__call__()`, but that reminds me of why your idea cannot work. I had it either call or fall back to more boxing. The end point of the chained attributes (or dict lookups) can perfectly well be a callable. E.g.
favorite = NoneAware(cfg, "Spam").user.profile.get_food()
Well, that's not going to work in my current toy code either since it would need to be:
favorite = NoneAware(cfg, "Spam").user.profile.get_food.unbox()()
But I could implement special stuff for "if the tail is a callable, unbox it automatically", I suppose.
However, it would be fundamentally ambiguous, I think, whether those final parens should mean "unbox" or "call". Or I guess if calling *always* meant "unbox" we could have:
favorite = NoneAware(cfg, "Spam").user.profile.get_food()()
That feels weird to me though.
Btw. I *do* realize that the semantics of my suggested NoneAware class is different from the coalescing syntax. I just look at whether attribute access succeeds or fails in that code, not whether the starting value is specifically None (nor any particular sentinel).
I believe that that behavior better fits the ACTUAL need underlying these ideas (i.e. potentially deeply nested but messy data; from JSON or similar sources). If we wanted to match the semantics of PEP 505 more exactly, it would be easy enough to compare a current level of the hierarchy to None specifically rather than check more generically "is this a thing that has that attribute?"
However, if the PEP 505 semantics really are the ones most desirable (i.e. including raising AttributeError on things that are not-None-but-don't-have-attribute), that's perfectly straightforward to implement using existing syntax and an API very close to what I suggest.
On Mon, Jul 23, 2018 at 11:12 AM David Mertz mertz@gnosis.cx wrote:
The need addressed by PEP 505 is real; it's also MUCH more niche and uncommon than something that would merit new syntax. Moreover, the actual legitimate purpose served by the PEP 505 syntax is easily served by existing Python simply by using a wrapper class.
Here is a way of solving the "deep attribute access to messy data" problem that is:
(1) Much more explicit (2) Requires no change in syntax (3) Will not be a bug magnet (4) Inasmuch as there are semantic traps, they are announced by the use of a class whose documentation would be pointed to for readers
The API that could be useful might be something like this:
In [1]: from none_aware import NoneAware In [2]: from types import SimpleNamespace In [3]: foo = SimpleNamespace() In [4]: foo.bar = SimpleNamespace() In [5]: foo.bar.baz = SimpleNamespace() In [6]: foo.bar.baz.blat = 42 In [7]: NoneAware(foo).bar.blim Out[7]: <none_aware.NoneAware at 0x11156a748> In [8]: NoneAware(foo).bar.blim.unbox() In [9]: NoneAware(foo).bar.baz.blat.unbox() Out[9]: 42 In [10]: NoneAware(foo).bar.baz.blat Out[10]: <none_aware.NoneAware at 0x11157d908> In [11]: NoneAware(foo).bar.baz.flam.unbox() In [12]: NoneAware(foo).bar.baz.flam Out[12]: <none_aware.NoneAware at 0x1115832b0>
The particular names I use are nothing special, and better ones might be found. I just called the class NoneAware and the "escape" method `.unbox()` because that seemed intuitive at first brush.
I don't disagree that needing to call .unbox() at the end of the chained attribute access is a little bit ugly. But it's a lot less ugly than large family of new operators. And honestly, it's a nice way of being explicit about the fact that we're entering then leaving a special world where attribute accesses don't fail.
I haven't implemented the equivalent dictionary lookups in the below. That would be straightforward, and I'm sure my 5 minute throwaway code could be improved in other ways also. But something better than this in the standard library would address ALL the actual needs described in PEP 505. Even the pattern Steve Dower is especially fond of like:
favorite = cfg?.user?.profile?.food ?? "Spam"
(i.e. a configuration may be incomplete at any level, if levels are missing default favorite food is Spam). We could simply spell that:
favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
I think that's 14 characters more in this example, but still compact. We could get that down to 2 characters if we used one-letter names for the class and method. I suppose down to zero characters if .unbox() was a property.
So completely toy implementation:
class NoneAware(object): def __init__(self, thing, sentinel=None): self.thing = thing self.sentinel = sentinal
def __getattr__(self, attr): try: return NoneAware(getattr(self.thing, attr)) except AttributeError: return NoneAware(self.sentinel) def unbox(self): return self.thing-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Mon, Jul 23, 2018 at 11:12:45AM -0400, David Mertz wrote:
Here is a way of solving the "deep attribute access to messy data" problem that is:
(1) Much more explicit (2) Requires no change in syntax (3) Will not be a bug magnet
That's one opinion.
(4) Inasmuch as there are semantic traps, they are announced by the use of a class whose documentation would be pointed to for readers
Operators have documentation too. If you think people won't read the documentation for the operator, what makes you think they're read the documentation for the class?
The API that could be useful might be something like this:
In [1]: from none_aware import NoneAware In [2]: from types import SimpleNamespace In [3]: foo = SimpleNamespace() In [4]: foo.bar = SimpleNamespace() In [5]: foo.bar.baz = SimpleNamespace() In [6]: foo.bar.baz.blat = 42 In [7]: NoneAware(foo).bar.blim Out[7]: <none_aware.NoneAware at 0x11156a748>
There's the first bug right there. foo.bar.blim ought to raise AttributeError, since there is no blim attribute defined on foo.bar.
Reminder: the proposal is for a null-coalescing operator, not an AttributeError suppressing operator.
The PEP is explicit that catching AttributeError is rejected:
https://www.python.org/dev/peps/pep-0505/#id21
In [8]: NoneAware(foo).bar.blim.unbox() In [9]: NoneAware(foo).bar.baz.blat.unbox() Out[9]: 42
How many bugs will be caused by people forgetting to unbox when they're done?
In [10]: NoneAware(foo).bar.baz.blat Out[10]: <none_aware.NoneAware at 0x11157d908> In [11]: NoneAware(foo).bar.baz.flam.unbox()
That ought to be AttributeError again, since there is no "flam" attribute defined on foo.bar.baz.
In [12]: NoneAware(foo).bar.baz.flam Out[12]: <none_aware.NoneAware at 0x1115832b0>
The particular names I use are nothing special, and better ones might be found. I just called the class NoneAware and the "escape" method `.unbox()` because that seemed intuitive at first brush.
I don't disagree that needing to call .unbox() at the end of the chained attribute access is a little bit ugly. But it's a lot less ugly than large family of new operators. And honestly, it's a nice way of being explicit about the fact that we're entering then leaving a special world where attribute accesses don't fail.
But we aren't entering such a world, at least not in PEP 505. Attribute access can fail.
spam.eggs = 42
spam?.eggs?.upper
is still going to raise AttributeError, because eggs is not None, it is an int, and ints don't have an attribute "upper".
I haven't implemented the equivalent dictionary lookups in the below. That would be straightforward, and I'm sure my 5 minute throwaway code could be improved in other ways also. But something better than this in the standard library would address ALL the actual needs described in PEP 505.
How does your class implement short-circuit behaviour?
Even the pattern Steve Dower is especially fond of like:
favorite = cfg?.user?.profile?.food ?? "Spam"
(i.e. a configuration may be incomplete at any level, if levels are missing default favorite food is Spam). We could simply spell that:
favorite = NoneAware(cfg, "Spam").user.profile.food.unbox()
I think that's 14 characters more in this example, but still compact. We could get that down to 2 characters if we used one-letter names for the class and method. I suppose down to zero characters if .unbox() was a property.
So we don't like operators like ?? because its too cryptic, but you're happy to have one-character class and property names.
favorite = N(cfg, "Spam").user.profile.food.u
What happens if you have an attribute that happens to be called "unbox" in your attribute look-up chain?
result = NoneAware(something, "default").spam.foo.unbox.eggs.unbox()
The proto here swallow and short circuit on attribute error. Changing to do it on Noneness is easy, and you can choose between the two behavior: it's a strength compared to the operator approach.
It's working in current python, another strength.
I think short-circuit behavior is similar: once stopping condition has been met, the gard object cascade till the end of the nested attribute lookup. Like None do for the operator.
More complex stuff when None-coalescing attribute access is mixed with normal access is better done with operators. Maybe complex operations fine of the middle of the access chain would also be easier with the operators.
How many times would you encounter that in real cases? I bet on very few. In those few case, would a one line ?. chain expression be better than a multiple line logic with temporaries? I think no.
That's my feeling, but in the absence of concrete examples, feelings are ok.
On Mon, Jul 23, 2018 at 10:53:11AM -0700, Grégory Lielens wrote:
The proto here swallow and short circuit on attribute error. Changing to do it on Noneness is easy, and you can choose between the two behavior: it's a strength compared to the operator approach.
Being able to choose between one desirable behaviour and one undesirable, rejected behaviour is *not* a strength.
PEP 505 has a section explaining why catching AttributeError is undesirable. I find the reasons it gives are compelling.
Can you explain why you reject the PEP's arguments against catching AttributeError?
On Tuesday, July 24, 2018 at 2:39:19 AM UTC+2, Steven D'Aprano wrote:
PEP 505 has a section explaining why catching AttributeError is undesirable. I find the reasons it gives are compelling.
Can you explain why you reject the PEP's arguments against catching AttributeError?
Yes, and people have done it multiple time already. I will try to make it as explicit as possible, with a terminology that some may find useful... when u have a object hierarchy linked through attributes, which -seems- the use pattern of ?. , the hierarchy is either - 1) fully regular (each attribute contains the same type of objects (same type=with the same attributes), then normal . access is what you want - 2) regular-or-None (contain same type of objects, or None). That's what ?. is for, and ?. is only for that. - 3) its partially regular (does not contain the same type of objects, some objects have more attributes than other, e.g. partial objects). - 4) its irregular (objects may be of completely different type (int, str,... and do not specailly share attributes).
?. is intended for 2). 2) and 3) can be dealed with swallowing AttributeError 4) can also be traversed swallowing AttributeError, but it's probably a very bad idea in almost all cases.
I've encountered all cases, from time to time, but none is really typical of most of my code. Which is more common? not sure, I would say 3). What is sure is that 2) do not stand as hugely more common than the others, so introducing a special syntax for 2) do not please me, so +0... Now as I find ?. unpleasant to parse (personal opinion), and having grown to share the general operaror / line noise allergy mainstream in the python community, i am a firm -1 on this: too small a niche, do not bring much compared to current approach, and visual line noise. firm -1 on ?? too, it's less polemic, less visually displeasing, but even less usefull I think: it does not gain that much compared to a more general ternary.
As the pymaybe/this thread cover 2) and 3), and do it without any change to python, I find it better. No change > lib change > backward compatible object change - like dict becoming ordered, or None triggering NoneAttributeError(AttributeError) > syntax change (expecially a bunch of linenoisy operators).
Now I will stop for a while here, arguments have become circular, proponents and opponents have exposed their view, but mostly proponents have provided very few additional information for a while, especially about real use cases, despite a few requests. We still do not know what motivated the proposal..We had to guess (JSON). Even the PEP do not say, it provide a set of real world example from the standard library, which is admirable but different. And personaly, while am very thankful for providing those concrete examples, I find them unconvicing: the code before is usually fine and improved little by the new operators. Sometimes it's worse: only slightly shorter and less clear.
On Mon, Jul 23, 2018 at 1:28 PM Steven D'Aprano steve@pearwood.info wrote:
There's the first bug right there. foo.bar.blim ought to raise AttributeError, since there is no blim attribute defined on foo.bar.
Note my observation that this is deliberately different semantics. I want to solve the underlying need, not simply emulate the less useful semantics of PEP 505. But those same semantics COULD be perfectly well emulated with a similar class.
In [8]: NoneAware(foo).bar.blim.unbox() In [9]: NoneAware(foo).bar.baz.blat.unbox() Out[9]: 42
How many bugs will be caused by people forgetting to unbox when they're done?
Far fewer and far more easily caught. When the object you get wherever the real failure happens is ` <none_aware.NoneAware at 0x11156a748>` or the like, the source of the problem become dead obvious. Having `None` somewhere unexpected is much more mysterious and difficult to diagnose.
That said, NoneAware is the wrong name for the class I wrote. Maybe GreedyAccess would be a better name. Perhaps I'll make slightly less toy implementations of an actual NoneCoalesce and GreedyAccess class and post in this thread and/or on PyPI.
It does look like PyMaybe does much of what I'm thinking of. I didn't think my toy was the first or best implementation.
But we aren't entering such a world, at least not in PEP 505. Attribute access can fail.
spam.eggs = 42
spam?.eggs?.upper
is still going to raise AttributeError, because eggs is not None, it is an int, and ints don't have an attribute "upper".
True enough. But it's extremely difficult to imagine a real-world case where those particular semantics are actually particularly useful. I guess my revulsion at the syntax blinded me to the fact that the semantics are actually pretty bad on their own.
How does your class implement short-circuit behaviour?
You mean if we want something like this?
favorite = GreedyAccess(cfg, sentinel=ExpensiveLookup()).user.profile.food.unbox()
It doesn't. Something else (like PyMaybe) could defer the computation using a lambda or other means. My toy code doesn't do that. We could also avoid the visible lambda by doing an 'iscallable(sentinel)' and only calling it if it turns out to be needed, but that might be too magical. E.g.:
# If we fail, favorite = ExpensiveLookup(), by magic favorite = GreedyAccess(cfg, sentinel=ExpensiveLookup).user.profile.food.unbox()
So we don't like operators like ?? because its too cryptic, but you're
happy to have one-character class and property names.
No, I'm not happy with that. I was just pointing out that code golf is possible for someone who really wants it. I'm happy to spend a few characters for readability. But one COULD write:
from nested import GreedyAccess as G G._ = G.unbox
What happens if you have an attribute that happens to be called "unbox" in your attribute look-up chain?
Don't use this class and/or modify the API slightly? This is getting trite. My 5 minute code isn't a final proposal, just a way of seeing a napkin-sketch of a better approach.
On Mon, Jul 23, 2018 at 01:55:48PM -0400, David Mertz wrote:
On Mon, Jul 23, 2018 at 1:28 PM Steven D'Aprano steve@pearwood.info wrote:
There's the first bug right there. foo.bar.blim ought to raise AttributeError, since there is no blim attribute defined on foo.bar.
Note my observation that this is deliberately different semantics. I want to solve the underlying need, not simply emulate the less useful semantics of PEP 505.
I know that "laziness and hubris" are virtues in a programmer, but the PEP authors describe use-cases where *testing for None* is precisely the semantics wanted. What are your use-cases for greedily swallowing AttributeError?
Before telling the PEP authors that they have misunderstood their own uses-cases and that their need isn't what they thought (coalescing None) but something radically different which they have already explicitly rejected (suppressing AttributeError), you better have some damn compelling use-cases.
"Laziness and hubris" are supposed to be virtues in programmers, but when you insist that people have their own use-cases wrong, and that the designers of C#, Dart and Swift made a serious mistake introducing this feature, you ought to back it up with more than just an assertion.
[...]
It does look like PyMaybe does much of what I'm thinking of. I didn't think my toy was the first or best implementation.
Are you aware that the PEP includes pymaybe as a rejected idea?
But we aren't entering such a world, at least not in PEP 505. Attribute access can fail. spam.eggs = 42 spam?.eggs?.upper is still going to raise AttributeError, because eggs is not None, it is an int, and ints don't have an attribute "upper".
True enough. But it's extremely difficult to imagine a real-world case where those particular semantics are actually particularly useful.
You can't imagine why it would be useful to NOT suppress the exception from a coding error like trying to uppercase an int.
On 23/07/18 16:12, David Mertz wrote:
Here is a way of solving the "deep attribute access to messy data" problem
Before we go too far, and /pace/ Steve's work, how real is this problem? I get that there is a use in accessing JSON trees, but this feels awfully like a rather specific issue about avoiding cleaning data before using it. As such, should be really be encouraging it?
(For the avoidance of doubt, this refers to the proposed ?. and ?[] operators.
that is:
(1) Much more explicit (2) Requires no change in syntax (3) Will not be a bug magnet (4) Inasmuch as there are semantic traps, they are announced by the use of a class whose documentation would be pointed to for readers
Worthy goals.
The API that could be useful might be something like this:
In [1]: from none_aware import NoneAware In [2]: from types import SimpleNamespace In [3]: foo = SimpleNamespace() In [4]: foo.bar = SimpleNamespace() In [5]: foo.bar.baz = SimpleNamespace() In [6]: foo.bar.baz.blat = 42 In [7]: NoneAware(foo).bar.blim Out[7]: <none_aware.NoneAware at 0x11156a748>
...and here's the first semantic trap. I was expecting to get None back there, and even your pre-announcement in goal (4) didn't stop me puzzling over it for a couple of minutes. The only reason I wasn't expecting an AttributeError was that I was confident you wouldn't propose this without doing something magic to effectively override the meaning of "." A naive user might be very surprised.
In [8]: NoneAware(foo).bar.blim.unbox() In [9]: NoneAware(foo).bar.baz.blat.unbox() Out[9]: 42
[snip]
I don't disagree that needing to call .unbox() at the end of the chained attribute access is a little bit ugly. But it's a lot less ugly than large family of new operators. And honestly, it's a nice way of being explicit about the fact that we're entering then leaving a special world where attribute accesses don't fail.
You and I have very different definitions of "nice" ;-) I have to say it's questionable which of "?." and "NoneAware()...unbox()" are uglier. Then again, I go back to my original comment: is this really a problem we want to solve?
How are you supposed to do method calling, the equivalent of "foo?.bar()" ? "NoneAware(foo).bar.unbox()()" looks downright weird. Is there more magic in NoneAware to cover this case? (Not that I think we should be encouraging people to do this, but...)
On Mon, Jul 23, 2018 at 2:12 PM Rhodri James rhodri@kynesim.co.uk wrote:
How are you supposed to do method calling, the equivalent of "foo?.bar()" ? "NoneAware(foo).bar.unbox()()" looks downright weird. Is there more magic in NoneAware to cover this case? (Not that I think we should be encouraging people to do this, but...)
Is there more magic? I don't know, and I don't really care that much. That's the point. This is just a plain old Python class that will work back to probably Python 1.4 or so. If you want to write a version that has just the right amount of magic, you are free to.
That said, I *DID* give it the wrong name in my first post of this thread. GreedyAccess is more accurate, and a NoneCoalesce class would behave somewhat differently.
In my opinion, the need at issue is worthwhile, but niche. Using a special class to deal with such a case is absolutely the right level of abstraction. Syntax is not! So sure, figure out how to tweak the API to be most useful, find a better name for the class, etc.
I wouldn't think it terrible if a class like this (but better) found a home in the standard library, but it doesn't deserve more prominence than that. Not even builtins, and *definitely* not syntax.
On 23/07/18 19:21, David Mertz wrote:
On Mon, Jul 23, 2018 at 2:12 PM Rhodri James rhodri@kynesim.co.uk wrote:
How are you supposed to do method calling, the equivalent of "foo?.bar()" ? "NoneAware(foo).bar.unbox()()" looks downright weird. Is there more magic in NoneAware to cover this case? (Not that I think we should be encouraging people to do this, but...)
Is there more magic? I don't know, and I don't really care that much. That's the point. This is just a plain old Python class that will work back to probably Python 1.4 or so. If you want to write a version that has just the right amount of magic, you are free to.
I care only in as much as you were proposing an incomplete solution (in my eyes at least). I'm pretty convinced by now I would never use it.
In my opinion, the need at issue is worthwhile, but niche. Using a special class to deal with such a case is absolutely the right level of abstraction. Syntax is not! So sure, figure out how to tweak the API to be most useful, find a better name for the class, etc.
I think part of my point was that a special class is not as readable or trap-free as writing the conditions out explicitly, given the unexpected "boxed" results, which makes me question whether the issue *is* worthwhile.
I wouldn't think it terrible if a class like this (but better) found a home in the standard library, but it doesn't deserve more prominence than that. Not even builtins, and *definitely* not syntax.
I think I might find it terrible.
On Mon, Jul 23, 2018 at 2:12 PM Rhodri James rhodri@kynesim.co.uk wrote:
How are you supposed to do method calling, the equivalent of "foo?.bar()" ? "NoneAware(foo).bar.unbox()()" looks downright weird. Is there more magic in NoneAware to cover this case? (Not that I think we should be encouraging people to do this, but...)
In PyMaybe, you would do this:
maybe(foo).bar().or_else(None)
Le 23/07/2018 à 17:12, David Mertz a écrit :
The need addressed by PEP 505 is real; it's also MUCH more niche and uncommon than something that would merit new syntax. Moreover, the actual legitimate purpose served by the PEP 505 syntax is easily served by existing Python simply by using a wrapper class.
Here is a way of solving the "deep attribute access to messy data" problem that is:
(1) Much more explicit (2) Requires no change in syntax (3) Will not be a bug magnet (4) Inasmuch as there are semantic traps, they are announced by the use of a class whose documentation would be pointed to for readers
The API that could be useful might be something like this:
In [1]: from none_aware import NoneAware In [2]: from types import SimpleNamespace In [3]: foo = SimpleNamespace() In [4]: foo.bar = SimpleNamespace() In [5]: foo.bar.baz = SimpleNamespace() In [6]: foo.bar.baz.blat = 42 In [7]: NoneAware(foo).bar.blim Out[7]: <none_aware.NoneAware at 0x11156a748> In [8]: NoneAware(foo).bar.blim.unbox() In [9]: NoneAware(foo).bar.baz.blat.unbox() Out[9]: 42 In [10]: NoneAware(foo).bar.baz.blat Out[10]: <none_aware.NoneAware at 0x11157d908> In [11]: NoneAware(foo).bar.baz.flam.unbox() In [12]: NoneAware(foo).bar.baz.flam Out[12]: <none_aware.NoneAware at 0x1115832b0>
This has existed as libs for a while:
https://github.com/ekampf/pymaybe
It's interest is much more limited than in Haskell because in Python, calls are not lazy, objects are dynamic and function calls are expensive.
Take the following:
foo().bar[0]
Doing it safely would be:
val = foo()
try: val = val.bar except AttributeError: val = None else: try: val = val[0] except KeyError: val = None
Clearly we see that the operation goes up to the end if everything goes right. Otherwise, it jumps to the default value.
With the operator proposal:
val = foo()?.bar?[0]
The "?" operator is like "or" / "and", and it will shotcircuit as well.
With the try / else proposal:
val = try foo().bar[0] else None
This is supposed to transform this ast to into the first try/except form. So it shortcircuit as well.
But with your proposal, it becomes:
val = maybe(foo()).bar[0].or_else(None)
Which means
foo() # call
maybe(foo()) # call
maybe(foo()).__getattr__('bar') # call
maybe(foo()).__getattr__('bar').__getitem__(0) # call
maybe(foo()).__getattr__('bar').__getitem__(0).or_else(None) # call
There is no shortcircuit, you get 5 calls in all cases, plus the maybe object proxing the call to the underlying value every time. So that's 7 calls, added to the try/excepts you still have behind the scene.
Plus, you don't get the same calls depending of the values, but the switch is implicit and behind the maybe object: no linter can save you from a typo.
So yes, it works right now. But it's a suboptimal solution.
Short circuit is indeed the main difference. Makes me re-think about the None trigger subclasses of invalid operations exceptions. Like None.a trigger a NoneAttributeError(AttributeError), and so on... I think it solves many issues without much problems nor syntax changes, but probably I miss something...
Sure, it's a big change, but maybe less so than 3 non-classic operators working only on None...
I agree a class can provide a very good alternative to PEP505. I built one a while ago, and still use it https://github.com/klahnakoski/mo-dots for most my data transformation code.
The library only boxes dicts and lists, which removes the need for .unbox() in many of my use cases. My point is, if a class is chosen instead of PEP505, I doubt the unboxing will be a big issue.
On 2018-07-23 11:12, David Mertz wrote:
Here is a way of solving the "deep attribute access to messy data" problem that is:
David Mertz schrieb am 23.07.2018 um 16:12:
The need addressed by PEP 505 is real; it's also MUCH more niche and uncommon than something that would merit new syntax. Moreover, the actual legitimate purpose served by the PEP 505 syntax is easily served by existing Python simply by using a wrapper class.
The discussion so far made it clear to me that
a) there is a use case for this feature, although I never needed it myself b) throwing new syntax at it is not the right solution
Especially since there seem to be slightly diverging ideas about the exact details in behaviour. Since this can be done in form of a library, people should just drop it into a couple of competing libraries and let users choose what they like better in their specific situation. And since we already have a PEP now, let's continue to use it as a basis for discussion about how these libraries should best behave in general and what mistakes they should avoid.
Stefan
On Tue, Jul 24, 2018 at 1:57 AM Stefan Behnel stefan_ml@behnel.de wrote:
David Mertz schrieb am 23.07.2018 um 16:12:
The need addressed by PEP 505 is real; it's also MUCH more niche and uncommon than something that would merit new syntax. Moreover, the actual legitimate purpose served by the PEP 505 syntax is easily served by existing Python simply by using a wrapper class.
The discussion so far made it clear to me that
a) there is a use case for this feature, although I never needed it myself b) throwing new syntax at it is not the right solution
Especially since there seem to be slightly diverging ideas about the exact details in behaviour. Since this can be done in form of a library, people should just drop it into a couple of competing libraries and let users choose what they like better in their specific situation. And since we already have a PEP now, let's continue to use it as a basis for discussion about how these libraries should best behave in general and what mistakes they should avoid.
Stefan
+1 There is still no proof that such a programming pattern would be widely used or is desirable in practice. A library on PYPI could help clarifying that (and the PEP should mention it).
participants (14)
-
Antoine Pitrou -
Brice Parent -
David Mertz -
Giampaolo Rodola' -
Grégory Lielens -
Jonathan Fine -
Kyle Lahnakoski -
Mark E. Haase -
Michel Desmoulin -
Paul Moore -
Rhodri James -
Robert Vanden Eynde -
Stefan Behnel -
Steven D'Aprano